Difference-in-differences
- Lesson 17
- 40 min
- Module 5: When you cannot randomise: natural experiments
Where we are
Our tests worked because we chose who got the change. Sometimes nobody chose at random: free delivery started in Quebec first, for business reasons. In this lesson, we compare Quebec's change with Ontario's change with difference-in-differences, and we check its main assumption.
Lesson 17 of 30 · Experiments and Causal Impact: Did Our Change Work?Difference-in-differences
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Estimate the effect of a change that started in one region first, without a test. Compare the change in that region with the change in a region that did not get it. Then check the assumption behind that comparison, and say how far you trust it.
NotebookDo this lesson in a notebook
In the notebook, you draw Quebec’s and Ontario’s weekly sales and compute the 2 × 2 table by hand. Then you run the regression with errors clustered by area, and a placebo test. Its setup cells install the packages, so it needs no local setup. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The counterfactual: what would have happened without the change, from The feature that “raised sales”.
- A 95% interval, from The result and its uncertainty. For a refresher, see Estimate uncertainty in the mathematics course.
- A regression with one extra variable, from Get more precise with CUPED.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m04, from Peeking, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.
Two towns and a new window display
Section titled “Two towns and a new window display”A garden centre has two shops: one in town A and one in town B. In May, the owner puts a new window display in shop A only. Shop A’s sales go from 200 to 236 plants a week. That is 36 more.
Was it the display? May is also the start of the planting season. Every garden shop sells more in May.
Shop B gives the answer. It did not get the display, but it had the same season. Its sales went from 150 to 165: 15 more. So the season alone added about 15 plants. Shop A had 36 more. The display probably added the rest: 36 − 15 = 21 plants a week.
The owner did not run a test. He used a shop without the change to see what the season did.
The problem: free delivery started in Quebec first
Section titled “The problem: free delivery started in Quebec first”Hana, the merchandising lead, wanted free delivery from 50 dollars instead of 75. TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield did not test it with a coin flip. It started the new rule in Quebec on Monday 2 June 2025, for business reasons. The other provinces kept the old rule.
Hana asks: “Did free delivery raise Quebec’s sales, and by how much?”
The data is synthetic: the course simulator makes it, so it knows the true effect. Each province has its own delivery areas, 60 in all. The chart shows Quebec’s and Ontario’s weekly sales. For each line, 100 is that province’s average of the 26 weeks before the change.
- Quebec
- Ontario
Text description of the chart
A line chart of 78 weeks, from 52 weeks before to 25 weeks after 2 June 2025. For each line, 100 is the province’s average of the 26 weeks before the change. Both provinces follow the same season. Sales are high in late spring 2025, up to about 125 for Quebec and 119 for Ontario. They are low in the autumn and winter, at about 87 to 89. Before week 0, the two lines move almost together. After week 0, Quebec’s line is a few points above Ontario’s line most weeks.
The season moves sales much more than free delivery does. Quebec’s sales in the 26 weeks after the change were 7.1% higher than in the 26 weeks before. But spring and summer are better weeks than winter, with or without free delivery. With 52 weeks before instead of 26, the same comparison says +9.3%. The answer changes with the window, so neither number is the effect.
The idea: compare the changes
Section titled “The idea: compare the changes”The garden centre’s answer has a name: difference-in-differences, often shortened to DiD. You compute two differences, then the difference between them:
- The change in the region that got the change (Quebec): after minus before.
- The change in a comparison region that did not get it (Ontario): after minus before.
- The effect: the first change minus the second change.
The second change shows what the season and everything else did. Subtract it, and the rest is the effect.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here is the garden centre again, as a table and a chart. We made up the numbers for this example.
| Before (April) | After (May) | Change | |
|---|---|---|---|
| Shop A (new display) | 200 | 236 | +36 |
| Shop B (no display) | 150 | 165 | +15 |
| Difference-in-differences | 36 − 15 = +21 |
- Shop A (new display)
- Shop B (no display)
- Shop A without the display (assumed)
Text description of the chart
Two periods, before and after. Shop A goes from 200 to 236 plants a week. Shop B goes from 150 to 165. A dashed line shows shop A without the display, if it had moved like shop B: from 200 to 215. The gap between 236 and 215 is 21 plants: the effect of the display.
The dashed line is the counterfactual: shop A’s sales without the display. Nobody saw it. We assumed that shop A would have moved like shop B. Everything depends on that assumption.
The assumption: parallel trends
Section titled “The assumption: parallel trends”Difference-in-differences assumes parallel trends: without the change, the two regions would have moved by the same amount. Their levels can differ: Quebec sells less than Ontario. Their paths must move together.
Larkfield’s sales grow and fall by a percentage of their size, so we compare on the log scale. A log turns “times 1.06” into “plus 0.058”. On that scale, parallel lines mean the same percentage change. The result comes back as a percentage.
You cannot check the assumption after the change: Quebec without free delivery does not exist. You can check it before the change. If the two regions moved together for a year, they probably would have continued. That is not a proof.
Larkfield: Quebec against Ontario
Section titled “Larkfield: Quebec against Ontario”First, update the project to the snapshot end-of-m05. It adds the tools of this module: xp did, xp synth, xp rdd, xp its, and the scenarios regions and loyalty. You copy its files over your project folder. Your own runs and plans stay.
-
Open a terminal in the folder that holds
larkfield-experimentsandnextia-learning-labs. If you cloned the labs repository with Git, update it first.Terminal window git -C nextia-learning-labs pull -
Copy the snapshot over your project.
Terminal window cp -R nextia-learning-labs/C40/snapshots/end-of-m05/. larkfield-experiments/cd larkfield-experimentssource .venv/bin/activateTerminal window Copy-Item -Recurse -Force nextia-learning-labs\C40\snapshots\end-of-m05\* larkfield-experimentscd larkfield-experiments.venv\Scripts\Activate.ps1The packages are the same as in Module 4, so you install nothing new.
-
Run the tests.
Terminal window python -m pytest -q....................... [100%]26 passed in 3.53sTwenty-six tests pass. Your time will differ.
-
Simulate the rollout.
Terminal window sim run --scenario regionsScenario regions (seed 40): Free delivery in Quebec first (from 2 June 2025): weekly sales of 60 areas in 10 provinces (no test).Wrote runs/regions-s40: areas_weekly (4,680 rows), provinces_weekly (780 rows), counterfactual (4,680 rows). Synthetic data. 0.2 scounterfactualholds Quebec’s areas without the change. Only the simulator knows it, andxp checkreads it. -
Run difference-in-differences, then compare with the truth.
Terminal window xp didxp checkDifference-in-differences: QC against ON (24 areas), change on 2025-06-02.Effect on weekly sales: +5.3% (95% interval +4.6% to +6.0%, p = 3.27e-56; errors clustered by area).Parallel trends before the change: weekly growth +0.157% (treated) against +0.174% (control).Run regions-s40: the last analysis was `xp did`.True effect (from the simulator): +6.0% on Quebec's weekly sales.Your estimate: +5.3% (+4.6% to +6.0%), -0.7 percentage points from the truth; the interval does NOT contain it.
Read the output
Section titled “Read the output”xp did says that free delivery raised Quebec’s weekly sales by 5.3%. The truth is 6.0%. The estimate is close, and much better than the naive +7.1% or +9.3%. But its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI ends at +5.96%, just below the truth.
The third line compares the trends before the change. Quebec’s sales grew by 0.157% a week and Ontario’s by 0.174%. They look parallel.
The regression behind it
Section titled “The regression behind it”xp did does not use one number per province. It uses the 24 delivery areas of the two provinces, week by week, in one regression:
- one term for each area: it absorbs each area’s own level;
- one term for each week: it absorbs the season and everything else that hit all areas that week;
- one yes-or-no term, “in Quebec, after the change”: its coefficient is the effect.
The week terms play Ontario’s role, and the regression uses all 78 weeks.
The weeks of one area are not independent: a good area stays good for months. So xp did uses errors clustered by area. A cluster is a group of rows that move together. The interval then counts 24 areas, not 1,872 area-weeks. Without clustering, the interval would be far too narrow.
The same answer by hand
Section titled “The same answer by hand”The notebook computes the 2 × 2 table with 26 weeks on each side. Here are its numbers: the mean weekly sales, and the change on the log scale.
| Before (26 weeks) | After (26 weeks) | Change (log × 100) | |
|---|---|---|---|
| Quebec | 254,237 dollars | 272,277 dollars | +6.89 |
| Ontario | 390,509 dollars | 395,530 dollars | +1.33 |
| Difference-in-differences | 6.89 − 1.33 = +5.56, so +5.7% |
The table says +5.7%, the regression +5.3%: they use different weeks. Both are near the truth on this run.
Move the windows yourself
Section titled “Move the windows yourself”The explorer shows Quebec’s weekly sales, and Ontario scaled to Quebec’s level in the window before. Its readout adds a simple estimate: Quebec’s change divided by Ontario’s change. It is the table’s idea, not the project’s regression.
Explore
Quebec against Ontario, and Quebec without free delivery
Predict first: Quebec's sales rose 7.1% from the 26 weeks before to the 26 weeks after. How much of that do you think is free delivery? Write a number, then show Quebec without free delivery.
- Average per week before
- 254,237
- Average per week after
- 272,277
- Naive change (after against before)
- +7.1%
- Against Ontario (difference-in-differences)
- +5.7%
- True effect (simulator)
- +6.0%
- Naive minus truth
- +1.1 points
Average sales per week: 254,237 dollars before, 272,277 dollars after. The naive change is +7.1%. The comparison with Ontario (difference-in-differences) gives +5.7%. Without the free delivery, the same 26 weeks after the change would have sold 6,678,494 dollars. The free delivery added +6.0% (+400,708 dollars). The naive answer is 1.1 points too high.
- Quebec (with free delivery)
- Quebec without free delivery (simulator)
- Ontario, scaled to Quebec before the rollout
Shaded: the two windows; the black lines are their averages
Text description of the explorer
The chart shows Quebec’s weekly sales from June 2024 to November 2025, in thousand dollars a week. A vertical line marks 2 June 2025. Shading marks the window before and the window after. With 26 weeks on each side, the average is 254,237 dollars a week before and 272,277 after. That is a naive change of +7.1%. A second line shows Ontario, scaled to Quebec’s level in the window before. Ontario’s change is much smaller, and the explorer’s difference-in-differences is +5.7%. The button shows a dashed line: Quebec without free delivery, from the simulator. The true effect is +6.0%. With 4 weeks on each side, the naive change and the explorer’s estimate are both +6.8%. With 52 weeks before and 26 after, the naive change is +9.3% and the estimate +5.4%.
Try two things. Set the window before to 52 weeks: the naive change jumps to +9.3%, but the estimate against Ontario stays near +5.4%. Then show Quebec without free delivery: the gap between the two Quebec lines is the true effect, about 6%.
The explorer computes a simple ratio of the window averages, as in the 2 × 2 table. So its numbers differ a little from xp did, which fits a regression on every week and area (+5.3%).
Check the assumption before you trust it
Section titled “Check the assumption before you trust it”There are two common checks. Both look only at the weeks before the change.
Compare the trends. xp did already printed them: +0.157% and +0.174% a week. A big gap would be a warning.
Run a placebo test. A placebo is a fake treatment. Pretend that the change started 26 weeks earlier, on 2 December 2024, and use only the weeks before the real change. Nothing changed on that date, so the “effect” should be near zero.
xp did --placeboDifference-in-differences: QC against ON (24 areas), change on 2025-06-02. Effect on weekly sales: +5.3% (95% interval +4.6% to +6.0%, p = 3.27e-56; errors clustered by area). Parallel trends before the change: weekly growth +0.157% (treated) against +0.174% (control). Placebo (a fake change on 2024-12-02, weeks before the real one only): -0.7% (-1.5% to +0.1%, p = 0.0718). A clear placebo effect means the trends were not parallel.The placebo effect is −0.7%, and its interval contains zero (p = 0.07). The check passes. Most teams would now report +5.3% with confidence.
An honest surprise: the checks passed, and the method was wrong
Section titled “An honest surprise: the checks passed, and the method was wrong”The simulator lets us do what no real team can: run the same rollout with 50 seeds, and compare each estimate with its truth.
- Against Ontario, the mean estimate was +3.0%, against a truth of +6.0%. The interval contained the truth in only 2% of the runs: 1 run in 50.
- Against all nine other provinces, the mean was +4.3%. The interval contained the truth in 10% of the runs.
Text description of the chart
Two panels with a vertical line at the true effect, +6.0%. Top panel, this run: against Ontario +5.3% (+4.6% to +6.0%); against all provinces +5.5% (+4.9% to +6.1%). Bottom panel, the mean over 50 seeds: against Ontario +3.0%; against all provinces +4.3%. Both means are far to the left of the truth.
So our run, seed 40, was a kind one. Why does the method miss? sim about regions says it: Quebec shares a slow eastern trend with the Atlantic provinces, and Ontario does not. The eastern trend drifts down on average. Without free delivery, Quebec would have fallen a little behind Ontario. Difference-in-differences assumes it would not, so it counts that loss against free delivery.
The next chart shows the gap between Quebec and Ontario, in 4-week blocks, with the gap before the change set to zero. The solid line is the real gap. The dashed line is the gap without free delivery, from the simulator. Difference-in-differences assumes that the dashed line stays at zero after the change.
- Real gap
- Gap without free delivery (simulator)
Text description of the chart
A line chart of 20 blocks of 4 weeks, from 52 weeks before to 26 weeks after the change. Before the change, both lines are the same and move between −2.6 and +1.4 around zero. After the change, the real gap is between +3.3 and +7.3 points. The gap without free delivery, from the simulator, is between −2.6 and +1.5. It is mostly below zero: −1.5, −1.3, −2.6, −0.6 and −0.7 in the last five blocks.
Before the change, the drift was too small to see. It hid in the week-to-week noise, and the checks passed. That is the main limit of the method. The pre-trend checks can only find a gap that is bigger than the noise.
TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, asks: “So difference-in-differences is useless?” No. It took the naive +7.1% or +9.3% down to +5.3%, with a clear argument. But the interval only covers the noise. It does not cover a wrong comparison region.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use difference-in-differences when:
- a change started in some units (regions, stores, teams) and not in others, at a known date;
- you have data from well before the change, so you can check the trends;
- the comparison units had nothing else new at the same time.
Do not trust it when:
- the trends before the change clearly differ, or the placebo test finds an effect;
- someone chose the units because they were doing well or badly. Larkfield chose Quebec “for business reasons”. If the reason was a falling Quebec, the method gives a wrong answer;
- something else changed in one group at the same time, for example a local holiday or a competitor’s sale.
The cost is time and care: a year of history, a good comparison group, and a clear statement of the assumption.
Guided practice: the 2 × 2 table and the regression
Section titled “Guided practice: the 2 × 2 table and the regression”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, predict whether you will see a jump in Quebec on 2 June. Then draw both provinces. The season moves both lines far more than the change.
- In section 2, compute the 2 × 2 table. You see
difference-in-differences: +5.7%. - In section 3, run the regression with errors clustered by area. You see
effect +5.28%and the truth+6.0%. - In section 4, run the placebo test. You see
placebo effect -0.7%, p = 0.072. - In section 5, run the regression with all 60 areas and put the effect in
did_all. Run the check cell. It printsCheck passed.
Show the answer to step 7
da = areas.copy()da["treated"] = (da.province == "QC").astype(int)da["log_revenue"] = np.log(da.revenue)ma = smf.ols("log_revenue ~ treated:post + C(area) + C(week)", data=da).fit( cov_type="cluster", cov_kwds={"groups": da["area"].astype("category").cat.codes})did_all = round(float(np.expm1(ma.params["treated:post"])), 4)print(did_all)The value is 0.0551: +5.5%. The same number as xp did --control all.
Your turn: all provinces as the comparison
Section titled “Your turn: all provinces as the comparison”Ontario alone may be a poor match for Quebec. Run difference-in-differences with all nine other provinces as the comparison, and compare with the truth. Before you run it, predict: will the estimate move toward the truth or away from it?
Show the answer
xp did --control allxp checkDifference-in-differences: QC against AB, BC, MB, NB, NL, NS, ON, PE, SK (60 areas), change on 2025-06-02. Effect on weekly sales: +5.5% (95% interval +4.9% to +6.1%, p = 6.25e-69; errors clustered by area). Parallel trends before the change: weekly growth +0.157% (treated) against +0.160% (control).Run regions-s40: the last analysis was `xp did`.True effect (from the simulator): +6.0% on Quebec's weekly sales.Your estimate: +5.5% (+4.9% to +6.1%), -0.5 percentage points from the truth; the interval contains it.The estimate moves toward the truth: +5.5%, and the interval now contains +6.0%. The comparison group now holds the Atlantic provinces, which share part of Quebec’s eastern trend. The trend gap before the change is also smaller (0.157% against 0.160% a week). But over 50 seeds, this comparison still finds only +4.3% on average. A mix of all provinces is better than Ontario, not good enough.
Common mistake
“The placebo test passed, so the estimate is right”
Symptom: a team compares Quebec with Ontario and gets +5.3% (+4.6% to +6.0%). The trends before the change look parallel, and the placebo test passes (p = 0.07). The team reports +5.3%. A later test with a coin flip finds about +6%.
Cause: the two regions had slightly different trends. The gap was too small to see in the year before the change, but it built up after. The placebo test can only find a gap bigger than the noise. The interval covers only the noise, not a wrong comparison region.
Fix: treat a passed check as “no warning”, not as proof. Try other comparison groups (xp did --control all) and a synthetic control. If the answers differ, report the range, and say which assumption each one needs. If the decision is large, run a real test in other provinces.
Check your understanding
Shop A's sales went from 400 to 460 after a new display. Shop B, without the display, went from 300 to 330 in the same weeks. What is the difference-in-differences estimate of the display's effect, and what does it assume?
Show the answer
Shop A changed by +60 and shop B by +30. The estimate is 60 − 30 = +30 a week. It assumes parallel trends: without the display, shop A would also have gained 30. On the log scale, shop A grew 15% and shop B 10%. If shops grow by percentages, the estimate is 1.15 ÷ 1.10 − 1, about +4.5%, instead. The scale you choose is part of the assumption.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does xp did cluster its errors by area? What would happen to the interval without clustering?
Show the answer
The weeks of one area are not independent: an area that sells well this week usually sells well next week. Without clustering, the regression treats 1,872 area-weeks as independent pieces of evidence. The interval would then be far too narrow. Clustering by area counts the 24 areas as the units, so the interval is wider and more honest.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Difference-in-differences against Ontario found +5.3% on seed 40, and the placebo test passed. Over 50 seeds, its mean was +3.0% against a truth of +6.0%. State the assumption that failed and one sign in the data that could show it.
Show the answer
The parallel-trends assumption failed: Quebec shares an eastern trend with the Atlantic provinces, and Ontario does not. Without free delivery, Quebec would have fallen slightly behind Ontario. One sign is a gap between the trends before the change, or a placebo effect. Here both were too small to see. Another sign is that the estimate moves when you change the comparison group: +5.3% against Ontario, +5.5% against all provinces.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- When a change starts in one region first, a before-and-after comparison mixes the change with the season. Quebec showed +7.1% or +9.3%, depending on the window.
- Difference-in-differences subtracts the change of a comparison region: (Quebec after − before) − (Ontario after − before). By hand: +5.7%. The regression with area and week terms: +5.3% (+4.6% to +6.0%). The truth: +6.0%.
- It assumes parallel trends: without the change, both regions would have moved the same way. Check the trends before the change and run a placebo test.
- Cluster the errors by unit (here, by area), or the interval is far too narrow.
- Over 50 seeds, Quebec against Ontario found +3.0% on average, and the interval contained the truth in 2% of the runs. The checks passed and the method was wrong: the eastern trend was too small to see before the change.
Next, instead of choosing one comparison region, we build one: a weighted mix of provinces that moved like Quebec.
Next: Synthetic control