Skip to content
Français

Difference-in-differences

  • Lesson 17
  • 40 min
  • Module 5: When you cannot randomise: natural experiments

Where we are

Our tests worked because we chose who got the change. Sometimes nobody chose at random: free delivery started in Quebec first, for business reasons. In this lesson, we compare Quebec's change with Ontario's change with difference-in-differences, and we check its main assumption.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Estimate the effect of a change that started in one region first, without a test. Compare the change in that region with the change in a region that did not get it. Then check the assumption behind that comparison, and say how far you trust it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you draw Quebec’s and Ontario’s weekly sales and compute the 2 × 2 table by hand. Then you run the regression with errors clustered by area, and a placebo test. Its setup cells install the packages, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m04, from Peeking, with .venv active.

Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.

A garden centre has two shops: one in town A and one in town B. In May, the owner puts a new window display in shop A only. Shop A’s sales go from 200 to 236 plants a week. That is 36 more.

Was it the display? May is also the start of the planting season. Every garden shop sells more in May.

Shop B gives the answer. It did not get the display, but it had the same season. Its sales went from 150 to 165: 15 more. So the season alone added about 15 plants. Shop A had 36 more. The display probably added the rest: 36 − 15 = 21 plants a week.

The owner did not run a test. He used a shop without the change to see what the season did.

The problem: free delivery started in Quebec first

Section titled “The problem: free delivery started in Quebec first”

Hana, the merchandising lead, wanted free delivery from 50 dollars instead of 75. TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield did not test it with a coin flip. It started the new rule in Quebec on Monday 2 June 2025, for business reasons. The other provinces kept the old rule.

Hana asks: “Did free delivery raise Quebec’s sales, and by how much?”

The data is synthetic: the course simulator makes it, so it knows the true effect. Each province has its own delivery areas, 60 in all. The chart shows Quebec’s and Ontario’s weekly sales. For each line, 100 is that province’s average of the 26 weeks before the change.

  • Quebec
  • Ontario
Weekly sales of the simulated Quebec and Ontario, June 2024 to November 2025. Both provinces rise and fall with the same garden season. After week 0, Quebec runs a little above Ontario. Synthetic data, seed 40.
Text description of the chart

A line chart of 78 weeks, from 52 weeks before to 25 weeks after 2 June 2025. For each line, 100 is the province’s average of the 26 weeks before the change. Both provinces follow the same season. Sales are high in late spring 2025, up to about 125 for Quebec and 119 for Ontario. They are low in the autumn and winter, at about 87 to 89. Before week 0, the two lines move almost together. After week 0, Quebec’s line is a few points above Ontario’s line most weeks.

The season moves sales much more than free delivery does. Quebec’s sales in the 26 weeks after the change were 7.1% higher than in the 26 weeks before. But spring and summer are better weeks than winter, with or without free delivery. With 52 weeks before instead of 26, the same comparison says +9.3%. The answer changes with the window, so neither number is the effect.

The garden centre’s answer has a name: difference-in-differences, often shortened to DiD. You compute two differences, then the difference between them:

  1. The change in the region that got the change (Quebec): after minus before.
  2. The change in a comparison region that did not get it (Ontario): after minus before.
  3. The effect: the first change minus the second change.

The second change shows what the season and everything else did. Subtract it, and the rest is the effect.

Here is the garden centre again, as a table and a chart. We made up the numbers for this example.

Before (April) After (May) Change
Shop A (new display) 200 236 +36
Shop B (no display) 150 165 +15
Difference-in-differences 36 − 15 = +21
  • Shop A (new display)
  • Shop B (no display)
  • Shop A without the display (assumed)
Difference-in-differences as two lines and the gap. The dashed line starts at shop A's 200 and rises by shop B's 15. Shop A reached 236, so the gap of 21 plants a week is the display's effect. A made-up example.
Text description of the chart

Two periods, before and after. Shop A goes from 200 to 236 plants a week. Shop B goes from 150 to 165. A dashed line shows shop A without the display, if it had moved like shop B: from 200 to 215. The gap between 236 and 215 is 21 plants: the effect of the display.

The dashed line is the counterfactual: shop A’s sales without the display. Nobody saw it. We assumed that shop A would have moved like shop B. Everything depends on that assumption.

Difference-in-differences assumes parallel trends: without the change, the two regions would have moved by the same amount. Their levels can differ: Quebec sells less than Ontario. Their paths must move together.

Larkfield’s sales grow and fall by a percentage of their size, so we compare on the log scale. A log turns “times 1.06” into “plus 0.058”. On that scale, parallel lines mean the same percentage change. The result comes back as a percentage.

You cannot check the assumption after the change: Quebec without free delivery does not exist. You can check it before the change. If the two regions moved together for a year, they probably would have continued. That is not a proof.

First, update the project to the snapshot end-of-m05. It adds the tools of this module: xp did, xp synth, xp rdd, xp its, and the scenarios regions and loyalty. You copy its files over your project folder. Your own runs and plans stay.

  1. Open a terminal in the folder that holds larkfield-experiments and nextia-learning-labs. If you cloned the labs repository with Git, update it first.

    Terminal window
    git -C nextia-learning-labs pull
  2. Copy the snapshot over your project.

    Terminal window
    cp -R nextia-learning-labs/C40/snapshots/end-of-m05/. larkfield-experiments/
    cd larkfield-experiments
    source .venv/bin/activate

    The packages are the same as in Module 4, so you install nothing new.

  3. Run the tests.

    Terminal window
    python -m pytest -q
    ....................... [100%]
    26 passed in 3.53s

    Twenty-six tests pass. Your time will differ.

  4. Simulate the rollout.

    Terminal window
    sim run --scenario regions
    Scenario regions (seed 40): Free delivery in Quebec first (from 2 June 2025): weekly sales of 60 areas in 10 provinces (no test).
    Wrote runs/regions-s40: areas_weekly (4,680 rows), provinces_weekly (780 rows), counterfactual (4,680 rows). Synthetic data. 0.2 s

    counterfactual holds Quebec’s areas without the change. Only the simulator knows it, and xp check reads it.

  5. Run difference-in-differences, then compare with the truth.

    Terminal window
    xp did
    xp check
    Difference-in-differences: QC against ON (24 areas), change on 2025-06-02.
    Effect on weekly sales: +5.3% (95% interval +4.6% to +6.0%, p = 3.27e-56; errors clustered by area).
    Parallel trends before the change: weekly growth +0.157% (treated) against +0.174% (control).
    Run regions-s40: the last analysis was `xp did`.
    True effect (from the simulator): +6.0% on Quebec's weekly sales.
    Your estimate: +5.3% (+4.6% to +6.0%), -0.7 percentage points from the truth; the interval does NOT contain it.

xp did says that free delivery raised Quebec’s weekly sales by 5.3%. The truth is 6.0%. The estimate is close, and much better than the naive +7.1% or +9.3%. But its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI ends at +5.96%, just below the truth.

The third line compares the trends before the change. Quebec’s sales grew by 0.157% a week and Ontario’s by 0.174%. They look parallel.

xp did does not use one number per province. It uses the 24 delivery areas of the two provinces, week by week, in one regression:

  • one term for each area: it absorbs each area’s own level;
  • one term for each week: it absorbs the season and everything else that hit all areas that week;
  • one yes-or-no term, “in Quebec, after the change”: its coefficient is the effect.

The week terms play Ontario’s role, and the regression uses all 78 weeks.

The weeks of one area are not independent: a good area stays good for months. So xp did uses errors clustered by area. A cluster is a group of rows that move together. The interval then counts 24 areas, not 1,872 area-weeks. Without clustering, the interval would be far too narrow.

The notebook computes the 2 × 2 table with 26 weeks on each side. Here are its numbers: the mean weekly sales, and the change on the log scale.

Before (26 weeks) After (26 weeks) Change (log × 100)
Quebec 254,237 dollars 272,277 dollars +6.89
Ontario 390,509 dollars 395,530 dollars +1.33
Difference-in-differences 6.89 − 1.33 = +5.56, so +5.7%

The table says +5.7%, the regression +5.3%: they use different weeks. Both are near the truth on this run.

The explorer shows Quebec’s weekly sales, and Ontario scaled to Quebec’s level in the window before. Its readout adds a simple estimate: Quebec’s change divided by Ontario’s change. It is the table’s idea, not the project’s regression.

Explore

Quebec against Ontario, and Quebec without free delivery

Average per week before
254,237
Average per week after
272,277
Naive change (after against before)
+7.1%
Against Ontario (difference-in-differences)
+5.7%
True effect (simulator)
+6.0%
Naive minus truth
+1.1 points

Average sales per week: 254,237 dollars before, 272,277 dollars after. The naive change is +7.1%. The comparison with Ontario (difference-in-differences) gives +5.7%. Without the free delivery, the same 26 weeks after the change would have sold 6,678,494 dollars. The free delivery added +6.0% (+400,708 dollars). The naive answer is 1.1 points too high.

  • Quebec (with free delivery)
  • Quebec without free delivery (simulator)
  • Ontario, scaled to Quebec before the rollout

Shaded: the two windows; the black lines are their averages

26 weeks (2 Dec 2024 to 1 Jun 2025)26 weeks (2 Jun to 30 Nov 2025)150200250300350Quebec sales per week (thousand dollars)3 Jun2 Sep2 Dec3 Mar2 Jun1 Seprollout 2 JunOntario, scaled to Quebec before the rolloutQuebec without free delivery (simulator)Quebec (with free delivery)Average per period before: 254,237Average per period after: 272,277
26 weeks (2 Dec 2024 to 1 Jun 2025)26 weeks (2 Jun to 30 Nov 2025)150200250300350Thousand $ a week3 Jun2 Dec2 Junrollout 2 JunOntario, scaled to Quebec before the rolloutQuebec without free delivery (simulator)Quebec (with free delivery)Average per period before: 254,237Average per period after: 272,277
Weekly sales of the simulated Quebec around free delivery (2 June 2025), with Ontario scaled to Quebec's level before the change. Synthetic data, seed 40.
Text description of the explorer

The chart shows Quebec’s weekly sales from June 2024 to November 2025, in thousand dollars a week. A vertical line marks 2 June 2025. Shading marks the window before and the window after. With 26 weeks on each side, the average is 254,237 dollars a week before and 272,277 after. That is a naive change of +7.1%. A second line shows Ontario, scaled to Quebec’s level in the window before. Ontario’s change is much smaller, and the explorer’s difference-in-differences is +5.7%. The button shows a dashed line: Quebec without free delivery, from the simulator. The true effect is +6.0%. With 4 weeks on each side, the naive change and the explorer’s estimate are both +6.8%. With 52 weeks before and 26 after, the naive change is +9.3% and the estimate +5.4%.

Try two things. Set the window before to 52 weeks: the naive change jumps to +9.3%, but the estimate against Ontario stays near +5.4%. Then show Quebec without free delivery: the gap between the two Quebec lines is the true effect, about 6%.

The explorer computes a simple ratio of the window averages, as in the 2 × 2 table. So its numbers differ a little from xp did, which fits a regression on every week and area (+5.3%).

There are two common checks. Both look only at the weeks before the change.

Compare the trends. xp did already printed them: +0.157% and +0.174% a week. A big gap would be a warning.

Run a placebo test. A placebo is a fake treatment. Pretend that the change started 26 weeks earlier, on 2 December 2024, and use only the weeks before the real change. Nothing changed on that date, so the “effect” should be near zero.

Terminal window
xp did --placebo
Difference-in-differences: QC against ON (24 areas), change on 2025-06-02.
Effect on weekly sales: +5.3% (95% interval +4.6% to +6.0%, p = 3.27e-56; errors clustered by area).
Parallel trends before the change: weekly growth +0.157% (treated) against +0.174% (control).
Placebo (a fake change on 2024-12-02, weeks before the real one only): -0.7% (-1.5% to +0.1%, p = 0.0718). A clear placebo effect means the trends were not parallel.

The placebo effect is −0.7%, and its interval contains zero (p = 0.07). The check passes. Most teams would now report +5.3% with confidence.

An honest surprise: the checks passed, and the method was wrong

Section titled “An honest surprise: the checks passed, and the method was wrong”

The simulator lets us do what no real team can: run the same rollout with 50 seeds, and compare each estimate with its truth.

  • Against Ontario, the mean estimate was +3.0%, against a truth of +6.0%. The interval contained the truth in only 2% of the runs: 1 run in 50.
  • Against all nine other provinces, the mean was +4.3%. The interval contained the truth in 10% of the runs.
Difference-in-differences against the simulator's truth. On this run (top), both estimates are near the truth. Over 50 seeds (bottom), both are too low on average: Ontario is not a good enough comparison for Quebec.
Text description of the chart

Two panels with a vertical line at the true effect, +6.0%. Top panel, this run: against Ontario +5.3% (+4.6% to +6.0%); against all provinces +5.5% (+4.9% to +6.1%). Bottom panel, the mean over 50 seeds: against Ontario +3.0%; against all provinces +4.3%. Both means are far to the left of the truth.

So our run, seed 40, was a kind one. Why does the method miss? sim about regions says it: Quebec shares a slow eastern trend with the Atlantic provinces, and Ontario does not. The eastern trend drifts down on average. Without free delivery, Quebec would have fallen a little behind Ontario. Difference-in-differences assumes it would not, so it counts that loss against free delivery.

The next chart shows the gap between Quebec and Ontario, in 4-week blocks, with the gap before the change set to zero. The solid line is the real gap. The dashed line is the gap without free delivery, from the simulator. Difference-in-differences assumes that the dashed line stays at zero after the change.

  • Real gap
  • Gap without free delivery (simulator)
The gap between Quebec and Ontario, in 4-week blocks. Difference-in-differences assumes that the dashed line stays at 0 after the change. After the change, the real gap rises by about 5 points. Without free delivery, the gap would have drifted about 1 point below zero. Difference-in-differences cannot see that drift, so it is too low. Synthetic data, seed 40.
Text description of the chart

A line chart of 20 blocks of 4 weeks, from 52 weeks before to 26 weeks after the change. Before the change, both lines are the same and move between −2.6 and +1.4 around zero. After the change, the real gap is between +3.3 and +7.3 points. The gap without free delivery, from the simulator, is between −2.6 and +1.5. It is mostly below zero: −1.5, −1.3, −2.6, −0.6 and −0.7 in the last five blocks.

Before the change, the drift was too small to see. It hid in the week-to-week noise, and the checks passed. That is the main limit of the method. The pre-trend checks can only find a gap that is bigger than the noise.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, asks: “So difference-in-differences is useless?” No. It took the naive +7.1% or +9.3% down to +5.3%, with a clear argument. But the interval only covers the noise. It does not cover a wrong comparison region.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use difference-in-differences when:

  • a change started in some units (regions, stores, teams) and not in others, at a known date;
  • you have data from well before the change, so you can check the trends;
  • the comparison units had nothing else new at the same time.

Do not trust it when:

  • the trends before the change clearly differ, or the placebo test finds an effect;
  • someone chose the units because they were doing well or badly. Larkfield chose Quebec “for business reasons”. If the reason was a falling Quebec, the method gives a wrong answer;
  • something else changed in one group at the same time, for example a local holiday or a competitor’s sale.

The cost is time and care: a year of history, a good comparison group, and a clear statement of the assumption.

Guided practice: the 2 × 2 table and the regression

Section titled “Guided practice: the 2 × 2 table and the regression”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, predict whether you will see a jump in Quebec on 2 June. Then draw both provinces. The season moves both lines far more than the change.
  4. In section 2, compute the 2 × 2 table. You see difference-in-differences: +5.7%.
  5. In section 3, run the regression with errors clustered by area. You see effect +5.28% and the truth +6.0%.
  6. In section 4, run the placebo test. You see placebo effect -0.7%, p = 0.072.
  7. In section 5, run the regression with all 60 areas and put the effect in did_all. Run the check cell. It prints Check passed.
Show the answer to step 7
da = areas.copy()
da["treated"] = (da.province == "QC").astype(int)
da["log_revenue"] = np.log(da.revenue)
ma = smf.ols("log_revenue ~ treated:post + C(area) + C(week)", data=da).fit(
cov_type="cluster", cov_kwds={"groups": da["area"].astype("category").cat.codes})
did_all = round(float(np.expm1(ma.params["treated:post"])), 4)
print(did_all)

The value is 0.0551: +5.5%. The same number as xp did --control all.

Your turn: all provinces as the comparison

Section titled “Your turn: all provinces as the comparison”

Ontario alone may be a poor match for Quebec. Run difference-in-differences with all nine other provinces as the comparison, and compare with the truth. Before you run it, predict: will the estimate move toward the truth or away from it?

Show the answer
Terminal window
xp did --control all
xp check
Difference-in-differences: QC against AB, BC, MB, NB, NL, NS, ON, PE, SK (60 areas), change on 2025-06-02.
Effect on weekly sales: +5.5% (95% interval +4.9% to +6.1%, p = 6.25e-69; errors clustered by area).
Parallel trends before the change: weekly growth +0.157% (treated) against +0.160% (control).
Run regions-s40: the last analysis was `xp did`.
True effect (from the simulator): +6.0% on Quebec's weekly sales.
Your estimate: +5.5% (+4.9% to +6.1%), -0.5 percentage points from the truth; the interval contains it.

The estimate moves toward the truth: +5.5%, and the interval now contains +6.0%. The comparison group now holds the Atlantic provinces, which share part of Quebec’s eastern trend. The trend gap before the change is also smaller (0.157% against 0.160% a week). But over 50 seeds, this comparison still finds only +4.3% on average. A mix of all provinces is better than Ontario, not good enough.

Common mistake

“The placebo test passed, so the estimate is right”

Symptom: a team compares Quebec with Ontario and gets +5.3% (+4.6% to +6.0%). The trends before the change look parallel, and the placebo test passes (p = 0.07). The team reports +5.3%. A later test with a coin flip finds about +6%.

Cause: the two regions had slightly different trends. The gap was too small to see in the year before the change, but it built up after. The placebo test can only find a gap bigger than the noise. The interval covers only the noise, not a wrong comparison region.

Fix: treat a passed check as “no warning”, not as proof. Try other comparison groups (xp did --control all) and a synthetic control. If the answers differ, report the range, and say which assumption each one needs. If the decision is large, run a real test in other provinces.

Check your understanding

Shop A's sales went from 400 to 460 after a new display. Shop B, without the display, went from 300 to 330 in the same weeks. What is the difference-in-differences estimate of the display's effect, and what does it assume?

Show the answer

Shop A changed by +60 and shop B by +30. The estimate is 60 − 30 = +30 a week. It assumes parallel trends: without the display, shop A would also have gained 30. On the log scale, shop A grew 15% and shop B 10%. If shops grow by percentages, the estimate is 1.15 ÷ 1.10 − 1, about +4.5%, instead. The scale you choose is part of the assumption.

Check your understanding

Why does xp did cluster its errors by area? What would happen to the interval without clustering?

Show the answer

The weeks of one area are not independent: an area that sells well this week usually sells well next week. Without clustering, the regression treats 1,872 area-weeks as independent pieces of evidence. The interval would then be far too narrow. Clustering by area counts the 24 areas as the units, so the interval is wider and more honest.

Check your understanding

Difference-in-differences against Ontario found +5.3% on seed 40, and the placebo test passed. Over 50 seeds, its mean was +3.0% against a truth of +6.0%. State the assumption that failed and one sign in the data that could show it.

Show the answer

The parallel-trends assumption failed: Quebec shares an eastern trend with the Atlantic provinces, and Ontario does not. Without free delivery, Quebec would have fallen slightly behind Ontario. One sign is a gap between the trends before the change, or a placebo effect. Here both were too small to see. Another sign is that the estimate moves when you change the comparison group: +5.3% against Ontario, +5.5% against all provinces.

Recap

  • When a change starts in one region first, a before-and-after comparison mixes the change with the season. Quebec showed +7.1% or +9.3%, depending on the window.
  • Difference-in-differences subtracts the change of a comparison region: (Quebec after − before) − (Ontario after − before). By hand: +5.7%. The regression with area and week terms: +5.3% (+4.6% to +6.0%). The truth: +6.0%.
  • It assumes parallel trends: without the change, both regions would have moved the same way. Check the trends before the change and run a placebo test.
  • Cluster the errors by unit (here, by area), or the interval is far too narrow.
  • Over 50 seeds, Quebec against Ontario found +3.0% on average, and the interval contained the truth in 2% of the runs. The checks passed and the method was wrong: the eastern trend was too small to see before the change.

Next, instead of choosing one comparison region, we build one: a weighted mix of provinces that moved like Quebec.

Next: Synthetic control