Skip to content
Français

Synthetic control

  • Lesson 18
  • 35 min
  • Module 5: When you cannot randomise: natural experiments

Where we are

Difference-in-differences needs a comparison region that moves like Quebec. Ontario is close, but not the same. In this lesson, we build a synthetic Quebec from a weighted mix of other provinces, and we test it with placebo regions.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Build the comparison region on purpose. Find a weighted mix of other regions that moved like the treated region before the change. Use the mix as its counterfactual after the change. Then test the result with placebo regions.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you find the weights of a synthetic Quebec with SciPy and draw it against the real Quebec. Then you compute the effect and run the placebo test for every other province. Its setup cells install the packages, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m05, with the regions run, from Difference-in-differences. Activate .venv.

Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.

A bakery chain opens a new café inside its bakery in town A. It wants to know what town A’s sales would have been without the café. No other town is a good match. Town B is smaller and grows slowly. Town C is bigger and grows fast.

The chain’s analyst has an idea. “Town A is somewhere between B and C. Let me mix them.” She tries 60% of town B and 40% of town C. Before the café, that mix follows town A almost exactly, week after week. So after the café opens, the mix is her best guess of town A without it.

She did not find a twin for town A. She built one.

The problem: Ontario is not Quebec’s twin

Section titled “The problem: Ontario is not Quebec’s twin”

In the last lesson, difference-in-differences compared Quebec with Ontario. On our run it found +5.3%, against a true +6.0%. Over 50 seeds it found only +3.0% on average. Quebec shares an eastern trend with the Atlantic provinces, and Ontario does not.

All nine other provinces together did better, but still not well: +4.3% on average. Their simple sum follows the biggest provinces most, not the ones that move like Quebec.

We need a comparison that moves like Quebec, by design.

A synthetic control is a weighted mix of regions that did not get the change. We choose the weights so that the mix follows the treated region as closely as possible before the change. After the change, the mix is the counterfactual. The gap between the real region and the mix is the effect.

The weights follow two rules. Each weight is zero or more, and they add up to 1. So the mix is a real average of real regions, not a strange formula. We call the regions that did not get the change the donors: they donate their data to the mix.

Alberto Abadie and his co-authors made the method well known in 2003 and 2010. They built a “California without its tobacco law” from other US states.

Here is the bakery’s mix, with made-up numbers. Weeks 1 to 4 are before the café; week 5 is after it.

Week Town A Town B Town C Mix: 0.6 × B + 0.4 × C
1 100 90 115 54 + 46 = 100
2 104 92 122 55.2 + 48.8 = 104
3 108 94 129 56.4 + 51.6 = 108
4 112 96 136 57.6 + 54.4 = 112
5 (café open) 126 98 143 58.8 + 57.2 = 116

The mix matches town A in all four weeks before. In week 5, town A sells 126 and the mix 116. The café’s estimated effect is 126 − 116 = +10 a week.

  • Town A
  • Town B
  • Town C
  • The mix (synthetic town A)
A synthetic control in five weeks. Neither town B nor town C moves like town A. The mix of 60% B and 40% C does, in every week before the café. In week 5, the gap between town A and the mix is the effect: +10. A made-up example.
Text description of the chart

A line chart over weeks 1 to 5. Town A: 100, 104, 108, 112, then 126 in week 5 with the café. Town B: 90, 92, 94, 96, 98. Town C: 115, 122, 129, 136, 143. The mix, 0.6 times B plus 0.4 times C: 100, 104, 108, 112, then 116 in week 5. The mix lies on top of town A for weeks 1 to 4. In week 5, town A is 10 above the mix.

In real data, no mix matches every week exactly. The computer tries many weights and keeps the ones with the smallest gaps before the change. This is a small optimisation: find the numbers that make an error as small as possible. The mathematics course explains how computers search for such numbers.

A synthetic control assumes that the mix would have kept moving like the treated region after the change, if nothing had changed. That is a weaker assumption than “Ontario moves like Quebec”. We chose the mix to follow Quebec’s ups and downs for a whole year. If Quebec has a trend that some donors share, the mix can share it too.

It is still an assumption. It fails if something new hits the treated region, or the donors, after the change. It also fails if no mix can follow the treated region before the change. Then the method has nothing to build on.

  1. Make sure that your project is at end-of-m05 and that the regions run exists. If you did the last lesson, both are ready. If not, run sim run --scenario regions.

  2. Build the synthetic Quebec, and compare with the truth.

    Terminal window
    xp synth
    xp check
    Synthetic QC: BC 0.38, NL 0.13, SK 0.13, AB 0.12, NB 0.07, MB 0.06, ON 0.06, PE 0.05
    Fit before the change: RMSPE 0.0128 (log sales). Effect after: +5.5%.
    Placebo test: post/pre gap ratio 4.4 for QC; rank 1 of 10 (p = 0.10). Placebo effects: AB +0.4%, BC +0.9%, MB -0.2%, NB -0.2%, NL -0.2%, NS -0.6%, ON +0.3%, PE -0.4%, SK -0.4%
    Run regions-s40: the last analysis was `xp synth`.
    True effect (from the simulator): +6.0% on Quebec's weekly sales.
    Your synthetic control: +5.5%, -0.5 percentage points from the truth (placebo p = 0.10).

The first line is the recipe of the synthetic Quebec. Nova Scotia gets a weight of 0 and is left out.

The weights of the synthetic Quebec. They add up to 1. British Columbia gets the most weight. Ontario, the comparison of the last lesson, gets only 0.06. Three Atlantic provinces (aqua) together get 0.25. Synthetic data, seed 40.
Text description of the chart

A column chart of the eight provinces with a weight above zero. British Columbia 0.38, Newfoundland and Labrador 0.13, Saskatchewan 0.13 and Alberta 0.12. New Brunswick 0.07, Manitoba 0.06, Ontario 0.06 and Prince Edward Island 0.05. Nova Scotia’s weight is 0. The weights add up to 1.

The weights are not a statement about the provinces’ culture or economy. They are the numbers that made the mix follow Quebec’s weekly sales best. The three Atlantic provinces in the mix carry part of the eastern trend that Ontario lacks.

xp synth matches the shape of the sales on the log scale, not their size: it compares each series with its own average before the change. So small provinces can have large weights.

The second line says how well the mix followed Quebec before the change. RMSPE is the root mean squared prediction error: roughly, the typical gap between Quebec and the mix in one week. On the log scale, 0.0128 means about 1.3%. Then comes the effect: the average gap after the change, +5.5%.

  • Quebec
  • Synthetic Quebec
  • Quebec without free delivery (simulator)
The real Quebec and the synthetic Quebec. Before the change, the mix follows Quebec within about 1.3% a week. After the change, Quebec runs about 5.5% above the mix. The grey line, which only the simulator knows, is Quebec without free delivery. It runs close to the mix. Synthetic data, seed 40.
Text description of the chart

A line chart of 78 weeks of Quebec’s sales, from June 2024 to November 2025, between about 226 and 317 thousand dollars a week. Before week 0, the real Quebec and the dashed synthetic Quebec are almost on top of each other through the whole season. After week 0, the real Quebec is above the synthetic one in every week, by 2 to 8%. A grey line from week 0 shows Quebec without free delivery, from the simulator; it runs close to the synthetic Quebec.

Is the gap real? A placebo test for every province

Section titled “Is the gap real? A placebo test for every province”

A gap after the change could be noise. How big is a gap when nothing happened? One week’s gap can be large by chance, so look at the gap over many weeks. The method answers with placebo tests. Pretend that each donor province got free delivery on 2 June. Build a synthetic version of it from the other donors, and measure its “effect”. These provinces got nothing, so their gaps show how big noise can be.

xp synth ran nine placebos. Their effects went from −0.6% (Nova Scotia) to +0.9% (British Columbia). Quebec’s is +5.5%.

  • Quebec
  • Each of the 9 other provinces (placebos)
The gap between each province and its synthetic version. The nine placebo provinces (grey) jump up and down around zero, before and after week 0. Quebec (orange) moves up at week 0 and stays above zero in every week after it. Synthetic data, seed 40.
Text description of the chart

A line chart of 10 lines over 78 weeks. Nine grey lines, one per placebo province, move up and down between about −8% and +7%, around zero, with no change at week 0. The orange line for Quebec moves between −2.5% and +3.1% before week 0. After week 0, it is above zero in every week, between +2.4% and +8.4%, and about +5.5% on average.

To turn the picture into one number, xp synth divides each province’s gap after the change by its gap before (both as RMSPE). Quebec’s ratio is 4.4: its gap after is 4.4 times its gap before. No placebo comes close, so Quebec ranks 1 of 10.

The p-value is the rank divided by the number of provinces. Here it is 1 ÷ 10 = 0.10. With 10 provinces, 0.10 is the smallest p-value possible. It is a different kind of p-value from those in Module 3: it asks how unusual Quebec is among the 10 provinces. So a synthetic control on few regions can never give “p < 0.05”. Report the rank and show the placebo chart. They are the evidence.

On our run, the synthetic control found +5.5%. Difference-in-differences found +5.3% against Ontario and +5.5% against all provinces. On this one run, the three are almost the same.

Over 50 seeds, they are not:

Method Mean estimate over 50 seeds True effect
Difference-in-differences, against Ontario +3.0% +6.0%
Difference-in-differences, against all provinces +4.3% +6.0%
Synthetic control +5.6% (standard deviation 0.7 points) +6.0%

The synthetic control lands closest. It still misses a little: some of the eastern trend is unique to Quebec, and no mix can copy it. Our version has no TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI: the placebo rank is its measure of doubt. The spread over seeds, 0.7 points, shows how much one run can move.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use a synthetic control when:

  • one unit, or a few, got the change: a province, a country, a big store, a market;
  • you have many donors that did not get it, and a long history before the change;
  • a mix of donors can follow the treated unit closely before the change.

Do not use it when:

  • the fit before the change is poor: a large RMSPE means that the mix is not a good counterfactual;
  • the treated unit is extreme, for example the biggest or the fastest-growing. No average of smaller units can reach it;
  • the change also reached the donors. If customers from Ontario started ordering through Quebec addresses, Ontario is no longer a clean donor.

The cost: more data and more checking than difference-in-differences. You need dozens of time points before the change. The placebo test needs a run for every donor. It is a few seconds on our 10 provinces, but it can take much longer on hundreds of stores.

Guided practice: build the weights yourself

Section titled “Guided practice: build the weights yourself”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, check the bakery’s mix by hand. You see the mix 100, 104, 108, 112 and the effect +10.
  4. In section 2, find the weights with scipy.optimize.minimize. You see BC 0.38 first and NS 0.00 last.
  5. In section 3, draw Quebec against the synthetic Quebec. Then compute the effect. You see effect +5.5%.
  6. In section 4, run the placebo test for every province, and find Quebec’s rank. Run the check cell. It prints Check passed.
Show the answer to step 6
def ratio(series, unit):
"""RMSPE after the change / RMSPE before, for one unit and its synthetic version."""
_, g = synth(series, unit, pre)
return np.sqrt((g[~pre] ** 2).mean()) / np.sqrt((g[pre] ** 2).mean())
donors = [p for p in series if p != "QC"]
ratios = {"QC": ratio(series, "QC")}
for p in donors:
others = {u: s for u, s in series.items() if u != "QC"}
ratios[p] = ratio(others, p)
order = sorted(ratios, key=ratios.get, reverse=True)
rank = order.index("QC") + 1
print(rank, round(ratios["QC"], 1))

Quebec ranks first, with a ratio of 4.4: 1 4.4. The p-value is 1 ÷ 10 = 0.10.

Hana wants one sentence for the free-delivery report. Put the three estimates of this module next to the truth, on this run and over 50 seeds. Use the outputs of xp did, xp did --control all and xp synth. Then write the sentence.

Show the answer
Method This run Mean over 50 seeds True effect
Difference-in-differences, Ontario +5.3% (+4.6% to +6.0%) +3.0% +6.0%
Difference-in-differences, all provinces +5.5% (+4.9% to +6.1%) +4.3% +6.0%
Synthetic control +5.5% (placebo rank 1 of 10) +5.6% +6.0%

A sentence Hana can use: “Free delivery from 50 dollars raised Quebec’s weekly sales by about 5 to 6%. Three methods agree on this run, and Quebec’s gap is larger than that of any province without the change. The methods assume that Quebec would have moved like the provinces we compared it with. The synthetic control is the most reliable of the three.”

In real life you would not have the last two columns. They come from the simulator. They are why we trust the synthetic control more than the comparison with Ontario.

Common mistake

“The synthetic control says +20%”

Symptom: a team builds a synthetic version of its biggest market after a price change. The estimate is +20%, much larger than anyone expected. The chart shows the real market above the mix after the change, and also above it before the change.

Cause: the fit before the change is poor. The biggest market grew faster than every donor, and no average of slower markets can follow a faster one. The gap after the change is mostly the old gap, not the effect. The RMSPE before the change was large, and nobody looked at it.

Fix: always look at the fit before the change, both in the chart and in the RMSPE. If the mix cannot follow the treated unit, do not use the method. Try matching on growth rates instead of levels, a shorter history, or difference-in-differences with a stated trend.

Check your understanding

Town A sold 50, 60 and 70 in three weeks before a change. Town B sold 40, 40 and 40. Town C sold 60, 80 and 100. Which weights for B and C make a mix that matches town A exactly? In week 4, after the change, A sells 90, B 40 and C 120. What is the effect?

Show the answer

Use a weight of 0.5 on B and 0.5 on C. The mix is then 50, 60 and 70, the same as town A in all three weeks. In week 4, the mix is 0.5 × 40 + 0.5 × 120 = 80. Town A sells 90, so the estimated effect is 90 − 80 = +10.

Check your understanding

The synthetic Quebec gives a placebo p-value of 0.10. Tomás says: “That is above 0.05, so the effect is not significant.” What would you answer?

Show the answer

The p-value is Quebec’s rank among the 10 provinces: 1 ÷ 10. With 10 provinces, 0.10 is the smallest value possible, so the 0.05 rule does not apply. Quebec’s gap is the largest of all 10 provinces. Its gap after the change is 4.4 times its gap before, and no placebo comes close. Report the rank and the placebo chart, not “significant” or “not significant”.

Check your understanding

State the assumption that a synthetic control needs, and one sign in the data that the assumption is false.

Show the answer

It assumes that the weighted mix of donors would have kept moving like the treated region after the change, if nothing had changed. One sign that it is false is a poor fit before the change. You see it as a large RMSPE, or as a gap between the region and the mix in the chart. Another sign: large placebo gaps, so noise alone makes gaps as big as the treated region’s.

Recap

  • A synthetic control is a weighted mix of donor regions that follows the treated region before the change. After the change, the mix is the counterfactual.
  • The weights are zero or more and add up to 1. They come from an optimisation that makes the gap before the change as small as possible.
  • The synthetic Quebec: British Columbia 0.38, Newfoundland and Labrador 0.13, Saskatchewan 0.13, Alberta 0.12, and four smaller weights. Ontario gets only 0.06. The fit before is within about 1.3% a week. The effect: +5.5%, against a true +6.0%.
  • Placebo tests run the method on every donor. Quebec’s gap ranks 1 of 10 (p = 0.10, the smallest possible with 10 provinces).
  • Over 50 seeds, the synthetic control found +5.6% on average: the closest of the methods in this module. Check the fit before the change, or the method can fail badly.

Next, a change that started at a threshold instead of a date: the loyalty tier.

Next: Regression discontinuity