Synthetic control
- Lesson 18
- 35 min
- Module 5: When you cannot randomise: natural experiments
Where we are
Difference-in-differences needs a comparison region that moves like Quebec. Ontario is close, but not the same. In this lesson, we build a synthetic Quebec from a weighted mix of other provinces, and we test it with placebo regions.
Lesson 18 of 30 · Experiments and Causal Impact: Did Our Change Work?Synthetic control
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Build the comparison region on purpose. Find a weighted mix of other regions that moved like the treated region before the change. Use the mix as its counterfactual after the change. Then test the result with placebo regions.
NotebookDo this lesson in a notebook
In the notebook, you find the weights of a synthetic Quebec with SciPy and draw it against the real Quebec. Then you compute the effect and run the placebo test for every other province. Its setup cells install the packages, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Difference-in-differences and parallel trends, from Difference-in-differences.
- A weighted average: each value times its weight, added up. For a refresher, see Describe a dataset in the mathematics course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m05, with theregionsrun, from Difference-in-differences. Activate.venv.
Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.
A recipe for a town that does not exist
Section titled “A recipe for a town that does not exist”A bakery chain opens a new café inside its bakery in town A. It wants to know what town A’s sales would have been without the café. No other town is a good match. Town B is smaller and grows slowly. Town C is bigger and grows fast.
The chain’s analyst has an idea. “Town A is somewhere between B and C. Let me mix them.” She tries 60% of town B and 40% of town C. Before the café, that mix follows town A almost exactly, week after week. So after the café opens, the mix is her best guess of town A without it.
She did not find a twin for town A. She built one.
The problem: Ontario is not Quebec’s twin
Section titled “The problem: Ontario is not Quebec’s twin”In the last lesson, difference-in-differences compared Quebec with Ontario. On our run it found +5.3%, against a true +6.0%. Over 50 seeds it found only +3.0% on average. Quebec shares an eastern trend with the Atlantic provinces, and Ontario does not.
All nine other provinces together did better, but still not well: +4.3% on average. Their simple sum follows the biggest provinces most, not the ones that move like Quebec.
We need a comparison that moves like Quebec, by design.
The idea: a weighted mix of other regions
Section titled “The idea: a weighted mix of other regions”A synthetic control is a weighted mix of regions that did not get the change. We choose the weights so that the mix follows the treated region as closely as possible before the change. After the change, the mix is the counterfactual. The gap between the real region and the mix is the effect.
The weights follow two rules. Each weight is zero or more, and they add up to 1. So the mix is a real average of real regions, not a strange formula. We call the regions that did not get the change the donors: they donate their data to the mix.
Alberto Abadie and his co-authors made the method well known in 2003 and 2010. They built a “California without its tobacco law” from other US states.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here is the bakery’s mix, with made-up numbers. Weeks 1 to 4 are before the café; week 5 is after it.
| Week | Town A | Town B | Town C | Mix: 0.6 × B + 0.4 × C |
|---|---|---|---|---|
| 1 | 100 | 90 | 115 | 54 + 46 = 100 |
| 2 | 104 | 92 | 122 | 55.2 + 48.8 = 104 |
| 3 | 108 | 94 | 129 | 56.4 + 51.6 = 108 |
| 4 | 112 | 96 | 136 | 57.6 + 54.4 = 112 |
| 5 (café open) | 126 | 98 | 143 | 58.8 + 57.2 = 116 |
The mix matches town A in all four weeks before. In week 5, town A sells 126 and the mix 116. The café’s estimated effect is 126 − 116 = +10 a week.
- Town A
- Town B
- Town C
- The mix (synthetic town A)
Text description of the chart
A line chart over weeks 1 to 5. Town A: 100, 104, 108, 112, then 126 in week 5 with the café. Town B: 90, 92, 94, 96, 98. Town C: 115, 122, 129, 136, 143. The mix, 0.6 times B plus 0.4 times C: 100, 104, 108, 112, then 116 in week 5. The mix lies on top of town A for weeks 1 to 4. In week 5, town A is 10 above the mix.
In real data, no mix matches every week exactly. The computer tries many weights and keeps the ones with the smallest gaps before the change. This is a small optimisation: find the numbers that make an error as small as possible. The mathematics course explains how computers search for such numbers.
What it assumes
Section titled “What it assumes”A synthetic control assumes that the mix would have kept moving like the treated region after the change, if nothing had changed. That is a weaker assumption than “Ontario moves like Quebec”. We chose the mix to follow Quebec’s ups and downs for a whole year. If Quebec has a trend that some donors share, the mix can share it too.
It is still an assumption. It fails if something new hits the treated region, or the donors, after the change. It also fails if no mix can follow the treated region before the change. Then the method has nothing to build on.
Larkfield: a synthetic Quebec
Section titled “Larkfield: a synthetic Quebec”-
Make sure that your project is at
end-of-m05and that theregionsrun exists. If you did the last lesson, both are ready. If not, runsim run --scenario regions. -
Build the synthetic Quebec, and compare with the truth.
Terminal window xp synthxp checkSynthetic QC: BC 0.38, NL 0.13, SK 0.13, AB 0.12, NB 0.07, MB 0.06, ON 0.06, PE 0.05Fit before the change: RMSPE 0.0128 (log sales). Effect after: +5.5%.Placebo test: post/pre gap ratio 4.4 for QC; rank 1 of 10 (p = 0.10). Placebo effects: AB +0.4%, BC +0.9%, MB -0.2%, NB -0.2%, NL -0.2%, NS -0.6%, ON +0.3%, PE -0.4%, SK -0.4%Run regions-s40: the last analysis was `xp synth`.True effect (from the simulator): +6.0% on Quebec's weekly sales.Your synthetic control: +5.5%, -0.5 percentage points from the truth (placebo p = 0.10).
The recipe
Section titled “The recipe”The first line is the recipe of the synthetic Quebec. Nova Scotia gets a weight of 0 and is left out.
Text description of the chart
A column chart of the eight provinces with a weight above zero. British Columbia 0.38, Newfoundland and Labrador 0.13, Saskatchewan 0.13 and Alberta 0.12. New Brunswick 0.07, Manitoba 0.06, Ontario 0.06 and Prince Edward Island 0.05. Nova Scotia’s weight is 0. The weights add up to 1.
The weights are not a statement about the provinces’ culture or economy. They are the numbers that made the mix follow Quebec’s weekly sales best. The three Atlantic provinces in the mix carry part of the eastern trend that Ontario lacks.
xp synth matches the shape of the sales on the log scale, not their size: it compares each series with its own average before the change. So small provinces can have large weights.
The fit before, and the gap after
Section titled “The fit before, and the gap after”The second line says how well the mix followed Quebec before the change. RMSPE is the root mean squared prediction error: roughly, the typical gap between Quebec and the mix in one week. On the log scale, 0.0128 means about 1.3%. Then comes the effect: the average gap after the change, +5.5%.
- Quebec
- Synthetic Quebec
- Quebec without free delivery (simulator)
Text description of the chart
A line chart of 78 weeks of Quebec’s sales, from June 2024 to November 2025, between about 226 and 317 thousand dollars a week. Before week 0, the real Quebec and the dashed synthetic Quebec are almost on top of each other through the whole season. After week 0, the real Quebec is above the synthetic one in every week, by 2 to 8%. A grey line from week 0 shows Quebec without free delivery, from the simulator; it runs close to the synthetic Quebec.
Is the gap real? A placebo test for every province
Section titled “Is the gap real? A placebo test for every province”A gap after the change could be noise. How big is a gap when nothing happened? One week’s gap can be large by chance, so look at the gap over many weeks. The method answers with placebo tests. Pretend that each donor province got free delivery on 2 June. Build a synthetic version of it from the other donors, and measure its “effect”. These provinces got nothing, so their gaps show how big noise can be.
xp synth ran nine placebos. Their effects went from −0.6% (Nova Scotia) to +0.9% (British Columbia). Quebec’s is +5.5%.
- Quebec
- Each of the 9 other provinces (placebos)
Text description of the chart
A line chart of 10 lines over 78 weeks. Nine grey lines, one per placebo province, move up and down between about −8% and +7%, around zero, with no change at week 0. The orange line for Quebec moves between −2.5% and +3.1% before week 0. After week 0, it is above zero in every week, between +2.4% and +8.4%, and about +5.5% on average.
To turn the picture into one number, xp synth divides each province’s gap after the change by its gap before (both as RMSPE). Quebec’s ratio is 4.4: its gap after is 4.4 times its gap before. No placebo comes close, so Quebec ranks 1 of 10.
The p-value is the rank divided by the number of provinces. Here it is 1 ÷ 10 = 0.10. With 10 provinces, 0.10 is the smallest p-value possible. It is a different kind of p-value from those in Module 3: it asks how unusual Quebec is among the 10 provinces. So a synthetic control on few regions can never give “p < 0.05”. Report the rank and show the placebo chart. They are the evidence.
How close is it? Fifty seeds
Section titled “How close is it? Fifty seeds”On our run, the synthetic control found +5.5%. Difference-in-differences found +5.3% against Ontario and +5.5% against all provinces. On this one run, the three are almost the same.
Over 50 seeds, they are not:
| Method | Mean estimate over 50 seeds | True effect |
|---|---|---|
| Difference-in-differences, against Ontario | +3.0% | +6.0% |
| Difference-in-differences, against all provinces | +4.3% | +6.0% |
| Synthetic control | +5.6% (standard deviation 0.7 points) | +6.0% |
The synthetic control lands closest. It still misses a little: some of the eastern trend is unique to Quebec, and no mix can copy it. Our version has no TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI: the placebo rank is its measure of doubt. The spread over seeds, 0.7 points, shows how much one run can move.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use a synthetic control when:
- one unit, or a few, got the change: a province, a country, a big store, a market;
- you have many donors that did not get it, and a long history before the change;
- a mix of donors can follow the treated unit closely before the change.
Do not use it when:
- the fit before the change is poor: a large RMSPE means that the mix is not a good counterfactual;
- the treated unit is extreme, for example the biggest or the fastest-growing. No average of smaller units can reach it;
- the change also reached the donors. If customers from Ontario started ordering through Quebec addresses, Ontario is no longer a clean donor.
The cost: more data and more checking than difference-in-differences. You need dozens of time points before the change. The placebo test needs a run for every donor. It is a few seconds on our 10 provinces, but it can take much longer on hundreds of stores.
Guided practice: build the weights yourself
Section titled “Guided practice: build the weights yourself”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, check the bakery’s mix by hand. You see the mix
100, 104, 108, 112and the effect+10. - In section 2, find the weights with
scipy.optimize.minimize. You seeBC 0.38first andNS 0.00last. - In section 3, draw Quebec against the synthetic Quebec. Then compute the effect. You see
effect +5.5%. - In section 4, run the placebo test for every province, and find Quebec’s rank. Run the check cell. It prints
Check passed.
Show the answer to step 6
def ratio(series, unit): """RMSPE after the change / RMSPE before, for one unit and its synthetic version.""" _, g = synth(series, unit, pre) return np.sqrt((g[~pre] ** 2).mean()) / np.sqrt((g[pre] ** 2).mean())
donors = [p for p in series if p != "QC"]ratios = {"QC": ratio(series, "QC")}for p in donors: others = {u: s for u, s in series.items() if u != "QC"} ratios[p] = ratio(others, p)order = sorted(ratios, key=ratios.get, reverse=True)rank = order.index("QC") + 1print(rank, round(ratios["QC"], 1))Quebec ranks first, with a ratio of 4.4: 1 4.4. The p-value is 1 ÷ 10 = 0.10.
Your turn: three methods, one truth
Section titled “Your turn: three methods, one truth”Hana wants one sentence for the free-delivery report. Put the three estimates of this module next to the truth, on this run and over 50 seeds. Use the outputs of xp did, xp did --control all and xp synth. Then write the sentence.
Show the answer
| Method | This run | Mean over 50 seeds | True effect |
|---|---|---|---|
| Difference-in-differences, Ontario | +5.3% (+4.6% to +6.0%) | +3.0% | +6.0% |
| Difference-in-differences, all provinces | +5.5% (+4.9% to +6.1%) | +4.3% | +6.0% |
| Synthetic control | +5.5% (placebo rank 1 of 10) | +5.6% | +6.0% |
A sentence Hana can use: “Free delivery from 50 dollars raised Quebec’s weekly sales by about 5 to 6%. Three methods agree on this run, and Quebec’s gap is larger than that of any province without the change. The methods assume that Quebec would have moved like the provinces we compared it with. The synthetic control is the most reliable of the three.”
In real life you would not have the last two columns. They come from the simulator. They are why we trust the synthetic control more than the comparison with Ontario.
Common mistake
“The synthetic control says +20%”
Symptom: a team builds a synthetic version of its biggest market after a price change. The estimate is +20%, much larger than anyone expected. The chart shows the real market above the mix after the change, and also above it before the change.
Cause: the fit before the change is poor. The biggest market grew faster than every donor, and no average of slower markets can follow a faster one. The gap after the change is mostly the old gap, not the effect. The RMSPE before the change was large, and nobody looked at it.
Fix: always look at the fit before the change, both in the chart and in the RMSPE. If the mix cannot follow the treated unit, do not use the method. Try matching on growth rates instead of levels, a shorter history, or difference-in-differences with a stated trend.
Check your understanding
Town A sold 50, 60 and 70 in three weeks before a change. Town B sold 40, 40 and 40. Town C sold 60, 80 and 100. Which weights for B and C make a mix that matches town A exactly? In week 4, after the change, A sells 90, B 40 and C 120. What is the effect?
Show the answer
Use a weight of 0.5 on B and 0.5 on C. The mix is then 50, 60 and 70, the same as town A in all three weeks. In week 4, the mix is 0.5 × 40 + 0.5 × 120 = 80. Town A sells 90, so the estimated effect is 90 − 80 = +10.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The synthetic Quebec gives a placebo p-value of 0.10. Tomás says: “That is above 0.05, so the effect is not significant.” What would you answer?
Show the answer
The p-value is Quebec’s rank among the 10 provinces: 1 ÷ 10. With 10 provinces, 0.10 is the smallest value possible, so the 0.05 rule does not apply. Quebec’s gap is the largest of all 10 provinces. Its gap after the change is 4.4 times its gap before, and no placebo comes close. Report the rank and the placebo chart, not “significant” or “not significant”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
State the assumption that a synthetic control needs, and one sign in the data that the assumption is false.
Show the answer
It assumes that the weighted mix of donors would have kept moving like the treated region after the change, if nothing had changed. One sign that it is false is a poor fit before the change. You see it as a large RMSPE, or as a gap between the region and the mix in the chart. Another sign: large placebo gaps, so noise alone makes gaps as big as the treated region’s.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A synthetic control is a weighted mix of donor regions that follows the treated region before the change. After the change, the mix is the counterfactual.
- The weights are zero or more and add up to 1. They come from an optimisation that makes the gap before the change as small as possible.
- The synthetic Quebec: British Columbia 0.38, Newfoundland and Labrador 0.13, Saskatchewan 0.13, Alberta 0.12, and four smaller weights. Ontario gets only 0.06. The fit before is within about 1.3% a week. The effect: +5.5%, against a true +6.0%.
- Placebo tests run the method on every donor. Quebec’s gap ranks 1 of 10 (p = 0.10, the smallest possible with 10 provinces).
- Over 50 seeds, the synthetic control found +5.6% on average: the closest of the methods in this module. Check the fit before the change, or the method can fail badly.
Next, a change that started at a threshold instead of a date: the loyalty tier.
Next: Regression discontinuity