Regression discontinuity
- Lesson 19
- 35 min
- Module 5: When you cannot randomise: natural experiments
Where we are
Our last two methods compared regions over time. Some changes start at a threshold instead: Larkfield's loyalty tier starts at 500 dollars of spending. In this lesson, we compare customers just above and just below the threshold with regression discontinuity.
Lesson 19 of 30 · Experiments and Causal Impact: Did Our Change Work?Regression discontinuity
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Estimate the effect of a rule that starts at a threshold. Compare the people just above the threshold with the people just below it. Choose how far from the line to look. Check whether people moved themselves over the line.
NotebookDo this lesson in a notebook
In the notebook, you simulate the loyalty tier with the project’s own code and draw the jump at 500 dollars. You fit a line on each side, find the customers who gamed the threshold, and leave them out with a donut. Its setup cells install the packages, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Difference-in-differences and its assumption, from Difference-in-differences.
- A straight line: a start value and a slope. For a refresher, see Functions and change in the mathematics course.
- A 95% interval, from The result and its uncertainty.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m05, from Difference-in-differences, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.
A pass mark at 60
Section titled “A pass mark at 60”A school gives a prize to every student who scores 60 or more on an exam. The prize is a summer course. Did the summer course help the students next year?
Comparing all prize winners with everyone else does not answer it. Winners are better students: they would do better next year anyway.
But look at the students who scored 59 and those who scored 61. They are almost the same students. One point is luck: one question read well or badly. Yet only the 61s got the summer course. If the 61s do much better next year than the 59s, the summer course is the likely reason.
The rule made a fair comparison by accident, right at the line.
The problem: Gold customers spend more, but why?
Section titled “The problem: Gold customers spend more, but why?”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield has a loyalty tier. Customers who spent 500 dollars or more in 2024 became Gold on 1 January 2025. Gold gives free returns and 5% back in points. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., the product manager, asks: “Does Gold make customers spend more?”
The simulator has this as the loyalty scenario, with 50,000 customers. All of it is synthetic. Gold customers spent 318.05 dollars in the first half of 2025, and the others 126.32 dollars. The naive answer is +191.73 dollars.
The simulator knows the truth: at the threshold, Gold adds 45 dollars in six months. The naive answer is more than four times too large. Gold customers spent more in 2024 too: that is how they became Gold. They are bigger customers, with or without the tier.
The idea: compare just below with just above
Section titled “The idea: compare just below with just above”Regression discontinuity compares the people on the two sides of a threshold. A discontinuity is a jump: a place where a line breaks. The idea:
- The running variable decides who gets the change. Here: 2024 spending. Larkfield must measure it before the change.
- The threshold is the value where the rule starts: 500 dollars.
- Without the tier, 2025 spending would rise smoothly with 2024 spending. There is no reason for a jump at exactly 500.
- So a jump at 500 is the effect of the tier, for customers near 500.
Customers far from the line are not alike: a 200-dollar customer and an 800-dollar customer differ in many ways. So the method uses only customers near the line. It fits a straight line on each side and reads the jump where the two lines meet the threshold.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Six customers near the line. The numbers are made up for this example.
| 2024 spending | Gold? | 2025 spending (six months) |
|---|---|---|
| 470 | no | 182 |
| 480 | no | 186 |
| 490 | no | 190 |
| 510 | yes | 240 |
| 520 | yes | 244 |
| 530 | yes | 248 |
On each side, 2025 spending rises by 4 dollars for every 10 dollars of 2024 spending. Extend each side’s line to 500:
- below: 190 + 4 = 194 dollars at 500;
- above: 240 − 4 = 236 dollars at 500.
The jump is 236 − 194 = +42 dollars: the estimated effect of Gold at the threshold. xp rdd does the same thing with thousands of customers and a regression on each side.
The bandwidth
Section titled “The bandwidth”How near is “near”? The bandwidth is how far from the threshold you look, in dollars, on each side. A bandwidth of 100 uses customers who spent 400 to 600 dollars.
- A narrow bandwidth uses customers who are most alike. But it has fewer of them, so the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI is wide.
- A wide bandwidth has more customers and a narrower interval. But a straight line may not fit far from the line, and the estimate can drift.
There is no perfect choice. Try several bandwidths. If the estimate stays about the same, you can trust it more.
Larkfield: the jump at 500 dollars
Section titled “Larkfield: the jump at 500 dollars”-
Make sure that your project is at
end-of-m05, with.venvactive. Simulate the loyalty tier.Terminal window sim run --scenario loyaltyScenario loyalty (seed 40): The Gold tier from 500 dollars of 2024 spending: spending in the first half of 2025 (no test).Wrote runs/loyalty-s40: customers (50,000 rows). Synthetic data. 0.6 s -
Run regression discontinuity, then compare with the truth.
Terminal window xp rddxp checkRegression discontinuity at spend_2024 = 500, outcome spend_2025_h1.Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it.Bandwidth 100: jump +17.54 dollars (95% interval +13.84 to +21.25), 10,125 customers.bandwidth 50: +17.76 (+12.31 to +23.22), 4,941 customersbandwidth 100: +17.54 (+13.84 to +21.25), 10,125 customersbandwidth 150: +23.83 (+20.92 to +26.73), 15,681 customersbandwidth 200: +25.49 (+23.02 to +27.96), 21,644 customersbandwidth 300: +28.75 (+26.75 to +30.76), 34,690 customersNaive (all Gold against all others): +191.73 dollars.Run loyalty-s40: the last analysis was `xp rdd`.True effect at the threshold (from the simulator): +45.00 dollars.Your estimate: +17.54 (+13.84 to +21.25): the interval does NOT contain it. Naive answer: +191.73.Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.
The estimate is +17.54 dollars, against a true +45. It is much better than the naive +191.73, and still badly wrong. Its interval does not contain the truth. And every bandwidth gives a different answer, from +17.76 to +28.75.
xp check printed the reason, because this is a practice problem we made on purpose. But the data already warned us, in the line before the estimate.
Check for people who game the threshold
Section titled “Check for people who game the threshold”The method has one strong assumption: nobody can choose their side of the line precisely. If customers can push themselves just over 500, the customers just above are not like those just below.
Larkfield’s customers could. A customer at 480 dollars in December could add a 25-dollar order and become Gold. The simulator planted this: 1,078 bargain hunters near 500 added a small order to reach Gold. They spend less afterwards, and they all sit between 500 and 535 dollars.
The check is simple: count the customers on each side of the line. Without gaming, the counts should change smoothly. xp rdd counted the 20 dollars on each side: 520 customers below, 1,534 above. With a smooth count, the two numbers would be close. The p-value of 1.49e-115 says that this gap is far too big for chance.
Text description of the chart
A histogram of 20 bands of 10 dollars, from 400 to 600 dollars of 2024 spending. From 400 to 500, the counts fall from 716 and 747 to 258 and 262 customers. Just above 500, the count jumps. There are 800 customers in 500 to 510 and 734 in 510 to 520. There are 755 in 520 to 530 and 593 in 530 to 540. From 540 to 600, the counts are between 326 and 435.
This check is often called a density test: it looks for a jump in the number of people, not in their outcome. A pile-up just above a line is the classic sign of gaming.
See the jump
Section titled “See the jump”The next chart shows the average 2025 spending in each 10-dollar band, with the two straight lines of xp rdd.
- Mean in each 10-dollar band
- Line below 500
- Line above, all customers
- Line above, without 500 to 540
Text description of the chart
A chart from 400 to 600 dollars of 2024 spending. Grey dots show the mean 2025 spending in each 10-dollar band. Below the line, they rise from 172.77 dollars at 405 to 203.63 at 495. Just above it, they are flat: 232.35 at 505, 233.50 at 515, 232.65 at 525 and 243.01 at 535. Further right, they rise from 265.12 at 545 to 281.91 at 595. A blue line fits the customers below 500 and reaches 206.87 dollars at 500. An orange line fits all the customers from 500 to 600. It starts at 224.41 at 500, a jump of +17.5, and rises steeply. A dashed aqua line fits only the customers from 540 to 600: it starts at 252.52 at 500, a jump of +45.7.
Look at the dots from 505 to 535. They are flat and low, below where the right-hand side should be. That is the bargain hunters. The orange line tries to fit them, so it starts too low at 500, and it must climb steeply to reach the dots further right.
The donut
Section titled “The donut”The fix is to leave out the customers closest to the line, on the side where they piled up. This is a donut: a hole in the middle of the data. With a 40-dollar donut, xp rdd uses customers from 400 to 500 and from 540 to 600.
xp rdd --donut 40xp checkRegression discontinuity at spend_2024 = 500, outcome spend_2025_h1. Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it. Bandwidth 100, donut 40: jump +45.66 dollars (95% interval +37.55 to +53.76), 7,243 customers. bandwidth 50: +50.50 (-15.66 to +116.66), 2,059 customers bandwidth 100: +45.66 (+37.55 to +53.76), 7,243 customers bandwidth 150: +49.97 (+45.04 to +54.90), 12,799 customers bandwidth 200: +45.96 (+42.15 to +49.78), 18,762 customers bandwidth 300: +46.00 (+43.17 to +48.83), 31,808 customers Naive (all Gold against all others): +191.73 dollars.Run loyalty-s40: the last analysis was `xp rdd`.True effect at the threshold (from the simulator): +45.00 dollars.Your estimate: +45.66 (+37.55 to +53.76): the interval contains it. Naive answer: +191.73.Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.The estimate is now +45.66 dollars (+37.55 to +53.76), and the interval contains the truth. The bandwidths now agree with each other too: from +45.66 to +49.97, except bandwidth 50.
Choose the bandwidth
Section titled “Choose the bandwidth”The chart puts both tables side by side.
Text description of the chart
Two panels with a vertical line at the true effect, +45 dollars. No donut: bandwidth 50 +17.8 (+12.3 to +23.2). Bandwidth 100 +17.5 (+13.8 to +21.2). Bandwidth 150 +23.8 (+20.9 to +26.7). Bandwidth 200 +25.5 (+23.0 to +28.0). Bandwidth 300 +28.8 (+26.7 to +30.8). All are far below +45. Donut of 40 dollars: bandwidth 50 +50.5 (−15.7 to +116.7). Bandwidth 100 +45.7 (+37.6 to +53.8). Bandwidth 150 +50.0 (+45.0 to +54.9). Bandwidth 200 +46.0 (+42.1 to +49.8). Bandwidth 300 +46.0 (+43.2 to +48.8).
Without the donut, the estimate grows with the bandwidth. A wider window adds normal customers, so the bargain hunters weigh less. When estimates move with the bandwidth, something is wrong.
With the donut, bandwidth 50 keeps only the customers from 450 to 500 and from 540 to 550: 2,059 customers. Its interval is 132 dollars wide. Bandwidths of 100 to 300 agree. Larkfield would report bandwidth 100 with the donut, and show the others as a check.
Over 50 seeds, the course measured both versions. Without the donut, the mean estimate was +17.3 dollars, and the interval never contained the truth (0%). With the 40-dollar donut, the mean was +44.5, and the interval contained the truth in 92% of the runs. That is close to the promised 95%.
A donut has a cost: you need to know how far the gaming goes. Here the density chart shows it: the pile-up ends at about 540. Too small a donut leaves gamers in; too large a donut leaves few customers near the line.
When to use it, when not, and what it costs
Section titled “When to use it, when not, and what it costs”Use regression discontinuity when:
- a rule gives a change to everyone above (or below) a number: a loyalty tier, a credit score, an age limit, a test score;
- the number was measured before the change, and people could not set it precisely;
- you have many people near the line.
Do not trust it when:
- the density check shows a pile-up at the line, and you cannot tell how far it goes;
- something else also changes at the same threshold. If Gold customers also got a new account manager at 500 dollars, the jump mixes both;
- the estimate moves a lot with the bandwidth.
Remember what it measures: the effect at the threshold, for customers near 500 dollars. It says nothing sure about Gold’s effect on a 2,000-dollar customer. In the simulator, the effect grows with spending, so a customer far above the line gets more than 45 dollars.
It costs little: data you already have, and one regression per side. The real cost is the check for gaming. Sometimes the fix is in the rule itself: Larkfield could set the threshold on spending before customers knew about the tier.
Guided practice: see the jump and the pile-up
Section titled “Guided practice: see the jump and the pile-up”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The notebook simulates the loyalty tier with the project’s own code, so the numbers match
xp rdd. - In section 1, predict how far the naive answer is from the truth. Then compute it. You see
naive +191.73. - In section 2, compute the six-customer example. You see the jump
+42. - In section 3, fit a line on each side with a bandwidth of 100. You see
jump +17.54. - In section 4, count the customers in the 20 dollars on each side. You see
520and1534. - In section 5, refit with a 40-dollar donut. Put the jump in
jump_donut. Run the check cell. It printsCheck passed.
Show the answer to step 7
d = near[(near.spend_2024 < 500) | (near.spend_2024 >= 540)]m = smf.ols("spend_2025_h1 ~ above * xc", data=d).fit(cov_type="HC1")jump_donut = round(float(m.params["above"]), 2)print(jump_donut)The value is 45.66: the same as xp rdd --donut 40.
Your turn: how big should the donut be?
Section titled “Your turn: how big should the donut be?”Mateo, the growth analyst, asks: “Why 40 dollars? Would 20 do?” Run xp rdd --donut 20 and xp check. Before you run it, predict: will the estimate be closer to +17.5 or to +45? Use the histogram above to explain the result.
Show the answer
xp rdd --donut 20xp checkRegression discontinuity at spend_2024 = 500, outcome spend_2025_h1. Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it. Bandwidth 100, donut 20: jump +13.21 dollars (95% interval +8.25 to +18.17), 8,591 customers. bandwidth 50: -14.46 (-24.73 to -4.19), 3,407 customers bandwidth 100: +13.21 (+8.25 to +18.17), 8,591 customers bandwidth 150: +26.44 (+22.79 to +30.10), 14,147 customers bandwidth 200: +29.09 (+26.06 to +32.12), 20,110 customers bandwidth 300: +33.46 (+31.07 to +35.85), 33,156 customers Naive (all Gold against all others): +191.73 dollars.Run loyalty-s40: the last analysis was `xp rdd`.True effect at the threshold (from the simulator): +45.00 dollars.Your estimate: +13.21 (+8.25 to +18.17): the interval does NOT contain it. Naive answer: +191.73.Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.The estimate is +13.21 dollars: even further from the truth than without a donut (+17.54). A 20-dollar donut removes the customers from 500 to 520. But the bargain hunters sit from 500 to 535, and the histogram shows the pile-up in the bands from 520 to 540 too. The ones that stay are now the first customers above the line, so they pull its start down even more. The bandwidths also disagree, from −14.46 to +33.46: a clear warning. The donut must cover the whole pile-up. Here that is about 40 dollars.
Common mistake
“The bandwidth that gives the answer we like”
Symptom: a team runs regression discontinuity on a loyalty tier. Bandwidth 100 gives +17.5 dollars, bandwidth 300 gives +28.8. The team reports +28.8, “the most precise one”. Its interval is narrow and does not contain the true effect.
Cause: customers gamed the threshold, so the customers just above it spend less than they should. Every bandwidth gives a wrong answer. The wide one is only less wrong, because it dilutes the gamers. A narrow interval says nothing about this bias. The density check showed the pile-up, and nobody read it.
Fix: run the density check first. If people pile up above the line, find how far the pile-up goes and leave it out with a donut. Then try several bandwidths: the estimates should agree. Choose the bandwidth before you see the estimates, and report the others as a check.
Check your understanding
Customers at 480 and 490 dollars spent 200 and 205 dollars later. Customers at 510 and 520 spent 245 and 250. Lines on each side have a slope of 0.5 dollars for each dollar. What is the jump at 500?
Show the answer
Below the line: 205 + 0.5 × 10 = 210 dollars at 500. Above the line: 245 − 0.5 × 10 = 240 dollars at 500. The jump is 240 − 210 = +30 dollars: the estimated effect of the tier for customers near 500.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why is the naive comparison of all Gold customers with all other customers (+191.73 dollars) so far from the true effect at 500 (+45 dollars)?
Show the answer
Gold customers became Gold because they spent more in 2024. They are bigger customers, and they would spend more in 2025 even without the tier. The naive answer mixes that difference with the tier’s effect. Near the threshold, the customers on both sides are alike, so the jump there shows only the tier’s effect.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
State the assumption that regression discontinuity needs, and one sign in the data that it is false.
Show the answer
It assumes that people cannot choose their side of the threshold precisely. Then the people just above and just below are alike. It also assumes that nothing else changes at the threshold. One sign that it is false is a pile-up just above the line in the density check. Here there were 1,534 customers above against 520 below. Another sign: an estimate that changes a lot with the bandwidth.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Regression discontinuity compares people just above and just below a threshold of a running variable. The jump at the line is the effect, for people near the line.
- The naive Gold against the rest: +191.73 dollars. The true effect at 500 dollars: +45.
- The bandwidth is how far from the line you look. Narrow: alike but few. Wide: many, but the straight lines may not fit.
- The main danger is gaming. A density check found 1,534 customers in the 20 dollars above the line against 520 below: the 1,078 bargain hunters we planted. They pulled the estimate down to +17.5 dollars (0% coverage over 50 seeds).
- A 40-dollar donut left them out: +45.7 dollars (+37.6 to +53.8), and 92% coverage over 50 seeds.
Next, the hardest case: no comparison group and no threshold, only one series and a date.
Next: Interrupted time series