Skip to content
Français

Regression discontinuity

  • Lesson 19
  • 35 min
  • Module 5: When you cannot randomise: natural experiments

Where we are

Our last two methods compared regions over time. Some changes start at a threshold instead: Larkfield's loyalty tier starts at 500 dollars of spending. In this lesson, we compare customers just above and just below the threshold with regression discontinuity.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Estimate the effect of a rule that starts at a threshold. Compare the people just above the threshold with the people just below it. Choose how far from the line to look. Check whether people moved themselves over the line.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate the loyalty tier with the project’s own code and draw the jump at 500 dollars. You fit a line on each side, find the customers who gamed the threshold, and leave them out with a donut. Its setup cells install the packages, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.

A school gives a prize to every student who scores 60 or more on an exam. The prize is a summer course. Did the summer course help the students next year?

Comparing all prize winners with everyone else does not answer it. Winners are better students: they would do better next year anyway.

But look at the students who scored 59 and those who scored 61. They are almost the same students. One point is luck: one question read well or badly. Yet only the 61s got the summer course. If the 61s do much better next year than the 59s, the summer course is the likely reason.

The rule made a fair comparison by accident, right at the line.

The problem: Gold customers spend more, but why?

Section titled “The problem: Gold customers spend more, but why?”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield has a loyalty tier. Customers who spent 500 dollars or more in 2024 became Gold on 1 January 2025. Gold gives free returns and 5% back in points. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., the product manager, asks: “Does Gold make customers spend more?”

The simulator has this as the loyalty scenario, with 50,000 customers. All of it is synthetic. Gold customers spent 318.05 dollars in the first half of 2025, and the others 126.32 dollars. The naive answer is +191.73 dollars.

The simulator knows the truth: at the threshold, Gold adds 45 dollars in six months. The naive answer is more than four times too large. Gold customers spent more in 2024 too: that is how they became Gold. They are bigger customers, with or without the tier.

The idea: compare just below with just above

Section titled “The idea: compare just below with just above”

Regression discontinuity compares the people on the two sides of a threshold. A discontinuity is a jump: a place where a line breaks. The idea:

  • The running variable decides who gets the change. Here: 2024 spending. Larkfield must measure it before the change.
  • The threshold is the value where the rule starts: 500 dollars.
  • Without the tier, 2025 spending would rise smoothly with 2024 spending. There is no reason for a jump at exactly 500.
  • So a jump at 500 is the effect of the tier, for customers near 500.

Customers far from the line are not alike: a 200-dollar customer and an 800-dollar customer differ in many ways. So the method uses only customers near the line. It fits a straight line on each side and reads the jump where the two lines meet the threshold.

Two boxes side by side: customers who spent 480 to 499 dollars in 2024 are not Gold; customers who spent 500 to 519 dollars are Gold from 1 January 2025. Between them, the line at 500 dollars. Below, a green box: almost the same customers, a few dollars apart, and the only big difference is the tier. The jump in 2025 spending at 500 is the tier's effect. Below that, a red box: some customers at 470 to 499 added a small order in December to reach Gold, so they moved over the line. An arrow leads to an orange box: they now sit at 500 to 535 and spend less later; leave that band out, a donut.
The idea of regression discontinuity, and its main danger. Customers on both sides of 500 dollars are alike, so the jump at the line is the effect. But if people can move themselves over the line, the customers just above it are no longer like those just below.

Six customers near the line. The numbers are made up for this example.

2024 spending Gold? 2025 spending (six months)
470 no 182
480 no 186
490 no 190
510 yes 240
520 yes 244
530 yes 248

On each side, 2025 spending rises by 4 dollars for every 10 dollars of 2024 spending. Extend each side’s line to 500:

  • below: 190 + 4 = 194 dollars at 500;
  • above: 240 − 4 = 236 dollars at 500.

The jump is 236 − 194 = +42 dollars: the estimated effect of Gold at the threshold. xp rdd does the same thing with thousands of customers and a regression on each side.

How near is “near”? The bandwidth is how far from the threshold you look, in dollars, on each side. A bandwidth of 100 uses customers who spent 400 to 600 dollars.

  • A narrow bandwidth uses customers who are most alike. But it has fewer of them, so the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI is wide.
  • A wide bandwidth has more customers and a narrower interval. But a straight line may not fit far from the line, and the estimate can drift.

There is no perfect choice. Try several bandwidths. If the estimate stays about the same, you can trust it more.

  1. Make sure that your project is at end-of-m05, with .venv active. Simulate the loyalty tier.

    Terminal window
    sim run --scenario loyalty
    Scenario loyalty (seed 40): The Gold tier from 500 dollars of 2024 spending: spending in the first half of 2025 (no test).
    Wrote runs/loyalty-s40: customers (50,000 rows). Synthetic data. 0.6 s
  2. Run regression discontinuity, then compare with the truth.

    Terminal window
    xp rdd
    xp check
    Regression discontinuity at spend_2024 = 500, outcome spend_2025_h1.
    Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it.
    Bandwidth 100: jump +17.54 dollars (95% interval +13.84 to +21.25), 10,125 customers.
    bandwidth 50: +17.76 (+12.31 to +23.22), 4,941 customers
    bandwidth 100: +17.54 (+13.84 to +21.25), 10,125 customers
    bandwidth 150: +23.83 (+20.92 to +26.73), 15,681 customers
    bandwidth 200: +25.49 (+23.02 to +27.96), 21,644 customers
    bandwidth 300: +28.75 (+26.75 to +30.76), 34,690 customers
    Naive (all Gold against all others): +191.73 dollars.
    Run loyalty-s40: the last analysis was `xp rdd`.
    True effect at the threshold (from the simulator): +45.00 dollars.
    Your estimate: +17.54 (+13.84 to +21.25): the interval does NOT contain it. Naive answer: +191.73.
    Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.

The estimate is +17.54 dollars, against a true +45. It is much better than the naive +191.73, and still badly wrong. Its interval does not contain the truth. And every bandwidth gives a different answer, from +17.76 to +28.75.

xp check printed the reason, because this is a practice problem we made on purpose. But the data already warned us, in the line before the estimate.

The method has one strong assumption: nobody can choose their side of the line precisely. If customers can push themselves just over 500, the customers just above are not like those just below.

Larkfield’s customers could. A customer at 480 dollars in December could add a 25-dollar order and become Gold. The simulator planted this: 1,078 bargain hunters near 500 added a small order to reach Gold. They spend less afterwards, and they all sit between 500 and 535 dollars.

The check is simple: count the customers on each side of the line. Without gaming, the counts should change smoothly. xp rdd counted the 20 dollars on each side: 520 customers below, 1,534 above. With a smooth count, the two numbers would be close. The p-value of 1.49e-115 says that this gap is far too big for chance.

Customers by 2024 spending, near the Gold threshold. The count falls smoothly toward 500 dollars, then triples just above it. The extra customers are the bargain hunters who added an order to reach Gold (a practice problem we made on purpose). Synthetic data, seed 40.
Text description of the chart

A histogram of 20 bands of 10 dollars, from 400 to 600 dollars of 2024 spending. From 400 to 500, the counts fall from 716 and 747 to 258 and 262 customers. Just above 500, the count jumps. There are 800 customers in 500 to 510 and 734 in 510 to 520. There are 755 in 520 to 530 and 593 in 530 to 540. From 540 to 600, the counts are between 326 and 435.

This check is often called a density test: it looks for a jump in the number of people, not in their outcome. A pile-up just above a line is the classic sign of gaming.

The next chart shows the average 2025 spending in each 10-dollar band, with the two straight lines of xp rdd.

  • Mean in each 10-dollar band
  • Line below 500
  • Line above, all customers
  • Line above, without 500 to 540
The jump at the Gold threshold. With all customers, the line above 500 starts low, because the bargain hunters sit just above the line and spend less. It gives a jump of +17.5 dollars. Without the band from 500 to 540 (a donut), the line above starts at the right height: +45.7 dollars, against a true +45. Synthetic data, seed 40.
Text description of the chart

A chart from 400 to 600 dollars of 2024 spending. Grey dots show the mean 2025 spending in each 10-dollar band. Below the line, they rise from 172.77 dollars at 405 to 203.63 at 495. Just above it, they are flat: 232.35 at 505, 233.50 at 515, 232.65 at 525 and 243.01 at 535. Further right, they rise from 265.12 at 545 to 281.91 at 595. A blue line fits the customers below 500 and reaches 206.87 dollars at 500. An orange line fits all the customers from 500 to 600. It starts at 224.41 at 500, a jump of +17.5, and rises steeply. A dashed aqua line fits only the customers from 540 to 600: it starts at 252.52 at 500, a jump of +45.7.

Look at the dots from 505 to 535. They are flat and low, below where the right-hand side should be. That is the bargain hunters. The orange line tries to fit them, so it starts too low at 500, and it must climb steeply to reach the dots further right.

The fix is to leave out the customers closest to the line, on the side where they piled up. This is a donut: a hole in the middle of the data. With a 40-dollar donut, xp rdd uses customers from 400 to 500 and from 540 to 600.

Terminal window
xp rdd --donut 40
xp check
Regression discontinuity at spend_2024 = 500, outcome spend_2025_h1.
Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it.
Bandwidth 100, donut 40: jump +45.66 dollars (95% interval +37.55 to +53.76), 7,243 customers.
bandwidth 50: +50.50 (-15.66 to +116.66), 2,059 customers
bandwidth 100: +45.66 (+37.55 to +53.76), 7,243 customers
bandwidth 150: +49.97 (+45.04 to +54.90), 12,799 customers
bandwidth 200: +45.96 (+42.15 to +49.78), 18,762 customers
bandwidth 300: +46.00 (+43.17 to +48.83), 31,808 customers
Naive (all Gold against all others): +191.73 dollars.
Run loyalty-s40: the last analysis was `xp rdd`.
True effect at the threshold (from the simulator): +45.00 dollars.
Your estimate: +45.66 (+37.55 to +53.76): the interval contains it. Naive answer: +191.73.
Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.

The estimate is now +45.66 dollars (+37.55 to +53.76), and the interval contains the truth. The bandwidths now agree with each other too: from +45.66 to +49.97, except bandwidth 50.

The chart puts both tables side by side.

The jump at 500 dollars for five bandwidths, without and with the 40-dollar donut. Without the donut, every bandwidth misses the truth (+45) by far. With it, every estimate is near the truth. Bandwidth 50 has too few customers left, so its interval is very wide.
Text description of the chart

Two panels with a vertical line at the true effect, +45 dollars. No donut: bandwidth 50 +17.8 (+12.3 to +23.2). Bandwidth 100 +17.5 (+13.8 to +21.2). Bandwidth 150 +23.8 (+20.9 to +26.7). Bandwidth 200 +25.5 (+23.0 to +28.0). Bandwidth 300 +28.8 (+26.7 to +30.8). All are far below +45. Donut of 40 dollars: bandwidth 50 +50.5 (−15.7 to +116.7). Bandwidth 100 +45.7 (+37.6 to +53.8). Bandwidth 150 +50.0 (+45.0 to +54.9). Bandwidth 200 +46.0 (+42.1 to +49.8). Bandwidth 300 +46.0 (+43.2 to +48.8).

Without the donut, the estimate grows with the bandwidth. A wider window adds normal customers, so the bargain hunters weigh less. When estimates move with the bandwidth, something is wrong.

With the donut, bandwidth 50 keeps only the customers from 450 to 500 and from 540 to 550: 2,059 customers. Its interval is 132 dollars wide. Bandwidths of 100 to 300 agree. Larkfield would report bandwidth 100 with the donut, and show the others as a check.

Over 50 seeds, the course measured both versions. Without the donut, the mean estimate was +17.3 dollars, and the interval never contained the truth (0%). With the 40-dollar donut, the mean was +44.5, and the interval contained the truth in 92% of the runs. That is close to the promised 95%.

A donut has a cost: you need to know how far the gaming goes. Here the density chart shows it: the pile-up ends at about 540. Too small a donut leaves gamers in; too large a donut leaves few customers near the line.

When to use it, when not, and what it costs

Section titled “When to use it, when not, and what it costs”

Use regression discontinuity when:

  • a rule gives a change to everyone above (or below) a number: a loyalty tier, a credit score, an age limit, a test score;
  • the number was measured before the change, and people could not set it precisely;
  • you have many people near the line.

Do not trust it when:

  • the density check shows a pile-up at the line, and you cannot tell how far it goes;
  • something else also changes at the same threshold. If Gold customers also got a new account manager at 500 dollars, the jump mixes both;
  • the estimate moves a lot with the bandwidth.

Remember what it measures: the effect at the threshold, for customers near 500 dollars. It says nothing sure about Gold’s effect on a 2,000-dollar customer. In the simulator, the effect grows with spending, so a customer far above the line gets more than 45 dollars.

It costs little: data you already have, and one regression per side. The real cost is the check for gaming. Sometimes the fix is in the rule itself: Larkfield could set the threshold on spending before customers knew about the tier.

Guided practice: see the jump and the pile-up

Section titled “Guided practice: see the jump and the pile-up”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The notebook simulates the loyalty tier with the project’s own code, so the numbers match xp rdd.
  3. In section 1, predict how far the naive answer is from the truth. Then compute it. You see naive +191.73.
  4. In section 2, compute the six-customer example. You see the jump +42.
  5. In section 3, fit a line on each side with a bandwidth of 100. You see jump +17.54.
  6. In section 4, count the customers in the 20 dollars on each side. You see 520 and 1534.
  7. In section 5, refit with a 40-dollar donut. Put the jump in jump_donut. Run the check cell. It prints Check passed.
Show the answer to step 7
d = near[(near.spend_2024 < 500) | (near.spend_2024 >= 540)]
m = smf.ols("spend_2025_h1 ~ above * xc", data=d).fit(cov_type="HC1")
jump_donut = round(float(m.params["above"]), 2)
print(jump_donut)

The value is 45.66: the same as xp rdd --donut 40.

Mateo, the growth analyst, asks: “Why 40 dollars? Would 20 do?” Run xp rdd --donut 20 and xp check. Before you run it, predict: will the estimate be closer to +17.5 or to +45? Use the histogram above to explain the result.

Show the answer
Terminal window
xp rdd --donut 20
xp check
Regression discontinuity at spend_2024 = 500, outcome spend_2025_h1.
Density check: 520 customers in the 20 dollars below, 1534 in the 20 above (p = 1.49e-115). Bunching above the threshold: people may game it.
Bandwidth 100, donut 20: jump +13.21 dollars (95% interval +8.25 to +18.17), 8,591 customers.
bandwidth 50: -14.46 (-24.73 to -4.19), 3,407 customers
bandwidth 100: +13.21 (+8.25 to +18.17), 8,591 customers
bandwidth 150: +26.44 (+22.79 to +30.10), 14,147 customers
bandwidth 200: +29.09 (+26.06 to +32.12), 20,110 customers
bandwidth 300: +33.46 (+31.07 to +35.85), 33,156 customers
Naive (all Gold against all others): +191.73 dollars.
Run loyalty-s40: the last analysis was `xp rdd`.
True effect at the threshold (from the simulator): +45.00 dollars.
Your estimate: +13.21 (+8.25 to +18.17): the interval does NOT contain it. Naive answer: +191.73.
Planted: Bargain hunters near 500 added a small order to reach Gold; they spend 60 dollars less later and sit between 500 and 535.

The estimate is +13.21 dollars: even further from the truth than without a donut (+17.54). A 20-dollar donut removes the customers from 500 to 520. But the bargain hunters sit from 500 to 535, and the histogram shows the pile-up in the bands from 520 to 540 too. The ones that stay are now the first customers above the line, so they pull its start down even more. The bandwidths also disagree, from −14.46 to +33.46: a clear warning. The donut must cover the whole pile-up. Here that is about 40 dollars.

Common mistake

“The bandwidth that gives the answer we like”

Symptom: a team runs regression discontinuity on a loyalty tier. Bandwidth 100 gives +17.5 dollars, bandwidth 300 gives +28.8. The team reports +28.8, “the most precise one”. Its interval is narrow and does not contain the true effect.

Cause: customers gamed the threshold, so the customers just above it spend less than they should. Every bandwidth gives a wrong answer. The wide one is only less wrong, because it dilutes the gamers. A narrow interval says nothing about this bias. The density check showed the pile-up, and nobody read it.

Fix: run the density check first. If people pile up above the line, find how far the pile-up goes and leave it out with a donut. Then try several bandwidths: the estimates should agree. Choose the bandwidth before you see the estimates, and report the others as a check.

Check your understanding

Customers at 480 and 490 dollars spent 200 and 205 dollars later. Customers at 510 and 520 spent 245 and 250. Lines on each side have a slope of 0.5 dollars for each dollar. What is the jump at 500?

Show the answer

Below the line: 205 + 0.5 × 10 = 210 dollars at 500. Above the line: 245 − 0.5 × 10 = 240 dollars at 500. The jump is 240 − 210 = +30 dollars: the estimated effect of the tier for customers near 500.

Check your understanding

Why is the naive comparison of all Gold customers with all other customers (+191.73 dollars) so far from the true effect at 500 (+45 dollars)?

Show the answer

Gold customers became Gold because they spent more in 2024. They are bigger customers, and they would spend more in 2025 even without the tier. The naive answer mixes that difference with the tier’s effect. Near the threshold, the customers on both sides are alike, so the jump there shows only the tier’s effect.

Check your understanding

State the assumption that regression discontinuity needs, and one sign in the data that it is false.

Show the answer

It assumes that people cannot choose their side of the threshold precisely. Then the people just above and just below are alike. It also assumes that nothing else changes at the threshold. One sign that it is false is a pile-up just above the line in the density check. Here there were 1,534 customers above against 520 below. Another sign: an estimate that changes a lot with the bandwidth.

Recap

  • Regression discontinuity compares people just above and just below a threshold of a running variable. The jump at the line is the effect, for people near the line.
  • The naive Gold against the rest: +191.73 dollars. The true effect at 500 dollars: +45.
  • The bandwidth is how far from the line you look. Narrow: alike but few. Wide: many, but the straight lines may not fit.
  • The main danger is gaming. A density check found 1,534 customers in the 20 dollars above the line against 520 below: the 1,078 bargain hunters we planted. They pulled the estimate down to +17.5 dollars (0% coverage over 50 seeds).
  • A 40-dollar donut left them out: +45.7 dollars (+37.6 to +53.8), and 92% coverage over 50 seeds.

Next, the hardest case: no comparison group and no threshold, only one series and a date.

Next: Interrupted time series