Interrupted time series
- Lesson 20
- 35 min
- Module 5: When you cannot randomise: natural experiments
Where we are
Sometimes there is no comparison group and no threshold: only one series and one change date. In this lesson, we go back to the recommendation block's launch and use the trend, the season and the holidays before the change to estimate what would have happened.
Lesson 20 of 30 · Experiments and Causal Impact: Did Our Change Work?Interrupted time series
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Estimate the effect of a change when you have only one series and the date of the change. Model what the series did before, extend it past the change as the counterfactual, and read the step. Know what the model must contain, and why this is the weakest of the module’s methods.
NotebookDo this lesson in a notebook
In the notebook, you fit the trend and a step on a tiny series. Then you do the same on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s daily orders around the block’s launch. You add the weekday, the holidays and the sale one by one and watch the estimate move. Its setup cells install the packages, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The block’s launch and the naive +8.3%, from The feature that “raised sales”.
- Difference-in-differences, from Difference-in-differences.
- Trend and season in a series. For a refresher, see Trend, season and noise in the forecasting course.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m05, from Difference-in-differences, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.
A café’s new opening hours
Section titled “A café’s new opening hours”A café opens one hour earlier from Monday 1 March. There is only one café, so there is no other café to compare with. There is no threshold either.
But the owner has a notebook with the number of customers on every day for a year. In February, the café gained about 2 customers a day, every week. Mondays are always quiet, and Saturdays are busy. In March, she draws the February line forward, with the same slope and the same weekly pattern. That is her guess of March without the early hour. Then she compares it with the real March.
She used the café’s own past as the comparison.
The problem: no comparison group for the block
Section titled “The problem: no comparison group for the block”In Module 1, TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s dashboard said that sales rose 8.3% after the recommendation block’s launch on 6 October 2025. That number is a practice problem we made on purpose. The simulator knows that the block’s true effect was only +2.0% of orders.
The block went live for everyone on the same day. There is no Ontario and no synthetic Quebec: every customer got it. There is no threshold. There is one series of daily orders, and one date.
The same weeks also had Thanksgiving (13 October) and an autumn sale (9 to 19 October). The naive comparison gives all of that to the block.
The idea: the past, extended, is the counterfactual
Section titled “The idea: the past, extended, is the counterfactual”An interrupted time series is a series with a change in the middle: the change “interrupts” it. The method has two parts:
- Model the series before the change: its trend, its weekly pattern, and the known events, such as holidays and sales.
- Extend the model past the change. It says what the days after would have been without the change. The gap between the real days and the extended model is the effect.
In practice, both parts are one regression on all the days. It has the trend, the weekday, the events and one extra yes-or-no term: “after the change”. That term adds a step on the change date. Its size is the effect. This form is often called segmented regression.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here is a made-up series of nine days. The trend adds 2 orders a day, and a change on day 6 adds a step.
| Day | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Orders | 102 | 104 | 106 | 108 | 110 | 122 | 124 | 126 | 128 |
| Trend from days 1 to 5, extended | 102 | 104 | 106 | 108 | 110 | 112 | 114 | 116 | 118 |
| Gap | 0 | 0 | 0 | 0 | 0 | 10 | 10 | 10 | 10 |
Days 1 to 5 fit a line: 100 + 2 × day. Extended, the line says 112 on day 6. The real day 6 had 122. The step is +10 orders a day, every day after the change.
- Orders
- The trend before, extended
Text description of the chart
A line chart over days 1 to 9. The orders rise by 2 a day: 102, 104, 106, 108 and 110. They jump to 122 on day 6 and keep rising by 2: 124, 126 and 128. A dashed line continues the trend of days 1 to 5: 112, 114, 116 and 118 on days 6 to 9. The real orders stay 10 above the dashed line from day 6.
What it assumes
Section titled “What it assumes”The method assumes that nothing else changed at the same date that the model does not know about. And it assumes that the model before the change, extended, is right after it. Both are strong. A comparison group protects you from surprises that hit both groups. Here, nothing does. If a competitor closed its shop the same week, the step would include it.
Larkfield: the block’s launch
Section titled “Larkfield: the block’s launch”-
Make sure that your project is at
end-of-m05, with.venvactive. Simulate the launch again.Terminal window sim run --scenario launchScenario launch (seed 40): Daily orders and sales around the recommendation block's launch (6 October 2025): no test, one series.Wrote runs/launch-s40: daily (427 rows), counterfactual (427 rows). Synthetic data. 0.1 s -
Run the interrupted time series, then compare with the truth.
Terminal window xp itsxp checkInterrupted time series (trend, weekday, holiday windows and the sale), 84 days (8 weeks before, 4 after), launch 2025-10-06:Effect of the launch on daily orders: +3.5% (95% interval +1.3% to +5.8%; errors robust to 7 days of autocorrelation).Run launch-s40: the last analysis was `xp its`.True effect (from the simulator): +2.0% on daily orders.Your estimate: +3.5% (+1.3% to +5.8%): the interval contains it.
From the naive +8.3% to +3.5%, against a true +2.0%, and the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains the truth. The model used 8 weeks before the launch and the 4 weeks after it.
What the model knows
Section titled “What the model knows”xp its fits the log of daily orders with:
- a trend: a straight line over the 84 days;
- the weekday: Mondays and Saturdays differ;
- the holiday windows: the shopping days around Labour Day and Thanksgiving;
- the sale: the days from 9 to 19 October;
- the step: 0 before 6 October, 1 from that day. Its coefficient is the effect.
The holidays and the sale are in the data as yes-or-no columns. Larkfield knows its own calendar. The model learns how much a holiday window adds from Labour Day, which was before the launch, and applies it to Thanksgiving.
The errors are robust to autocorrelation. Autocorrelation means that neighbouring days are alike: a busy Tuesday is often followed by a busy Wednesday. So 84 days are not 84 independent pieces of evidence. xp its widens the interval for this, looking up to 7 days apart. It is the same idea as clustering by area in difference-in-differences.
- Daily orders
- Full model, without the step
- Trend and weekday only
- Without the block (simulator)
Text description of the chart
A line chart of 84 days, from 56 days before to 27 days after 6 October 2025. The orders are between about 1,350 and 2,370 a day. The orders rise and fall with the weekday. Just after the launch, they rise sharply during Thanksgiving and the sale, then fall back. In the 28 days after the launch, the orders average 1,816.5 a day. The simulator’s world without the block averages 1,780.8. The full model without its step averages 1,751.7 and follows the bump. The simple model without its step averages 1,624.2: it shows no bump at all.
A model without the calendar gives the calendar to the change
Section titled “A model without the calendar gives the calendar to the change”What if the model did not know the holiday and the sale? xp its --simple keeps only the trend and the weekday.
xp its --simplexp checkInterrupted time series (trend and weekday only), 84 days (8 weeks before, 4 after), launch 2025-10-06: Effect of the launch on daily orders: +11.3% (95% interval +4.1% to +19.1%; errors robust to 7 days of autocorrelation).Run launch-s40: the last analysis was `xp its`.True effect (from the simulator): +2.0% on daily orders.Your estimate: +11.3% (+4.1% to +19.1%): the interval does NOT contain it.+11.3%: worse than the naive +8.3% on sales. Thanksgiving and the sale all fall in the 4 weeks after the launch. The simple model has no term for them, so the step takes their orders. The step answers the question “what was different after 6 October?”, and the answer is: the block, a holiday and a sale.
The course measured both models over 50 seeds:
- trend and weekday only: mean +12.7%, the interval contained the truth in 14% of the runs;
- trend, weekday, holidays and sale: mean +3.3%, the interval contained the truth in 84% of the runs.
Even the full model is a little too high on average, and 84% is below the promised 95%. The model’s straight trend and its holiday term are simple guesses about a complex calendar. The simulator’s season is not a straight line.
More history is not always better
Section titled “More history is not always better”A longer history sounds safer. xp its --before 52 uses a whole year before the launch. It gives −2.2% (−6.2% to +2.1%). The interval contains the truth, but the estimate now says that the block may have lowered orders. A year holds a whole garden season: a spring peak and a winter low. A straight trend line cannot follow it. So the extended line is wrong in October, and the step absorbs the error.
A long history needs a model of the season. The forecasting course teaches how to model trend, season and holidays, in Trend, season and noise and Calendars, holidays and events. A forecast made before the change, from a model you checked on past data, is a strong counterfactual.
Text description of the chart
One panel with a vertical line at the true effect, +2.0%. Naive before and after, on sales: +8.3%, with no interval. Trend and weekday only: +11.3% (+4.1% to +19.1%). A year before, with no season: −2.2% (−6.2% to +2.1%). Trend, weekday, holidays and sale: +3.5% (+1.3% to +5.8%).
When this is the only evidence you have
Section titled “When this is the only evidence you have”An interrupted time series is the weakest of the four methods in this module. Every other method has a group that did not get the change: Ontario, a mix of provinces, the customers just below 500. Here, the comparison is a model, and the model is only as good as your knowledge of the calendar.
Use it when:
- the change went to everyone at once, and you cannot find a comparison group;
- you have a long, regular history, and you know the events in it: holidays, sales, price changes, outages;
- the effect is quick and large compared with the day-to-day noise.
Do not trust it when:
- something else changed near the same date, and it is not in the model;
- the estimate moves a lot when you change the history length or the model;
- the effect builds up slowly. A slow effect looks like a change in trend, and the model may give it to the trend.
The cost is low in data and high in judgement. Write down the model before you see the result: the history length, the terms, the events. Otherwise you can find a model for any answer you want. Then say clearly that this is the weakest kind of evidence.
Which method, and what it needs
Section titled “Which method, and what it needs”The diagram puts the module together. It starts from the best option: a test with a coin flip.
| Method | It assumes | A sign that the assumption is false |
|---|---|---|
| Difference-in-differences | Without the change, both groups would have moved the same way (parallel trends) | Different trends before the change; a placebo effect; an estimate that moves with the comparison group |
| Synthetic control | The weighted mix would have kept following the treated unit | A poor fit before the change (large RMSPE); placebo gaps as large as the treated unit’s |
| Regression discontinuity | Nobody chose their side of the threshold; nothing else changes there | A pile-up just above the line (density check); an estimate that moves with the bandwidth |
| Interrupted time series | Nothing else changed at that date; the model before, extended, holds after | An estimate that moves with the model or the history length; a known event near the date |
Guided practice: add the calendar one term at a time
Section titled “Guided practice: add the calendar one term at a time”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, fit the nine-day example. You see the step
+10.0. - In section 2, compute the naive change of orders over 4 weeks on each side. You see
+10.2%, against a true+2.0%. - In section 3, fit the model with the trend and the weekday only. You see
+11.3%. - In section 4, add the holiday windows and the sale. Put the effect in
its_full. Run the check cell. It printsCheck passed. - In section 5, draw the real orders against the model without its step.
Show the answer to step 6
m = smf.ols("log_orders ~ post + t + C(weekday) + holiday_window + sale", data=w).fit( cov_type="HAC", cov_kwds={"maxlags": 7})its_full = round(float(np.expm1(m.params["post"])), 4)print(its_full)The value is 0.0349: +3.5%, the same as xp its.
Your turn: more history
Section titled “Your turn: more history”Mateo says: “Eight weeks is too short. Use 16 weeks before the launch.” Run xp its --before 16 and xp check. Before you run it, look at the garden season in the first chart of Difference-in-differences. Predict whether the estimate will go up or down.
Show the answer
xp its --before 16xp checkInterrupted time series (trend, weekday, holiday windows and the sale), 140 days (16 weeks before, 4 after), launch 2025-10-06: Effect of the launch on daily orders: +10.5% (95% interval +5.7% to +15.5%; errors robust to 7 days of autocorrelation).Run launch-s40: the last analysis was `xp its`.True effect (from the simulator): +2.0% on daily orders.Your estimate: +10.5% (+5.7% to +15.5%): the interval does NOT contain it.The estimate goes up to +10.5%, and the interval misses the truth. Sixteen weeks before 6 October start in mid-June, near the top of the garden season. Sales then fall through the summer. The straight trend learns “sales fall”, draws that fall forward into October, and expects too few orders after the launch. So the step looks bigger. Eight weeks were a better choice here, but only because their straight line happened to fit. The safer fix is a model of the season, not a different window.
Common mistake
“The new homepage raised orders 11%”
Symptom: a team launches a new homepage for everyone on a Monday. They fit a trend and a weekday pattern on the 8 weeks before and find a step of +11%, with an interval that excludes zero. A month later, orders are back where they were.
Cause: a holiday and a sale fell in the weeks after the launch, and the model had no term for them. The step took their orders. A clear interval does not help: it measures the noise around the wrong model.
Fix: list every event near the change date before you fit, and add each one to the model. Fit the model on several history lengths and model forms, and report the range. If the estimate moves a lot, say that the evidence is weak. Where you can, keep a small group without the change (a holdout), so that the next launch has a comparison group.
Check your understanding
A shop sold 50, 53, 56 and 59 items on four days, then a change started. On the next two days it sold 70 and 73. With a straight trend, what is the estimated effect a day?
Show the answer
The trend adds 3 items a day: 50, 53, 56, 59. Extended, it says 62 and 65 for the next two days. The shop sold 70 and 73. The step is 70 − 62 = 8 and 73 − 65 = 8: about +8 items a day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why did the model with only a trend and the weekday find +11.3% for the block, when the truth is +2.0%?
Show the answer
Thanksgiving and the autumn sale fell in the 4 weeks after the launch, and the model had no term for them. The model’s counterfactual stayed low through the holiday and the sale. So the step, the “after the launch” term, absorbed their extra orders as if they were the block’s effect. The model with the holiday windows and the sale found +3.5%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
State the assumption that an interrupted time series needs, and one sign in the data that it is false.
Show the answer
It assumes that nothing else changed at the same date that the model does not know about. It also assumes that the model of the past, extended, is right after the change. One sign that it is false is an estimate that moves a lot with the history length or the model terms. Here it was +3.5% with 8 weeks, +10.5% with 16 weeks and −2.2% with 52. Another sign: a known event, such as a holiday or a sale, near the change date.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An interrupted time series models one series before a change and extends the model past it. The step on the change date is the effect (segmented regression).
- The model must know the trend, the weekday and every event near the change. With them, the block’s effect was +3.5% (+1.3% to +5.8%) against a true +2.0%. Coverage over 50 seeds: 84%.
- Without the holiday and the sale, the step took them: +11.3%, worse than the naive +8.3%. With a year of history and no season: −2.2%. With 16 weeks: +10.5%.
- Errors robust to autocorrelation count neighbouring days as alike.
- It is the weakest method of this module. The comparison is a model, not a group. Write the model down before you see the result.
- Every method in this module needs an assumption you cannot fully test: name it, and look for its warning sign.
Next module: no change date at all. Only logs of who used the block, and what they spent.
Next: Regression adjustment