Skip to content
Français

Interrupted time series

  • Lesson 20
  • 35 min
  • Module 5: When you cannot randomise: natural experiments

Where we are

Sometimes there is no comparison group and no threshold: only one series and one change date. In this lesson, we go back to the recommendation block's launch and use the trend, the season and the holidays before the change to estimate what would have happened.

What you will be able to do

Estimate the effect of a change when you have only one series and the date of the change. Model what the series did before, extend it past the change as the counterfactual, and read the step. Know what the model must contain, and why this is the weakest of the module’s methods.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you fit the trend and a step on a tiny series. Then you do the same on TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s daily orders around the block’s launch. You add the weekday, the holidays and the sale one by one and watch the estimate move. Its setup cells install the packages, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

Tested on macOS with Python 3.12 and the end-of-m05 snapshot of the project. Windows was not tested.

A café opens one hour earlier from Monday 1 March. There is only one café, so there is no other café to compare with. There is no threshold either.

But the owner has a notebook with the number of customers on every day for a year. In February, the café gained about 2 customers a day, every week. Mondays are always quiet, and Saturdays are busy. In March, she draws the February line forward, with the same slope and the same weekly pattern. That is her guess of March without the early hour. Then she compares it with the real March.

She used the café’s own past as the comparison.

The problem: no comparison group for the block

Section titled “The problem: no comparison group for the block”

In Module 1, TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop.’s dashboard said that sales rose 8.3% after the recommendation block’s launch on 6 October 2025. That number is a practice problem we made on purpose. The simulator knows that the block’s true effect was only +2.0% of orders.

The block went live for everyone on the same day. There is no Ontario and no synthetic Quebec: every customer got it. There is no threshold. There is one series of daily orders, and one date.

The same weeks also had Thanksgiving (13 October) and an autumn sale (9 to 19 October). The naive comparison gives all of that to the block.

The idea: the past, extended, is the counterfactual

Section titled “The idea: the past, extended, is the counterfactual”

An interrupted time series is a series with a change in the middle: the change “interrupts” it. The method has two parts:

  1. Model the series before the change: its trend, its weekly pattern, and the known events, such as holidays and sales.
  2. Extend the model past the change. It says what the days after would have been without the change. The gap between the real days and the extended model is the effect.

In practice, both parts are one regression on all the days. It has the trend, the weekday, the events and one extra yes-or-no term: “after the change”. That term adds a step on the change date. Its size is the effect. This form is often called segmented regression.

Here is a made-up series of nine days. The trend adds 2 orders a day, and a change on day 6 adds a step.

Day 1 2 3 4 5 6 7 8 9
Orders 102 104 106 108 110 122 124 126 128
Trend from days 1 to 5, extended 102 104 106 108 110 112 114 116 118
Gap 0 0 0 0 0 10 10 10 10

Days 1 to 5 fit a line: 100 + 2 × day. Extended, the line says 112 on day 6. The real day 6 had 122. The step is +10 orders a day, every day after the change.

  • Orders
  • The trend before, extended
An interrupted time series in nine days. The dashed line is the trend of days 1 to 5, drawn forward. From day 6, the orders run 10 above it: the step is the effect. A made-up example.
Text description of the chart

A line chart over days 1 to 9. The orders rise by 2 a day: 102, 104, 106, 108 and 110. They jump to 122 on day 6 and keep rising by 2: 124, 126 and 128. A dashed line continues the trend of days 1 to 5: 112, 114, 116 and 118 on days 6 to 9. The real orders stay 10 above the dashed line from day 6.

The method assumes that nothing else changed at the same date that the model does not know about. And it assumes that the model before the change, extended, is right after it. Both are strong. A comparison group protects you from surprises that hit both groups. Here, nothing does. If a competitor closed its shop the same week, the step would include it.

  1. Make sure that your project is at end-of-m05, with .venv active. Simulate the launch again.

    Terminal window
    sim run --scenario launch
    Scenario launch (seed 40): Daily orders and sales around the recommendation block's launch (6 October 2025): no test, one series.
    Wrote runs/launch-s40: daily (427 rows), counterfactual (427 rows). Synthetic data. 0.1 s
  2. Run the interrupted time series, then compare with the truth.

    Terminal window
    xp its
    xp check
    Interrupted time series (trend, weekday, holiday windows and the sale), 84 days (8 weeks before, 4 after), launch 2025-10-06:
    Effect of the launch on daily orders: +3.5% (95% interval +1.3% to +5.8%; errors robust to 7 days of autocorrelation).
    Run launch-s40: the last analysis was `xp its`.
    True effect (from the simulator): +2.0% on daily orders.
    Your estimate: +3.5% (+1.3% to +5.8%): the interval contains it.

From the naive +8.3% to +3.5%, against a true +2.0%, and the TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains the truth. The model used 8 weeks before the launch and the 4 weeks after it.

xp its fits the log of daily orders with:

  • a trend: a straight line over the 84 days;
  • the weekday: Mondays and Saturdays differ;
  • the holiday windows: the shopping days around Labour Day and Thanksgiving;
  • the sale: the days from 9 to 19 October;
  • the step: 0 before 6 October, 1 from that day. Its coefficient is the effect.

The holidays and the sale are in the data as yes-or-no columns. Larkfield knows its own calendar. The model learns how much a holiday window adds from Labour Day, which was before the launch, and applies it to Thanksgiving.

The errors are robust to autocorrelation. Autocorrelation means that neighbouring days are alike: a busy Tuesday is often followed by a busy Wednesday. So 84 days are not 84 independent pieces of evidence. xp its widens the interval for this, looking up to 7 days apart. It is the same idea as clustering by area in difference-in-differences.

  • Daily orders
  • Full model, without the step
  • Trend and weekday only
  • Without the block (simulator)
Daily orders 8 weeks before and 4 weeks after the block's launch. The full model, drawn without its step, follows the holiday and the sale and runs close to the simulator's world without the block. The simple model does not know the holiday or the sale: it stays low after the launch, so the whole bump looks like the block's effect. Synthetic data, seed 40.
Text description of the chart

A line chart of 84 days, from 56 days before to 27 days after 6 October 2025. The orders are between about 1,350 and 2,370 a day. The orders rise and fall with the weekday. Just after the launch, they rise sharply during Thanksgiving and the sale, then fall back. In the 28 days after the launch, the orders average 1,816.5 a day. The simulator’s world without the block averages 1,780.8. The full model without its step averages 1,751.7 and follows the bump. The simple model without its step averages 1,624.2: it shows no bump at all.

A model without the calendar gives the calendar to the change

Section titled “A model without the calendar gives the calendar to the change”

What if the model did not know the holiday and the sale? xp its --simple keeps only the trend and the weekday.

Terminal window
xp its --simple
xp check
Interrupted time series (trend and weekday only), 84 days (8 weeks before, 4 after), launch 2025-10-06:
Effect of the launch on daily orders: +11.3% (95% interval +4.1% to +19.1%; errors robust to 7 days of autocorrelation).
Run launch-s40: the last analysis was `xp its`.
True effect (from the simulator): +2.0% on daily orders.
Your estimate: +11.3% (+4.1% to +19.1%): the interval does NOT contain it.

+11.3%: worse than the naive +8.3% on sales. Thanksgiving and the sale all fall in the 4 weeks after the launch. The simple model has no term for them, so the step takes their orders. The step answers the question “what was different after 6 October?”, and the answer is: the block, a holiday and a sale.

The course measured both models over 50 seeds:

  • trend and weekday only: mean +12.7%, the interval contained the truth in 14% of the runs;
  • trend, weekday, holidays and sale: mean +3.3%, the interval contained the truth in 84% of the runs.

Even the full model is a little too high on average, and 84% is below the promised 95%. The model’s straight trend and its holiday term are simple guesses about a complex calendar. The simulator’s season is not a straight line.

A longer history sounds safer. xp its --before 52 uses a whole year before the launch. It gives −2.2% (−6.2% to +2.1%). The interval contains the truth, but the estimate now says that the block may have lowered orders. A year holds a whole garden season: a spring peak and a winter low. A straight trend line cannot follow it. So the extended line is wrong in October, and the step absorbs the error.

A long history needs a model of the season. The forecasting course teaches how to model trend, season and holidays, in Trend, season and noise and Calendars, holidays and events. A forecast made before the change, from a model you checked on past data, is a strong counterfactual.

Four answers to one question on the same data: what did the block do to orders? Only the model with the holidays and the sale, on 8 weeks before, lands near the truth (+2.0%). Without the calendar, the model is even worse than the naive comparison. Synthetic data, seed 40.
Text description of the chart

One panel with a vertical line at the true effect, +2.0%. Naive before and after, on sales: +8.3%, with no interval. Trend and weekday only: +11.3% (+4.1% to +19.1%). A year before, with no season: −2.2% (−6.2% to +2.1%). Trend, weekday, holidays and sale: +3.5% (+1.3% to +5.8%).

An interrupted time series is the weakest of the four methods in this module. Every other method has a group that did not get the change: Ontario, a mix of provinces, the customers just below 500. Here, the comparison is a model, and the model is only as good as your knowledge of the calendar.

Use it when:

  • the change went to everyone at once, and you cannot find a comparison group;
  • you have a long, regular history, and you know the events in it: holidays, sales, price changes, outages;
  • the effect is quick and large compared with the day-to-day noise.

Do not trust it when:

  • something else changed near the same date, and it is not in the model;
  • the estimate moves a lot when you change the history length or the model;
  • the effect builds up slowly. A slow effect looks like a change in trend, and the model may give it to the trend.

The cost is low in data and high in judgement. Write down the model before you see the result: the history length, the terms, the events. Otherwise you can find a model for any answer you want. Then say clearly that this is the weakest kind of evidence.

The diagram puts the module together. It starts from the best option: a test with a coin flip.

A decision diagram. First question: can you choose at random who gets the change? Yes leads to an A/B test, Modules 2 to 4. No leads to the second question: did a rule decide, with a threshold? Yes leads to regression discontinuity, just below against just above. No leads to the third question: did some regions, stores or groups get it, and others not? Yes leads to difference-in-differences with many similar units, or a synthetic control with one unit and many others. No leads to an interrupted time series: only one series and a date. A note says that every method needs an assumption you cannot fully test.
Choose a method for a change that was not randomised. Every path ends in an assumption: write it down, and look for the sign that it is false.
Method It assumes A sign that the assumption is false
Difference-in-differences Without the change, both groups would have moved the same way (parallel trends) Different trends before the change; a placebo effect; an estimate that moves with the comparison group
Synthetic control The weighted mix would have kept following the treated unit A poor fit before the change (large RMSPE); placebo gaps as large as the treated unit’s
Regression discontinuity Nobody chose their side of the threshold; nothing else changes there A pile-up just above the line (density check); an estimate that moves with the bandwidth
Interrupted time series Nothing else changed at that date; the model before, extended, holds after An estimate that moves with the model or the history length; a known event near the date

Guided practice: add the calendar one term at a time

Section titled “Guided practice: add the calendar one term at a time”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, fit the nine-day example. You see the step +10.0.
  4. In section 2, compute the naive change of orders over 4 weeks on each side. You see +10.2%, against a true +2.0%.
  5. In section 3, fit the model with the trend and the weekday only. You see +11.3%.
  6. In section 4, add the holiday windows and the sale. Put the effect in its_full. Run the check cell. It prints Check passed.
  7. In section 5, draw the real orders against the model without its step.
Show the answer to step 6
m = smf.ols("log_orders ~ post + t + C(weekday) + holiday_window + sale", data=w).fit(
cov_type="HAC", cov_kwds={"maxlags": 7})
its_full = round(float(np.expm1(m.params["post"])), 4)
print(its_full)

The value is 0.0349: +3.5%, the same as xp its.

Mateo says: “Eight weeks is too short. Use 16 weeks before the launch.” Run xp its --before 16 and xp check. Before you run it, look at the garden season in the first chart of Difference-in-differences. Predict whether the estimate will go up or down.

Show the answer
Terminal window
xp its --before 16
xp check
Interrupted time series (trend, weekday, holiday windows and the sale), 140 days (16 weeks before, 4 after), launch 2025-10-06:
Effect of the launch on daily orders: +10.5% (95% interval +5.7% to +15.5%; errors robust to 7 days of autocorrelation).
Run launch-s40: the last analysis was `xp its`.
True effect (from the simulator): +2.0% on daily orders.
Your estimate: +10.5% (+5.7% to +15.5%): the interval does NOT contain it.

The estimate goes up to +10.5%, and the interval misses the truth. Sixteen weeks before 6 October start in mid-June, near the top of the garden season. Sales then fall through the summer. The straight trend learns “sales fall”, draws that fall forward into October, and expects too few orders after the launch. So the step looks bigger. Eight weeks were a better choice here, but only because their straight line happened to fit. The safer fix is a model of the season, not a different window.

Common mistake

“The new homepage raised orders 11%”

Symptom: a team launches a new homepage for everyone on a Monday. They fit a trend and a weekday pattern on the 8 weeks before and find a step of +11%, with an interval that excludes zero. A month later, orders are back where they were.

Cause: a holiday and a sale fell in the weeks after the launch, and the model had no term for them. The step took their orders. A clear interval does not help: it measures the noise around the wrong model.

Fix: list every event near the change date before you fit, and add each one to the model. Fit the model on several history lengths and model forms, and report the range. If the estimate moves a lot, say that the evidence is weak. Where you can, keep a small group without the change (a holdout), so that the next launch has a comparison group.

Check your understanding

A shop sold 50, 53, 56 and 59 items on four days, then a change started. On the next two days it sold 70 and 73. With a straight trend, what is the estimated effect a day?

Show the answer

The trend adds 3 items a day: 50, 53, 56, 59. Extended, it says 62 and 65 for the next two days. The shop sold 70 and 73. The step is 70 − 62 = 8 and 73 − 65 = 8: about +8 items a day.

Check your understanding

Why did the model with only a trend and the weekday find +11.3% for the block, when the truth is +2.0%?

Show the answer

Thanksgiving and the autumn sale fell in the 4 weeks after the launch, and the model had no term for them. The model’s counterfactual stayed low through the holiday and the sale. So the step, the “after the launch” term, absorbed their extra orders as if they were the block’s effect. The model with the holiday windows and the sale found +3.5%.

Check your understanding

State the assumption that an interrupted time series needs, and one sign in the data that it is false.

Show the answer

It assumes that nothing else changed at the same date that the model does not know about. It also assumes that the model of the past, extended, is right after the change. One sign that it is false is an estimate that moves a lot with the history length or the model terms. Here it was +3.5% with 8 weeks, +10.5% with 16 weeks and −2.2% with 52. Another sign: a known event, such as a holiday or a sale, near the change date.

Recap

  • An interrupted time series models one series before a change and extends the model past it. The step on the change date is the effect (segmented regression).
  • The model must know the trend, the weekday and every event near the change. With them, the block’s effect was +3.5% (+1.3% to +5.8%) against a true +2.0%. Coverage over 50 seeds: 84%.
  • Without the holiday and the sale, the step took them: +11.3%, worse than the naive +8.3%. With a year of history and no season: −2.2%. With 16 weeks: +10.5%.
  • Errors robust to autocorrelation count neighbouring days as alike.
  • It is the weakest method of this module. The comparison is a model, not a group. Write the model down before you see the result.
  • Every method in this module needs an assumption you cannot fully test: name it, and look for its warning sign.

Next module: no change date at all. Only logs of who used the block, and what they spent.

Next: Regression adjustment