Skip to content
Français

Forecast and compare

  • Lesson 22
  • 35 min
  • Module 5: Anomalies in time

Where we are

STL looks at the whole series, including the days after the one we check. A live alert must decide with the past only. In this lesson, we forecast each day from the days before it, alert when the real value falls outside the prediction interval, and compare with STL.

Lesson 22 of 38 · Anomaly Detection: Find What Is UnusualForecast and compare
Course overview

What you will be able to do

Alert on a day when the real value falls outside what the past predicts for it. A forecast uses only the days before, so it can run every morning, and its interval says how sure it is.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and statsmodels. They download the daily orders, the daily tickets and the calendar (less than 1 MB). About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

The notebook route (Colab or Kaggle) or the local route, as in the first lesson of this module.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and statsmodels 0.15.0. The Windows commands and Colab were not tested.

The weather forecast says: “tomorrow, between 18 and 24 degrees”. Tomorrow, it is 35 degrees. You do not need to know how the forecast was made to see that something unusual happened. The real value fell far outside the range.

That range has a name: the prediction interval. It is where the real value should fall, most of the time. A 99% interval is wide enough for the real value to fall inside it on 99 days in 100. That holds if the forecast’s errors behave as they did in the past.

So a forecast gives a detector for free: forecast each day, and alert when the real value is outside the interval.

In the last lesson, STL computed each day’s trend from the days before and after it. On the morning of 6 July 2025, Grace wants to know if yesterday (5 July) was unusual. STL cannot tell her yet: the days after 5 July do not exist. When they arrive, STL changes its split of 5 July.

A live alert needs a method that uses the past only. A forecast does exactly that.

Five Saturdays had 800, 820, 790, 810 and 830 orders. The sixth Saturday has 1,150. Is it unusual?

  1. Forecast the sixth Saturday from the five before it. The plainest forecast is their mean: 810.
  2. Measure how far the past Saturdays were from the forecast. Their standard deviation is 15.8 orders.
  3. Build a 99% interval: the forecast ± 2.58 × 15.8. The number 2.58 gives 99% for errors with a normal (bell-shaped) distribution. The interval is 769 to 851.
  4. Compare the real value with the interval. 1,150 is outside: an alert.

The notebook prints the same:

forecast 810, spread 15.8, 99% interval 769 to 851
the real value 1150 is outside

The width of the interval comes from the past errors. If the past Saturdays were far apart, the interval is wide, and only a big surprise is outside. If they were close, the interval is narrow. The forecast knows how sure it is.

The mean of five Saturdays is too plain for two years of orders: it does not follow the year. We use one standard model, SARIMA (Seasonal AutoRegressive Integrated Moving Average). In plain words, it forecasts each day from three things:

  • the level of the last few days (yesterday’s orders say a lot about today’s);
  • the same weekday a week before (the weekly season);
  • its own recent errors (if it was too low for three days, it corrects itself).

The model has a few numbers (parameters) that set how much each of these counts. We learn them on the first year only (1 April 2024 to 31 March 2025). Then the model steps through all two years, one day at a time. Each day’s forecast uses only the days before it. The model works on the log of the orders, so an error of 10% has the same size on a quiet day and on a busy day.

The forecasting course explains SARIMA and its settings, and how to choose between models with a backtest. Here it is only a tool: any model with an honest prediction interval works the same way.

A loop. The days before today, only the past, go to the forecast model, learned on the first year. The model gives a forecast and a 99% interval: for 5 July 2025, 971, from 694 to 1,359. A question: is today's real value inside the interval? If yes: normal, no alert. If no: is it a known event? If no: an alert, because 1,372 is above 1,359. Then, tomorrow, today joins the past, and the loop starts again.
A forecast detector, one day at a time. It uses only the past, so it can run every morning. On 5 July 2025, the real orders (1,372) are just above the interval.
Text description of the diagram

On the left, a box: the days before today, only the past. An arrow goes to the forecast model, which was learned on the first year. The model gives a forecast and a 99% interval. For 5 July 2025, the forecast is 971 orders, and the interval is 694 to 1,359. A question follows: is today’s real value inside the interval? If yes, the day is normal, with no alert. If no, a second question: is it a known event? If not, an alert: 1,372 is above 1,359. From both ends, an arrow says “tomorrow: today joins the past”, and a dashed arrow goes back to the first box.

The notebook counts the alarm days after the warm-up, and prints two days:

alarm days: 16 | on or next to a holiday: 15
orders forecast low high
2025-07-05 1372.0 971.2 694.2 1358.8
2025-11-28 1224.0 526.7 376.5 736.9

On Black Friday 2025, the forecast was 526.7 orders, and the real value (1,224) is far above the interval. On 5 July 2025, the peak of the planted heat wave, the forecast was already higher (971.2), because the days before were busy. The real value (1,372) is just above the interval (1,358.8).

  • Real orders
  • Forecast
  • 99% interval
  • Outside the interval
The forecast and its 99% interval in June and July 2025. The real orders stay inside the interval, also on most heat-wave days, because each forecast follows the busy days before it. Only 5 July (1,372) is outside, just above the high limit of 1,358.8.
Text description of the chart

Daily orders from 2 June to 3 August 2025, with the forecast line and two dashed lines for the 99% interval. The interval is wide: the high line is about 40% above the forecast, and the low line about 30% below it. In the heat-wave week (30 June to 6 July), the real orders rise from 828 to 1,372. The forecast and the interval rise with them, one day later. Only 5 July is outside: 1,372 orders, just above the high limit of 1,358.8. After the heat wave, the interval stays high for a few days, then comes down.

  • Real orders
  • Forecast
  • 99% interval
  • Outside the interval
The forecast and its interval from November 2025 to January 2026. Black Friday (1,224), Cyber Monday (1,136), the day after it, Christmas Eve, Christmas Day, Boxing Day and New Year's Day are outside. All are holidays or next to one.
Text description of the chart

Daily orders from 3 November 2025 to 11 January 2026, with the forecast and its 99% interval. Most days are inside. Seven dots mark the days outside. Three are above the interval: Black Friday (1,224 orders, interval up to 736.9), Cyber Monday (1,136, up to 1,117.4) and Boxing Day (496, up to 440.9). Four are below it: the day after Cyber Monday (586, from 629.6), Christmas Eve (337, from 452.2), Christmas Day (157, from 351.9) and New Year’s Day 2026 (299, from 353.2).

The second chart shows a useful habit of a forecast. After Cyber Monday, the forecast for the next day is high (880.9). The real orders (586) fall back to normal, and that is below the interval. The model expected the busy days to go on. An event confuses a forecast for a day or two after it, too. That is why we mark the day before and the day after each holiday as known events.

The interval is wide. Over the two years, its median width is 68.4% of the forecast. The model’s errors on the first year were large, so it is careful. That is good for Grace’s time, and it means a forecast misses smaller problems.

Now we can compare every detector of this module on the same two years, after the same warm-up, with the same known events. The notebook builds all five:

alarm days on or next to a holiday reported after marking holidays
global threshold 9 4 5
rolling median, 28 days 19 9 10
seasonal baseline, same weekday 43 11 32
STL remainder 91 16 75
forecast, 99% interval 16 15 1
The five detectors of this module on two years of Larkfield's daily orders, after the holidays and the days next to them are marked. The forecast reports 1 day (the heat-wave peak); STL reports 75.
Text description of the chart

Five columns: global threshold 5 days, rolling median 10, seasonal baseline 32, STL remainder 75, forecast with a 99% interval 1. The forecast’s column is in another colour.

The forecast reports the fewest days. 15 of its 16 alarm days are holidays or next to one. The other one is the peak of the heat wave. The global threshold also reports few days, but for a bad reason. Its limits are so wide that it is blind to a Tuesday that halves. STL reports the most, because its limit on a clean remainder is narrow.

Which is “right”? The data has no list of real incidents for the daily orders, only the planted situations of the shared series. So the right choice depends on what Grace’s team wants to see:

  • The forecast when the team wants few alerts, only big surprises, every morning.
  • The seasonal baseline when the weekly pattern is stable. The team gets a reason they can check by hand: “40% fewer orders than the last four Thursdays”. It needs no model.
  • STL to study last year, not for a morning alert.

The tickets tell a different story. In the heat wave, the help desk had 537.4 tickets a day, against 349.8 in the 4 weeks before. The same five detectors on the daily tickets (all channels together) alarm on these days of the heat-wave week:

Detector Heat-wave days alarmed (of 7)
Global threshold 3
Rolling median, 28 days 2
Seasonal baseline, same weekday 5
STL remainder 4
Forecast, 99% interval 3

The seasonal baseline sees the most days: it compares with four normal weeks, and it does not adapt within the week. The forecast alarms on 1, 2 and 4 July. Then the busy days are in its past, and it follows the new level.

When a forecast is better, and what it costs

Section titled “When a forecast is better, and what it costs”

Use a forecast for a live alert on a series with a trend and a season: daily orders, tickets, requests. It uses only the past. Its interval comes from its own errors, so you do not set a spread by hand.

A seasonal baseline is enough when the pattern is stable and the history is short. It is a few lines of pandas, and anyone can check its reason.

What a forecast costs:

  • A model to choose and to learn. Here it takes a fraction of a second, but the choice of model is a skill of its own: the forecasting course teaches it.
  • A first year of history to learn from.
  • It follows a long event: after a few busy days, the next busy day is not a surprise. On the heat wave, it alarmed only on the peak.
  • An event confuses the next day’s forecast, too. Mark the days around known events.

Common mistake

A 95% interval because it sounds strict

Symptom: you choose a 95% interval instead of 99%, because 95% sounds more careful. Grace’s team gets more alerts on ordinary days.

Diagnose: a 95% interval, by its own definition, leaves out 1 normal day in 20. The notebook counts what it does on the orders:

95% interval: alarm days 24 | reported after marking holidays 7

The 99% interval reported 1 day; the 95% interval reports 7. That is still fewer than 1 day in 20. One likely reason: the model learned the size of its errors on a first year with holidays in it. So its interval is wide on normal days.

Recover: choose the interval’s level as you choose a threshold. Start from the number of alerts the team can read in a week, and from the cost of a missed day. Write the choice down with its reason.

The notebook’s section 4 has two tasks.

  1. List the days that the forecast reports after the warm-up, when the holidays and the days next to them are marked.
  2. Write them as strings "YYYY-MM-DD" in the list forecast_other_days, then run its check cell.
  3. Build the same forecast on the daily tickets. Use the same model, learned on the first year.
  4. List the heat-wave days (2025-06-30 to 2025-07-06) outside its 99% interval, as strings, in ticket_heat_wave_alarms.
  5. Run its check cell.

Check your understanding

On the day after Cyber Monday 2025, the real orders (586) were below the forecast's interval (from 629.6). Nothing was wrong. Why did the forecast alarm, and what is the fix?

Show the answer

The forecast uses the days just before, and Cyber Monday was very busy (1,136 orders). So the model expected the next day to be busy too (forecast 880.9), and the normal day looked too low. An event confuses the forecast for a day or two after it. The fix is to mark the days around a known event (the day before and the day after a holiday) as known events too.

Check your understanding

Predict: you change the interval from 99% to 99.9%. Will the forecast report more or fewer days, and what will it miss?

Show the answer

Fewer days, because the interval becomes wider. It will miss smaller surprises. With 99%, the heat-wave peak was only just outside (1,372 against a high limit of 1,358.8). A wider interval would probably miss it. The interval level is a trade between the team’s time and the cost of a missed day.

Check your understanding

Grace asks which detector to put on the daily ticket count, as a morning alert for the help desk. Name one, and give one reason from this lesson and one cost.

Show the answer

A forecast with a 99% interval: it uses only the past, so it can run every morning, and its interval comes from its own errors. It found the start of the heat wave on the tickets. Its cost: a model learned on a year of history. It also follows a long event: it did not alarm on the last two days of the heat wave. A seasonal baseline is a good second choice: simpler, and it saw 5 of the 7 heat-wave days.

Recap

  • A forecast with a prediction interval asks: is today far from what the past says today should be? It uses only the past, so it can run live.
  • The interval’s width comes from the model’s own past errors. Its level (95%, 99%) is a choice, like a threshold.
  • A forecast follows long events and is confused for a day after a big one: mark the days around known events.
  • On two years of orders, after marking holidays: forecast 1 day, global threshold 5, rolling median 10, seasonal baseline 32, STL 75.
  • A seasonal baseline is enough when the pattern is stable; STL is for history.

You can now alert on a single day that falls outside its forecast. But some changes do not go away: a memory leak that grows for days, or an app launch that moves normal for good. Next, you find those change points, and you see why a change point is a new normal, not an alert.

Next: Change points