Forecast and compare
- Lesson 22
- 35 min
- Module 5: Anomalies in time
Where we are
STL looks at the whole series, including the days after the one we check. A live alert must decide with the past only. In this lesson, we forecast each day from the days before it, alert when the real value falls outside the prediction interval, and compare with STL.
Lesson 22 of 38 · Anomaly Detection: Find What Is UnusualForecast and compare
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Alert on a day when the real value falls outside what the past predicts for it. A forecast uses only the days before, so it can run every morning, and its interval says how sure it is.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and statsmodels. They download the daily orders, the daily tickets and the calendar (less than 1 MB). About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Seasonal baselines, known events and the five-week warm-up, from Normal changes with the hour and the season.
- STL and its remainder, from STL residuals.
- What a forecast is, in one sentence: a guess of a future value from past values. This lesson uses one forecasting model as a tool. The forecasting course, Forecasting and Time Series: Predict What Comes Next, explains how forecasting models work. It also shows how to choose one and how to test it with backtests.
What you need on your computer
The notebook route (Colab or Kaggle) or the local route, as in the first lesson of this module.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and statsmodels 0.15.0. The Windows commands and Colab were not tested.
A weather forecast
Section titled “A weather forecast”The weather forecast says: “tomorrow, between 18 and 24 degrees”. Tomorrow, it is 35 degrees. You do not need to know how the forecast was made to see that something unusual happened. The real value fell far outside the range.
That range has a name: the prediction interval. It is where the real value should fall, most of the time. A 99% interval is wide enough for the real value to fall inside it on 99 days in 100. That holds if the forecast’s errors behave as they did in the past.
So a forecast gives a detector for free: forecast each day, and alert when the real value is outside the interval.
The problem: STL looks at tomorrow
Section titled “The problem: STL looks at tomorrow”In the last lesson, STL computed each day’s trend from the days before and after it. On the morning of 6 July 2025, Grace wants to know if yesterday (5 July) was unusual. STL cannot tell her yet: the days after 5 July do not exist. When they arrive, STL changes its split of 5 July.
A live alert needs a method that uses the past only. A forecast does exactly that.
The idea, by hand
Section titled “The idea, by hand”Five Saturdays had 800, 820, 790, 810 and 830 orders. The sixth Saturday has 1,150. Is it unusual?
- Forecast the sixth Saturday from the five before it. The plainest forecast is their mean: 810.
- Measure how far the past Saturdays were from the forecast. Their standard deviation is 15.8 orders.
- Build a 99% interval: the forecast ± 2.58 × 15.8. The number 2.58 gives 99% for errors with a normal (bell-shaped) distribution. The interval is 769 to 851.
- Compare the real value with the interval. 1,150 is outside: an alert.
The notebook prints the same:
forecast 810, spread 15.8, 99% interval 769 to 851the real value 1150 is outsideThe width of the interval comes from the past errors. If the past Saturdays were far apart, the interval is wide, and only a big surprise is outside. If they were close, the interval is narrow. The forecast knows how sure it is.
A forecast model for Larkfield’s orders
Section titled “A forecast model for Larkfield’s orders”The mean of five Saturdays is too plain for two years of orders: it does not follow the year. We use one standard model, SARIMA (Seasonal AutoRegressive Integrated Moving Average). In plain words, it forecasts each day from three things:
- the level of the last few days (yesterday’s orders say a lot about today’s);
- the same weekday a week before (the weekly season);
- its own recent errors (if it was too low for three days, it corrects itself).
The model has a few numbers (parameters) that set how much each of these counts. We learn them on the first year only (1 April 2024 to 31 March 2025). Then the model steps through all two years, one day at a time. Each day’s forecast uses only the days before it. The model works on the log of the orders, so an error of 10% has the same size on a quiet day and on a busy day.
The forecasting course explains SARIMA and its settings, and how to choose between models with a backtest. Here it is only a tool: any model with an honest prediction interval works the same way.
Text description of the diagram
On the left, a box: the days before today, only the past. An arrow goes to the forecast model, which was learned on the first year. The model gives a forecast and a 99% interval. For 5 July 2025, the forecast is 971 orders, and the interval is 694 to 1,359. A question follows: is today’s real value inside the interval? If yes, the day is normal, with no alert. If no, a second question: is it a known event? If not, an alert: 1,372 is above 1,359. From both ends, an arrow says “tomorrow: today joins the past”, and a dashed arrow goes back to the first box.
Days outside the interval
Section titled “Days outside the interval”The notebook counts the alarm days after the warm-up, and prints two days:
alarm days: 16 | on or next to a holiday: 15 orders forecast low high2025-07-05 1372.0 971.2 694.2 1358.82025-11-28 1224.0 526.7 376.5 736.9On Black Friday 2025, the forecast was 526.7 orders, and the real value (1,224) is far above the interval. On 5 July 2025, the peak of the planted heat wave, the forecast was already higher (971.2), because the days before were busy. The real value (1,372) is just above the interval (1,358.8).
- Real orders
- Forecast
- 99% interval
- Outside the interval
Text description of the chart
Daily orders from 2 June to 3 August 2025, with the forecast line and two dashed lines for the 99% interval. The interval is wide: the high line is about 40% above the forecast, and the low line about 30% below it. In the heat-wave week (30 June to 6 July), the real orders rise from 828 to 1,372. The forecast and the interval rise with them, one day later. Only 5 July is outside: 1,372 orders, just above the high limit of 1,358.8. After the heat wave, the interval stays high for a few days, then comes down.
- Real orders
- Forecast
- 99% interval
- Outside the interval
Text description of the chart
Daily orders from 3 November 2025 to 11 January 2026, with the forecast and its 99% interval. Most days are inside. Seven dots mark the days outside. Three are above the interval: Black Friday (1,224 orders, interval up to 736.9), Cyber Monday (1,136, up to 1,117.4) and Boxing Day (496, up to 440.9). Four are below it: the day after Cyber Monday (586, from 629.6), Christmas Eve (337, from 452.2), Christmas Day (157, from 351.9) and New Year’s Day 2026 (299, from 353.2).
The second chart shows a useful habit of a forecast. After Cyber Monday, the forecast for the next day is high (880.9). The real orders (586) fall back to normal, and that is below the interval. The model expected the busy days to go on. An event confuses a forecast for a day or two after it, too. That is why we mark the day before and the day after each holiday as known events.
The interval is wide. Over the two years, its median width is 68.4% of the forecast. The model’s errors on the first year were large, so it is careful. That is good for Grace’s time, and it means a forecast misses smaller problems.
Five detectors on the same days
Section titled “Five detectors on the same days”Now we can compare every detector of this module on the same two years, after the same warm-up, with the same known events. The notebook builds all five:
alarm days on or next to a holiday reported after marking holidaysglobal threshold 9 4 5rolling median, 28 days 19 9 10seasonal baseline, same weekday 43 11 32STL remainder 91 16 75forecast, 99% interval 16 15 1Text description of the chart
Five columns: global threshold 5 days, rolling median 10, seasonal baseline 32, STL remainder 75, forecast with a 99% interval 1. The forecast’s column is in another colour.
The forecast reports the fewest days. 15 of its 16 alarm days are holidays or next to one. The other one is the peak of the heat wave. The global threshold also reports few days, but for a bad reason. Its limits are so wide that it is blind to a Tuesday that halves. STL reports the most, because its limit on a clean remainder is narrow.
Which is “right”? The data has no list of real incidents for the daily orders, only the planted situations of the shared series. So the right choice depends on what Grace’s team wants to see:
- The forecast when the team wants few alerts, only big surprises, every morning.
- The seasonal baseline when the weekly pattern is stable. The team gets a reason they can check by hand: “40% fewer orders than the last four Thursdays”. It needs no model.
- STL to study last year, not for a morning alert.
The tickets in the heat wave
Section titled “The tickets in the heat wave”The tickets tell a different story. In the heat wave, the help desk had 537.4 tickets a day, against 349.8 in the 4 weeks before. The same five detectors on the daily tickets (all channels together) alarm on these days of the heat-wave week:
| Detector | Heat-wave days alarmed (of 7) |
|---|---|
| Global threshold | 3 |
| Rolling median, 28 days | 2 |
| Seasonal baseline, same weekday | 5 |
| STL remainder | 4 |
| Forecast, 99% interval | 3 |
The seasonal baseline sees the most days: it compares with four normal weeks, and it does not adapt within the week. The forecast alarms on 1, 2 and 4 July. Then the busy days are in its past, and it follows the new level.
When a forecast is better, and what it costs
Section titled “When a forecast is better, and what it costs”Use a forecast for a live alert on a series with a trend and a season: daily orders, tickets, requests. It uses only the past. Its interval comes from its own errors, so you do not set a spread by hand.
A seasonal baseline is enough when the pattern is stable and the history is short. It is a few lines of pandas, and anyone can check its reason.
What a forecast costs:
- A model to choose and to learn. Here it takes a fraction of a second, but the choice of model is a skill of its own: the forecasting course teaches it.
- A first year of history to learn from.
- It follows a long event: after a few busy days, the next busy day is not a surprise. On the heat wave, it alarmed only on the peak.
- An event confuses the next day’s forecast, too. Mark the days around known events.
Common mistake
A 95% interval because it sounds strict
Symptom: you choose a 95% interval instead of 99%, because 95% sounds more careful. Grace’s team gets more alerts on ordinary days.
Diagnose: a 95% interval, by its own definition, leaves out 1 normal day in 20. The notebook counts what it does on the orders:
95% interval: alarm days 24 | reported after marking holidays 7The 99% interval reported 1 day; the 95% interval reports 7. That is still fewer than 1 day in 20. One likely reason: the model learned the size of its errors on a first year with holidays in it. So its interval is wide on normal days.
Recover: choose the interval’s level as you choose a threshold. Start from the number of alerts the team can read in a week, and from the cost of a missed day. Write the choice down with its reason.
Your turn
Section titled “Your turn”The notebook’s section 4 has two tasks.
- List the days that the forecast reports after the warm-up, when the holidays and the days next to them are marked.
- Write them as strings
"YYYY-MM-DD"in the listforecast_other_days, then run its check cell. - Build the same forecast on the daily tickets. Use the same model, learned on the first year.
- List the heat-wave days (2025-06-30 to 2025-07-06) outside its 99% interval, as strings, in
ticket_heat_wave_alarms. - Run its check cell.
Check your understanding
On the day after Cyber Monday 2025, the real orders (586) were below the forecast's interval (from 629.6). Nothing was wrong. Why did the forecast alarm, and what is the fix?
Show the answer
The forecast uses the days just before, and Cyber Monday was very busy (1,136 orders). So the model expected the next day to be busy too (forecast 880.9), and the normal day looked too low. An event confuses the forecast for a day or two after it. The fix is to mark the days around a known event (the day before and the day after a holiday) as known events too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you change the interval from 99% to 99.9%. Will the forecast report more or fewer days, and what will it miss?
Show the answer
Fewer days, because the interval becomes wider. It will miss smaller surprises. With 99%, the heat-wave peak was only just outside (1,372 against a high limit of 1,358.8). A wider interval would probably miss it. The interval level is a trade between the team’s time and the cost of a missed day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace asks which detector to put on the daily ticket count, as a morning alert for the help desk. Name one, and give one reason from this lesson and one cost.
Show the answer
A forecast with a 99% interval: it uses only the past, so it can run every morning, and its interval comes from its own errors. It found the start of the heat wave on the tickets. Its cost: a model learned on a year of history. It also follows a long event: it did not alarm on the last two days of the heat wave. A seasonal baseline is a good second choice: simpler, and it saw 5 of the 7 heat-wave days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A forecast with a prediction interval asks: is today far from what the past says today should be? It uses only the past, so it can run live.
- The interval’s width comes from the model’s own past errors. Its level (95%, 99%) is a choice, like a threshold.
- A forecast follows long events and is confused for a day after a big one: mark the days around known events.
- On two years of orders, after marking holidays: forecast 1 day, global threshold 5, rolling median 10, seasonal baseline 32, STL 75.
- A seasonal baseline is enough when the pattern is stable; STL is for history.
You can now alert on a single day that falls outside its forecast. But some changes do not go away: a memory leak that grows for days, or an app launch that moves normal for good. Next, you find those change points, and you see why a change point is a new normal, not an alert.
Next: Change points