Skip to content
Français

STL residuals

  • Lesson 21
  • 35 min
  • Module 5: Anomalies in time

Where we are

Seasonal baselines remove the weekly pattern, but they still react to the trend and to the yearly season. In this lesson, we split the daily orders into trend, season and remainder with STL, detect on the remainder only, and mark the known events so that holidays are not reported.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 21 of 38 · Anomaly Detection: Find What Is UnusualSTL residuals
Course overview

What you will be able to do

Take the trend and the season out of a series, and look for anomalies in what is left. Then the threshold does not have to know about the year, the week or the slow growth.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and statsmodels. They download the daily orders, the calendar and the ticket API’s metrics (less than 1 MB), with compare.py. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

The notebook route (Colab or Kaggle) or the local route, as in the last lesson.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and statsmodels 0.15.0. The Windows commands and Colab were not tested.

Your electricity bill changes every month. Three things move it:

  1. Prices rise slowly over the years. That is the trend.
  2. Every winter the bill is higher, because you heat the house. That repeats every year. That is the season.
  3. One month is strange: the heater was broken and ran all day. That is what is left when you take away the trend and the season. It is the remainder.

To spot the broken heater, you do not compare January with July. You ask: after the usual rise and the usual winter, how much is left? A large remainder is the anomaly.

The problem: baselines still see the trend

Section titled “The problem: baselines still see the trend”

In the last lesson, the seasonal baseline compared each day with the same weekday in the 4 weeks before. It still reported 32 days after the holidays were marked. Part of that is noise from only four values. Part is the year: in spring, the orders rise every week, so each day is higher than the four before it. A detector that does not know the trend reports normal growth as “unusual”.

So we want to take the trend and the season out first, and then look at what is left.

We write every value as a sum of three parts:

orders = trend + season + remainder

Here is a small example. Three made-up weeks have a level that rises slowly, a weekly pattern, and one low Thursday (500 orders) in week 3. We split them by hand:

  1. Trend: the mean of the 7 days around each day (a moving average).
  2. Season: for each weekday, the mean of what is left after the trend. All Thursdays share one value.
  3. Remainder: orders − trend − season.
Week Day Orders Trend Season Remainder
1 Thu 600 652.9 −92.9 40.0
1 Sat 800 658.6 144.3 −2.9
2 Thu 620 672.9 −92.9 40.0
2 Sat 820 678.6 144.3 −2.9
3 Wed 630 670.0 −48.3 8.3
3 Thu 500 672.9 −92.9 −80.0
3 Fri 680 678.3 −9.9 11.6

Against the mean of all 21 days, the low Thursday has a z-score of only −1.76, because the busy weekends make the spread large. In the remainder, it stands out: −80.0, while every other day is between −23.9 and 40.0.

Look at the other two Thursdays. Their remainder is +40.0, not 0. The low Thursday pulled the “Thursday effect” down to −92.9. So the normal Thursdays now look a little high. One odd day spoiled the season for every day like it. A good method must not be pulled by odd days.

STL means Seasonal and Trend decomposition using Loess. Loess is a way to draw a smooth line through noisy points. For each point, it fits a short line through its neighbours, and gives near neighbours more weight. STL uses loess twice, once for the trend and once for the season, and repeats until both are stable.

STL has three strengths over our hand split:

  • The trend can bend: it follows a spring rise and a winter fall.
  • The season can change slowly over the years.
  • With robust=True, STL gives odd days less weight, so one Black Friday does not spoil every Friday.

You choose one number: the period, the length of one cycle. For daily orders with a weekly pattern, the period is 7. For hourly data with a daily pattern, it is 24. The forecasting course, Forecasting and Time Series: Predict What Comes Next, explains decomposition in depth, with more than one season.

Daily orders, 1,224 on Black Friday 2025, split into three boxes: trend 533.8 (the slow level), season minus 34.6 (the usual Friday effect) and remainder 724.8 (what is left). An arrow from the remainder goes to a question: is the robust z of the remainder above 3.5? If yes, a second question: is it a known event, a holiday or a campaign? If yes, mark it, not reported. If no, report the day.
STL splits each day into three parts. We detect on the remainder only, and we do not report the known events.
Text description of the diagram

On the left, the daily orders: 1,224 on Black Friday 2025. Three arrows lead to three boxes. The trend, the slow level, is 533.8. The season, the usual Friday effect, is minus 34.6. The remainder, what is left, is 724.8. The three parts add up to 1,224. From the remainder, an arrow goes to a question: is its robust z-score above 3.5? If yes, a second question: is it a known event, such as a holiday or a campaign? If yes, the day is marked and not reported. If no, the day is reported.

The notebook fills the missing day (19 October 2025) from its neighbours, because STL needs a value on every day. Then it runs STL(filled, period=7, robust=True).

  • Orders a day
  • STL trend
Larkfield's daily orders and the STL trend. The trend follows the garden season: high in spring and early summer, low in winter. It also rises in the heat-wave week, to 953 orders on 5 July 2025.
Text description of the chart

A noisy line of daily orders from April 2024 to March 2026, and a smooth line through it: the STL trend. The trend is about 730 in September 2024 and falls to about 500 in January and February 2025. It rises in spring 2025 to about 915 in mid-May and falls in June. It has a sharp bump in the heat-wave week, where it reaches 953 on 5 July. It is about 600 to 670 in the second winter. Black Friday and Christmas stand out of the noisy line, but the trend does not follow them.

Here is one ordinary week, as STL splits it:

Day Orders Trend Season Remainder
Mon 9 Sep 2024 743 733.6 −6.9 16.3
Tue 10 Sep 2024 675 732.5 −51.1 −6.4
Wed 11 Sep 2024 734 731.2 −19.2 22.0
Thu 12 Sep 2024 698 729.4 −21.4 −10.0
Fri 13 Sep 2024 697 728.0 −17.9 −13.1
Sat 14 Sep 2024 772 727.4 55.6 −11.0
Sun 15 Sep 2024 792 727.9 41.4 22.7

Each row adds up: 727.4 + 55.6 − 11.0 = 772 on the Saturday. Every remainder of this week is small, between −13.1 and 22.7.

The STL weekly season for four weeks of September 2024. Saturday is 40 to 59 orders above the trend, and Tuesday 31 to 54 below. The shape repeats, but its size changes from week to week: STL lets the season change.
Text description of the chart

Twenty-eight dots joined by a line, over four weeks from Monday 2 September 2024. The same shape repeats each week. It is low on Tuesday (minus 31 to minus 54) and around minus 10 to minus 30 on Wednesday to Friday. It is high on Saturday (plus 40 to plus 59) and Sunday (plus 20 to plus 59). The size changes a little from week to week. The weekly pattern is small next to the yearly trend.

Now we detect on the remainder, with the robust z-score from Module 2. The remainder has a median of 0.4 and a MAD of 14.6. A robust z above 3.5 means a remainder more than 3.5 × 1.4826 × 14.6 = 75.6 orders from the median.

Black Friday 2025 shows the split at work:

  • orders 1,224 = trend 533.8 + season −34.6 + remainder 724.8

The trend did not follow Black Friday: one day is too short for a trend. So almost all of it is in the remainder, far above the limit.

The notebook counts the alarms after the warm-up of the last lesson:

limit: remainder beyond 75.6 orders from 0.4
alarm days: 91 | on or next to a holiday: 16

Then it marks the holidays and the days next to them as known events:

reported after marking holidays: 75
the first ten: ['2024-05-06', '2024-05-21', '2024-05-23', '2024-06-08', '2024-06-12', '2024-06-23', '2024-07-01', '2024-07-04', '2024-07-10', '2024-07-18']
  • STL remainder
  • Limits (−75.2 and 76.0)
  • Reported day (75)
The STL remainder of the daily orders, with the limits at 75.6 orders from the median. The 75 dots are the days still reported after the holidays are marked. Black Friday has the largest remainder, but it is a known event, so it is not a dot.
Text description of the chart

The remainder line stays mostly between minus 50 and plus 50 orders, around zero. Two dashed lines mark the limits at minus 75.2 and plus 76.0. Seventy-five dots mark the reported days: they are spread over both years, some above and some below the limits. Black Friday 2025 has the largest remainder, 724.8, marked with a label; it is a holiday, so it is not reported. The heat-wave peak on 5 July 2025 has a remainder of 348.0 and is reported.

STL reports 75 days, more than the seasonal baseline (32). Why?

On normal days, STL’s remainder is small, so its limit is narrow: 75.6 orders. Any day that the trend and the weekly season do not explain goes over it. That includes the planted situations of the shared series, such as the heat wave. It also includes many ordinary busy or quiet days. A narrow limit on a clean remainder is sensitive. That is good for finding things, and bad for Grace’s time.

Look at the heat wave, too. On 5 July 2025, the trend rose to 953.0, so the remainder is only 348.0. A week of high orders is long enough for the trend to follow it. STL then reports the peak day, but it treats part of the heat wave as “the new level”.

And there is a second limit. STL computes each day’s trend from the days around it, before and after. On the morning of 6 July, you do not know 7 July yet. So STL is a tool to study history, not a live alert. The next lesson forecasts each day from the past only.

Amira’s ticket API has two planted incidents with very different shapes:

  • a slow memory leak, from 22 September 2025, 03:00, until the restart on 27 September, 04:30: the latency rises slowly for five days;
  • an error spike after a release, on 8 October 2025, 14:05 to 14:45: errors, and slower requests, for 40 minutes.

The notebook takes the hourly mean of the p95 latency (95% of the requests were faster than this), and runs STL with a daily season: period=24. It alarms when the robust z of the remainder is above 4.

  • p95 latency, hourly mean
  • STL trend
The ticket API's hourly latency and the STL trend. During the memory leak, the trend rises with the latency, from 201.7 ms to 255.2 ms, then drops at the restart. The trend takes the leak in.
Text description of the chart

The hourly latency moves up and down every day between about 180 and 220 milliseconds. From 22 September, it rises steadily for five days, up to above 350 milliseconds, then drops back at the restart on 27 September. The smooth trend line rises with it, from 201.7 to 255.2 milliseconds, and drops with it. From 20 October, after the configuration change, the latency and the trend stay higher.

  • Robust z of the remainder
  • Limits ±4
  • Planted leak (start and restart)
The STL remainder of the hourly latency as a robust z, with the limits ±4. Inside the leak, the remainder stays inside the band for four days; the first alarm comes on 26 September at 07:00, 6,000 minutes after the start. The release spike is one sharp hour, z 14.54. The restart gives a large negative remainder, z −31.23.
Text description of the chart

A line of robust z values, mostly between minus 4 and plus 4, with dashed limits at plus and minus 4. On weekend mornings and evenings, it dips below minus 4 many times. Two vertical dashed lines mark the start of the planted leak (22 September, 03:00) and the restart (27 September, 04:30). Between them, the line stays inside the band for most of four days. Then it rises above 4 from 26 September, 07:00, up to about 34. After the restart it falls to minus 31.23. On 8 October at 14:00, one sharp peak reaches 14.54: the release spike. From 20 October, 10:00, the line jumps above 4 for some hours: the configuration change.

The notebook’s comparison-table row agrees:

{'alert events': 57, 'false alert events': 54, 'minutes to detection': {'memory_leak': 6000, 'config_change': 0}}
Detector Alert events a day Memory leak found after Configuration change found after
STL residual, hourly latency 0.95 6,000 minutes 0 minutes
Seasonal baseline, latency 1.53 506 minutes 3 minutes

STL has 54 false alert events. Where do they come from? Outside the planted incidents, the detector alarms in 163 hours. 121 of them are on Saturdays and Sundays, and in all 121 the latency is lower than expected. A period of 24 hours teaches STL one day, the same for all seven days. Weekends have about half the requests, so their busy hours are quieter and faster than a weekday’s. STL with one period cannot see two cycles. statsmodels has MSTL for several seasons (a day and a week); we did not run it here.

The trend absorbs the leak. A slow rise is exactly what a trend is for. So STL moves the leak into the trend, and the remainder stays normal. On the leak’s first day, the largest |z| is 0.96. The first alarm comes on 26 September at 07:00, 6,000 minutes (100 hours) after the start. A seasonal baseline on the latency compares with the same minute one and two weeks before. It sees the rise after 506 minutes. Many metrics at once builds it.

The release spike is the opposite. One sudden hour is too short for any trend, so it goes straight into the remainder: z 14.54 at 14:00. The configuration change on 20 October is a sudden step, so STL also sees it at once.

And look at the restart. Amira restarted the service, and the latency dropped at once. The remainder fell to z −31.23. A fix can look like an anomaly too. Count the restart as a known event, as you count a holiday.

Use it to study a finished stretch of history: “which days of last year were unusual, after the season?”. It is fast and easy to explain: “the orders were 724.8 above the trend and the usual Friday”.

Do not use it as a live alert, because it uses the days after each day. And do not use it alone for slow changes: the trend takes them in.

What it costs: one setting (the period), and the care to fill missing values first. A narrow limit on a clean remainder reports many days: mark the known events.

Common mistake

STL with a missing day: no error, and silent days

Symptom: you run STL on the orders without filling the missing day (19 October 2025). Nothing fails. But fewer days alarm than you expect.

Diagnose: the notebook runs STL on the series with the gap left as NaN, and counts the days that have no remainder:

broken = STL(orders, period=7, robust=True).fit() # the missing day left as NaN, not filled
print("days with no remainder:", int(broken.resid.isna().sum()), "of", len(orders))
days with no remainder: 386 of 728

One missing value spreads through the loess fits. A robust z of NaN is never above 3.5, so 386 days can never alarm.

Recover: fill a missing value before STL (here, from its neighbours with interpolate). Keep the row count of each day as a separate check, so the gap is still reported to Mei as a data problem.

The notebook’s section 5 has two tasks.

  1. Count the days that STL reports on the daily orders after the warm-up. Leave out the holidays and the days next to them.
  2. Write the count as stl_reported, then run its check cell.
  3. Find the first hour inside the memory leak (2025-09-22 03:00 to 2025-09-27 04:30) where the hourly STL detector alarms.
  4. Write it as a string like "2025-09-22 03:00:00" in first_leak_alarm, then run its check cell.

Check your understanding

STL on the hourly latency finds the release spike at once, but the memory leak only after 6,000 minutes. Explain the difference in one or two sentences.

Show the answer

STL puts slow changes into the trend and sudden changes into the remainder. The leak rises slowly over five days, so the trend follows it and the remainder stays small. The release spike lasts less than an hour, which is too short for the trend, so it goes straight into the remainder (z 14.54).

Check your understanding

Predict: Grace wants a list, every morning, of yesterday's unusual order count. Is STL on the daily orders a good tool for that? What would you use instead?

Show the answer

Not on its own. STL computes each day’s trend from the days before and after it. Yesterday has no “after” yet, so its split changes as new days arrive. STL is a tool to study history. For a daily alert, use a method that needs only the past: a seasonal baseline, or a forecast with a prediction interval (the next lesson).

Check your understanding

On Black Friday 2025, the remainder is 724.8 orders. On the heat-wave peak (5 July 2025), it is only 348.0, although that day had more orders (1,372 against 1,224). Why?

Show the answer

The trend. Black Friday is one day, too short for the trend to follow, so almost all of its jump is in the remainder. The heat wave lasted a week, so the trend rose with it, to 953.0 on 5 July. The longer an event lasts, the more of it STL treats as the level, and the less is left in the remainder.

Recap

  • STL splits a series into trend, season and remainder. You choose the period: 7 for daily data with a week, 24 for hourly data with a day.
  • Detect on the remainder with a robust z-score, and mark known events (holidays, campaigns, restarts).
  • A narrow limit on a clean remainder is sensitive: on the orders, STL reports 75 days after the holidays are marked.
  • Slow changes go into the trend: STL found the memory leak after 6,000 minutes. Sudden spikes stand out at once.
  • STL uses the days after each day: use it for history, not for live alerts. Fill missing values first.

You can now remove the trend and the season and see what is left. But STL looks at the future, and a live alert cannot. Next, you forecast each day from the days before it, and alert when the real value falls outside the prediction interval. Then you compare five detectors on the same days.

Next: Forecast and compare