STL residuals
- Lesson 21
- 35 min
- Module 5: Anomalies in time
Where we are
Seasonal baselines remove the weekly pattern, but they still react to the trend and to the yearly season. In this lesson, we split the daily orders into trend, season and remainder with STL, detect on the remainder only, and mark the known events so that holidays are not reported.
Lesson 21 of 38 · Anomaly Detection: Find What Is UnusualSTL residuals
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Take the trend and the season out of a series, and look for anomalies in what is left. Then the threshold does not have to know about the year, the week or the slow growth.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and statsmodels. They download the daily orders, the calendar and the ticket API’s metrics (less than 1 MB), with compare.py. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Seasonal baselines and known events, from Normal changes with the hour and the season.
- The robust z-score (median and MAD), from Median, MAD and IQR.
What you need on your computer
The notebook route (Colab or Kaggle) or the local route, as in the last lesson.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and statsmodels 0.15.0. The Windows commands and Colab were not tested.
Your electricity bill
Section titled “Your electricity bill”Your electricity bill changes every month. Three things move it:
- Prices rise slowly over the years. That is the trend.
- Every winter the bill is higher, because you heat the house. That repeats every year. That is the season.
- One month is strange: the heater was broken and ran all day. That is what is left when you take away the trend and the season. It is the remainder.
To spot the broken heater, you do not compare January with July. You ask: after the usual rise and the usual winter, how much is left? A large remainder is the anomaly.
The problem: baselines still see the trend
Section titled “The problem: baselines still see the trend”In the last lesson, the seasonal baseline compared each day with the same weekday in the 4 weeks before. It still reported 32 days after the holidays were marked. Part of that is noise from only four values. Part is the year: in spring, the orders rise every week, so each day is higher than the four before it. A detector that does not know the trend reports normal growth as “unusual”.
So we want to take the trend and the season out first, and then look at what is left.
The idea: trend + season + remainder
Section titled “The idea: trend + season + remainder”We write every value as a sum of three parts:
orders = trend + season + remainder
Here is a small example. Three made-up weeks have a level that rises slowly, a weekly pattern, and one low Thursday (500 orders) in week 3. We split them by hand:
- Trend: the mean of the 7 days around each day (a moving average).
- Season: for each weekday, the mean of what is left after the trend. All Thursdays share one value.
- Remainder: orders − trend − season.
| Week | Day | Orders | Trend | Season | Remainder |
|---|---|---|---|---|---|
| 1 | Thu | 600 | 652.9 | −92.9 | 40.0 |
| 1 | Sat | 800 | 658.6 | 144.3 | −2.9 |
| 2 | Thu | 620 | 672.9 | −92.9 | 40.0 |
| 2 | Sat | 820 | 678.6 | 144.3 | −2.9 |
| 3 | Wed | 630 | 670.0 | −48.3 | 8.3 |
| 3 | Thu | 500 | 672.9 | −92.9 | −80.0 |
| 3 | Fri | 680 | 678.3 | −9.9 | 11.6 |
Against the mean of all 21 days, the low Thursday has a z-score of only −1.76, because the busy weekends make the spread large. In the remainder, it stands out: −80.0, while every other day is between −23.9 and 40.0.
Look at the other two Thursdays. Their remainder is +40.0, not 0. The low Thursday pulled the “Thursday effect” down to −92.9. So the normal Thursdays now look a little high. One odd day spoiled the season for every day like it. A good method must not be pulled by odd days.
STL: the same split, done well
Section titled “STL: the same split, done well”STL means Seasonal and Trend decomposition using Loess. Loess is a way to draw a smooth line through noisy points. For each point, it fits a short line through its neighbours, and gives near neighbours more weight. STL uses loess twice, once for the trend and once for the season, and repeats until both are stable.
STL has three strengths over our hand split:
- The trend can bend: it follows a spring rise and a winter fall.
- The season can change slowly over the years.
- With
robust=True, STL gives odd days less weight, so one Black Friday does not spoil every Friday.
You choose one number: the period, the length of one cycle. For daily orders with a weekly pattern, the period is 7. For hourly data with a daily pattern, it is 24. The forecasting course, Forecasting and Time Series: Predict What Comes Next, explains decomposition in depth, with more than one season.
Text description of the diagram
On the left, the daily orders: 1,224 on Black Friday 2025. Three arrows lead to three boxes. The trend, the slow level, is 533.8. The season, the usual Friday effect, is minus 34.6. The remainder, what is left, is 724.8. The three parts add up to 1,224. From the remainder, an arrow goes to a question: is its robust z-score above 3.5? If yes, a second question: is it a known event, such as a holiday or a campaign? If yes, the day is marked and not reported. If no, the day is reported.
STL on Larkfield’s daily orders
Section titled “STL on Larkfield’s daily orders”The notebook fills the missing day (19 October 2025) from its neighbours, because STL needs a value on every day. Then it runs STL(filled, period=7, robust=True).
The trend
Section titled “The trend”- Orders a day
- STL trend
Text description of the chart
A noisy line of daily orders from April 2024 to March 2026, and a smooth line through it: the STL trend. The trend is about 730 in September 2024 and falls to about 500 in January and February 2025. It rises in spring 2025 to about 915 in mid-May and falls in June. It has a sharp bump in the heat-wave week, where it reaches 953 on 5 July. It is about 600 to 670 in the second winter. Black Friday and Christmas stand out of the noisy line, but the trend does not follow them.
Here is one ordinary week, as STL splits it:
| Day | Orders | Trend | Season | Remainder |
|---|---|---|---|---|
| Mon 9 Sep 2024 | 743 | 733.6 | −6.9 | 16.3 |
| Tue 10 Sep 2024 | 675 | 732.5 | −51.1 | −6.4 |
| Wed 11 Sep 2024 | 734 | 731.2 | −19.2 | 22.0 |
| Thu 12 Sep 2024 | 698 | 729.4 | −21.4 | −10.0 |
| Fri 13 Sep 2024 | 697 | 728.0 | −17.9 | −13.1 |
| Sat 14 Sep 2024 | 772 | 727.4 | 55.6 | −11.0 |
| Sun 15 Sep 2024 | 792 | 727.9 | 41.4 | 22.7 |
Each row adds up: 727.4 + 55.6 − 11.0 = 772 on the Saturday. Every remainder of this week is small, between −13.1 and 22.7.
The season
Section titled “The season”Text description of the chart
Twenty-eight dots joined by a line, over four weeks from Monday 2 September 2024. The same shape repeats each week. It is low on Tuesday (minus 31 to minus 54) and around minus 10 to minus 30 on Wednesday to Friday. It is high on Saturday (plus 40 to plus 59) and Sunday (plus 20 to plus 59). The size changes a little from week to week. The weekly pattern is small next to the yearly trend.
The remainder and its limit
Section titled “The remainder and its limit”Now we detect on the remainder, with the robust z-score from Module 2. The remainder has a median of 0.4 and a MAD of 14.6. A robust z above 3.5 means a remainder more than 3.5 × 1.4826 × 14.6 = 75.6 orders from the median.
Black Friday 2025 shows the split at work:
- orders 1,224 = trend 533.8 + season −34.6 + remainder 724.8
The trend did not follow Black Friday: one day is too short for a trend. So almost all of it is in the remainder, far above the limit.
The notebook counts the alarms after the warm-up of the last lesson:
limit: remainder beyond 75.6 orders from 0.4alarm days: 91 | on or next to a holiday: 16Then it marks the holidays and the days next to them as known events:
reported after marking holidays: 75the first ten: ['2024-05-06', '2024-05-21', '2024-05-23', '2024-06-08', '2024-06-12', '2024-06-23', '2024-07-01', '2024-07-04', '2024-07-10', '2024-07-18']- STL remainder
- Limits (−75.2 and 76.0)
- Reported day (75)
Text description of the chart
The remainder line stays mostly between minus 50 and plus 50 orders, around zero. Two dashed lines mark the limits at minus 75.2 and plus 76.0. Seventy-five dots mark the reported days: they are spread over both years, some above and some below the limits. Black Friday 2025 has the largest remainder, 724.8, marked with a label; it is a holiday, so it is not reported. The heat-wave peak on 5 July 2025 has a remainder of 348.0 and is reported.
STL reports more days, not fewer
Section titled “STL reports more days, not fewer”STL reports 75 days, more than the seasonal baseline (32). Why?
On normal days, STL’s remainder is small, so its limit is narrow: 75.6 orders. Any day that the trend and the weekly season do not explain goes over it. That includes the planted situations of the shared series, such as the heat wave. It also includes many ordinary busy or quiet days. A narrow limit on a clean remainder is sensitive. That is good for finding things, and bad for Grace’s time.
Look at the heat wave, too. On 5 July 2025, the trend rose to 953.0, so the remainder is only 348.0. A week of high orders is long enough for the trend to follow it. STL then reports the peak day, but it treats part of the heat wave as “the new level”.
And there is a second limit. STL computes each day’s trend from the days around it, before and after. On the morning of 6 July, you do not know 7 July yet. So STL is a tool to study history, not a live alert. The next lesson forecasts each day from the past only.
STL on the ticket API
Section titled “STL on the ticket API”Amira’s ticket API has two planted incidents with very different shapes:
- a slow memory leak, from 22 September 2025, 03:00, until the restart on 27 September, 04:30: the latency rises slowly for five days;
- an error spike after a release, on 8 October 2025, 14:05 to 14:45: errors, and slower requests, for 40 minutes.
The notebook takes the hourly mean of the p95 latency (95% of the requests were faster than this), and runs STL with a daily season: period=24. It alarms when the robust z of the remainder is above 4.
- p95 latency, hourly mean
- STL trend
Text description of the chart
The hourly latency moves up and down every day between about 180 and 220 milliseconds. From 22 September, it rises steadily for five days, up to above 350 milliseconds, then drops back at the restart on 27 September. The smooth trend line rises with it, from 201.7 to 255.2 milliseconds, and drops with it. From 20 October, after the configuration change, the latency and the trend stay higher.
- Robust z of the remainder
- Limits ±4
- Planted leak (start and restart)
Text description of the chart
A line of robust z values, mostly between minus 4 and plus 4, with dashed limits at plus and minus 4. On weekend mornings and evenings, it dips below minus 4 many times. Two vertical dashed lines mark the start of the planted leak (22 September, 03:00) and the restart (27 September, 04:30). Between them, the line stays inside the band for most of four days. Then it rises above 4 from 26 September, 07:00, up to about 34. After the restart it falls to minus 31.23. On 8 October at 14:00, one sharp peak reaches 14.54: the release spike. From 20 October, 10:00, the line jumps above 4 for some hours: the configuration change.
The notebook’s comparison-table row agrees:
{'alert events': 57, 'false alert events': 54, 'minutes to detection': {'memory_leak': 6000, 'config_change': 0}}| Detector | Alert events a day | Memory leak found after | Configuration change found after |
|---|---|---|---|
| STL residual, hourly latency | 0.95 | 6,000 minutes | 0 minutes |
| Seasonal baseline, latency | 1.53 | 506 minutes | 3 minutes |
Most false events are weekends
Section titled “Most false events are weekends”STL has 54 false alert events. Where do they come from? Outside the planted incidents, the detector alarms in 163 hours. 121 of them are on Saturdays and Sundays, and in all 121 the latency is lower than expected. A period of 24 hours teaches STL one day, the same for all seven days. Weekends have about half the requests, so their busy hours are quieter and faster than a weekday’s. STL with one period cannot see two cycles. statsmodels has MSTL for several seasons (a day and a week); we did not run it here.
Why STL finds the leak late
Section titled “Why STL finds the leak late”The trend absorbs the leak. A slow rise is exactly what a trend is for. So STL moves the leak into the trend, and the remainder stays normal. On the leak’s first day, the largest |z| is 0.96. The first alarm comes on 26 September at 07:00, 6,000 minutes (100 hours) after the start. A seasonal baseline on the latency compares with the same minute one and two weeks before. It sees the rise after 506 minutes. Many metrics at once builds it.
The release spike is the opposite. One sudden hour is too short for any trend, so it goes straight into the remainder: z 14.54 at 14:00. The configuration change on 20 October is a sudden step, so STL also sees it at once.
And look at the restart. Amira restarted the service, and the latency dropped at once. The remainder fell to z −31.23. A fix can look like an anomaly too. Count the restart as a known event, as you count a holiday.
When to use STL, and what it costs
Section titled “When to use STL, and what it costs”Use it to study a finished stretch of history: “which days of last year were unusual, after the season?”. It is fast and easy to explain: “the orders were 724.8 above the trend and the usual Friday”.
Do not use it as a live alert, because it uses the days after each day. And do not use it alone for slow changes: the trend takes them in.
What it costs: one setting (the period), and the care to fill missing values first. A narrow limit on a clean remainder reports many days: mark the known events.
Common mistake
STL with a missing day: no error, and silent days
Symptom: you run STL on the orders without filling the missing day (19 October 2025). Nothing fails. But fewer days alarm than you expect.
Diagnose: the notebook runs STL on the series with the gap left as NaN, and counts the days that have no remainder:
broken = STL(orders, period=7, robust=True).fit() # the missing day left as NaN, not filledprint("days with no remainder:", int(broken.resid.isna().sum()), "of", len(orders))days with no remainder: 386 of 728One missing value spreads through the loess fits. A robust z of NaN is never above 3.5, so 386 days can never alarm.
Recover: fill a missing value before STL (here, from its neighbours with interpolate). Keep the row count of each day as a separate check, so the gap is still reported to Mei as a data problem.
Your turn
Section titled “Your turn”The notebook’s section 5 has two tasks.
- Count the days that STL reports on the daily orders after the warm-up. Leave out the holidays and the days next to them.
- Write the count as
stl_reported, then run its check cell. - Find the first hour inside the memory leak (2025-09-22 03:00 to 2025-09-27 04:30) where the hourly STL detector alarms.
- Write it as a string like
"2025-09-22 03:00:00"infirst_leak_alarm, then run its check cell.
Check your understanding
STL on the hourly latency finds the release spike at once, but the memory leak only after 6,000 minutes. Explain the difference in one or two sentences.
Show the answer
STL puts slow changes into the trend and sudden changes into the remainder. The leak rises slowly over five days, so the trend follows it and the remainder stays small. The release spike lasts less than an hour, which is too short for the trend, so it goes straight into the remainder (z 14.54).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Grace wants a list, every morning, of yesterday's unusual order count. Is STL on the daily orders a good tool for that? What would you use instead?
Show the answer
Not on its own. STL computes each day’s trend from the days before and after it. Yesterday has no “after” yet, so its split changes as new days arrive. STL is a tool to study history. For a daily alert, use a method that needs only the past: a seasonal baseline, or a forecast with a prediction interval (the next lesson).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On Black Friday 2025, the remainder is 724.8 orders. On the heat-wave peak (5 July 2025), it is only 348.0, although that day had more orders (1,372 against 1,224). Why?
Show the answer
The trend. Black Friday is one day, too short for the trend to follow, so almost all of its jump is in the remainder. The heat wave lasted a week, so the trend rose with it, to 953.0 on 5 July. The longer an event lasts, the more of it STL treats as the level, and the less is left in the remainder.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- STL splits a series into trend, season and remainder. You choose the period: 7 for daily data with a week, 24 for hourly data with a day.
- Detect on the remainder with a robust z-score, and mark known events (holidays, campaigns, restarts).
- A narrow limit on a clean remainder is sensitive: on the orders, STL reports 75 days after the holidays are marked.
- Slow changes go into the trend: STL found the memory leak after 6,000 minutes. Sudden spikes stand out at once.
- STL uses the days after each day: use it for history, not for live alerts. Fill missing values first.
You can now remove the trend and the season and see what is left. But STL looks at the future, and a live alert cannot. Next, you forecast each day from the days before it, and alert when the real value falls outside the prediction interval. Then you compare five detectors on the same days.
Next: Forecast and compare