Skip to content
Français

Normal changes with the hour and the season

  • Lesson 20
  • 35 min
  • Module 5: Anomalies in time

Where we are

Our detectors so far treat every payment the same, whatever the day. Daily orders are different: Mondays are busy, Sundays are quiet and Black Friday is huge. In this lesson, we see why one threshold fails on a series, and we build rolling and seasonal baselines.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 20 of 38 · Anomaly Detection: Find What Is UnusualNormal changes with the hour and the season
Course overview

What you will be able to do

Compare each day with the right “normal”. A threshold that is the same for every day either misses a bad Tuesday or alarms on every Black Friday.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib. They download the daily orders, the daily tickets and the calendar (less than 1 MB). About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The notebook route (recommended): a browser and a free Google account for Colab, or a free Kaggle account. Nothing to install.
  • The local route: Python 3.12 and VS Code with its Python and Jupyter extensions, set up as in Meet the Larkfield data. Download the notebook from the box above and choose your .venv kernel. The first cell installs the rest.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. The Windows commands and Colab were not tested.

A bakery sells more bread on Saturday than on Tuesday. Every week, the same. Now the baker wants a simple warning: “tell me when a day is strange”.

If she compares every day with the average day, every Saturday looks strange. And a very bad Tuesday looks almost normal, because it is still close to the average. The right question is not “is this day far from the average?”. It is “is this Saturday unusual for a Saturday?”.

This is a contextual anomaly, from Module 1: a value that is normal in one context and strange in another. In a time series, the context is the time: the hour, the weekday, the month, the holiday.

Look at two made-up weeks of orders. The second Thursday is low: 400 orders. Every other day is normal.

Day Orders Global z Change against the same day last week
Mon 610 −0.33 +10
Tue 590 −0.51 +10
Wed 600 −0.42 +10
Thu 400 −2.16 −210
Fri 660 0.11 +10
Sat 830 1.59 +10
Sun 770 1.07 +10

The global z compares each day with the mean of all 14 days (647.9) and their standard deviation (114.6). The low Thursday has a z of −2.16. But the Saturday has 1.59: not far away. A threshold that catches the Thursday is close to catching every Saturday too.

The last column asks a better question. Against the same day of the week before, the Saturday changed by 10 orders, and the Thursday by −210. Only the Thursday is strange. This comparison is called a seasonal baseline: we compare each value with the same moment of the cycle. The cycle is called the season. Here it is one week, so “the same moment” is the same weekday.

Larkfield’s orders have a week and a year

Section titled “Larkfield’s orders have a week and a year”

Larkfield’s daily orders are the shared series of the forecasting course. They cover two years, from 1 April 2024 to 29 March 2026, with all sales channels together. They have a mean of 676.0 orders a day. Each weekday has its own level:

Larkfield's mean orders on each weekday over two years. Saturday (711.9), Sunday (707.0) and Monday (703.3) are busy. Wednesday (637.0) is the quietest day. The axis starts at 600 to show the differences.
Text description of the chart

Seven columns, one per weekday, on an axis from 600 to 720 orders. Monday 703.3, Tuesday 657.5, Wednesday 637.0, Thursday 650.2, Friday 665.1, Saturday 711.9, Sunday 707.0. The weekend columns are in another colour. The axis does not start at zero.

The weekly pattern is small next to the year. Larkfield sells garden products, so spring and early summer are busy, and winter is quiet. Holidays change the day completely: 109 orders on Christmas Day 2024, and 1,224 on Black Friday 2025.

The day with the most orders is not Black Friday. It is 5 July 2025, with 1,372 orders, in the heat wave that the shared series planted (30 June to 6 July 2025). Parasol sales jumped, and the coast ran out of parasols. The 4 weeks before had 762.3 orders a day.

The same idea works at every scale. The ticket API (the help desk’s service, owned by Amira) counts its requests every minute. Look at the median requests a minute for each hour of the day:

  • Monday to Friday
  • Saturday and Sunday
The ticket API's median requests a minute for each hour, over 60 days. On weekdays, 23 a minute at night, 179 at 10:00 and 135 in the evening. Weekends have about half.
Text description of the chart

Two lines over the 24 hours of the day. The weekday line starts at 23 requests a minute at midnight and rises from 6:00 to a peak of 179 at 10:00. It falls to 41 at 15:00, and rises again to 134 and 135 at 19:00 and 20:00. The weekend line has the same shape at about half the height: 13 at night and 99 at 10:00.

At 10:00 on a weekday, 50 requests a minute would be a problem: something stops the customers. At 3:00 on a Sunday, 50 would be a lot. A threshold must know the hour. In the rest of this lesson, we use the daily orders, because they have the longest history.

A baseline is what we call “normal” for one moment. Each detector in this lesson compares today with a different baseline.

A box for today, Thursday 3 July 2025 with 874 orders, with three arrows. The first goes to the global threshold: compare with every day, the mean of the first year plus or minus 3 standard deviations. The second goes to the rolling median: compare with the last month, the median of the 28 days just before today plus or minus 3.5 robust sigma. The third goes to the seasonal baseline: compare with the same weekday, the median of the 4 Thursdays before today plus or minus 3.5 robust sigma.
Three baselines for the same day. Each one answers a different question: far from every day, far from the last month, or far from the same weekday.
Text description of the diagram

On the left, a box for today: Thursday 3 July 2025, with 874 orders. Three arrows leave it. The first arrow, “compare with every day”, goes to the global threshold: the mean of the first year plus or minus 3 standard deviations. The second arrow, “compare with the last month”, goes to the rolling median. It uses the median of the 28 days just before today, plus or minus 3.5 robust sigma. The third arrow, “compare with the same weekday”, goes to the seasonal baseline. It uses the median of the 4 Thursdays before today, plus or minus 3.5 robust sigma.

The global threshold uses one mean and one standard deviation for all days: those of the first year (1 April 2024 to 31 March 2025). The first year has a mean of 642.9 and a standard deviation of 155.4. So the limits are 176.7 and 1,109.1 orders, on every day.

  • Orders a day
  • Global limits (176.7 and 1,109.1)
  • Alarm day (9)
Two years of Larkfield's daily orders and the global limits. The global threshold alarms on 9 days only: two Christmas Days, Black Friday and Cyber Monday 2025, two days of the heat wave and two days just after it, and 17 May 2025. A Tuesday that halved would stay inside the limits.
Text description of the chart

A line of daily orders from April 2024 to March 2026, mostly between 400 and 900. It is higher in spring and summer and lower in winter. Two dashed lines mark the global limits, at 176.7 and 1,109.1 orders, flat for the whole two years. Nine dots mark the alarm days. Below the low limit: Christmas Day 2024 (109) and 2025 (157). Above the high limit: 17 May 2025 (1,174); 5, 6, 7 and 9 July 2025 (1,372, 1,200, 1,313 and 1,150); Black Friday 2025 (1,224) and Cyber Monday 2025 (1,136).

The limits are so far apart that only the biggest days pass them. After a warm-up of 5 weeks, the global threshold alarms on 9 days, and 4 of them are on or next to a holiday. It looks calm. But it is calm because it is blind. A normal Tuesday has about 650 orders. A Tuesday with 330 orders (half of normal) stays inside the limits.

The rolling median follows the level. For each day, it takes the 28 days before it: their median is the baseline, and their MAD is the spread. It alarms when the robust z-score is above 3.5. Because it uses only the days before, it can run every morning on yesterday’s data.

It follows the year: in May it compares with April, and in December with November. But it mixes all weekdays together, so a quiet Wednesday and a busy Saturday share one baseline.

The seasonal baseline compares each day with the same weekday in the 4 weeks before: four Thursdays for a Thursday. Their median is the baseline. Four values give a small MAD, so the spread has a floor: at least 5% of the baseline.

  • Orders a day
  • Seasonal baseline limits
  • Alarm day
June and July 2025 with the seasonal baseline's limits. The limits follow the weekly pattern because each day is compared with its own weekday. The heat wave (30 June to 6 July) goes above them. From one week later, the heat-wave days are in the baseline, and for four weeks the limits are much wider: up to 1,872 orders.
Text description of the chart

Daily orders from 2 June to 3 August 2025, with two dashed lines for the high and low limits of the seasonal baseline. The limits move up and down with the weekdays. Before the heat wave, the orders stay between the limits. From 4 July, the orders cross the high limit and reach 1,372 on 5 July. Dots mark the alarm days on 4, 5, 6, 7 and 9 July. In the next four weeks, the high limit jumps up to between 1,100 and 1,872. The low limit falls, once below zero. The reason: the heat-wave days are now among the four same weekdays of the baseline.

The chart shows a weakness too. A week after the heat wave, the baseline contains the heat-wave days. So the limits are wide for 4 weeks. Every baseline that uses the past also learns the past’s anomalies. The robust median protects against one odd day out of four, not against two.

The notebook builds the three detectors. It counts their alarm days after a warm-up of 5 weeks, from 6 May 2024, when every baseline has 4 weeks of history:

alarm days on or next to a holiday
global threshold 9 4
rolling median, 28 days 19 9
seasonal baseline, same weekday 43 11

Which detector is right? None of them, yet. Each one is a different idea of normal, so each one reports different days.

Holidays are known in advance. Grace’s team knows that Black Friday is busy. A detector that reports Black Friday every year teaches the team to ignore it. So we mark them as known events: the holidays of the calendar and the day before and after each one. A known event is not reported.

The notebook marks them and counts again:

{'global threshold': 5, 'rolling median, 28 days': 10, 'seasonal baseline, same weekday': 32}

The seasonal baseline still reports 32 days. Some are the planted situations of the shared series, such as the heat wave. Many are its own noise: four past values are a small reference, and one odd day among them moves the baseline. The next two lessons use more history: STL looks at the whole series, and a forecast learns from a full year.

Baseline Use it when It costs
Global threshold The series has no week, no season and no trend, for example a sensor at a fixed setting It is blind on a series that moves
Rolling median The level moves slowly, and there is no strong weekly pattern It mixes weekdays; it learns recent anomalies
Seasonal baseline The pattern repeats each week (or each day, for hours) It needs several cycles of history; four values are noisy

All three are cheap: a few lines of pandas. And a person can explain every alarm: “this Thursday has 40% fewer orders than the last four Thursdays”. Start with them before any model.

Common mistake

The missing day becomes zero

Symptom: on 19 October 2025, the orders feed sent no rows, as you saw in Meet the Larkfield data. Mei, the data engineer, owns the feed. If you read the orders with groupby only, the day is not there at all, and no detector can alarm. If some code fills the gap with 0, every detector alarms on a “day with zero orders”. Then the team looks for a sales problem that never happened.

Diagnose: the notebook asks every detector about the day, and counts the day’s rows in the file:

alarms on 2025-10-19: {'global threshold': False, 'rolling median, 28 days': False, 'seasonal baseline, same weekday': False}
rows of 2025-10-19: 0

Recover: read the series with .reindex() on the full list of days, so a gap becomes NaN (missing), not 0. Then check the rows of each day as a separate check, before any detector. A missing day is a data problem for Mei, not a business problem for Grace.

The notebook’s section 5 has two tasks. Each one has a check cell.

  1. Read the summary table. Write the rolling median’s alarm days (before marking holidays) as rolling_alarms.
  2. Run the check cell below it.
  3. Load the daily tickets from data/tickets_daily.csv. Add all channels together.
  4. Build the same seasonal baseline on the tickets.
  5. List the days of the heat-wave week (2025-06-30 to 2025-07-06) on which it alarms, as strings "YYYY-MM-DD". Write the list as heat_wave_alarms.
  6. Run the check cell. It prints Check passed when your list is right.

The heat wave is a real event for the help desk: 537.4 tickets a day, against 349.8 a day in the 4 weeks before. The roster did not change. Grace wants to know about such a week, even though nothing is broken.

Check your understanding

The global threshold reports only 5 days after the holidays are marked, and the seasonal baseline reports 32. Is the global threshold the better detector? Explain.

Show the answer

No. It reports few days because its limits (176.7 and 1,109.1 orders) are so wide that only the biggest days pass them. A Tuesday with half the normal orders stays inside. Few alarms is good only when the detector also finds what matters. The seasonal baseline asks a sharper question (“unusual for this weekday?”); its problem is noise from only four past values, which more history can fix.

Check your understanding

Predict: for the four Thursdays after the heat wave, will the seasonal baseline's limits be wider or narrower than before it? Why?

Show the answer

Wider. The baseline for each day is the same weekday in the 4 weeks before. For 4 weeks after the heat wave, one of those four days is a heat-wave day, with many more orders. The spread (MAD) grows, so the limits move apart. A baseline that learns from the past also learns the past’s anomalies. The median protects against one odd value out of four, but the spread still grows.

Check your understanding

Larkfield's ticket API has about 179 requests a minute at 10:00 on a weekday and 23 at night. Amira wants an alert when requests drop. Which baseline would you choose, and why not the global threshold?

Show the answer

A seasonal baseline: compare each minute with the same minute of the week in the weeks before. A global threshold low enough not to alarm every night (below 23) cannot see a drop at 10:00 from 179 to 50. The context (hour and weekday) decides what normal is, so the baseline must use it.

Recap

  • In a time series, “normal” depends on the hour, the weekday, the season and the holidays. A strange value is often a contextual anomaly.
  • A global threshold is wide and blind. A rolling median follows the level. A seasonal baseline compares with the same moment of the cycle.
  • Mark known events (holidays and the days next to them), or they fill the report.
  • Every baseline from the past also learns the past’s anomalies.
  • A missing day is invisible to every detector of values. Count the rows.

You can now compare each day with a baseline that knows the week. But the seasonal baseline still reports 32 days, many of them noise, and it reacts late to a slow trend. Next, you split the series into trend, season and remainder with STL, and detect on the remainder only.

Next: STL residuals