Skip to content
Français

Many metrics at once

  • Lesson 24
  • 35 min
  • Module 5: Anomalies in time

Where we are

We have detectors for one series at a time. The ticket API has latency, errors and requests, and one incident moves all three at once. In this lesson, we compare one detector per metric with one for all, and we group alerts so that one incident gives one alert, not forty.

Lesson 24 of 38 · Anomaly Detection: Find What Is UnusualMany metrics at once
Course overview

What you will be able to do

Watch several metrics of one service without drowning the team in alerts. One incident should give one alert that says what moved, not forty alerts from four detectors.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the ticket API’s metrics (less than 1 MB) with compare.py. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

The notebook route (Colab or Kaggle) or the local route, as in the first lesson of this module.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and scikit-learn 1.9.1. The Windows commands and Colab were not tested.

A kitchen has a smoke alarm, a heat alarm and a gas alarm. A pan catches fire. All three alarms ring, and each one rings again every minute until the fire is out.

The fire brigade needs one call that says “kitchen fire, smoke and heat”. Forty calls in ten minutes would not help. They would hide the second fire, if there was one.

A service with three metrics is the same. One incident moves all of them at once.

Amira’s ticket API writes one row every minute, for 60 days (1 September to 30 October 2025, 86,400 minutes):

  • requests: how many requests arrived in the minute (mean 70.57);
  • error rate: the share of requests that failed (mean 0.004, so 0.4%);
  • p95 latency: 95% of the requests were faster than this (median 198.2 ms).

We planted three incidents, and each one moves a different metric:

Planted incident When What moves
A slow memory leak 22 September, 03:00, to the restart on 27 September, 04:30 Latency, slowly, for five days
An error spike after a release 8 October, 14:05 to 14:45 Errors, at once; latency a little
A configuration change from 20 October, 10:00 Latency, one step up, for good

The release on 8 October at 14:05 went wrong. For 40 minutes, many requests failed: up to 14.75% in one minute. Then Amira rolled it back. Here is what the error-rate threshold (alert when more than 3% fail in a minute) saw:

  • Error rate
  • Threshold 3%
  • Planted spike
  • Alert minute (34)
The error rate of the ticket API, minute by minute, around the planted release spike. In the 40 minutes of the spike, the error-rate threshold alerts in 34 minutes. Before and after it, most minutes have no error at all.
Text description of the chart

A line of the error rate per minute from 13:30 to 15:29 on 8 October 2025, with a dashed threshold at 3%. Before 14:05, the line is at 0% in 30 of the 35 minutes. Between the two dashed vertical lines at 14:05 and 14:45, it jumps to between 1.47% and 14.75%. Thirty-four dots mark the minutes above 3%. After 14:45, the line goes back to 0%.

Thirty-four alerts from one detector, for one incident. The error count test (more errors than 0.4% of the requests can explain) adds 31 more. The seasonal baseline of latency adds 3, because the latency rose a little too:

  • p95 latency
  • Latency threshold 350 ms
  • Planted spike
  • Seasonal baseline alert (3)
The p95 latency in the same two hours. During the spike it is between 200 and 275 ms, against 170 to 228 ms before. That stays far below the 350 ms threshold, but the seasonal baseline alerts in 3 minutes.
Text description of the chart

A line of the p95 latency per minute from 13:30 to 15:29, between 170 and 228 milliseconds before 14:05. Between 14:05 and 14:45, it is a little higher, between 200 and 275 milliseconds, then falls back. A dashed line at 350 milliseconds, the latency threshold, is far above the line all the time. Three dots, near 14:23 to 14:25, mark the minutes where the seasonal baseline alerts.

The notebook counts the raw alert minutes of the four per-metric detectors together:

alert minutes, four detectors: 14474
alert minutes in the release spike: 34

Over 60 days, the four detectors give 14,474 alert minutes. Nobody can read that. We need two things: a better way to watch the metrics, and a way to turn alert minutes into incidents.

The simplest design gives each metric its own detector. The notebook builds four, and scores each one on the planted incidents of its own metric:

detector alert events false alert events recall minutes to detection
latency threshold 17 0 0.5 {'memory_leak': 3342, 'config_change': None}
error-rate threshold 526 525 1.0 {'release_error_spike': 0}
seasonal baseline, latency 92 78 1.0 {'memory_leak': 506, 'config_change': 3}
error count test 328 327 1.0 {'release_error_spike': 0}

An alert event is a group of alerts less than 30 minutes apart. Recall is the share of planted incidents of that metric with at least one event.

The error-rate threshold finds the release spike at once (0 minutes). But it has 525 false events in 60 days. Why so many? In 2,130 of its 2,736 alert minutes, only one request failed. On a quiet minute, one error in 30 requests is already 3.3%. A share is noisy when the count under it is small.

The error count test asks a better question: are there more errors than the number of requests can explain? It expects 0.4% of the requests to fail. It alerts when the real count is 4 standard deviations above that, and only in minutes with 20 or more requests. It still has 327 false events: better, but not good.

The latency threshold (350 ms) has no false event at all. But it finds the leak only after 3,342 minutes, more than two days. It never sees the configuration change: a step of about 14% stays below 350 ms. The seasonal baseline compares with the same minute 1 and 2 weeks before. It finds the leak after 506 minutes and the configuration change after 3.

Predict. An Isolation Forest sees the three metrics together, in every minute. Will it have more or fewer alert events than the per-metric detectors?

The notebook builds the features first. It scales each metric against its usual value at that hour of the day (its median and MAD), so it compares 10:00 with 10:00. Then an Isolation Forest (200 trees) learns the first two weeks as “normal”. It alerts on the minutes whose score is in the top 0.1% of those two weeks.

detector alert events false alert events recall minutes to detection
Isolation Forest, 3 metrics 61 48 1.0 {'memory_leak': 2752, 'release_error_spike': 0, 'config_change': 1126}
two per-metric detectors 337 286 1.0 {'memory_leak': 192, 'release_error_spike': 0, 'config_change': 404}
Alert events in 60 days, for each per-metric detector and for one Isolation Forest on all three metrics. The Isolation Forest has 61 events and finds all three planted incidents.
Text description of the chart

Five columns, from left to right: latency threshold (Latency) 17 events, error-rate threshold 526, seasonal baseline of latency 92 and error count test 328. The fifth column, in another colour, is the Isolation Forest on three metrics: 61.

The Isolation Forest has 61 events and finds all three incidents. The latency threshold OR the error count test (the “two per-metric detectors” row) has 337 events. One detector for all metrics sees a minute where several things are a little odd at once. So it needs fewer alerts to find the same incidents.

The “two per-metric detectors” row says it finds the memory leak after 192 minutes, much faster than the Isolation Forest (2,752). Do not believe it yet. Its first alert inside the leak is an error alert, at 06:12 on 22 September. The leak does not cause errors. The error count test fires in about 7.9 minutes a day during the leak, and in 7.1 a day overall. That alert was chance. The row’s latency part (the 350 ms threshold) first alerts on 24 September at 10:42. A scorer that counts any alert inside a window rewards noisy detectors. That is why each per-metric detector above was scored on its own metric’s incidents only.

Design Strength Cost
One detector per metric Each alert says what moved: “error rate 12% at 14:06”. Easy to tune one metric at a time. Many detectors, many alerts for one incident.
One detector for all metrics Fewer alerts; sees “everything a little odd at once”. The alert says only “this minute is strange”. A person must find which metric moved.

The comparison table records this cost as the time to explain one alert. It is 2 minutes for a per-metric alert and 5 minutes for the Isolation Forest. Both are assumptions of the course, not measurements. A false alert event costs 3 dollars (also an assumption). So the cost per week is 100 dollars for the two per-metric detectors and 17 dollars for the Isolation Forest.

A common, practical answer is both. One detector for all metrics decides when to alert. The per-metric scores go into the alert’s text to say what moved.

Even a good detector gives several alert minutes for one incident. So the last step is alert grouping: join alerts that are close in time into one alert event, and send one message per event.

Four detectors, the latency threshold, the error-rate threshold, the seasonal baseline and the error count test, give 14,474 alert minutes in 60 days. A box, group, joins alerts less than 30 minutes apart. The result is 496 alert events, 8.3 a day. A dashed arrow from the group box shows the release spike: 34 alert minutes become 1 event.
Alert grouping on the ticket API. The four per-metric detectors give 14,474 alert minutes in 60 days. Joined with a 30-minute gap, they are 496 events, and the whole release spike is one event.
Text description of the diagram

On the left, a box with the four per-metric detectors. They are the latency threshold, the error-rate threshold, the seasonal baseline of latency and the error count test. An arrow goes to “14,474 alert minutes in 60 days”. The next box, “group”, joins alerts that are less than 30 minutes apart. The result is 496 alert events, about 8.3 a day. A dashed arrow from the group box goes to a note: the release spike, 34 alert minutes, becomes 1 event.

Here is the idea on a small example. Three detectors watch 12 minutes; 1 means an alert in that minute:

14:00 14:01 14:02 14:03 14:04 14:05 14:06 14:07 14:08 14:09 14:10 14:11
latency 0 0 1 1 1 0 0 0 0 0 0 1
errors 0 1 1 1 1 1 0 0 0 0 0 0
requests 0 0 0 1 0 0 0 0 0 0 0 0
raw alerts: 10 | minutes with an alert: 6 | events (5-minute gap): 2
  1. Count the raw alerts: one per detector and minute. There are 10.
  2. Join the detectors: a minute with any alert is one alert minute. There are 6 (14:01 to 14:05, and 14:11).
  3. Walk through the alert minutes in time order. Start a new event when the gap since the last alert minute is more than 5 minutes.
  4. 14:01 to 14:05 are one event. 14:11 comes 6 minutes after 14:05, so it starts a second event.

On the ticket API, with a 30-minute gap:

events after grouping (30 minutes): 496
events that overlap the spike: 1
The same 14,474 alert minutes of the four per-metric detectors, grouped with a longer and longer gap. With a 1-minute gap they are 2,931 events, with 5 minutes 2,014, and with 30 minutes 496.
Text description of the chart

Four columns. With no grouping, 14,474 alert minutes. With a 1-minute gap, 2,931 events. With a 5-minute gap, 2,014 events. With a 30-minute gap, 496 events. The 30-minute column is in another colour.

The gap is a choice:

  • Too short, and one incident is many events: the release spike would still be several messages.
  • Too long, and two incidents become one event. A second problem that starts during the first one is hidden inside it.

A good gap is a little longer than the quiet minutes inside a real incident, and much shorter than the time between two incidents. Grouping does not remove false alerts: 496 events in 60 days is still 8.3 a day. Grouping makes each incident one message; a better detector makes fewer messages.

  • Start with one detector per metric, with the right test for each: a seasonal baseline for latency, a count test for errors. Their alerts explain themselves.
  • Add one detector for all metrics when the per-metric detectors send too many alerts, or when incidents show as several small moves at once.
  • Always group the alerts of all detectors of one service into events, and send one message per event, with every metric’s state in it.
  • For dashboards, alert routing and incident response, the MLOps course, MLOps and LLMOps: Operating AI in Production, goes further.

Common mistake

An error-rate threshold on quiet minutes

Symptom: the error-rate threshold sends hundreds of alerts at night and at weekends, and the team starts to ignore it.

Diagnose: count the errors behind each alert minute. On the ticket API, 2,130 of the 2,736 alert minutes have exactly one failed request. The median alert minute has only 27 requests. One error in 27 is 3.7%, above the 3% threshold.

Recover: test the count, not the share. Compare the number of errors with what the number of requests can explain. Skip minutes with very few requests (the error count test uses 20). Or group the minutes into 5 or 15 minutes before you compute the rate, so the count under it is larger.

The notebook’s section 6 has two tasks.

  1. Group the same alert minutes (any_alarm) with a gap of 2 hours instead of 30 minutes.
  2. Count the events. Write the number as events_2h, then run its check cell.
  3. Find the Isolation Forest’s row in table.
  4. Read how many minutes after the start of the memory leak its first event comes. Write it as iso_leak_minutes, then run its check cell.

Check your understanding

The error-rate threshold has 525 false alert events in 60 days; the error count test has 327. Both find the release spike at once. Why is the count test better, and why is it still not good enough?

Show the answer

A share of a few requests is noisy: one error in 30 requests is already 3.3%. And 2,130 of the threshold’s 2,736 alert minutes had exactly one error. The count test compares the errors with what the number of requests can explain, and skips minutes with fewer than 20 requests. It still alarms 327 times because single minutes are small samples. Grouping minutes, or a higher limit, would cut it further.

Check your understanding

Predict: you group the alerts with a gap of 1 day. What happens to the number of events, and what could go wrong?

Show the answer

The number of events falls a lot, because most alerts are less than a day apart. But two incidents that happen on the same day, or one right after the other, become one event. A second incident hidden inside the first one is not reported. The gap must be shorter than the usual time between two real incidents.

Check your understanding

Omar asks: the Isolation Forest has 61 events and the two per-metric detectors 337. Why not use only the Isolation Forest?

Show the answer

Because its alert says only “this minute is strange”. The person on duty must then look at each metric to find what moved, which the course assumes takes 5 minutes, not 2. It also found the leak late (2,752 minutes) and the configuration change after 1,126 minutes. The seasonal baseline found them after 506 and 3. A good design uses one detector for all metrics to decide when to alert, and puts the per-metric scores in the alert.

Recap

  • One incident moves several metrics, and every per-metric detector sends its own alerts: 34 alert minutes for one 40-minute release spike.
  • Use the right test for each metric: a count test for errors, a seasonal baseline for latency.
  • One detector for all metrics (here an Isolation Forest on scaled metrics) needs fewer alerts, but its alerts are harder to explain.
  • Group alerts that are close in time into events: 14,474 alert minutes became 496 events, and the release spike one event. The gap is a choice.
  • A scorer that counts any alert in a window rewards noise: score each detector on the incidents it can see.

In this module, you found anomalies in time: seasonal baselines, STL, forecasts, change points and many metrics. The detectors so far are rules, statistics and trees. Next, in Module 6, a small neural network learns to rebuild normal data, and what it cannot rebuild is strange.

Next: Autoencoders and reconstruction error