Skip to content
Français

Faults in machine or server metrics

  • Case study 2
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, we found the memory leak and the release spike in the ticket API's metrics. Now we try the same detectors on real machines and servers, with faults that really happened. In this case study, we compare three methods by the faults they find, how late, and the false pages a day.

Case study 2 of 4 · Anomaly Detection: Find What Is UnusualFaults in machine or server metrics
Course overview

What you will be able to do

Watch 38 metrics of real servers for faults, and choose a detector that you can defend. You will set every threshold the same way, with no label, for a page budget. You will compare a robust z-score, Isolation Forest and a live change-point score. You score each by the faults it finds, how late, and the false pages a day. Then you run the choice once on a server that no step has seen.

Case studyPortfolio piece

Licence
MIT
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Fault windows as labels
  • Page budget from normal data
  • Robust z per metric
  • Isolation Forest on windows
  • Live change-point score
  • Time to detection
  • Explain a page
  • One run on new input

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and ruptures. They download three servers from the original source (about 55 MB) and check each file. After the setup, the notebook runs in under a minute on a laptop.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12 and the versions in the setup cell (pandas 3.1.1, NumPy 2.5.4, scikit-learn 1.9.1, ruptures 1.1.10). The setup cell makes a new folder server-metrics. To start again, delete that folder.

Tested on macOS (Apple silicon) with Python 3.12.13, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

A smoke alarm that rings every time you make toast is soon switched off. One that rings ten minutes after the fire starts is too late. Server alerts fail in the same two ways.

Hiroshi leads the on-call rota of a team that runs many servers. When an alert starts, the engineer on call gets a page: a message to the phone, day or night. Kwame is the team’s data scientist. We invented both people for this case study. The servers and their faults are real.

Hiroshi’s rules for Kwame are the course’s kind of rules. Each is an assumption, not a measurement:

  • The page budget: on normal days, at most 1 page a day per server.
  • A false page costs 15 minutes of an engineer’s time, to look, decide and go back to sleep.
  • An alert that stays on pages again after 30 minutes. Many paging tools work like this.

Hiroshi asks Kwame: “Which detector should page us? How many real faults will it find, how late, and how many false pages will it send?”

You will build three detectors on 38 metrics per server. You will set each one’s threshold for the same page budget, with no label. Then you score them on 21 real faults of two servers, choose one, and run it once on a third server with 4 faults.

The data is SMD, the Server Machine Dataset. The research group NetManAIOps collected it from 28 servers of “a large Internet company” and published it with their OmniAnomaly method. Each server has 38 metrics, one value per minute, for about five weeks.

The files of one server:

File What it holds
train-<m>.txt The first half: one row per minute, 38 values. No labels.
test-<m>.txt The second half, in the same form.
label-<m>.txt One 0 or 1 per minute of the test half: 1 means “inside a fault”.
interp-<m>.txt For each fault, the metrics that the dataset’s authors found behind it.

The metrics have no names. They are measures such as CPU, memory, network and disk, but the dataset does not say which is which. The authors scaled every metric to the range 0 to 1. There are no dates, only minute numbers.

We compared two other datasets. NAB, the Numenta Anomaly Benchmark, has server metrics too, but one metric per file. Its data’s terms are also unclear. The repository was under the AGPL licence from 2015 to December 2024, then MIT, and much of its data comes from third parties. SKAB is under the GPL, which is not on our list of allowed licences.

Kwame takes three servers, chosen by a rule before any detector runs: the first server of each of the three groups.

  • machine-1-1 and machine-2-1 are for the comparison. Together they have 21 faults in 36.2 test days.
  • machine-3-1 is the new input. It is used once, after the choice. It has 4 faults in 19.9 test days.

A rule chosen in advance stops you from picking the servers where your favourite method looks good.

machine-1-1: train (28479, 38), test (28479, 38) = 19.8 days, 9.5% of test minutes in a fault, 8 fault windows
window lengths (minutes): [546, 554, 457, 721, 409, 3, 2, 2]
machine-2-1: train (23693, 38), test (23694, 38) = 16.5 days, 4.9% of test minutes in a fault, 13 fault windows
window lengths (minutes): [22, 54, 46, 26, 8, 452, 17, 33, 75, 16, 12, 21, 388]
missing values: 0
smallest and largest value: 0.0 0.0 | 1.0 1.0
metrics that never change in the training half: 8 of 38
metrics that never change in the test half: 6 of 38

What this tells us. The two servers are different. machine-1-1 has five long faults of 7 to 12 hours, and three tiny ones of 2 or 3 minutes. machine-2-1 has 13 faults, most of them under an hour. There are no missing values. Eight metrics never move in the training half. Their spread is 0, so a z-score would divide by zero. Kwame gives every metric a spread of at least 1% of its range.

  • Metric 1
  • Start and end of the labelled fault
Metric 1 of machine-1-1 around its first labelled fault (546 minutes). It jumps from 0.086 to 0.495 as the fault starts. At minute 270 it drops back to about 0.1, then climbs again until the fault ends.
Text description of the chart

The line shows metric 1 of machine-1-1, minute by minute. It starts 240 minutes before the first labelled fault and ends 60 minutes after it. Before the fault, it is low, between 0.04 and 0.13. At minute 0, it jumps to about 0.5 and stays near 0.5 until minute 269. At minute 270, it drops to about 0.1 for about half an hour, then climbs to 0.68 with peaks and falls. In the hour after the label ends, it is between 0.19 and 0.34. Two dashed vertical lines mark the start and the end of the label, 545 minutes apart.

What this tells us. This fault starts as a step: the level moves at once and stays for hours. Any detector that compares this minute with normal should see it. Other faults last 2 minutes, or move several metrics a little.

In the course’s comparison table, a payment was one unit. A server fault is different. It is a window of time, from its first labelled minute to its last. One fault gives hundreds of unusual minutes, but Hiroshi needs one page, early.

So we score each detector by three numbers, as for the ticket API in Many metrics at once:

  • Faults found: a fault is found when the detector alerts inside its window.
  • Minutes to detection: from the start of the window to the first alert inside it.
  • False pages a day: pages that fall outside every fault window.
A flow from left to right. Score each minute: above the threshold means an alert minute. Page the engineer at the first alert minute, and again after 30 minutes if alerts go on. A question: is the page inside a labelled fault? Yes: a true page, the fault is found, and the minutes to detection are the first alert minus the fault start. No: a false page, which costs 15 minutes (an assumption).
How a detector is scored in this case study. A page is sent at the first alert minute, and again every 30 minutes while the alerts go on.
Text description of the diagram

The diagram is a flow from left to right. The first box says: score each minute; a score above the threshold makes an alert minute. The second box says: page the engineer at the first alert minute, and again after 30 minutes if the alerts go on. A diamond asks: is the page inside a labelled fault? The yes arrow goes to “true page: the fault is found; minutes to detection are the first alert minus the fault start”. The no arrow goes to “false page: it costs 15 minutes (an assumption)”.

Why pages, and not the course’s alert events (alerts less than 30 minutes apart, joined into one)? A detector that alerts every 20 minutes for ten days makes one event, and Hiroshi’s phone rings all the time. A failure case below shows it.

Take 120 made-up minutes. One fault runs from minute 50 to minute 59. A detector alerts at minute 10, at minutes 52 to 54, at minutes 70 to 85, and at minute 100.

pages at minutes: [10, 52, 82]
{'pages': 3, 'false pages': 2, 'faults found': 1, 'minutes to detection': [2]}
  • Minute 10 is the first alert, so it pages. It is outside the fault: a false page.
  • Minute 52 is 42 minutes after the last page, so it pages. It is inside the fault: a true page. The fault is found 2 minutes after its start.
  • The alert from 70 to 85 starts only 18 minutes after the page at 52. It pages once, at minute 82, 30 minutes after that page. That is a false page.
  • Minute 100 is 18 minutes after the page at 82, so it does not page.

The same threshold rule for every detector

Section titled “The same threshold rule for every detector”

Each detector gives a score per minute, and a threshold turns the score into alerts. So every threshold is chosen the same way, with no label:

The training half of each server, with no labels, is split into the fit part, the first 80 percent, where each detector learns normal, and the calibration part, the last 20 percent, where the threshold is set for 1 page a day. The test half, with labels, is scored: pages, faults found, minutes to detection. A note: machine-1-1 and machine-2-1 are compared, 21 faults; machine-3-1 is the new input, used once, 4 faults.
The split for every detector and every server. No label is used before the test half.
Text description of the diagram

On the left is the training half of a server, about two and a half weeks, with no labels. Its first 80% is the fit part: each detector learns normal there. Its last 20% is the calibration part: the threshold is set there for 1 page a day. An arrow goes to the test half, with labels, where the pages, the faults found and the minutes to detection are scored. A note says: machine-1-1 and machine-2-1 are compared, with 21 faults; machine-3-1 is the new input, used once, with 4 faults.

  1. Learn normal on the first 80% of the training half (the fit minutes).
  2. On the last 20% (the calibration minutes, about 3 to 4 days), choose the lowest threshold that pages at most once a day.
  3. Run on the test half, and score it with the labels.

The training half may contain faults. We treat it as “mostly normal”, as a real team would. This is the rule of The alert budget, with pages a day in place of alerts a day.

For each metric, its median and MAD on the fit minutes give a robust z-score, as in Median, MAD and IQR. The detector’s score is the largest |z| of the 38 metrics in that minute.

def robust_scale(fit):
"""Each metric's median and spread (1.4826 x MAD, at least MIN_SCALE) on the fit minutes."""
med = np.median(fit, axis=0)
spread = np.maximum(1.4826 * np.median(np.abs(fit - med), axis=0), MIN_SCALE)
return med, spread
def robust_z_score(x, med, spread):
"""The baseline: the largest |robust z| over the 38 metrics, minute by minute."""
return np.abs((x - med) / spread).max(axis=1)
machine-1-1: threshold |z| > 20.3; {'pages': 156, 'false pages': 83, 'faults found': 8, 'faults': 8, 'minutes to detection': [1, 0, 0, 0, 0, 0, 0, 0]}
machine-2-1: threshold |z| > 49.6; {'pages': 23, 'false pages': 14, 'faults found': 7, 'faults': 13, 'minutes to detection': [None, None, 45, 0, None, 449, 5, 28, None, None, None, 14, 8]}

What this tells us. The thresholds are very high: |z| above 20 and above 50. Many metrics hardly move on normal minutes: 17 of the 38 metrics of machine-1-1 have a spread at the 1% floor. On such a metric, an ordinary small spike gives a huge z, so the budget pushes the threshold far up. On machine-1-1, the baseline finds all 8 faults within a minute, the tiny ones too. On machine-2-1, it finds only 7 of 13, and one of them 449 minutes late. In the 6 faults that it misses, no metric reaches |z| = 49.6.

We did not run the course’s other baseline, STL residuals. STL with a one-day season (1,440 minutes) took 73 seconds for one metric of one test half, on our laptop. At that speed, 38 metrics on three servers take more than two hours, far too slow for a notebook.

Isolation Forest sees all metrics at once, as in Isolation Forest. A single minute is noisy, so Kwame gives it short windows. For each minute, the features are the mean and the standard deviation of each metric over the last 10 minutes: 76 features. The forest has 200 trees and learns the fit minutes. Its score is how fast a minute is cut free.

machine-1-1 (0.3 s): {'pages': 126, 'false pages': 84, 'faults found': 5, 'faults': 8, 'minutes to detection': [254, 239, 226, 340, 258, None, None, None]}
machine-2-1 (0.2 s): {'pages': 25, 'false pages': 15, 'faults found': 8, 'faults': 13, 'minutes to detection': [0, None, 45, None, None, 449, 5, 30, None, 6, None, 14, 8]}

What this tells us. On machine-1-1, Isolation Forest finds the five long faults 226 to 340 minutes late, and misses the three tiny ones. The step in metric 1 that you saw above is one feature of 76. A likely reason: the trees rarely cut on that one feature. So a minute where only it moved is not cut free fast. Features decide more than the algorithm made the same point with payments. On machine-2-1, it is about as good as the baseline.

Many faults here are steps: a level moves and stays. Change points found steps with PELT and binary segmentation. But those methods cut a finished series. They need the minutes after the change. A page must come now.

Think of a bathroom scale. You do not need a year of weighings to see that something changed. You compare the last few days with the days before them. Kwame does the same at every minute. He takes only the last 60 minutes. He asks: where is the best single cut in this hour, and how much does it lower the squared error? If one level fits the hour well, no cut helps much. If the level moved inside the hour, the best cut lowers the squared error a lot. That drop is the score.

A tiny example. Twelve minutes of one scaled metric: 0.1, −0.2, 0.0, 0.2, −0.1, 0.0, then 2.1, 1.9, 2.0, 2.2, 1.8, 2.0. With no cut, the squared error is 12.20. In this tiny example, each cut leaves at least 3 minutes on each side (the real score uses 5):

cut before minute 4: squared error 7.93, lowered by 4.27
cut before minute 5: squared error 6.50, lowered by 5.70
cut before minute 6: squared error 3.63, lowered by 8.57
cut before minute 7: squared error 0.20, lowered by 12.00
cut before minute 8: squared error 3.97, lowered by 8.23
cut before minute 9: squared error 6.20, lowered by 6.00
cut before minute 10: squared error 8.20, lowered by 4.00

The best cut is before minute 7, where the level moved. It lowers the squared error by 12.00, almost all of it.

A flow from left to right. Every minute, take the last 60 minutes of the 38 scaled metrics. Try every cut with at least 5 minutes on each side. The score is how much the best cut lowers the squared error. A question: above the threshold for 1 page a day? Yes: page, and name the metric with the largest robust z.
The live change-point score. It uses only the past hour, so it can page at once.
Text description of the diagram

The diagram is a flow from left to right. The first box: every minute, take the last 60 minutes of the 38 scaled metrics. The second box: try every cut with at least 5 minutes on each side. The third box: the score is how much the best cut lowers the squared error. A diamond asks: is the score above the threshold for 1 page a day? The yes arrow goes to the last box: page, and name the metric with the largest robust z.

change_score does this at every minute for all 38 metrics at once. Each metric is scaled by its median and spread first, as for the robust z. Running sums make it fast: one test half takes less than a second.

def change_score(z, w=60, m=5):
"""A live change-point score: in the last w minutes, the best single cut with at least m minutes on each
side, and how much it lowers the squared error (the l2 cost of the ruptures package). z: scaled metrics."""
n, d = z.shape
S = np.vstack([np.zeros(d), np.cumsum(z, axis=0)])
out = np.zeros(n)
t = np.arange(w - 1, n) # the window is a = t - w + 1 ... t
a = t - w + 1
best = np.zeros(len(t))
for k in range(m, w - m + 1): # the cut: k minutes before it, w - k after it
c = a + k
left = (S[c] - S[a]) / k
right = (S[t + 1] - S[c]) / (w - k)
gain = (k * (w - k) / w) * ((left - right) ** 2).sum(axis=1)
best = np.maximum(best, gain)
out[w - 1:] = best
return out

Is it really the cut that the course’s tool finds? The notebook compares one minute with ruptures’ binary segmentation, with one cut and the l2 cost:

ruptures: cut after minute 37 of 60; squared error lowered by 945.3
change_score at that minute: 945.3

The measurement script checked 200 random minutes the same way. The largest difference was 1.2 × 10⁻¹², a rounding error.

machine-1-1: {'pages': 185, 'false pages': 116, 'faults found': 8, 'faults': 8, 'minutes to detection': [2, 1, 0, 0, 0, 0, 0, 0]}
machine-2-1: {'pages': 38, 'false pages': 22, 'faults found': 10, 'faults': 13, 'minutes to detection': [0, None, 45, 0, None, 2, 5, 30, None, 6, 8, 14, 8]}

What this tells us. The change-point score finds all 8 faults of machine-1-1 within 2 minutes, and 10 of the 13 of machine-2-1. Look at the sixth fault of machine-2-1: the baseline found it after 449 minutes, the change-point score after 2. A likely reason: the dataset lists four metrics behind this fault. A small step in several metrics at once adds up in the squared error, while no single metric’s z reaches the baseline’s threshold.

  • Robust z
  • Isolation Forest
  • Change point
  • Threshold, start and end of the fault
Each detector's score divided by its threshold, around the first fault of machine-1-1. Above 1 is an alert. The robust z alerts after 1 minute and the change-point score after 2; Isolation Forest only after 254 minutes.
Text description of the chart

Three lines show the score of each detector divided by its own threshold, so that 1 is the threshold for all of them. A dashed line marks 1. Before the fault, all three lines stay mostly below 1. At minute 0, the robust z and the change-point lines jump far above 1 within 2 minutes. The robust z stays above 1 for 82% of the fault. The change-point line falls below 1 after about an hour, then rises again each time the level moves. It is above 1 for 66% of the fault. The Isolation Forest line stays below 1 until minute 254, then rises above it for part of the rest.

Why does the change-point line fall below 1 about an hour after the start? Then the last 60 minutes have one level again: nothing changed inside the hour. A change-point score says “it moved”, not “it is still high”. For a page, that is enough.

Predict. At a budget of 1 page a day per server, which detector finds the most faults within 30 minutes? The simple robust z, Isolation Forest, or the change-point score?

The notebook scores the three detectors on both comparison servers and adds them up. “Engineer hours” are the false pages × 15 minutes, per server and week (an assumption).

pages false pages per day precision faults found found within 30 min median minutes to detection engineer hours a week on false pages
robust z 179 2.68 0.458 15 13 0.0 4.7
Isolation Forest 151 2.73 0.344 13 6 45.0 4.8
change point 223 3.81 0.381 18 17 1.5 6.7

Which column decides? Hiroshi’s question was: find the faults early, within the page budget. So the measure is faults found within 30 minutes, at the same page budget. The other columns explain it:

  • The change-point score finds the most: 17 of 21 faults within 30 minutes, against 13 for the robust z and 6 for Isolation Forest.
  • The robust z has the best precision: 45.8% of its pages are true. It sends fewer false pages (2.68 a day) than the change-point score (3.81).
  • Isolation Forest loses on every column except the number of pages. Its median time to detection is 45 minutes.

The budget was 1 page a day. On the test halves, every detector sends 2.7 to 3.8 false pages a day. The calibration days were calmer than the test days. The worst is machine-1-1, with 4.2 to 5.9 false pages a day. Only 6 to 14 of its 83 to 116 false pages came within an hour of a fault. So the edges of the labels do not explain them.

A threshold set on three or four days is a guess about the next three weeks, as in When “normal” moves. And the detectors do not cost the same, so the table is not quite fair. The chart below makes it fair. For ten budgets, it plots the false pages that each detector really sent against the faults it found within 30 minutes:

  • Robust z
  • Isolation Forest
  • Change point
Faults found within 30 minutes against the false pages a day, for ten budgets (0.25 to 8 pages a day on the calibration minutes). Up and to the left is better. Up to about 7 false pages a day, the change-point line is above the others.
Text description of the chart

Three lines with dots, one per detector, for ten budgets. The horizontal axis is the false pages a day that the detector really sent on the test halves, from 0 to 11. The vertical axis is the faults found within 30 minutes, out of 21. The change-point line starts at 15 faults with 2.1 false pages a day. It is at 17 from 3.8 to 7 false pages, and reaches 19 at about 9. The robust z line starts at 12 faults with 0.44 false pages a day. It is at 13 to 16 between 1 and 5 false pages, and reaches 18 at about 7.6. Isolation Forest stays at 6 faults until about 3 false pages a day and reaches 17 only at about 8.

What this tells us. Between 2 and 5 false pages a day, the change-point score finds 1 to 3 more faults within 30 minutes than the robust z. Above about 7, the robust z catches up. The robust z alone can go very quiet: 12 faults with 0.44 false pages a day. Isolation Forest on these windows is far behind until about 8 false pages a day.

Kwame chooses the change-point score for the page. He chooses it on these two servers only, before he looks at the third.

A page that says “server 1-1 is unusual” makes the engineer search 38 metrics at 3 a.m. Explain an alert asked for the reason in the alert itself. Here the reason is simple: at the first alert minute, name the metric with the largest |robust z|. The change-point score decides when to page, and the robust z says what moved. Many metrics at once ended with the same design.

SMD lets us check it. For each fault, the dataset’s authors listed the metrics behind it.

machine-1-1: the top metric is in the dataset's label for 8 of 8 found faults
[(1, True), (9, True), (9, True), (14, True), (9, True), (9, True), (9, True), (9, True)]
machine-2-1: the top metric is in the dataset's label for 6 of 10 found faults
[(16, False), (10, True), (34, True), (26, True), (18, True), (9, True), (18, False), (11, False), (16, False), (16, True)]

What this tells us. On machine-1-1, the named metric is always one of the authors’ metrics. On machine-2-1, it is right for 6 of 10. Metric 16 is named three times and is wrong twice. The explanation is a good first place to look, not a diagnosis.

The choice is made: the change-point score for the page, the top |z| metric for the explanation. Now Kwame runs all three detectors, with the same rules and no change, on machine-3-1. No step has seen it, and it comes from the third group of servers.

Predict. Will the chosen detector keep its lead on a server from another group?

machine-3-1: 19.9 test days, 4 fault windows
pages false pages per day faults found minutes to detection alert minutes
robust z 35 1.51 2 [None, 31, None, 16] 57
Isolation Forest 490 24.18 4 [3, 116, 0, 0] 8785
change point 56 2.31 4 [12, 21, 20, 6] 514
The new server, machine-3-1, run once with the same rules. The change-point score finds all 4 faults with 2.31 false pages a day. Isolation Forest also finds all 4, but with 24.18 false pages a day. The robust z finds 2.
Text description of the chart

A column chart with three columns, the false pages a day on machine-3-1. Robust z: 1.51, and it finds 2 of 4 faults. Isolation Forest: 24.18, by far the highest, and it finds 4 of 4. Change point: 2.31, highlighted, and it finds 4 of 4.

What this tells us. The choice holds. The change-point score finds all 4 faults, 6 to 21 minutes after their start, with 2.31 false pages a day. The robust z misses half of them. Isolation Forest also finds all 4, but it sends 24.18 false pages a day. It is in alert for 8,785 of the 28,700 test minutes. Its normal, learned on the training half, did not fit the test half of this server. “Found all 4” means little when the detector alerts almost all the time.

Look at the minutes to detection, too: 6 to 21 minutes, slower than on most faults of the comparison servers. The dataset lists 3 to 15 metrics behind each of these four faults. We did not study why they are found later.

Common mistake

The alert that lasted ten days

Symptom: Kwame first set the budget in alert events, the course’s rule: alerts less than 30 minutes apart are one event. On machine-3-1, the robust z kept its budget: only 7 alert events in 20 days. Hiroshi was pleased. Then her phone kept ringing.

Diagnose: the notebook counts the alert minutes and the pages of the same threshold:

cal, st, _ = new_scores["robust z"]
flags = st > threshold_for_budget(cal, BUDGET, count="events") # the budget counts alert events, not pages
events = events_of(flags)
print("alert events:", len(events), "| alert minutes:", int(flags.sum()), "| pages:", len(pages_of(flags)))
print("the longest alert event:", max(e - s + 1 for s, e in events), "minutes")
alert events: 7 | alert minutes: 3095 | pages: 928
the longest alert event: 14967 minutes

One metric alerts every few minutes, and the gaps never reach 30 minutes. So the alerts join into one event of 14,967 minutes, more than 10 days. The event count stays low, and the engineer gets 928 pages.

Fix: count what reaches the person: pages, with a repeat time. Or cap the length of an event, for example “a new event after 60 minutes”. Then set the budget again.

Consequence in AI work: a measure that hides cost makes a bad detector look good. An alert that never ends is the fastest way to have it muted, as in Alert fatigue and ownership.

Kwame writes to Hiroshi in her words:

  • Page with the change-point score on the last 60 minutes, at 1 page a day on normal days. On the two comparison servers, it found 17 of 21 faults within 30 minutes. On the new server, it found all 4.
  • Put the top metric in the page. It matched the dataset’s own list for 17 of 22 found faults on the three servers. Treat it as the first place to look.
  • Expect more pages than the budget. On the comparison servers’ test days, it sent 3.8 false pages a day on average, not 1 (5.9 on machine-1-1). On the new server, it sent 2.3. That is about 6.7 engineer hours a week per server (at 15 minutes a page, an assumption). Check the real rate each week, and set the threshold again from a longer normal period.
  • Do not use Isolation Forest on these windows to page. It was late on the comparison servers and flooded the new one.

And what not to use it for:

  • Not for slow drifts. A change that takes more than an hour never shows as a step inside the last 60 minutes. A memory leak like the ticket API’s needs a longer window, such as the seasonal baseline of Many metrics at once.
  • Not as the only check for a fleet. 1 page a day per server is 28 pages a day for 28 servers. A fleet needs its alerts grouped across servers too, not only in time as in Many metrics at once.
  • The simple method came close. The robust z found 13 faults within 30 minutes against 17, with fewer false pages, the best precision and no setting but the threshold. If the team must send fewer pages, it is the better choice.
  • The step is the common fault shape here. That is why a change-point score wins. On servers whose faults are slow drifts or short spikes, the ranking could change.
  • The budget was set on 3 to 4 days. The test days were busier, so every detector broke it. A longer normal period would help.
  • The labels are judgements. A detector that alerts 1 minute after a window ends gets no credit, and its page counts as false.
  • Three servers, 25 faults. That is a small sample, and the order of the detectors could change on other servers.
  • No names, no clock. The metrics are anonymous and scaled. A real team would add the hour of the day, as in Normal changes with the hour and the season.
  • The data is about real systems of a real company. It holds no personal data. Do not try to find out which company or which servers it describes.
  • Credit the source. The MIT licence allows you to use, change and share the data. Keep the copyright and licence notice with any copy.
  • A page wakes a person up. False pages cost sleep and trust, not only minutes. State the false pages a day next to every claim about faults found.
  • Label the assumptions. The page budget, the 15 minutes per page and the 30-minute repeat are choices. Say so wherever a number depends on them.

This case study shows an employer that you can take real operations data to a paging decision. It also shows that you score detectors honestly, with the cost next to the gain. Make it yours with one change, for example more servers, another window length, or the hour of the day. Then make it public and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • scoring.md: the scoring rules (pages, the 30-minute repeat, found, minutes to detection) and the threshold rule.
  • comparison.csv: the three detectors at the budget, and curves.csv with the ten budgets; new_server.csv, kept apart.
  • The data source and licence, with the attribution text above. A link and the checksums are enough; you do not need to copy the data.

A README template:

# Paging on server faults: robust z vs Isolation Forest vs a live change point
## Problem
An on-call team watches 38 metrics per server. Budget: 1 page a day per server on normal
days; a false page costs 15 minutes; an alert that stays on pages again after 30 minutes
(assumptions). Find faults early.
## Data and licence
SMD (Server Machine Dataset), NetManAIOps, MIT licence,
https://github.com/NetManAIOps/OmniAnomaly. Servers machine-1-1 and machine-2-1
(comparison, 21 faults) and machine-3-1 (new input, 4 faults), chosen by a rule.
## Approach
Fit on the first 80% of the unlabelled training half, threshold for 1 page a day on the
last 20%, score on the labelled test half. Robust z per metric (max of 38), Isolation
Forest on 10-minute means and standard deviations, best single cut in the last 60 minutes.
## Results
Faults found within 30 minutes: change point 17/21, robust z 13/21, Isolation Forest 6/21.
False pages a day: 3.81, 2.68, 2.73. New server: change point 4/4 at 2.31 false pages a day;
Isolation Forest 4/4 at 24.18.
## Limits
Budget broken on the test days; 3 servers; anonymous metrics; no time of day; label edges.

How to describe it in an interview, in two sentences: “I compared three detectors on real server metrics with one no-label threshold rule, by faults found within 30 minutes and false pages a day. A live change-point score on the last hour won and held on a new server, and I showed that counting events instead of pages hid a ten-day alert.”

Go further:

  • Run all 28 servers (about 470 MB). Does the change-point score still win on most of them?
  • Add the robust z and the change-point score into one rule: page when either is above its threshold. Set the budget for the pair, and compare.
  • Compare with the other case studies, Card fraud with a review budget and Control charts on a real production line: where does a simple method win?

The notebook’s section 9 has two tasks.

Task 1. The change-point score needs at least 5 minutes on each side of the cut. Before you run it, predict: with at least 10 minutes, will it find more or fewer faults within 30 minutes? Run it with change_score(z, m=10) on the two comparison servers, with the same rules, and write the count in found_30_m10.

Show the answer
rows = []
for m in DEV:
train, test, label = data[m]
n_fit = int(len(train) * FIT_SHARE)
med, spread = robust_scale(train[:n_fit])
z_train, z_test = (train - med) / spread, (test - med) / spread
threshold = threshold_for_budget(change_score(z_train, m=10)[n_fit:])
rows.append(score_windows(change_score(z_test, m=10) > threshold, windows_of(label)))
found_30_m10 = pooled(rows, dev_days)["found within 30 min"]
print(found_30_m10)
16

One fewer than with 5 minutes (17). A longer side waits for more minutes after the change before the cut counts. It also misses steps that are short.

Task 2. Run the comparison at a budget of 2 pages a day with compare(2.0). How many false pages a day does the robust z give on the comparison servers? Write it in robust_false_per_day_2.

Show the answer
robust_false_per_day_2 = compare(2.0).loc["robust z", "false pages per day"]
print(robust_false_per_day_2)
3.62

From 2.68 to 3.62 false pages a day, for 1 more fault within 30 minutes (14 instead of 13). Doubling the budget does not double the faults found.

Check your understanding

With the course's rule for alert events, the robust z kept its budget on the new server: 7 alert events in 20 days. Why was that a bad measure here, and what did Kwame count instead?

Show the answer

Alerts less than 30 minutes apart join into one event. One metric alerted every few minutes, so the gaps never reached 30 minutes. The alerts joined into one event of 14,967 minutes, more than 10 days. The event count stayed low, but the engineer would get 928 pages. Kwame counted pages instead: a page at the first alert minute, and again after 30 minutes if the alerts go on. A budget must count what reaches the person.

Check your understanding

On the new server, Isolation Forest found all 4 faults, the same as the change-point score. Why is it still the worst choice?

Show the answer

It found them by alerting almost all the time: in 8,785 of 28,700 minutes. It sent 24.18 false pages a day, against 2.31 for the change-point score. A detector that alerts all the time finds every fault, and nobody can act on it. Faults found mean something only next to the false pages it cost. Its normal, learned on the training half, did not fit the test half of this server.

Check your understanding

The change-point methods of the course cut a finished series. How does the change-point score of this case study page live, and what kind of fault can it not see?

Show the answer

At every minute, it looks only at the last 60 minutes. It finds the best single cut there, with at least 5 minutes on each side, and scores how much that cut lowers the squared error. It needs no minute from the future, so it can page at once, a few minutes after a step. It cannot see a slow drift that takes more than an hour, such as a memory leak: inside any one hour, the level hardly moves. And after a step, the score falls again, because the last hour has one level.

Recap

  • Server faults are windows. Score a detector by the faults it finds, the minutes to detection, and the false pages a day, together.
  • Set every threshold the same way, with no label: learn normal on the fit minutes, set the budget on the calibration minutes.
  • The live change-point score (the best cut in the last 60 minutes) found 17 of 21 faults within 30 minutes. The simple robust z found 13 with fewer false pages. Isolation Forest on windows found 6.
  • The budget of 1 page a day broke on the test days, for every detector. Compare at the same false pages a day too.
  • On the new server, the choice found all 4 faults. Count pages, not alert events: one event lasted 10 days.

Next, a case study with no labels at all: a real public series, where you plant your own anomalies to measure the detectors.

Next: Unusual readings in public open data