Faults in machine or server metrics
- Case study 2
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we found the memory leak and the release spike in the ticket API's metrics. Now we try the same detectors on real machines and servers, with faults that really happened. In this case study, we compare three methods by the faults they find, how late, and the false pages a day.
Case study 2 of 4 · Anomaly Detection: Find What Is UnusualFaults in machine or server metrics
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Watch 38 metrics of real servers for faults, and choose a detector that you can defend. You will set every threshold the same way, with no label, for a page budget. You will compare a robust z-score, Isolation Forest and a live change-point score. You score each by the faults it finds, how late, and the false pages a day. Then you run the choice once on a server that no step has seen.
The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and ruptures. They download three servers from the original source (about 55 MB) and check each file. After the setup, the notebook runs in under a minute on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The robust z-score (median and MAD), from Median, MAD and IQR.
- Isolation Forest, and why the features matter more than the algorithm, from Isolation Forest and Features decide more than the algorithm.
- Change points, the squared-error cost and the penalty, from Change points.
- Alert events, windows found and minutes to detection, from Many metrics at once.
- Choosing a threshold from a budget, from The alert budget.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12 and the versions in the setup cell (pandas 3.1.1, NumPy 2.5.4, scikit-learn 1.9.1, ruptures 1.1.10). The setup cell makes a new folder server-metrics. To start again, delete that folder.
Tested on macOS (Apple silicon) with Python 3.12.13, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”A smoke alarm that rings every time you make toast is soon switched off. One that rings ten minutes after the fire starts is too late. Server alerts fail in the same two ways.
Hiroshi leads the on-call rota of a team that runs many servers. When an alert starts, the engineer on call gets a page: a message to the phone, day or night. Kwame is the team’s data scientist. We invented both people for this case study. The servers and their faults are real.
Hiroshi’s rules for Kwame are the course’s kind of rules. Each is an assumption, not a measurement:
- The page budget: on normal days, at most 1 page a day per server.
- A false page costs 15 minutes of an engineer’s time, to look, decide and go back to sleep.
- An alert that stays on pages again after 30 minutes. Many paging tools work like this.
Hiroshi asks Kwame: “Which detector should page us? How many real faults will it find, how late, and how many false pages will it send?”
What you will build
Section titled “What you will build”You will build three detectors on 38 metrics per server. You will set each one’s threshold for the same page budget, with no label. Then you score them on 21 real faults of two servers, choose one, and run it once on a third server with 4 faults.
The data
Section titled “The data”The data is SMD, the Server Machine Dataset. The research group NetManAIOps collected it from 28 servers of “a large Internet company” and published it with their OmniAnomaly method. Each server has 38 metrics, one value per minute, for about five weeks.
The files of one server:
| File | What it holds |
|---|---|
train-<m>.txt |
The first half: one row per minute, 38 values. No labels. |
test-<m>.txt |
The second half, in the same form. |
label-<m>.txt |
One 0 or 1 per minute of the test half: 1 means “inside a fault”. |
interp-<m>.txt |
For each fault, the metrics that the dataset’s authors found behind it. |
The metrics have no names. They are measures such as CPU, memory, network and disk, but the dataset does not say which is which. The authors scaled every metric to the range 0 to 1. There are no dates, only minute numbers.
We compared two other datasets. NAB, the Numenta Anomaly Benchmark, has server metrics too, but one metric per file. Its data’s terms are also unclear. The repository was under the AGPL licence from 2015 to December 2024, then MIT, and much of its data comes from third parties. SKAB is under the GPL, which is not on our list of allowed licences.
Which servers
Section titled “Which servers”Kwame takes three servers, chosen by a rule before any detector runs: the first server of each of the three groups.
- machine-1-1 and machine-2-1 are for the comparison. Together they have 21 faults in 36.2 test days.
- machine-3-1 is the new input. It is used once, after the choice. It has 4 faults in 19.9 test days.
A rule chosen in advance stops you from picking the servers where your favourite method looks good.
First checks
Section titled “First checks”machine-1-1: train (28479, 38), test (28479, 38) = 19.8 days, 9.5% of test minutes in a fault, 8 fault windows window lengths (minutes): [546, 554, 457, 721, 409, 3, 2, 2]machine-2-1: train (23693, 38), test (23694, 38) = 16.5 days, 4.9% of test minutes in a fault, 13 fault windows window lengths (minutes): [22, 54, 46, 26, 8, 452, 17, 33, 75, 16, 12, 21, 388]missing values: 0smallest and largest value: 0.0 0.0 | 1.0 1.0metrics that never change in the training half: 8 of 38metrics that never change in the test half: 6 of 38What this tells us. The two servers are different. machine-1-1 has five long faults of 7 to 12 hours, and three tiny ones of 2 or 3 minutes. machine-2-1 has 13 faults, most of them under an hour. There are no missing values. Eight metrics never move in the training half. Their spread is 0, so a z-score would divide by zero. Kwame gives every metric a spread of at least 1% of its range.
One fault, minute by minute
Section titled “One fault, minute by minute”- Metric 1
- Start and end of the labelled fault
Text description of the chart
The line shows metric 1 of machine-1-1, minute by minute. It starts 240 minutes before the first labelled fault and ends 60 minutes after it. Before the fault, it is low, between 0.04 and 0.13. At minute 0, it jumps to about 0.5 and stays near 0.5 until minute 269. At minute 270, it drops to about 0.1 for about half an hour, then climbs to 0.68 with peaks and falls. In the hour after the label ends, it is between 0.19 and 0.34. Two dashed vertical lines mark the start and the end of the label, 545 minutes apart.
What this tells us. This fault starts as a step: the level moves at once and stays for hours. Any detector that compares this minute with normal should see it. Other faults last 2 minutes, or move several metrics a little.
How we score a detector
Section titled “How we score a detector”Faults are windows, not points
Section titled “Faults are windows, not points”In the course’s comparison table, a payment was one unit. A server fault is different. It is a window of time, from its first labelled minute to its last. One fault gives hundreds of unusual minutes, but Hiroshi needs one page, early.
So we score each detector by three numbers, as for the ticket API in Many metrics at once:
- Faults found: a fault is found when the detector alerts inside its window.
- Minutes to detection: from the start of the window to the first alert inside it.
- False pages a day: pages that fall outside every fault window.
Text description of the diagram
The diagram is a flow from left to right. The first box says: score each minute; a score above the threshold makes an alert minute. The second box says: page the engineer at the first alert minute, and again after 30 minutes if the alerts go on. A diamond asks: is the page inside a labelled fault? The yes arrow goes to “true page: the fault is found; minutes to detection are the first alert minus the fault start”. The no arrow goes to “false page: it costs 15 minutes (an assumption)”.
Why pages, and not the course’s alert events (alerts less than 30 minutes apart, joined into one)? A detector that alerts every 20 minutes for ten days makes one event, and Hiroshi’s phone rings all the time. A failure case below shows it.
A tiny example
Section titled “A tiny example”Take 120 made-up minutes. One fault runs from minute 50 to minute 59. A detector alerts at minute 10, at minutes 52 to 54, at minutes 70 to 85, and at minute 100.
pages at minutes: [10, 52, 82]{'pages': 3, 'false pages': 2, 'faults found': 1, 'minutes to detection': [2]}- Minute 10 is the first alert, so it pages. It is outside the fault: a false page.
- Minute 52 is 42 minutes after the last page, so it pages. It is inside the fault: a true page. The fault is found 2 minutes after its start.
- The alert from 70 to 85 starts only 18 minutes after the page at 52. It pages once, at minute 82, 30 minutes after that page. That is a false page.
- Minute 100 is 18 minutes after the page at 82, so it does not page.
The same threshold rule for every detector
Section titled “The same threshold rule for every detector”Each detector gives a score per minute, and a threshold turns the score into alerts. So every threshold is chosen the same way, with no label:
Text description of the diagram
On the left is the training half of a server, about two and a half weeks, with no labels. Its first 80% is the fit part: each detector learns normal there. Its last 20% is the calibration part: the threshold is set there for 1 page a day. An arrow goes to the test half, with labels, where the pages, the faults found and the minutes to detection are scored. A note says: machine-1-1 and machine-2-1 are compared, with 21 faults; machine-3-1 is the new input, used once, with 4 faults.
- Learn normal on the first 80% of the training half (the fit minutes).
- On the last 20% (the calibration minutes, about 3 to 4 days), choose the lowest threshold that pages at most once a day.
- Run on the test half, and score it with the labels.
The training half may contain faults. We treat it as “mostly normal”, as a real team would. This is the rule of The alert budget, with pages a day in place of alerts a day.
The baseline: a robust z-score per metric
Section titled “The baseline: a robust z-score per metric”For each metric, its median and MAD on the fit minutes give a robust z-score, as in Median, MAD and IQR. The detector’s score is the largest |z| of the 38 metrics in that minute.
def robust_scale(fit): """Each metric's median and spread (1.4826 x MAD, at least MIN_SCALE) on the fit minutes.""" med = np.median(fit, axis=0) spread = np.maximum(1.4826 * np.median(np.abs(fit - med), axis=0), MIN_SCALE) return med, spread
def robust_z_score(x, med, spread): """The baseline: the largest |robust z| over the 38 metrics, minute by minute.""" return np.abs((x - med) / spread).max(axis=1)machine-1-1: threshold |z| > 20.3; {'pages': 156, 'false pages': 83, 'faults found': 8, 'faults': 8, 'minutes to detection': [1, 0, 0, 0, 0, 0, 0, 0]}machine-2-1: threshold |z| > 49.6; {'pages': 23, 'false pages': 14, 'faults found': 7, 'faults': 13, 'minutes to detection': [None, None, 45, 0, None, 449, 5, 28, None, None, None, 14, 8]}What this tells us. The thresholds are very high: |z| above 20 and above 50. Many metrics hardly move on normal minutes: 17 of the 38 metrics of machine-1-1 have a spread at the 1% floor. On such a metric, an ordinary small spike gives a huge z, so the budget pushes the threshold far up. On machine-1-1, the baseline finds all 8 faults within a minute, the tiny ones too. On machine-2-1, it finds only 7 of 13, and one of them 449 minutes late. In the 6 faults that it misses, no metric reaches |z| = 49.6.
We did not run the course’s other baseline, STL residuals. STL with a one-day season (1,440 minutes) took 73 seconds for one metric of one test half, on our laptop. At that speed, 38 metrics on three servers take more than two hours, far too slow for a notebook.
Isolation Forest on short windows
Section titled “Isolation Forest on short windows”Isolation Forest sees all metrics at once, as in Isolation Forest. A single minute is noisy, so Kwame gives it short windows. For each minute, the features are the mean and the standard deviation of each metric over the last 10 minutes: 76 features. The forest has 200 trees and learns the fit minutes. Its score is how fast a minute is cut free.
machine-1-1 (0.3 s): {'pages': 126, 'false pages': 84, 'faults found': 5, 'faults': 8, 'minutes to detection': [254, 239, 226, 340, 258, None, None, None]}machine-2-1 (0.2 s): {'pages': 25, 'false pages': 15, 'faults found': 8, 'faults': 13, 'minutes to detection': [0, None, 45, None, None, 449, 5, 30, None, 6, None, 14, 8]}What this tells us. On machine-1-1, Isolation Forest finds the five long faults 226 to 340 minutes late, and misses the three tiny ones. The step in metric 1 that you saw above is one feature of 76. A likely reason: the trees rarely cut on that one feature. So a minute where only it moved is not cut free fast. Features decide more than the algorithm made the same point with payments. On machine-2-1, it is about as good as the baseline.
A live change-point score
Section titled “A live change-point score”The problem: change points look back
Section titled “The problem: change points look back”Many faults here are steps: a level moves and stays. Change points found steps with PELT and binary segmentation. But those methods cut a finished series. They need the minutes after the change. A page must come now.
The idea: cut only the last hour
Section titled “The idea: cut only the last hour”Think of a bathroom scale. You do not need a year of weighings to see that something changed. You compare the last few days with the days before them. Kwame does the same at every minute. He takes only the last 60 minutes. He asks: where is the best single cut in this hour, and how much does it lower the squared error? If one level fits the hour well, no cut helps much. If the level moved inside the hour, the best cut lowers the squared error a lot. That drop is the score.
A tiny example. Twelve minutes of one scaled metric: 0.1, −0.2, 0.0, 0.2, −0.1, 0.0, then 2.1, 1.9, 2.0, 2.2, 1.8, 2.0. With no cut, the squared error is 12.20. In this tiny example, each cut leaves at least 3 minutes on each side (the real score uses 5):
cut before minute 4: squared error 7.93, lowered by 4.27cut before minute 5: squared error 6.50, lowered by 5.70cut before minute 6: squared error 3.63, lowered by 8.57cut before minute 7: squared error 0.20, lowered by 12.00cut before minute 8: squared error 3.97, lowered by 8.23cut before minute 9: squared error 6.20, lowered by 6.00cut before minute 10: squared error 8.20, lowered by 4.00The best cut is before minute 7, where the level moved. It lowers the squared error by 12.00, almost all of it.
Text description of the diagram
The diagram is a flow from left to right. The first box: every minute, take the last 60 minutes of the 38 scaled metrics. The second box: try every cut with at least 5 minutes on each side. The third box: the score is how much the best cut lowers the squared error. A diamond asks: is the score above the threshold for 1 page a day? The yes arrow goes to the last box: page, and name the metric with the largest robust z.
The code
Section titled “The code”change_score does this at every minute for all 38 metrics at once. Each metric is scaled by its median and spread first, as for the robust z. Running sums make it fast: one test half takes less than a second.
def change_score(z, w=60, m=5): """A live change-point score: in the last w minutes, the best single cut with at least m minutes on each side, and how much it lowers the squared error (the l2 cost of the ruptures package). z: scaled metrics.""" n, d = z.shape S = np.vstack([np.zeros(d), np.cumsum(z, axis=0)]) out = np.zeros(n) t = np.arange(w - 1, n) # the window is a = t - w + 1 ... t a = t - w + 1 best = np.zeros(len(t)) for k in range(m, w - m + 1): # the cut: k minutes before it, w - k after it c = a + k left = (S[c] - S[a]) / k right = (S[t + 1] - S[c]) / (w - k) gain = (k * (w - k) / w) * ((left - right) ** 2).sum(axis=1) best = np.maximum(best, gain) out[w - 1:] = best return outIs it really the cut that the course’s tool finds? The notebook compares one minute with ruptures’ binary segmentation, with one cut and the l2 cost:
ruptures: cut after minute 37 of 60; squared error lowered by 945.3change_score at that minute: 945.3The measurement script checked 200 random minutes the same way. The largest difference was 1.2 × 10⁻¹², a rounding error.
machine-1-1: {'pages': 185, 'false pages': 116, 'faults found': 8, 'faults': 8, 'minutes to detection': [2, 1, 0, 0, 0, 0, 0, 0]}machine-2-1: {'pages': 38, 'false pages': 22, 'faults found': 10, 'faults': 13, 'minutes to detection': [0, None, 45, 0, None, 2, 5, 30, None, 6, 8, 14, 8]}What this tells us. The change-point score finds all 8 faults of machine-1-1 within 2 minutes, and 10 of the 13 of machine-2-1. Look at the sixth fault of machine-2-1: the baseline found it after 449 minutes, the change-point score after 2. A likely reason: the dataset lists four metrics behind this fault. A small step in several metrics at once adds up in the squared error, while no single metric’s z reaches the baseline’s threshold.
Three detectors on one fault
Section titled “Three detectors on one fault”- Robust z
- Isolation Forest
- Change point
- Threshold, start and end of the fault
Text description of the chart
Three lines show the score of each detector divided by its own threshold, so that 1 is the threshold for all of them. A dashed line marks 1. Before the fault, all three lines stay mostly below 1. At minute 0, the robust z and the change-point lines jump far above 1 within 2 minutes. The robust z stays above 1 for 82% of the fault. The change-point line falls below 1 after about an hour, then rises again each time the level moves. It is above 1 for 66% of the fault. The Isolation Forest line stays below 1 until minute 254, then rises above it for part of the rest.
Why does the change-point line fall below 1 about an hour after the start? Then the last 60 minutes have one level again: nothing changed inside the hour. A change-point score says “it moved”, not “it is still high”. For a page, that is enough.
The fair comparison
Section titled “The fair comparison”Predict. At a budget of 1 page a day per server, which detector finds the most faults within 30 minutes? The simple robust z, Isolation Forest, or the change-point score?
The notebook scores the three detectors on both comparison servers and adds them up. “Engineer hours” are the false pages × 15 minutes, per server and week (an assumption).
pages false pages per day precision faults found found within 30 min median minutes to detection engineer hours a week on false pagesrobust z 179 2.68 0.458 15 13 0.0 4.7Isolation Forest 151 2.73 0.344 13 6 45.0 4.8change point 223 3.81 0.381 18 17 1.5 6.7One measure for the choice
Section titled “One measure for the choice”Which column decides? Hiroshi’s question was: find the faults early, within the page budget. So the measure is faults found within 30 minutes, at the same page budget. The other columns explain it:
- The change-point score finds the most: 17 of 21 faults within 30 minutes, against 13 for the robust z and 6 for Isolation Forest.
- The robust z has the best precision: 45.8% of its pages are true. It sends fewer false pages (2.68 a day) than the change-point score (3.81).
- Isolation Forest loses on every column except the number of pages. Its median time to detection is 45 minutes.
The budget broke
Section titled “The budget broke”The budget was 1 page a day. On the test halves, every detector sends 2.7 to 3.8 false pages a day. The calibration days were calmer than the test days. The worst is machine-1-1, with 4.2 to 5.9 false pages a day. Only 6 to 14 of its 83 to 116 false pages came within an hour of a fault. So the edges of the labels do not explain them.
A threshold set on three or four days is a guess about the next three weeks, as in When “normal” moves. And the detectors do not cost the same, so the table is not quite fair. The chart below makes it fair. For ten budgets, it plots the false pages that each detector really sent against the faults it found within 30 minutes:
- Robust z
- Isolation Forest
- Change point
Text description of the chart
Three lines with dots, one per detector, for ten budgets. The horizontal axis is the false pages a day that the detector really sent on the test halves, from 0 to 11. The vertical axis is the faults found within 30 minutes, out of 21. The change-point line starts at 15 faults with 2.1 false pages a day. It is at 17 from 3.8 to 7 false pages, and reaches 19 at about 9. The robust z line starts at 12 faults with 0.44 false pages a day. It is at 13 to 16 between 1 and 5 false pages, and reaches 18 at about 7.6. Isolation Forest stays at 6 faults until about 3 false pages a day and reaches 17 only at about 8.
What this tells us. Between 2 and 5 false pages a day, the change-point score finds 1 to 3 more faults within 30 minutes than the robust z. Above about 7, the robust z catches up. The robust z alone can go very quiet: 12 faults with 0.44 false pages a day. Isolation Forest on these windows is far behind until about 8 false pages a day.
Kwame chooses the change-point score for the page. He chooses it on these two servers only, before he looks at the third.
Explain a page
Section titled “Explain a page”A page that says “server 1-1 is unusual” makes the engineer search 38 metrics at 3 a.m. Explain an alert asked for the reason in the alert itself. Here the reason is simple: at the first alert minute, name the metric with the largest |robust z|. The change-point score decides when to page, and the robust z says what moved. Many metrics at once ended with the same design.
SMD lets us check it. For each fault, the dataset’s authors listed the metrics behind it.
machine-1-1: the top metric is in the dataset's label for 8 of 8 found faults [(1, True), (9, True), (9, True), (14, True), (9, True), (9, True), (9, True), (9, True)]machine-2-1: the top metric is in the dataset's label for 6 of 10 found faults [(16, False), (10, True), (34, True), (26, True), (18, True), (9, True), (18, False), (11, False), (16, False), (16, True)]What this tells us. On machine-1-1, the named metric is always one of the authors’ metrics. On machine-2-1, it is right for 6 of 10. Metric 16 is named three times and is wrong twice. The explanation is a good first place to look, not a diagnosis.
The new server, once
Section titled “The new server, once”The choice is made: the change-point score for the page, the top |z| metric for the explanation. Now Kwame runs all three detectors, with the same rules and no change, on machine-3-1. No step has seen it, and it comes from the third group of servers.
Predict. Will the chosen detector keep its lead on a server from another group?
machine-3-1: 19.9 test days, 4 fault windows pages false pages per day faults found minutes to detection alert minutesrobust z 35 1.51 2 [None, 31, None, 16] 57Isolation Forest 490 24.18 4 [3, 116, 0, 0] 8785change point 56 2.31 4 [12, 21, 20, 6] 514Text description of the chart
A column chart with three columns, the false pages a day on machine-3-1. Robust z: 1.51, and it finds 2 of 4 faults. Isolation Forest: 24.18, by far the highest, and it finds 4 of 4. Change point: 2.31, highlighted, and it finds 4 of 4.
What this tells us. The choice holds. The change-point score finds all 4 faults, 6 to 21 minutes after their start, with 2.31 false pages a day. The robust z misses half of them. Isolation Forest also finds all 4, but it sends 24.18 false pages a day. It is in alert for 8,785 of the 28,700 test minutes. Its normal, learned on the training half, did not fit the test half of this server. “Found all 4” means little when the detector alerts almost all the time.
Look at the minutes to detection, too: 6 to 21 minutes, slower than on most faults of the comparison servers. The dataset lists 3 to 15 metrics behind each of these four faults. We did not study why they are found later.
Common mistake
The alert that lasted ten days
Symptom: Kwame first set the budget in alert events, the course’s rule: alerts less than 30 minutes apart are one event. On machine-3-1, the robust z kept its budget: only 7 alert events in 20 days. Hiroshi was pleased. Then her phone kept ringing.
Diagnose: the notebook counts the alert minutes and the pages of the same threshold:
cal, st, _ = new_scores["robust z"]flags = st > threshold_for_budget(cal, BUDGET, count="events") # the budget counts alert events, not pagesevents = events_of(flags)print("alert events:", len(events), "| alert minutes:", int(flags.sum()), "| pages:", len(pages_of(flags)))print("the longest alert event:", max(e - s + 1 for s, e in events), "minutes")alert events: 7 | alert minutes: 3095 | pages: 928the longest alert event: 14967 minutesOne metric alerts every few minutes, and the gaps never reach 30 minutes. So the alerts join into one event of 14,967 minutes, more than 10 days. The event count stays low, and the engineer gets 928 pages.
Fix: count what reaches the person: pages, with a repeat time. Or cap the length of an event, for example “a new event after 60 minutes”. Then set the budget again.
Consequence in AI work: a measure that hides cost makes a bad detector look good. An alert that never ends is the fastest way to have it muted, as in Alert fatigue and ownership.
Recommendation
Section titled “Recommendation”Kwame writes to Hiroshi in her words:
- Page with the change-point score on the last 60 minutes, at 1 page a day on normal days. On the two comparison servers, it found 17 of 21 faults within 30 minutes. On the new server, it found all 4.
- Put the top metric in the page. It matched the dataset’s own list for 17 of 22 found faults on the three servers. Treat it as the first place to look.
- Expect more pages than the budget. On the comparison servers’ test days, it sent 3.8 false pages a day on average, not 1 (5.9 on machine-1-1). On the new server, it sent 2.3. That is about 6.7 engineer hours a week per server (at 15 minutes a page, an assumption). Check the real rate each week, and set the threshold again from a longer normal period.
- Do not use Isolation Forest on these windows to page. It was late on the comparison servers and flooded the new one.
And what not to use it for:
- Not for slow drifts. A change that takes more than an hour never shows as a step inside the last 60 minutes. A memory leak like the ticket API’s needs a longer window, such as the seasonal baseline of Many metrics at once.
- Not as the only check for a fleet. 1 page a day per server is 28 pages a day for 28 servers. A fleet needs its alerts grouped across servers too, not only in time as in Many metrics at once.
Insights and limits
Section titled “Insights and limits”- The simple method came close. The robust z found 13 faults within 30 minutes against 17, with fewer false pages, the best precision and no setting but the threshold. If the team must send fewer pages, it is the better choice.
- The step is the common fault shape here. That is why a change-point score wins. On servers whose faults are slow drifts or short spikes, the ranking could change.
- The budget was set on 3 to 4 days. The test days were busier, so every detector broke it. A longer normal period would help.
- The labels are judgements. A detector that alerts 1 minute after a window ends gets no credit, and its page counts as false.
- Three servers, 25 faults. That is a small sample, and the order of the detectors could change on other servers.
- No names, no clock. The metrics are anonymous and scaled. A real team would add the hour of the day, as in Normal changes with the hour and the season.
Responsible use
Section titled “Responsible use”- The data is about real systems of a real company. It holds no personal data. Do not try to find out which company or which servers it describes.
- Credit the source. The MIT licence allows you to use, change and share the data. Keep the copyright and licence notice with any copy.
- A page wakes a person up. False pages cost sleep and trust, not only minutes. State the false pages a day next to every claim about faults found.
- Label the assumptions. The page budget, the 15 minutes per page and the 30-minute repeat are choices. Say so wherever a number depends on them.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can take real operations data to a paging decision. It also shows that you score detectors honestly, with the cost next to the gain. Make it yours with one change, for example more servers, another window length, or the hour of the day. Then make it public and easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
scoring.md: the scoring rules (pages, the 30-minute repeat, found, minutes to detection) and the threshold rule.comparison.csv: the three detectors at the budget, andcurves.csvwith the ten budgets;new_server.csv, kept apart.- The data source and licence, with the attribution text above. A link and the checksums are enough; you do not need to copy the data.
A README template:
# Paging on server faults: robust z vs Isolation Forest vs a live change point
## ProblemAn on-call team watches 38 metrics per server. Budget: 1 page a day per server on normaldays; a false page costs 15 minutes; an alert that stays on pages again after 30 minutes(assumptions). Find faults early.
## Data and licenceSMD (Server Machine Dataset), NetManAIOps, MIT licence,https://github.com/NetManAIOps/OmniAnomaly. Servers machine-1-1 and machine-2-1(comparison, 21 faults) and machine-3-1 (new input, 4 faults), chosen by a rule.
## ApproachFit on the first 80% of the unlabelled training half, threshold for 1 page a day on thelast 20%, score on the labelled test half. Robust z per metric (max of 38), IsolationForest on 10-minute means and standard deviations, best single cut in the last 60 minutes.
## ResultsFaults found within 30 minutes: change point 17/21, robust z 13/21, Isolation Forest 6/21.False pages a day: 3.81, 2.68, 2.73. New server: change point 4/4 at 2.31 false pages a day;Isolation Forest 4/4 at 24.18.
## LimitsBudget broken on the test days; 3 servers; anonymous metrics; no time of day; label edges.How to describe it in an interview, in two sentences: “I compared three detectors on real server metrics with one no-label threshold rule, by faults found within 30 minutes and false pages a day. A live change-point score on the last hour won and held on a new server, and I showed that counting events instead of pages hid a ten-day alert.”
Go further:
- Run all 28 servers (about 470 MB). Does the change-point score still win on most of them?
- Add the robust z and the change-point score into one rule: page when either is above its threshold. Set the budget for the pair, and compare.
- Compare with the other case studies, Card fraud with a review budget and Control charts on a real production line: where does a simple method win?
Your turn
Section titled “Your turn”The notebook’s section 9 has two tasks.
Task 1. The change-point score needs at least 5 minutes on each side of the cut. Before you run it, predict: with at least 10 minutes, will it find more or fewer faults within 30 minutes? Run it with change_score(z, m=10) on the two comparison servers, with the same rules, and write the count in found_30_m10.
Show the answer
rows = []for m in DEV: train, test, label = data[m] n_fit = int(len(train) * FIT_SHARE) med, spread = robust_scale(train[:n_fit]) z_train, z_test = (train - med) / spread, (test - med) / spread threshold = threshold_for_budget(change_score(z_train, m=10)[n_fit:]) rows.append(score_windows(change_score(z_test, m=10) > threshold, windows_of(label)))found_30_m10 = pooled(rows, dev_days)["found within 30 min"]print(found_30_m10)16One fewer than with 5 minutes (17). A longer side waits for more minutes after the change before the cut counts. It also misses steps that are short.
Task 2. Run the comparison at a budget of 2 pages a day with compare(2.0). How many false pages a day does the robust z give on the comparison servers? Write it in robust_false_per_day_2.
Show the answer
robust_false_per_day_2 = compare(2.0).loc["robust z", "false pages per day"]print(robust_false_per_day_2)3.62From 2.68 to 3.62 false pages a day, for 1 more fault within 30 minutes (14 instead of 13). Doubling the budget does not double the faults found.
Knowledge check
Section titled “Knowledge check”Check your understanding
With the course's rule for alert events, the robust z kept its budget on the new server: 7 alert events in 20 days. Why was that a bad measure here, and what did Kwame count instead?
Show the answer
Alerts less than 30 minutes apart join into one event. One metric alerted every few minutes, so the gaps never reached 30 minutes. The alerts joined into one event of 14,967 minutes, more than 10 days. The event count stayed low, but the engineer would get 928 pages. Kwame counted pages instead: a page at the first alert minute, and again after 30 minutes if the alerts go on. A budget must count what reaches the person.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the new server, Isolation Forest found all 4 faults, the same as the change-point score. Why is it still the worst choice?
Show the answer
It found them by alerting almost all the time: in 8,785 of 28,700 minutes. It sent 24.18 false pages a day, against 2.31 for the change-point score. A detector that alerts all the time finds every fault, and nobody can act on it. Faults found mean something only next to the false pages it cost. Its normal, learned on the training half, did not fit the test half of this server.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The change-point methods of the course cut a finished series. How does the change-point score of this case study page live, and what kind of fault can it not see?
Show the answer
At every minute, it looks only at the last 60 minutes. It finds the best single cut there, with at least 5 minutes on each side, and scores how much that cut lowers the squared error. It needs no minute from the future, so it can page at once, a few minutes after a step. It cannot see a slow drift that takes more than an hour, such as a memory leak: inside any one hour, the level hardly moves. And after a step, the score falls again, because the last hour has one level.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Server faults are windows. Score a detector by the faults it finds, the minutes to detection, and the false pages a day, together.
- Set every threshold the same way, with no label: learn normal on the fit minutes, set the budget on the calibration minutes.
- The live change-point score (the best cut in the last 60 minutes) found 17 of 21 faults within 30 minutes. The simple robust z found 13 with fewer false pages. Isolation Forest on windows found 6.
- The budget of 1 page a day broke on the test days, for every detector. Compare at the same false pages a day too.
- On the new server, the choice found all 4 faults. Count pages, not alert events: one event lasted 10 days.
Next, a case study with no labels at all: a real public series, where you plant your own anomalies to measure the detectors.