Unusual readings in public open data
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we always had an answer key of planted anomalies. Public open data has none. In this case study, we take a real public series, plant our own anomalies to measure the detectors, separate real events from data faults, and write the alert policy.
Case study 3 of 4 · Anomaly Detection: Find What Is UnusualUnusual readings in public open data
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Find the faults in a real public data feed that has no answer key. You will compare a station with its neighbours, plant faults to measure recall, review a random sample by hand, and read the agreement between detectors. Then you write an alert policy for a review budget, run it on new weeks, and say honestly what it found.
The notebook runs every step of this page, with the same code and output, plus two tasks with checks. Its setup cells install pandas and scikit-learn and download the data (8.2 MB) from UCI with a checksum. After the install, it runs in under a minute.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The median, the MAD and the robust z-score, from Median, MAD and IQR, and when a rule is enough, from Rules, and when a rule is the right answer.
- Isolation Forest, from Isolation Forest, and why features matter more than the algorithm, from Features decide more than the algorithm.
- The weekly budget and precision at the budget, from The alert budget.
- Planted anomalies, a reviewed sample with its 95% interval, and agreement, from Evaluate without a full answer key.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, NumPy 2.5.4 and scikit-learn 1.9.1. The setup cell makes a new folder open-air-readings. To start again, delete that folder.
Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”A small team runs a public air map. Every hour, it shows the readings of a city’s 12 air-quality stations. People check it before a run or a trip to the park.
Mina keeps the map’s data correct. Kai, an analyst, builds the detectors. We invented both people and the map. The readings are real: 12 stations in Beijing, 2013 to 2017.
The map can go wrong in two ways:
- It shows a broken reading. A station reports 3 while the air is dirty. People see “good air”, and later stop trusting the map.
- It hides a real event. On a night of heavy smog, every station reads 600. If a detector hides that as a fault, people get no warning. This is the worse mistake.
Mina can review 5 station-days a week. A station-day is one station on one day. A review takes about 6 minutes: 30 minutes a week. Both numbers are our assumptions.
Mina asks Kai: “Which readings should I check each week? And can you tell me how good your detector is, when nobody knows which readings were wrong?”
What you will build
Section titled “What you will build”You will build an alert policy. It says which readings the map hides with no review, and which ones go to Mina’s 5 weekly reviews, in what order. You will measure it three ways with no answer key. Then you run it on 8 weeks that no step has seen.
The data
Section titled “The data”The data is Beijing Multi-Site Air Quality from the UCI Machine Learning Repository. It has one row per station and hour, from 1 March 2013 to 28 February 2017. The air readings come from the Beijing Municipal Environmental Monitoring Center.
The case study uses these columns:
| Column | What it means |
|---|---|
station |
One of the 12 monitoring stations, for example Dongsi or Huairou. |
year, month, day, hour |
The hour of the reading. |
PM2.5 |
Fine particles in the air, in micrograms per cubic metre (µg/m³). This is the number on the map. |
PM10 |
Particles up to a slightly larger size, measured by a second instrument at the same station. In physics, PM2.5 is a part of PM10. |
There are no labels. Nobody marked which readings are wrong.
We compared two other open feeds. UCI’s Air Quality data from an Italian city has one device, so it has no neighbours. Seattle’s Fremont Bridge bicycle counter is a live feed whose file keeps changing, with only two counters.
First checks
Section titled “First checks”The notebook makes two tables, pm25 and pm10, with one column per station and one row per hour. Then it counts the problems that you can see without a detector:
values = pm25.stack() # every PM2.5 reading that is not missingprint(f"missing PM2.5 readings: {pm25.isna().mean().mean():.1%}; hours with no station at all: {pm25.isna().all(axis=1).sum()}")print(f"smallest value: {values.min():.0f}, read {(values == values.min()).sum():,} times; " f"the value 3, read {(values == 3).sum():,} times; largest: {values.max():.0f}")print(f"PM2.5 equal to PM10: {(pm25 == pm10).stack().mean():.1%}; PM2.5 above PM10: {(pm25 > pm10).stack().mean():.1%}")decimals = {c: int((readings[c].dropna() % 1 != 0).sum()) for c in ["PM2.5", "PM10"]}print("readings that are not whole numbers:", decimals)same_as_before = pm25.eq(pm25.shift()) & pm25.notna()print("stations whose value did not change from 05:00 to 09:00 on 18 September 2013:", int(same_as_before.loc["2013-09-18 05:00":"2013-09-18 09:00"].all().sum()), "of 12")420,768 readings: 12 stations x 35,064 hours, 2013-03-01 to 2017-02-28missing PM2.5 readings: 2.1%; hours with no station at all: 81smallest value: 2, read 17 times; the value 3, read 8,810 times; largest: 999PM2.5 equal to PM10: 23.5%; PM2.5 above PM10: 5.1%readings that are not whole numbers: {'PM2.5': 181, 'PM10': 229}stations whose value did not change from 05:00 to 09:00 on 18 September 2013: 12 of 12What this tells us.
- About 2% of the readings are missing. The map already shows them as missing, so we do not flag them.
- The value 3 appears 8,810 times, and smaller values only 17 times. So 3 is the floor: the lowest value that the instruments normally report.
- PM2.5 is above PM10 in 5% of the hours, which physics does not allow, and exactly equal to it in almost a quarter. So the two instruments do not always agree, and “PM2.5 above PM10” alone cannot be a fault rule.
- A few hundred readings are not whole numbers, unlike all the others. They look filled in.
- On 18 September 2013, every hour from 05:00 to 09:00 repeated the hour before, at all 12 stations. The whole feed froze.
The data has faults, and no notes about them. Every verdict about a fault here is a judgement from the data.
Frame the problem
Section titled “Frame the problem”Kai writes down the frame first:
- The unit: a station-day. Mina reviews days, not hours. A sensor that is stuck for 8 hours is one problem, not 8.
- The score of a station-day: the highest hourly score of that day.
- The budget: each week, the 5 station-days with the highest scores go to Mina. Ties go in time order.
- The periods. The history, March 2013 to 1 March 2015, teaches the detectors what normal looks like. The evaluation weeks, 2 March 2015 to 28 February 2016, are 52 weeks, Monday to Sunday: all comparisons use them. The new weeks, 2 January to 26 February 2017, are kept for the end-to-end run.
- What counts as a fault: a PM2.5 reading that is wrong by about 10 µg/m³ or more. A smaller error does not change what a person sees on the map.
The idea: ask the neighbours
Section titled “The idea: ask the neighbours”A thermometer on your balcony
Section titled “A thermometer on your balcony”Your balcony thermometer shows 35 °C on a cool spring morning. Is it a heat wave, or is the thermometer in the sun? You check the weather for your city. If the whole city is at 12 °C, your thermometer is wrong.
Air is the same. Dirty air covers the whole city, and every station rises. When one instrument fails, only that station moves. So we compare each reading with the median of the other stations in the same hour.
A real event and a fault
Section titled “A real event and a fault”The New Year fireworks of 8 February 2016 are a real event. In the chart below, all the stations rise together after midnight.
- Highest station
- Median of the 12 stations
- Lowest station
Text description of the chart
The chart shows three lines from 18:00 on 7 February to 12:00 on 8 February 2016. The median of the 12 stations is about 20 to 70 in the evening and 172 at midnight. It jumps to 627 at 01:00 and stays above 480 until 07:00. It falls to 58 at 09:00 and 9 at noon. The highest station reaches 999 at 02:00. The lowest station is 28 at midnight and 296 at 01:00. All three lines rise and fall together.
The second chart shows a fault. On 16 January 2017 at 11:00 and 12:00, Wanshouxigong reads 10 and 3, while the other 11 stations stay near 90. At 13:00 it reads 72 again.
- Wanshouxigong, PM2.5
- Median of the other 11 stations
Text description of the chart
The chart shows 06:00 to 18:00 on 16 January 2017. The median of the other 11 stations goes smoothly from 101 to 118, with values between 83 and 92 around noon. The Wanshouxigong line follows it at 64 to 83 until 10:00. It drops to 10 at 11:00 and 3 at 12:00. Then it returns to 72 at 13:00 and rises with the others to 114.
A tiny example: the neighbour gap
Section titled “A tiny example: the neighbour gap”Five stations in one hour read 80, 85, 78, 90 and 240. For each station, take the median of the other four, and the gap between them on a log scale, log(1 + reading). The log scale makes 80 against 90 and 400 against 450 the same small gap.
reading median of the others gap (log)A 80 87.5 -0.09B 85 85.0 0.00C 78 87.5 -0.11D 90 82.5 0.09E 240 82.5 1.06Stations A to D are within 0.11 of their neighbours. Station E is 1.06 above them. In the history, 95% of all gaps are between −1.04 and 0.7, so E is unusual.
Each station has its own normal gap: a country station is often cleaner than the centre. So each gap becomes a robust z-score, with the station’s own median and MAD from the history, as in Median, MAD and IQR. We call it the neighbour z-score:
def neighbour_gap(pm25, min_others=6): """log(1 + reading) minus the median of the other stations in the same hour.""" x = np.log1p(pm25) gap = {} for s in x.columns: others = x.drop(columns=s) gap[s] = (x[s] - others.median(axis=1)).where(others.notna().sum(axis=1) >= min_others) return pd.DataFrame(gap)
def neighbour_z(pm25, history=HISTORY): """The robust z-score of the neighbour gap, with each station's median and MAD from the history.""" g = neighbour_gap(pm25) return pd.DataFrame({s: robust_z(g[s], g.loc[history, s]) for s in g.columns}).abs()With fewer than 6 other stations in the hour, there is no fair median, and the hour gets no score.
Six detectors and a weekly budget
Section titled “Six detectors and a weekly budget”Kai compares six detectors. Each one gives every station and hour a score.
- Own MAD (the baseline). The robust z-score of a reading against its own station’s last 7 days. It knows nothing about the other stations.
- Rules, written before anyone looked at the flags. A rule fires on the same value for 6 hours or more (above the floor), on PM2.5 more than twice PM10 plus 50, or on the maximum, 999. A rule says 1 or 0. See Rules, and when a rule is the right answer.
- Neighbour z, as above.
- Isolation Forest on five numbers per station and hour: the level, the neighbour gap, the change since the last hour, PM2.5 against PM10, and the hours without a change. It learns from 60,000 random hours of the history, with 200 trees.
- Rules, then neighbour z, and rules, then Isolation Forest. A rule flag comes first, and the score orders the rest.
Then the budget turns scores into Mina’s weekly list:
def weekly_flags(hourly, period, budget=BUDGET, start=None): """The `budget` highest station-days of each week (ties in time order). A 0/1 rule sends only its 1s.""" day = station_days(hourly, period) day["week"] = (day["day"] - pd.Timestamp(start or period.start)).dt.days // 7 day = day.sort_values(["week", "score", "day", "station"], ascending=[True, False, True, True]) flags = day.groupby("week").head(budget) return flags[flags["score"] > 0].reset_index(drop=True)Each detector sends 260 flags in the 52 evaluation weeks, 5 a week. The rules alone fired on only 4 station-days in the whole year.
Which detector’s 260 flags are better, when nobody knows which readings were wrong? We use the three ways of Evaluate without a full answer key.
Way 1: plant faults that you know
Section titled “Way 1: plant faults that you know”The planted faults
Section titled “The planted faults”We make a small answer key of our own. We copy the readings and plant 104 faults in the evaluation weeks: 2 a week, at random stations and days, 26 of each kind:
| Kind | What we change |
|---|---|
| Spike | One hour reads 3 × the real value + 150. |
| Stuck | The same value for 8 hours. |
| Scale | Every reading of the day is 60% too high, like a badly calibrated instrument. |
| Clock | The day shows the readings of 6 hours earlier. |
Each detector scores the copy, and Mina’s weekly list takes the top 5 as before. Recall at the budget is the share of planted faults that are on the list:
kind clock scale spike stuck allown MAD (baseline) 0.04 0.04 0.27 0.04 0.10rules 0.12 0.04 0.85 1.00 0.50neighbour z 0.23 0.12 0.77 0.08 0.30Isolation Forest 0.15 0.08 1.00 0.58 0.45rules, then neighbour z 0.31 0.15 1.00 1.00 0.62rules, then Isolation Forest 0.23 0.08 1.00 1.00 0.58Text description of the chart
A column chart with six columns, the planted faults found of 104. Own MAD (the baseline): 10. Rules: 52. Neighbour z: 31. Isolation Forest: 47. Rules, then neighbour z: 64, the highest. Rules, then Isolation Forest: 60.
What this tells us
Section titled “What this tells us”- The baseline finds almost nothing: 10 of 104. Real changes of the air fill its budget.
- The simple rules are strong: they find every stuck sensor and 85% of the spikes, with about one flag a week.
- Rules, then neighbour z are the best, 64 of 104, a little ahead of rules, then Isolation Forest, 60. The difference is 4 faults in a year. The simple method is as good, and much easier to explain: “this station reads 3, its neighbours read 90”.
- Nobody finds the scale errors: at most 15%. A reading 60% too high still looks like air. A slow calibration error needs a method for small lasting shifts, such as CUSUM on the neighbour gap, from Small, lasting shifts: CUSUM and EWMA.
Planted faults give exact numbers, but only for the four kinds that we imagined. To see the real faults, someone must look.
Way 2: review a random sample by hand
Section titled “Way 2: review a random sample by hand”The checklist
Section titled “The checklist”Before the review, Kai and Mina write down how to decide. Without a checklist, one reviewer decides differently on Monday and on Friday.
Text description of the diagram
The diagram is a decision tree. The first question: is PM2.5 at the floor, stuck, or a one-hour jump, while its own PM10 or the other stations are 10 µg/m³ or more away? If yes, the verdict is fault. For example: 3 for three hours, while its PM10 reads 53 to 104 and the others 70 to 78. If no, the second question: does its PM10, or another station, show the same change at about the same time? If yes, the verdict is real, for example clean air that reaches this station an hour earlier. If no, the third question: is the gap under 10 µg/m³, too small to change what the map shows? If yes, the verdict is normal, for example 3 against 9 in clean air. If no, the verdict is unclear: write down why, and count it apart.
The sample
Section titled “The sample”The notebook takes 20 random flags of the baseline, of neighbour z and of Isolation Forest, from the evaluation weeks with no planted faults. The rules fired only 4 times, so we review all 4.
64 picked flags, 61 different station-days to review (366 minutes at 6 minutes each)Two detectors picked the same three station-days. 61 reviews are about 12 weeks of Mina’s budget, for one evaluation.
For each flag, the reviewer reads an evidence card: the worst hour of the day, with 3 hours on each side. Here is one:
PM2.5 PM10 others, median others, highest same as before10-19 16:00 NaN NaN 79.5 85.0 010-19 17:00 NaN NaN 79.0 89.0 010-19 18:00 3.0 3.0 82.0 90.0 010-19 19:00 3.0 3.0 92.0 104.0 110-19 20:00 69.0 69.0 98.0 120.0 010-19 21:00 86.0 86.0 100.0 135.0 010-19 22:00 84.0 84.0 106.0 117.0 0Wanliu, 19 October 2015. After two missing hours, both instruments read 3, while the other stations read 82 to 92. The verdict: fault. The file reviews.csv has a verdict and a one-line reason for each of the 61 station-days.
The verdicts
Section titled “The verdicts”{'fault': 23, 'real': 18, 'normal': 16, 'unclear': 4}own MAD (baseline) faults 5 of 20: 0.25 (95% interval 0.06 to 0.44) | real 7, normal 7, unclear 1neighbour z faults 10 of 20: 0.50 (95% interval 0.28 to 0.72) | real 7, normal 0, unclear 3Isolation Forest faults 9 of 20: 0.45 (95% interval 0.23 to 0.67) | real 3, normal 7, unclear 1rules faults 1 of 4: 0.25 (95% interval 0.00 to 0.67) | real 1, normal 2, unclear 0Text description of the chart
Five rows, each a share of faults with a 95% interval, on an axis from 0 to 100%. Own MAD (the baseline), 5 of 20: 25%, from 6% to 44%. Neighbour z, 10 of 20: 50%, from 28% to 72%. Isolation Forest, 9 of 20: 45%, from 23% to 67%. The floor rule, 13 of 20: 65%, from 44% to 86%. The policy on the new weeks, 7 of 40: 17.5%, from 6% to 29%.
What this tells us
Section titled “What this tells us”- Neighbour z and Isolation Forest are about equal. 10 and 9 faults of 20. Their intervals overlap almost completely. Twenty reviews cannot separate them.
- The baseline is worse: 5 faults of 20, and 14 of its 20 flags were clean air or real changes of the air.
- The rules’ 4 flags taught us about the data. One was a real fault: a station stuck at 11 for 7 hours. One was the New Year night, at 999. Two were PM10 problems: PM2.5 agreed with the neighbours, but PM10 was too low and not a whole number. So “PM2.5 far above PM10” sometimes points at the wrong instrument.
- Isolation Forest flags clean air (7 normal of 20). Neighbour z flags real local changes (7 real of 20), for example clean air that reaches one station first.
A reviewed sample says nothing about faults that got no flag, so it gives no recall. And the verdicts are our judgement.
Way 3: agreement between detectors
Section titled “Way 3: agreement between detectors”Agreement needs no answer key. The Jaccard index of two lists is the flags in both, divided by the flags in either. 1 means the same flags, 0 means none in common.
flagged by all three: 32 own MAD (baseline) neighbour z Isolation Forestown MAD (baseline) 1.00 0.13 0.12neighbour z 0.13 1.00 0.23Isolation Forest 0.12 0.23 1.00What this tells us. The detectors agree little: at most 0.23, and only 32 station-days are on all three lists. Low agreement warns against one detector alone, but it does not say which one is right.
Common mistake
The baseline flags clean air
Symptom: Mina opens a baseline flag for Wanliu on 22 January 2016. The station reads 3 to 8. The other stations read 5 to 9. Nothing is wrong.
Cause: the baseline compares each reading with its own last 7 days, on a log scale. In very clean air, a step from 8 to 3 is a big step on that scale. But it is 5 µg/m³: nobody who reads the map would see it.
Fix: compare a station with its neighbours, not only with its own past. Then a whole clean city looks normal. Also write down, before the review, how big an error must be to matter: here, about 10 µg/m³.
Consequence in AI work: a detector that is right “on its scale” can still waste the budget. 14 of the baseline’s 20 reviewed flags had no fault, and while Mina closed them, real faults waited.
What the review taught us: drops to the floor
Section titled “What the review taught us: drops to the floor”One kind of fault, again and again
Section titled “One kind of fault, again and again”Most of the 23 faults in the review look the same. PM2.5 falls to the floor for an hour or two, while its PM10 or the other stations stay higher. Nobody planted this kind, so only the review could find it.
A new rule can catch it: a reading of 4 or less, while the median of the other stations is 15 or more. Then the reading is at least 11 below its neighbours.
def floor_drops(pm25, low=4, others_at_least=15): """Readings at the floor (4 or less) while the median of the other stations is 15 or more.""" others = pd.DataFrame({s: pm25.drop(columns=s).median(axis=1) for s in pm25.columns}) return (pm25 <= low) & (others >= others_at_least)floor-drop station-days in the evaluation weeks: 224 (4.3 a week)floor drop False Trueverdictfault 5 18normal 15 1real 17 1unclear 4 018 of the 23 reviewed faults are floor drops. But there are 4.3 floor-drop station-days a week: almost Mina’s whole budget of 5.
Is the rule right?
Section titled “Is the rule right?”The 18 faults came from the sampled flags, not from the rule. To know how often the rule is right, we review a new random sample: 20 of its 224 station-days.
floor drops reviewed: {'fault': 13, 'unclear': 3, 'real': 2, 'normal': 2}faults 13 of 20: 0.65 (95% interval 0.44 to 0.86)About two in three floor drops are faults. The others are clean air at one station, for example a northern station that clears an hour before the city.
Let the rule act alone
Section titled “Let the rule act alone”When the rule is wrong, it hides a low reading: the map shows “no reading” instead of “good air” for an hour. It never hides pollution.
So this rule can act with no review, and Mina’s budget goes to the rest. It is a written rule with an owner, as in Alert fatigue and ownership. Mina counts the floor drops each week and sends the count to the data provider.
The comparison under one measure
Section titled “The comparison under one measure”Kai puts every detector in one table. The measure that decides is the one tied to Mina’s budget: planted faults found in 5 flags a week. The review column is from 20 reviews each, so read it with its interval. The costs are our assumptions: 6 minutes per review.
| Detector | Flags a week | Planted found | Planted missed a week | Reviewed: faults | Review minutes a week |
|---|---|---|---|---|---|
| Own MAD (baseline) | 5.0 | 0.10 | 1.8 | 0.25 (5 of 20) | 30 |
| Rules | 1.1 | 0.50 | 1.0 | 0.25 (1 of 4) | 6 |
| Neighbour z | 5.0 | 0.30 | 1.4 | 0.50 (10 of 20) | 30 |
| Isolation Forest | 5.0 | 0.45 | 1.1 | 0.45 (9 of 20) | 30 |
| Rules, then neighbour z | 5.0 | 0.62 | 0.8 | not reviewed | 30 |
| Rules, then Isolation Forest | 5.0 | 0.58 | 0.8 | not reviewed | 30 |
| Policy: floor drops removed, rules, then neighbour z | 5.0 | 0.62 | 0.8 | on new weeks: 0.17 (7 of 40) | 30 |
The rules’ 1.1 flags a week are on the copy with planted faults. “Rules, then neighbour z” shares 258 of its 260 real flags with neighbour z, so its reviewed share is close to 0.50.
The simple method wins. Rules, then the neighbour z-score, find the most planted faults, and they are the easiest to explain. Isolation Forest is close, and the review cannot separate the two.
The recommendation: the alert policy
Section titled “The recommendation: the alert policy”Text description of the diagram
The diagram is a flow from left to right. Every hour, the readings of 12 stations, PM2.5 and PM10, arrive. Step 1 asks: is it a floor drop, a reading of 4 or less while the median of the others is 15 or more? If yes, the policy marks the reading missing, with no review, and counts it each week: 54 station-days in the 8 new weeks. If no, step 2 scores it: a rule first, then the neighbour z-score. Step 3 keeps the 5 highest station-days of the week. Step 4: Mina reviews the evidence card of each one. If it is a fault, she holds the reading and tells the data provider: 7 in the 8 new weeks. If it is real or normal, she keeps the reading and notes the reason: 33 in the 8 new weeks.
Kai writes to Mina:
- Hide floor drops automatically. About two in three are faults, and a wrong one only hides clean air. Count them every week for the data provider.
- Review the 5 highest station-days a week, ranked by the three rules first, then by the neighbour z-score. Use the checklist, and write one reason per verdict.
- Never hide a high reading without a review. A city-wide high is real air, and the neighbour z-score does not flag it.
- Do not use Isolation Forest for now. It found 4 fewer planted faults, and its reasons are harder to explain.
And what not to use the policy for:
- Not for slow calibration errors. It found at most 15% of the planted scale errors.
- Not for a fault of the whole feed. When all 12 stations freeze together, as on 18 September 2013, the neighbours agree with each other. Only the stuck rule sees it.
- Not as a statement about a station’s quality. A verdict is our judgement from the data, not a fact from the provider.
End to end: 8 new weeks
Section titled “End to end: 8 new weeks”The run
Section titled “The run”Kai runs the policy once on 2 January to 26 February 2017. No step has seen these weeks, and we plant no faults. Each flag gets a reason from the data of its worst hour.
new_scores = policy(pm25, pm10)new_flags = weekly_flags(new_scores, NEW_INPUT)new_drops = drop_days.loc[NEW_INPUT]new_flags["reason"] = [reason(s, d) for s, d in zip(new_flags["station"], new_flags["day"])]8 weeks: 54 floor-drop station-days removed with no review; 40 flags to review, 0 of them from a rule day station reason0 2017-01-02 Wanshouxigong 10:00 362, above the median of the others, 23; its PM10 4571 2017-01-02 Tiantan 11:00 314, above the median of the others, 29; its PM10 4792 2017-01-02 Gucheng 04:00 13, below the median of the others, 269; its PM10 133 2017-01-08 Wanshouxigong 18:00 154, above the median of the others, 18; its PM10 1644 2017-01-08 Tiantan 18:00 125, above the median of the others, 18; its PM10 1455 2017-01-10 Aotizhongxin 19:00 13, below the median of the others, 69; its PM10 506 2017-01-10 Wanshouxigong 17:00 57, above the median of the others, 8; its PM10 977 2017-01-11 Nongzhanguan 15:00 7, below the median of the others, 50; its PM10 58 2017-01-13 Wanliu 18:00 2, below the median of the others, 14; its PM10 229 2017-01-11 Wanshouxigong 22:00 119, above the median of the others, 26; its PM10 119The reasons explain the first flags. On 2 January, Wanshouxigong and Tiantan, two central stations, read over 300 while the others read about 25. Their PM10 agrees, and the city follows two hours later. Gucheng reads 13 while the others read 269: clean air reaches it first.
The review of all 40 flags
Section titled “The review of all 40 flags”Mina reviews all 40 flags (new-reviews.csv):
{'real': 27, 'fault': 7, 'normal': 6}faults 7 of 40: 0.17 (95% interval 0.06 to 0.29)What this tells us. Only 7 of the 40 flags are faults, far fewer than the 10 of 20 of neighbour z in the evaluation weeks. There are two reasons:
- The floor rule took the easy faults first. 54 station-days never reached the queue.
- Winter fills the queue with real local differences. Clean air from the north reaches the northern stations an hour or two before the centre. The neighbour z-score cannot tell a front that arrives an hour early from a fault that lasts an hour.
Mina keeps the 27 real readings. But the budget buys fewer faults than the evaluation weeks promised.
Three of the 7 faults are on 16 January 2017: Wanshouxigong at 11:00, Shunyi at 15:00 and Tiantan at 16:00. One station after another drops and comes back: a question for the data provider.
The New Year night
Section titled “The New Year night”The New Year night of 28 January 2017 is the dirtiest night in these weeks. Does the policy call it a fault?
median of the 12 stations, 00:00 to 06:00: [342, 566, 655, 613, 582, 574, 538]own MAD (baseline): 1 flag on 28 Januarypolicy: 0 flags on 28 JanuaryNo. Every station is high, so no station is far from its neighbours. The baseline spends one of its 5 weekly flags on that night. The policy keeps every reading on the map.
Insights and limits
Section titled “Insights and limits”- No labels, three kinds of evidence. Planted faults gave exact recall for the kinds that we imagined. The review found the most common real fault. Agreement showed that the detectors see different things. None alone was enough.
- The review budget sets how much you know. 20 reviews give an interval about 40 points wide.
- The evaluation was too kind. In the new weeks, the share of faults in the queue fell from about one in two to about one in six.
- The verdicts are ours. The course author wrote each verdict from an evidence card with a written checklist, not from a field visit, and could see which detector picked each flag. 4 of 61 first verdicts were “unclear”. A blind second reviewer would disagree on some. Reviewers make mistakes, as Evaluate without a full answer key warns.
- The policy is blind to two kinds of fault: slow calibration errors (scale) and faults of the whole feed.
- Old data, one city. Instruments change, so check the floor rule again on newer data.
Responsible use
Section titled “Responsible use”- People act on public data. A hidden real reading is the worst mistake, so the policy only hides low readings automatically.
- Do not publish a verdict as a fact. “Station X was faulty on this day” is a judgement from the data. Call it that, and give the evidence card.
- Credit the source (CC BY 4.0), with the attribution text above and a note of what you changed.
- Label the assumptions. The budget of 5, the 6 minutes, the 10 µg/m³ limit and the floor rule’s numbers are choices. Say so wherever a number depends on them.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can evaluate a detector honestly with no labels, and report a result that went against you. Make it yours with one change, for example PM10 instead of PM2.5, a new kind of planted fault, or a second reviewer. Then make it public and easy to read in five minutes.
What to put in the repository:
README.md(template below), and the notebook with its outputs.checklist.md(the review checklist, one example per verdict) andreviews.csv(your verdicts, each with a reason).comparison.csv, with the review of the new weeks kept apart.- The data source and licence, with the attribution text. A link and a checksum are enough.
A README template:
# Faults in a public air-quality feed, with no labels
## Problem12 stations report PM2.5 every hour to a public map. One reviewer can check 5 station-daysa week (assumption). Hiding a real high reading is the worst mistake.
## Data and licenceUCI "Beijing Multi-Site Air Quality" (Chen), hourly, Mar 2013 - Feb 2017, CC BY 4.0,https://doi.org/10.24432/C5RK5G. No labels. Unchanged.
## ApproachUnit: a station-day. Detectors: own-station MAD (baseline), 3 rules, neighbour z-score(robust z of the gap to the other stations' median), Isolation Forest, rules + each score.Evaluation without labels: 104 planted faults (recall at 5 a week), a random hand review(20 flags each, 95% intervals), Jaccard agreement.
## ResultsRules then neighbour z: 64 of 104 planted faults; Isolation Forest 47; baseline 10.Review: neighbour z 10/20 faults, Isolation Forest 9/20. The review found floor drops(reading <= 4, others >= 15): 65% faults, now hidden with no review.New weeks (Jan-Feb 2017): 7 faults in 40 reviewed flags, 27 real local differences.
## LimitsOur verdicts; wide intervals; blind to slow calibration errors and feed-wide faults;2013-2017 data.How to describe it in an interview, in two sentences: “I built a fault detector for a public air-quality feed with no labels, and evaluated it with planted faults, a hand review and agreement. The review found the most common real fault, which became an automatic rule, and new weeks showed more real local differences than faults.”
Go further:
- Add CUSUM on the neighbour gap, from Small, lasting shifts: CUSUM and EWMA, and plant the scale errors again.
- Flag a dip that comes back: far from both the hour before and the hour after, while the neighbours do not move. Does it separate faults from fronts?
- Compare with the other case studies, Card fraud with a review budget and Faults in machine or server metrics: where did a label exist, and what did it change?
Your turn
Section titled “Your turn”Task 1: a bigger budget for the scale errors
Section titled “Task 1: a bigger budget for the scale errors”Neighbour z found 12% of the planted scale errors with 5 flags a week. Before you run it, predict: with a budget of 10 flags a week, does it find twice as many? In the notebook, section 11, use weekly_flags(planted_scores["neighbour z"], EVALUATION, budget=10) and found(), and put the share of scale errors found in share_scale_neighbour_10, rounded to 2 decimals.
Show the answer
budget_10 = weekly_flags(planted_scores["neighbour z"], EVALUATION, budget=10)share_scale_neighbour_10 = round(float(found(budget_10)[planted.kind == "scale"].mean()), 2)Check passed: share_scale_neighbour_10 is 0.23.Yes, about twice as many: 23% instead of 12%. But it costs twice the review time, and 77% of the scale errors are still missed. A bigger budget does not fix a detector that cannot see a fault.
Task 2: a stricter floor rule
Section titled “Task 2: a stricter floor rule”The floor rule asks for a median of 15 or more at the other stations. Predict: with 30 or more, how many of the 224 floor-drop station-days remain? Use floor_drops(pm25, others_at_least=30) and put the count of station-days in the evaluation weeks in floor_days_30.
Show the answer
drops_30 = floor_drops(pm25, others_at_least=30)floor_days_30 = int(drops_30.loc[EVALUATION].groupby(drops_30.loc[EVALUATION].index.normalize()).any().sum().sum())Check passed: floor_days_30 is 84.84 station-days, 1.6 a week. A stricter rule hides less clean air by mistake, but the drops while the others read 15 to 29 come back on the map. To choose, review a sample of those 140 station-days.
Knowledge check
Section titled “Knowledge check”Check your understanding
Kai's planted faults say that rules, then neighbour z find 62% of faults in the budget. Mina says: 'So 62% of the real faults are found.' Explain why that is not right, and what the planted number does tell her.
Show the answer
The 62% is recall on the 104 faults that Kai planted. They are of four kinds that he imagined: spikes, stuck values, scale errors and clock errors. Real faults are of other kinds too. The most common one in this feed, the drop to the floor, was not planted at all. So the number says nothing exact about real faults. It does tell her how the detectors compare on the same, known test. Rules, then neighbour z found 64 of 104, Isolation Forest 47 and the baseline 10. Use planted faults to compare and choose, not to promise a real recall.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the hand review, neighbour z had 10 faults of 20 flags and Isolation Forest 9 of 20. Can Kai say that neighbour z is more precise? What would he need to be sure?
Show the answer
No. With 20 reviews, the 95% interval of each share is about 0.4 wide. It is 0.28 to 0.72 for neighbour z, and 0.23 to 0.67 for Isolation Forest. The intervals overlap almost completely, so the difference of one fault could be chance. To separate two shares that differ by about 0.05, he would need hundreds of reviews per detector. That is months of Mina’s budget. So he decides with other evidence: the planted faults, and which detector is easier to explain.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The floor rule is right about two times in three, and it acts with no review. A rule that hid every reading above 500 might be right just as often. Why is the first rule acceptable and the second not?
Show the answer
Because the cost of a mistake is very different. When the floor rule is wrong, it hides a low reading. The map shows no reading instead of clean air for an hour, and nobody is at risk. When a rule that hides high readings is wrong, it hides real pollution, such as the New Year fireworks at 999. Then people get no warning, the worst mistake for the map. An automatic action is acceptable only when its mistakes are cheap; a rule whose mistakes are expensive must go to a person.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- With no labels, you used three kinds of evidence. Planted faults are exact, for the kinds you planted. A random hand review is honest, with wide intervals. Agreement is free, but blind to what all miss.
- Comparing a station with its neighbours separates real events from faults. On 104 planted faults, rules, then the neighbour z-score found 64, Isolation Forest 47 and the baseline 10: the simple method won.
- The review found the most common real fault, drops to the floor. A rule now hides them with no review, because its mistakes only hide clean air.
- On 8 new weeks, the policy hid 54 floor-drop station-days and sent 40 flags: 7 faults, 27 real local differences, 6 normal. The New Year night got no flag.
- A number from the evaluation weeks is not a promise for the next season: check the policy on new data and say what it found.
Next, a real production line: control charts and drift checks on a factory’s sensors, compared with Isolation Forest.