Control charts on a real production line
- Case study 4
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we watched Larkfield's refunds with control charts and its payments with drift tests. Factories invented these methods. In this case study, we build control charts and drift checks on the sensors of a real production line, compare them with Isolation Forest, and look for early warnings.
Case study 4 of 4 · Anomaly Detection: Find What Is UnusualControl charts on a real production line
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Take a real production line’s sensor readings to a weekly inspection list, and say honestly what it is worth. You will set phase I limits and find out how many of 590 sensors are noise. You will run a drift report with the false discovery rate and chart the sensors that moved. Then you compare the detectors under one inspection budget, with the pass/fail results only as a check.
The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install pandas, SciPy and scikit-learn, and download the data (2 MB) from the UCI repository with a checksum. It needs no account, and it runs in under a minute after the install.
Before you startWhat to know and to have ready
What you need to know
These lessons matter most:
- Phase I and phase II, and the I-MR chart, from Control charts: is this variation normal?.
- The EWMA chart and its settings, from Small, lasting shifts: CUSUM and EWMA.
- KS, PSI and why a big sample always says “changed”, from Has the distribution changed?.
- Bonferroni, the false discovery rate and the drift report, from Drift in many features, and what to do about it.
- Isolation Forest, from Isolation Forest.
- Precision at a budget and cost per week, from The alert budget.
What you need on your computer
Nothing: the notebook runs in Colab, Kaggle or VS Code. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, NumPy 2.5.4, SciPy 1.18.1, scikit-learn 1.9.1 and matplotlib 3.11.2. The setup cell makes a new folder secom-case-study. To start again, delete that folder.
Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”A factory makes semiconductor chips on a long line of machines. Each unit (one item of production) passes many process steps. At each step, sensors and measurement points record how the machines run. At the end of the line, a test says pass or fail.
Beatriz is the quality engineer of the line. Ivan is a data analyst. We invented both people for this case study. The line and its data are real.
Beatriz has two questions:
- “The line has 590 sensors. Which of them should my team watch, and with which charts?”
- “My team can take 5 units a week out of the line for a full inspection. Which 5 should we take?”
The costs are our assumptions:
- An inspection takes an engineer about an hour, with the tool time: 80 dollars (assumption).
- A fail that reaches the line test has been through all the later steps for nothing: 400 dollars of wasted work (assumption).
A failed unit found early costs 80 dollars instead of 400. A passed unit that we inspect costs 80 dollars for nothing.
What you will build
Section titled “What you will build”You will build a weekly job for Beatriz. It reads the new units of the week. It reports which sensors drifted, and the state of the charts on the chosen sensors. It lists the 5 units to inspect, with the sensors that are furthest out on each one. The pass/fail results never train anything. They only check each step afterwards.
The data
Section titled “The data”The data is SECOM from the UCI Machine Learning Repository. It has 1,567 units of one semiconductor line, from 19 July to 17 October 2008. Each unit has 590 sensor readings, and the result and time of its line test.
| File | What it holds |
|---|---|
secom.data |
One line per unit, 590 numbers separated by spaces. NaN is a missing reading. The sensors have no names: the factory removed them. |
secom_labels.data |
One line per unit: -1 (passed) or 1 (failed), and the time of the test. |
We compared two other datasets. The Tennessee Eastman Process runs are public domain, but simulated, and 1.4 GB. The Bosch production line data is under a competition’s rules, not an open licence, and about 14 GB.
First checks
Section titled “First checks”The notebook loads both files and checks them:
sensors = pd.read_csv("secom.data", sep=r"\s+", header=None)sensors.columns = [f"s{i:03d}" for i in range(sensors.shape[1])] # the sensors have numbers, not nameslabels = pd.read_csv("secom_labels.data", sep=" ", header=None, names=["result", "time"])labels["time"] = pd.to_datetime(labels["time"], format="%d/%m/%Y %H:%M:%S")fail = (labels["result"] == 1).to_numpy() # 1 = failed the line test, -1 = passed(1567, 590) | fails: 104 (6.6%)from 2008-07-19 11:55:00 to 2008-10-17 06:07:00 | in time order: Truemissing readings: 4.5% | units a day: median 16, most 62What this tells us. The units are already in time order, so a control chart can follow them one by one. 104 units failed, 6.6%. The line makes a different number of units each day, from 1 to 62. So we chart units, not days, and we count the budget per week.
Four periods
Section titled “Four periods”Control charts need a phase I: a period that sets the limits. Then phase II draws every new unit against those limits, as in Control charts. We split the 13 weeks (19 July to 17 October) in time:
phase I 19 Jul to 31 Aug: 618 units, 65 fails (10.5%)first week 01 Sep to 07 Sep: 133 units, 2 fails (1.5%)comparison 08 Sep to 12 Oct: 722 units, 36 fails (5.0%)new week 13 Oct to 17 Oct: 94 units, 1 fails (1.1%)What this tells us. Phase I sets the limits. The first week after it is the first look: its drift report chooses the sensors to chart. The comparison weeks compare every detector under the same budget. The new week stays unused until the end, for the end-to-end run.
The fail rate is not stable. It is 10.5% in phase I and 5.0% in the comparison weeks. A phase I with many fails is not a “good” period, but it is the only one we have.
How many sensors are noise?
Section titled “How many sensors are noise?”Sensors that cannot carry a signal
Section titled “Sensors that cannot carry a signal”Some sensors cannot tell us anything. The notebook counts three kinds:
constant = sensors.nunique() <= 1 # the same value on every unitmostly_empty = sensors.isna().mean() > 0.5 # more than half of the readings missingtop_share = sensors[phase1].apply(lambda s: s.value_counts(normalize=True).max() if s.notna().any() else 1.0)stuck_in_phase1 = (top_share > 0.95) & ~constant & ~mostly_empty # one value in more than 95% of phase Iconstant: 116 | more than half empty: 28 | stuck on one value in phase I: 14 | kept: 432A stuck sensor reads the same value on almost every phase I unit. Four of them read one value on 616 of the 618 phase I units, and they vary only later. Their moving range in phase I is close to 0, so their limits are almost 0 wide. Every later change then looks like a huge alarm. The failure case below tells the story.
The labels as a check
Section titled “The labels as a check”Now the question that the dataset is famous for: which sensors are related to the fails? We use the labels only to check, not to choose anything.
For each sensor, the notebook runs a rank test (the Mann-Whitney test). It sorts all the readings of the sensor and asks: do the failed units sit higher or lower in the order than the passed units? It looks at the order, not the size, so a few huge readings cannot decide it. With 432 tests, chance alone gives about 22 below 0.05. So we use Bonferroni and the false discovery rate (BH), as in Drift in many features.
Before you read the output, predict: how many sensors does BH keep over all the units? And inside one period only?
432 tests | p < 0.05: 85 (chance alone: about 22) | Bonferroni: 9 | FDR (BH): 23phase I only: FDR (BH) keeps 2comparison weeks only: FDR (BH) keeps 0Over all the units, BH keeps 23 sensors. Inside phase I, it keeps 2. Inside the comparison weeks, it keeps none. Why does the number fall so much?
Most of the signal is the calendar
Section titled “Most of the signal is the calendar”The notebook tests which sensors moved between phase I and the later weeks (the KS test, with BH):
sensors that moved between phase I and the rest (KS, FDR): 364 of 432of the 23 related to fails, moved too: 23fail rate: phase I 10.5%, after it 4.1%What this tells us. The fail rate fell after phase I, and 364 of the 432 sensors moved at the same time. A sensor that only follows the calendar is “related to fails” over all the units: its phase I values come with more fails. All 23 sensors that BH kept also moved with time. Inside one period, where the calendar cannot help, almost nothing is left.
Text description of the diagram
The diagram starts with 590 sensors on 1,567 units. One branch goes to 158 sensors that cannot carry a signal: 116 constant, 28 more than half empty and 14 stuck on one value in phase I. The other branch goes to the 432 sensors that we keep. From the 432, one box says that 364 moved between phase I and the later weeks (KS test, false discovery rate 5%). Another box says that 23 are related to the fails over all the units (rank test, false discovery rate 5%). All 23 also moved with time. Below it, a last box says: inside one period, 2 in phase I and 0 in the comparison weeks.
So, honestly: 158 sensors are certainly noise. Of the other 432, we cannot show that any single sensor tells a failed unit from a passed one inside a period. That does not prove that they are useless: 36 fails is a small sample, and a sensor can matter together with others.
Phase I limits for 432 sensors
Section titled “Phase I limits for 432 sensors”The notebook builds the I chart of every sensor, as in Control charts. The centre line is the phase I mean. Sigma comes from the moving range: the mean change from one unit to the next, divided by 1.128. A missing reading takes the phase I median, so it gives no alarm.
centre = ref.mean()sigma = ref.diff().abs().mean() / 1.128 # the I-MR chart's sigma, from the moving rangefilled = S.fillna(ref.median()) # a missing reading takes phase I's median: it gives no alarmz = (filled - centre) / sigma # each reading, in sigmas from its centre linebeyond = z.abs() > 3 # outside the I chart's limitsPredict first. If every sensor were in control and bell-shaped, 0.27% of the readings would be outside the limits by chance. With 432 sensors, that is 1.2 sensors on each unit. How many do you expect in phase I itself?
sensors outside their limits on one unit, phase I: 12.1 on average (chance alone, if every sensor were in control and bell-shaped: 1.2)the same in the comparison weeks: 15.3units with no sensor outside: phase I 3 of 618What this tells us. Even in phase I, a unit has 12.1 sensors outside their own limits, ten times more than chance. Only 3 of the 618 phase I units have none. Real sensors have steps, slow drifts and long tails. So phase I is not “in control” for many of them.
This is the hotel with a hundred smoke alarms, from Drift in many features, with 432 alarms. One sensor outside its limits means almost nothing. The number of sensors outside, on one unit, may mean more. We test that below.
Multivariate drift with the false discovery rate
Section titled “Multivariate drift with the false discovery rate”The first week
Section titled “The first week”The drift report compares the first week after phase I with phase I, sensor by sensor. It gives the KS test’s p-value, the PSI and the BH decision, as the report of Drift in many features did for Larkfield’s payments:
432 sensors | p < 0.05: 265 | FDR (BH): 246 | PSI 0.25 or more: 205BH keeps 246 sensors, and 205 have a large PSI (0.25 or more). With 133 units, this is not only the “big sample” effect of Has the distribution changed?. Half of the line really moved.
Every week
Section titled “Every week”The same report for every week after phase I:
week units fails fail rate sensors drifted (BH)2008-09-01 133 2 0.015 2462008-09-08 95 4 0.042 3232008-09-15 154 5 0.032 3122008-09-22 166 4 0.024 3102008-09-29 169 15 0.089 3312008-10-06 138 8 0.058 3292008-10-13 94 1 0.011 336The week of 29 September has the most fails: 15 of 169 units, 8.9%. But its drift report looks like the others. Every week, BH keeps 246 to 336 sensors. Here is the fail rate of every week, phase I included. It is the line’s real quality signal, from the labels:
- Phase I weeks
- Weeks after phase I
- Phase I fail rate (10.5%)
Text description of the chart
The chart shows one dot per week, from the week of 14 July to the week of 13 October 2008. A dashed line marks the phase I fail rate, 10.5%. The first seven dots are phase I: 23.1% (only 13 units), 9.5%, 20.8%, 13.0%, 13.7%, 10.6% and 4.1%. Then come the weeks after phase I: 1.5%, 4.2%, 3.2%, 2.4%, 8.9% (the week of 29 September, the highest after phase I), 5.8% and 1.1%.
What this tells us. Drift is everywhere and all the time, and it does not follow the fails. When the fail rate went down after phase I, hundreds of sensors drifted. When it went up on 29 September, the drift report did not change. On this line, the drift report says “look” every week, and it cannot say where a quality problem is.
Chart the sensors that moved
Section titled “Chart the sensors that moved”Which sensors to chart
Section titled “Which sensors to chart”Beatriz cannot read 432 charts. A control chart also makes sense only on a sensor that was in control in phase I. So Ivan first keeps the chartable sensors, with four rules that need no label:
- fewer than 5% of the readings missing;
- at least 50 different values in phase I (not a counter with few steps);
- at most 1% of the phase I units outside the limits;
- little memory from one unit to the next (lag-1 autocorrelation below 0.3).
151 sensors pass. From those, he takes the 3 with the largest PSI in the first week’s drift report:
chartable sensors: 151 of 432chosen: ['s059', 's416', 's051'] p PSI BHs059 1.5e-18 2.254 Trues416 3.6e-11 1.914 Trues051 5.6e-06 1.409 TrueThe I chart of s059
Section titled “The I chart of s059”- s059, each unit
- Centre line, phase I mean (7.66)
- Limits from phase I (−18.68 and 34.00)
- End of phase I
Text description of the chart
The chart shows the reading of sensor s059 on each of the 1,567 units as a thin, noisy line. A solid centre line is at 7.66, the phase I mean. Dashed limits are at −18.68 and 34.00. A dashed vertical line marks the end of phase I, at unit 618. In phase I, the line is high and noisy: 90% of the readings are between −8.4 and 26.9. Six phase I readings are outside the limits. Two of them, 87.0 and 168.1, are above the top of the chart. After phase I, the line is lower: 90% of the readings are between −5.7 and 5.2, with a mean of −0.08. Only two later readings are outside the limits, both on 1 September.
What this tells us. The level of s059 fell by about 7.7, which is 0.9 sigma, and stayed low. Single units rarely cross a 3-sigma limit after a shift that small. That is the job of the EWMA, from Small, lasting shifts: CUSUM and EWMA.
EWMA on the chosen sensors
Section titled “EWMA on the chosen sensors”The notebook runs an EWMA on each chosen sensor, with the settings of that lesson: λ = 0.1 and L = 2.6. Each EWMA starts at 0 at the start of phase I, and again on the first unit after it. Then the labels check the alarms: is the fail rate higher while the EWMA is in alarm?
- EWMA of s059
- EWMA limits
- End of phase I
Text description of the chart
The chart shows the EWMA of sensor s059, in sigmas, for the 1,567 units. Two dashed limits are at +0.596 and −0.596. A dashed vertical line marks the end of phase I. In phase I, the EWMA goes up to 1.67 on 5 August. It comes down to −0.89 on 31 August, so it crosses both limits. After phase I, it starts again at 0 and falls below the lower limit within ten units, on 1 September. It stays below the limit for 93% of the comparison units, and its lowest value is −1.22.
sensor share in alarm, phase I first alarm after phase I share in alarm, comparison fail rate in alarm fail rate not in alarm s059 0.388 01 Sep 09:06 0.927 0.048 0.075 s416 0.209 01 Sep 05:32 0.191 0.036 0.053 s051 0.286 01 Sep 23:05 0.108 0.038 0.051What this tells us. Three things.
- The EWMA found the shifts fast. All three charts alarmed on 1 September, the first day after phase I.
- The shifts did not hurt the yield. For each sensor, the units made during an alarm failed less often than the others: 4.8% against 7.5% for s059. The process moved; the quality did not get worse.
- Phase I was not stable. The EWMAs were in alarm in 21% to 39% of phase I itself. The I chart’s rule (1% outside) passed these sensors, but a slow drift inside phase I hides from an I chart. Check phase I with an EWMA too.
One measure: fails found in five inspections a week
Section titled “One measure: fails found in five inspections a week”The detectors
Section titled “The detectors”Now Beatriz’s second question: which 5 units a week should her team inspect? Each detector gives every unit a score. Each week, the team inspects the 5 units with the highest scores. Ivan compares six options in the five comparison weeks:
- No inspection: the floor.
- Count outside limits: how many of the 432 sensors are outside their phase I limits on this unit. This is the simple rule, the baseline.
- Robust z count: how many sensors are more than 4 robust z (MAD) from their phase I median, as in Median, MAD and IQR.
- I chart, 3 chosen sensors: the largest |z| of s059, s416 and s051.
- EWMA, 3 chosen sensors: the largest |EWMA| of the same three.
- Isolation Forest: 200 trees on all 432 sensors in z units, trained on phase I.
A perfect row, which knows the answers, shows the ceiling. Some weeks have fewer than 5 fails, so even perfect cannot reach a precision of 1.
The comparison
Section titled “The comparison”The notebook scores every detector with the same function. The last column is the chance that random picks would find as many fails (a binomial test):
def at_budget(score, mask=compare_weeks, budget=BUDGET): """Each week, inspect the `budget` units with the highest scores (ties: the earlier unit first).""" found = inspected = 0 for w in sorted(set(week[mask])): units = np.nonzero(mask & (week == w))[0] top = units[np.argsort(-score[units], kind="stable")][:budget] inspected, found = inspected + len(top), found + int(fail[top].sum()) ...Predict first: which detector finds the most fails? Does any detector cost less than no inspection at all?
comparison weeks: 5 | units: 722 | fails: 36 | a random pick finds a fail 5.0% of the time detector inspected fails found precision recall cost a week chance of this by luck no inspection 0 0 0.00 0.000 2880 1.000 count outside limits (all sensors) 25 3 0.12 0.083 3040 0.126 count above 4 robust z (all sensors) 25 1 0.04 0.028 3200 0.722I chart, 3 chosen sensors (largest |z|) 25 0 0.00 0.000 3280 1.000EWMA, 3 chosen sensors (largest |EWMA|) 25 0 0.00 0.000 3280 1.000 Isolation Forest (all sensors) 25 2 0.08 0.056 3120 0.356 perfect (knows the answers) 25 23 0.92 0.639 1440 0.000Text description of the chart
Seven rows, one dot each: the fails found in 25 inspections. No inspection: 0. Count outside limits: 3, highlighted. Robust z count: 1. I chart on 3 sensors: 0. EWMA on 3 sensors: 0. Isolation Forest: 2. Perfect: 23, far to the right of all the others.
What the comparison tells us
Section titled “What the comparison tells us”- The simplest score wins. “How many sensors are outside their limits” finds 3 fails in 25 inspections, a precision of 0.12. Random picks would find a fail 5.0% of the time. Isolation Forest, on the same 432 sensors, finds 2.
- The charts on the chosen sensors find nothing at the unit level. An EWMA watches the process: it says “the level moved since 1 September”, and that is true for weeks. It cannot say which unit will fail.
- But all the numbers are small. 3 fails in 25 inspections would happen by luck 12.6% of the time. So we cannot say that the count is better than random. We can say that nothing here is clearly better than it.
- No detector pays for its inspections. With our costs, an inspection pays only if more than 80 ÷ 400 = 20% of the inspected units fail. The best precision is 12%. So inspecting costs more each week (3,040 dollars) than not inspecting (2,880 dollars).
Tomás, the course’s AI engineer, would ask for a bigger model. But more model is not more signal: Isolation Forest, with all 432 sensors, did not beat a count. When not to use machine learning showed the same on Larkfield’s payments.
The recommendation
Section titled “The recommendation”Ivan writes to Beatriz in her words:
- Do not choose units for inspection with these sensors yet. The best score found 3 fails in 25 inspections, where random picks find about 1, and that can be luck. It costs more than it saves.
- Use the count of sensors outside their limits as the one number to watch per unit. It is simple to explain, and it was the best of the six. Check it again when there are more fails.
- Chart the process, not the unit. EWMA charts on a few chartable sensors show fast when the process moves (here, on the first day after phase I). For each EWMA alarm, check the yield before you act.
- Make a new phase I. The old one had a 10.5% fail rate and drifting sensors. Take a few weeks with a stable fail rate and stable EWMAs, check them, and set new limits.
- Remove or repair the noise. 158 sensors carry no signal. 14 of them were stuck in phase I. Ask the equipment team whether they are broken, replaced, or reset.
And what not to use it for:
- Not to stop the line. Almost every unit has a sensor outside its limits. Hold units only after the alarm steps below.
- Not as a finding about physics. The sensors have no names.
- Not as a fail predictor. Nothing trained on the labels, and the checks show that the readings do not separate fails well.
The end-to-end run on a new week
Section titled “The end-to-end run on a new week”The last week, 13 to 17 October, was not used by any step. Ivan runs the weekly job on it. It does the drift report, the state of the EWMAs and the inspection list, with the 3 sensors that are furthest out on each unit:
13 Oct to 17 Oct: 94 unitsdrift against phase I: 336 sensors (BH); largest PSI: s116 7.38, s539 6.19, s247 5.98EWMA of the chosen sensors, share of the week in alarm: {'s059': 1.0, 's416': 0.32, 's051': 0.02} time sensors outside furthest outunit1489 13 Oct 20:53 49 s338, s202, s3331482 13 Oct 15:42 32 s572, s576, s5741525 15 Oct 07:36 30 s032, s036, s0341533 15 Oct 14:08 30 s222, s494, s3601539 15 Oct 19:24 29 s572, s574, s576Then the labels of the new week, as a check:
fails among the 5 inspected: 0 | failed units this week: [1519] | their rank by the count: [8] of 94What this tells us. The job works on new data. Its result is modest. One unit of 94 failed, and the count ranked it 8th. With a budget of 5, the team would have missed it. With a budget of 10, it would have found it, at twice the cost.
Sensors s572, s574 and s576 are furthest out on two of the five units. Only the factory’s sensor list can say whether they are on one machine.
What a quality engineer does with an alarm
Section titled “What a quality engineer does with an alarm”An alarm is the start of the work, not the end. Beatriz’s team follows four steps:
Text description of the diagram
The diagram starts with an alarm: a unit with many sensors outside its limits, or an EWMA alarm. Step 1 asks: is it the sensor or the process? Look for a stuck sensor, missing readings or a recalibration. If it is the sensor, fix the data feed or the sensor and write it down, with no hold. If it is the process, step 2 asks: is it one unit or a lasting shift? The I chart shows one point, the EWMA shows a run. Step 3 asks: what changed at that time? Look at the tool log, the maintenance, the material batch and the recipe. Step 4 asks: does it hurt the yield? Look at the fail rate since the change. If not, normal moved: write it down and set new phase I limits after the change. If it does, hold and inspect the units made since the change, fix the cause, then set new limits.
On this line, the EWMA alarm of s059 on 1 September goes through all four steps. It is not a stuck sensor (step 1). It is a lasting shift, not one unit (step 2). The data cannot answer step 3: only the factory’s logs can. At step 4, the fail rate went down after the shift. So Beatriz writes it down, keeps watching, and sets new limits for s059 after a stable period. She holds no unit.
The yield check of step 4 can be a p-chart of the weekly fail rate. It comes late, at the end of the line, but it decides whether a sensor alarm matters.
Common mistake
The four sensors that were stuck
Symptom: in Ivan’s first run, the inspection list for the new week named the same sensors on every unit: s542, s543, s544 and s545. The five units had 33 to 57 sensors “outside their limits”.
Cause: these four sensors read one value on 616 of the 618 phase I units, for example 0.0026 for s544. Their moving range in phase I was close to 0 (exactly 0 for s544), so their limits were almost 0 wide. After phase I, they read real, changing values, and every value was “outside”. This was step 1 of the alarm steps: the sensor, not the process.
Fix: a rule before any limit. A sensor that reads one value on more than 95% of phase I has no usable limits. The notebook now counts 14 such sensors and removes them, so 432 sensors are left, not 440.
Consequence in AI work: a detector that adds up many features follows its worst feature. One feature with a scale near 0 can fill every alert. Check the spread of each feature in the reference window before you trust a combined score.
Common mistake
Choose the sensors with the labels
Symptom: a common way to use SECOM is to pick the sensors that differ most between fails and passes, and then to chart those. In phase I, BH keeps 2 such sensors.
Cause: in the comparison weeks, BH keeps none of the 432 sensors as related to fails. The sensors that phase I’s labels pick are not reliable later. Over all the units, 23 sensors look related, but all 23 also moved with time, together with the fail rate.
Fix: choose the sensors without the labels (chartable, and moved most in the drift report), and use the labels only to check afterwards. If you must choose with labels, choose in one period and check in a later one, as here.
Consequence in AI work: a feature chosen on the same data that you then evaluate always looks good. On a process that drifts, it often only measures the date. This is why the machine learning course keeps a separate test set, in Train, validation and test sets: keep the choice and the check apart.
Insights and limits
Section titled “Insights and limits”- Many sensors, little signal. 158 of 590 sensors carry none. Inside one period, no single sensor of the other 432 separated fails from passes. Over all units, the 23 that look related move with the calendar.
- Phase I was not in control.
- Drift is not damage. Hundreds of sensors drifted every week while the fail rate went down.
- Small numbers. Five comparison weeks, 36 fails, 25 inspections per detector. A difference of one or two fails is noise, so the comparison cannot rank the detectors: none is clearly useful.
- Our assumptions. The budget (5 a week) and the costs (80 and 400 dollars) are ours, not the factory’s. With a missed fail at 1,000 dollars, the break-even precision falls to 8%. Then the count would pay, if its precision held.
- One line, 13 weeks in 2008, anonymous sensors. We cannot check the physics, the tool logs or the maintenance, which an engineer would use first.
Responsible use
Section titled “Responsible use”- The data is about a real factory. It has no personal data. The factory removed the sensor names, so do not try to find out which company or machines it describes.
- Credit the source. CC BY 4.0 allows you to share and change the data, with the attribution text above and a note of your changes.
- A hold affects people. Holding units slows the line, and an operator may have to explain an alarm that was a broken sensor. Check step 1 before you involve people, and say how sure the alarm is.
- Label the assumptions. The budget, the costs and the phase I period are choices. Say so wherever a number depends on them, and report the “chance by luck” next to small counts.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can work with real, messy sensor data and that you report a weak result honestly. Make it yours with one change. For example, try a new phase I, a Hotelling T² chart, or another budget. Then make it public and easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
sensor_triage.csv: each sensor with its rule (constant, empty, stuck, kept, chartable).drift_by_week.csvandcomparison.csv: the weekly drift counts, and the six detectors with fails found, precision, recall, cost and the chance by luck.- The data source and licence, with the attribution text above. A link and the checksum are enough.
A README template:
# Control charts and drift checks on a real semiconductor line (UCI SECOM)
## Problem590 sensors, 1,567 units, pass/fail at the end of the line. Which sensors to chart,and which 5 units a week to inspect? Inspection 80 dollars, a fail at the line test400 dollars (assumptions).
## Data and licenceUCI "SECOM" (McCann, Johnston), July-October 2008, CC BY 4.0,https://doi.org/10.24432/C54305. Labels used only as a check.
## ApproachPhase I = 19 Jul - 31 Aug. Removed 158 constant, empty or stuck sensors.I chart limits for 432 sensors; weekly drift report (KS, PSI, BH); EWMA on 3 chartablesensors that moved most; 6 detectors under 5 inspections a week, 5 comparison weeks.
## ResultsNo sensor separates fails inside one period (BH: 0 of 432). Hundreds drift every week,but the yield improved. Best detector: count of sensors outside limits, 3 fails in25 inspections (random: about 1; chance by luck 13%). No detector pays at these costs.
## LimitsUnstable phase I; 36 fails; anonymous sensors; assumed costs.How to describe it in an interview, in four sentences: “I built phase I limits and a weekly drift report for the 590 sensors of a real semiconductor line. I compared six detectors under one inspection budget, with the pass/fail results only as a check. 158 sensors carried no signal, and most of the apparent link to fails was time. No detector beat random picks with confidence, and I reported that with the numbers.”
Go further:
- Build a Hotelling T² chart: one multivariate chart on the main components of phase I. Does it beat the count?
- Choose a new phase I, for example 1 to 28 September, and repeat the comparison on October. Do the EWMAs stay quiet?
- Compare with the other case studies, Card fraud with a review budget and Faults in machine or server metrics: where does a simple rule win there?
Your turn: a bigger budget
Section titled “Your turn: a bigger budget”Change one thing: the budget. Before you run it, predict: with 10 inspections a week, is the precision of the count higher or lower than 0.12? In the notebook, section 8, use at_budget() with budget=10 for scores["count outside limits (all sensors)"], and put the precision in precision_10.
Show the answer
precision_10 = at_budget(scores["count outside limits (all sensors)"], budget=10)["precision"]Check passed: precision_10 is 0.12.The precision stays at 0.12: 6 fails in 50 inspections, so the count finds twice as many fails at twice the cost. It is still far below the 20% that an inspection needs to pay. A larger budget finds more, but it does not make each inspection more useful.
Knowledge check
Section titled “Knowledge check”Check your understanding
Over all 1,567 units, the false discovery rate keeps 23 sensors as related to the fails. Inside the comparison weeks, it keeps none. Explain why, and what it means for choosing sensors.
Show the answer
The fail rate changed with time: 10.5% in phase I and about 4% after it. At the same time, 364 of the 432 sensors moved. A sensor that only follows the calendar has different values in the period with many fails. So, over all the units, it looks related to the fails. All 23 kept sensors also moved with time. Inside one period, the calendar cannot help, and no sensor separates fails from passes. So choosing sensors by their link to fails over all the data would mostly choose clocks. Choose without the labels, and check later, in a separate period.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The EWMA of sensor s059 alarms on the first day after phase I and stays in alarm for weeks. The units made during the alarm failed less often than the others. What should Beatriz do, and why?
Show the answer
She should follow the alarm steps. It is not a stuck sensor, and it is a lasting shift, not one unit, so the process really moved. She should ask what changed at that time (the tool log, maintenance, the material). Then she checks the yield: the fail rate did not go up (4.8% in alarm, 7.5% not in alarm). So the change does not hurt the quality. She writes it down, holds no units, and sets new limits for s059 after a stable period. An EWMA says that the process moved, not that it got worse; only the yield check can say that.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The count of sensors outside their limits finds 3 fails in 25 inspections; random picks would find about 1. Ivan does not recommend using it to choose units yet. Give two reasons from the numbers.
Show the answer
First, the difference can be luck. With a fail rate of 5.0%, random picks would find 3 or more fails in 25 inspections about 12.6% of the time. With only 36 fails in five weeks, one or two fails more is noise. Second, it does not pay. An inspection costs 80 dollars and a fail 400 dollars, so an inspection pays only if more than 20% of the inspected units fail. The count’s precision is 12%. Inspecting with it costs 3,040 dollars a week, more than the 2,880 dollars of no inspection.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Of 590 sensors, 158 carry no signal (constant, mostly empty or stuck in phase I). Of the other 432, the false discovery rate keeps 23 as related to fails over all units. It keeps 0 inside the comparison weeks: most of the signal is time.
- Phase I was not in control: 12.1 sensors outside their limits per unit, against 1.2 by chance.
- The weekly drift report kept 246 to 336 sensors every week, and it did not follow the fails.
- The EWMA on the chosen sensors found the shift on the first day, and the yield check showed that it did not hurt the quality.
- Under one budget of 5 inspections a week, the simplest score (sensors outside their limits) found the most fails, 3 of 36. But it could be luck, and no detector paid for its inspections.
Next, the final assignment: Larkfield’s refunds per agent, with a new region that goes live in the middle of the month. You choose the charts and the checks, and you write the alert policy.
Next: Final assignment