Skip to content
Français

Control charts on a real production line

  • Case study 4
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, we watched Larkfield's refunds with control charts and its payments with drift tests. Factories invented these methods. In this case study, we build control charts and drift checks on the sensors of a real production line, compare them with Isolation Forest, and look for early warnings.

Case study 4 of 4 · Anomaly Detection: Find What Is UnusualControl charts on a real production line
Course overview

What you will be able to do

Take a real production line’s sensor readings to a weekly inspection list, and say honestly what it is worth. You will set phase I limits and find out how many of 590 sensors are noise. You will run a drift report with the false discovery rate and chart the sensors that moved. Then you compare the detectors under one inspection budget, with the pass/fail results only as a check.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Phase I and phase II
  • I chart for hundreds of sensors
  • Noise sensors
  • Drift report with FDR
  • EWMA on chosen sensors
  • Labels as a check
  • One budget for all detectors
  • Isolation Forest
  • An alarm runbook

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install pandas, SciPy and scikit-learn, and download the data (2 MB) from the UCI repository with a checksum. It needs no account, and it runs in under a minute after the install.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

These lessons matter most:

What you need on your computer

Nothing: the notebook runs in Colab, Kaggle or VS Code. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, NumPy 2.5.4, SciPy 1.18.1, scikit-learn 1.9.1 and matplotlib 3.11.2. The setup cell makes a new folder secom-case-study. To start again, delete that folder.

Tested on macOS with Python 3.12.13, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

A factory makes semiconductor chips on a long line of machines. Each unit (one item of production) passes many process steps. At each step, sensors and measurement points record how the machines run. At the end of the line, a test says pass or fail.

Beatriz is the quality engineer of the line. Ivan is a data analyst. We invented both people for this case study. The line and its data are real.

Beatriz has two questions:

  • “The line has 590 sensors. Which of them should my team watch, and with which charts?”
  • “My team can take 5 units a week out of the line for a full inspection. Which 5 should we take?”

The costs are our assumptions:

  • An inspection takes an engineer about an hour, with the tool time: 80 dollars (assumption).
  • A fail that reaches the line test has been through all the later steps for nothing: 400 dollars of wasted work (assumption).

A failed unit found early costs 80 dollars instead of 400. A passed unit that we inspect costs 80 dollars for nothing.

You will build a weekly job for Beatriz. It reads the new units of the week. It reports which sensors drifted, and the state of the charts on the chosen sensors. It lists the 5 units to inspect, with the sensors that are furthest out on each one. The pass/fail results never train anything. They only check each step afterwards.

The data is SECOM from the UCI Machine Learning Repository. It has 1,567 units of one semiconductor line, from 19 July to 17 October 2008. Each unit has 590 sensor readings, and the result and time of its line test.

File What it holds
secom.data One line per unit, 590 numbers separated by spaces. NaN is a missing reading. The sensors have no names: the factory removed them.
secom_labels.data One line per unit: -1 (passed) or 1 (failed), and the time of the test.

We compared two other datasets. The Tennessee Eastman Process runs are public domain, but simulated, and 1.4 GB. The Bosch production line data is under a competition’s rules, not an open licence, and about 14 GB.

The notebook loads both files and checks them:

sensors = pd.read_csv("secom.data", sep=r"\s+", header=None)
sensors.columns = [f"s{i:03d}" for i in range(sensors.shape[1])] # the sensors have numbers, not names
labels = pd.read_csv("secom_labels.data", sep=" ", header=None, names=["result", "time"])
labels["time"] = pd.to_datetime(labels["time"], format="%d/%m/%Y %H:%M:%S")
fail = (labels["result"] == 1).to_numpy() # 1 = failed the line test, -1 = passed
(1567, 590) | fails: 104 (6.6%)
from 2008-07-19 11:55:00 to 2008-10-17 06:07:00 | in time order: True
missing readings: 4.5% | units a day: median 16, most 62

What this tells us. The units are already in time order, so a control chart can follow them one by one. 104 units failed, 6.6%. The line makes a different number of units each day, from 1 to 62. So we chart units, not days, and we count the budget per week.

Control charts need a phase I: a period that sets the limits. Then phase II draws every new unit against those limits, as in Control charts. We split the 13 weeks (19 July to 17 October) in time:

phase I 19 Jul to 31 Aug: 618 units, 65 fails (10.5%)
first week 01 Sep to 07 Sep: 133 units, 2 fails (1.5%)
comparison 08 Sep to 12 Oct: 722 units, 36 fails (5.0%)
new week 13 Oct to 17 Oct: 94 units, 1 fails (1.1%)

What this tells us. Phase I sets the limits. The first week after it is the first look: its drift report chooses the sensors to chart. The comparison weeks compare every detector under the same budget. The new week stays unused until the end, for the end-to-end run.

The fail rate is not stable. It is 10.5% in phase I and 5.0% in the comparison weeks. A phase I with many fails is not a “good” period, but it is the only one we have.

Some sensors cannot tell us anything. The notebook counts three kinds:

constant = sensors.nunique() <= 1 # the same value on every unit
mostly_empty = sensors.isna().mean() > 0.5 # more than half of the readings missing
top_share = sensors[phase1].apply(lambda s: s.value_counts(normalize=True).max() if s.notna().any() else 1.0)
stuck_in_phase1 = (top_share > 0.95) & ~constant & ~mostly_empty # one value in more than 95% of phase I
constant: 116 | more than half empty: 28 | stuck on one value in phase I: 14 | kept: 432

A stuck sensor reads the same value on almost every phase I unit. Four of them read one value on 616 of the 618 phase I units, and they vary only later. Their moving range in phase I is close to 0, so their limits are almost 0 wide. Every later change then looks like a huge alarm. The failure case below tells the story.

Now the question that the dataset is famous for: which sensors are related to the fails? We use the labels only to check, not to choose anything.

For each sensor, the notebook runs a rank test (the Mann-Whitney test). It sorts all the readings of the sensor and asks: do the failed units sit higher or lower in the order than the passed units? It looks at the order, not the size, so a few huge readings cannot decide it. With 432 tests, chance alone gives about 22 below 0.05. So we use Bonferroni and the false discovery rate (BH), as in Drift in many features.

Before you read the output, predict: how many sensors does BH keep over all the units? And inside one period only?

432 tests | p < 0.05: 85 (chance alone: about 22) | Bonferroni: 9 | FDR (BH): 23
phase I only: FDR (BH) keeps 2
comparison weeks only: FDR (BH) keeps 0

Over all the units, BH keeps 23 sensors. Inside phase I, it keeps 2. Inside the comparison weeks, it keeps none. Why does the number fall so much?

The notebook tests which sensors moved between phase I and the later weeks (the KS test, with BH):

sensors that moved between phase I and the rest (KS, FDR): 364 of 432
of the 23 related to fails, moved too: 23
fail rate: phase I 10.5%, after it 4.1%

What this tells us. The fail rate fell after phase I, and 364 of the 432 sensors moved at the same time. A sensor that only follows the calendar is “related to fails” over all the units: its phase I values come with more fails. All 23 sensors that BH kept also moved with time. Inside one period, where the calendar cannot help, almost nothing is left.

A tree from 590 sensors. 158 cannot carry a signal: 116 constant, 28 more than half empty, 14 stuck on one value in phase I. 432 are kept. Of the 432, 364 moved between phase I and later. 23 are related to fails over all units, and all 23 also moved with time. Inside one period: 2 in phase I, 0 in the comparison weeks.
How many sensors are noise? 158 of 590 cannot carry a signal. Of the 432 others, the false discovery rate keeps 23 as related to fails over all units, and 0 inside the comparison weeks.
Text description of the diagram

The diagram starts with 590 sensors on 1,567 units. One branch goes to 158 sensors that cannot carry a signal: 116 constant, 28 more than half empty and 14 stuck on one value in phase I. The other branch goes to the 432 sensors that we keep. From the 432, one box says that 364 moved between phase I and the later weeks (KS test, false discovery rate 5%). Another box says that 23 are related to the fails over all the units (rank test, false discovery rate 5%). All 23 also moved with time. Below it, a last box says: inside one period, 2 in phase I and 0 in the comparison weeks.

So, honestly: 158 sensors are certainly noise. Of the other 432, we cannot show that any single sensor tells a failed unit from a passed one inside a period. That does not prove that they are useless: 36 fails is a small sample, and a sensor can matter together with others.

The notebook builds the I chart of every sensor, as in Control charts. The centre line is the phase I mean. Sigma comes from the moving range: the mean change from one unit to the next, divided by 1.128. A missing reading takes the phase I median, so it gives no alarm.

centre = ref.mean()
sigma = ref.diff().abs().mean() / 1.128 # the I-MR chart's sigma, from the moving range
filled = S.fillna(ref.median()) # a missing reading takes phase I's median: it gives no alarm
z = (filled - centre) / sigma # each reading, in sigmas from its centre line
beyond = z.abs() > 3 # outside the I chart's limits

Predict first. If every sensor were in control and bell-shaped, 0.27% of the readings would be outside the limits by chance. With 432 sensors, that is 1.2 sensors on each unit. How many do you expect in phase I itself?

sensors outside their limits on one unit, phase I: 12.1 on average (chance alone, if every sensor were in control and bell-shaped: 1.2)
the same in the comparison weeks: 15.3
units with no sensor outside: phase I 3 of 618

What this tells us. Even in phase I, a unit has 12.1 sensors outside their own limits, ten times more than chance. Only 3 of the 618 phase I units have none. Real sensors have steps, slow drifts and long tails. So phase I is not “in control” for many of them.

This is the hotel with a hundred smoke alarms, from Drift in many features, with 432 alarms. One sensor outside its limits means almost nothing. The number of sensors outside, on one unit, may mean more. We test that below.

Multivariate drift with the false discovery rate

Section titled “Multivariate drift with the false discovery rate”

The drift report compares the first week after phase I with phase I, sensor by sensor. It gives the KS test’s p-value, the PSI and the BH decision, as the report of Drift in many features did for Larkfield’s payments:

432 sensors | p < 0.05: 265 | FDR (BH): 246 | PSI 0.25 or more: 205

BH keeps 246 sensors, and 205 have a large PSI (0.25 or more). With 133 units, this is not only the “big sample” effect of Has the distribution changed?. Half of the line really moved.

The same report for every week after phase I:

week units fails fail rate sensors drifted (BH)
2008-09-01 133 2 0.015 246
2008-09-08 95 4 0.042 323
2008-09-15 154 5 0.032 312
2008-09-22 166 4 0.024 310
2008-09-29 169 15 0.089 331
2008-10-06 138 8 0.058 329
2008-10-13 94 1 0.011 336

The week of 29 September has the most fails: 15 of 169 units, 8.9%. But its drift report looks like the others. Every week, BH keeps 246 to 336 sensors. Here is the fail rate of every week, phase I included. It is the line’s real quality signal, from the labels:

  • Phase I weeks
  • Weeks after phase I
  • Phase I fail rate (10.5%)
The weekly fail rate. It is high in phase I (10.5% over the period), low in early September, and up again in the week of 29 September (8.9%).
Text description of the chart

The chart shows one dot per week, from the week of 14 July to the week of 13 October 2008. A dashed line marks the phase I fail rate, 10.5%. The first seven dots are phase I: 23.1% (only 13 units), 9.5%, 20.8%, 13.0%, 13.7%, 10.6% and 4.1%. Then come the weeks after phase I: 1.5%, 4.2%, 3.2%, 2.4%, 8.9% (the week of 29 September, the highest after phase I), 5.8% and 1.1%.

What this tells us. Drift is everywhere and all the time, and it does not follow the fails. When the fail rate went down after phase I, hundreds of sensors drifted. When it went up on 29 September, the drift report did not change. On this line, the drift report says “look” every week, and it cannot say where a quality problem is.

Beatriz cannot read 432 charts. A control chart also makes sense only on a sensor that was in control in phase I. So Ivan first keeps the chartable sensors, with four rules that need no label:

  • fewer than 5% of the readings missing;
  • at least 50 different values in phase I (not a counter with few steps);
  • at most 1% of the phase I units outside the limits;
  • little memory from one unit to the next (lag-1 autocorrelation below 0.3).

151 sensors pass. From those, he takes the 3 with the largest PSI in the first week’s drift report:

chartable sensors: 151 of 432
chosen: ['s059', 's416', 's051']
p PSI BH
s059 1.5e-18 2.254 True
s416 3.6e-11 1.914 True
s051 5.6e-06 1.409 True
  • s059, each unit
  • Centre line, phase I mean (7.66)
  • Limits from phase I (−18.68 and 34.00)
  • End of phase I
The I chart of s059. After phase I, the level falls from about 7.7 to about 0 and stays there, but only 2 later units are outside the limits. Two phase I readings, 87.0 and 168.1, are drawn at the top edge.
Text description of the chart

The chart shows the reading of sensor s059 on each of the 1,567 units as a thin, noisy line. A solid centre line is at 7.66, the phase I mean. Dashed limits are at −18.68 and 34.00. A dashed vertical line marks the end of phase I, at unit 618. In phase I, the line is high and noisy: 90% of the readings are between −8.4 and 26.9. Six phase I readings are outside the limits. Two of them, 87.0 and 168.1, are above the top of the chart. After phase I, the line is lower: 90% of the readings are between −5.7 and 5.2, with a mean of −0.08. Only two later readings are outside the limits, both on 1 September.

What this tells us. The level of s059 fell by about 7.7, which is 0.9 sigma, and stayed low. Single units rarely cross a 3-sigma limit after a shift that small. That is the job of the EWMA, from Small, lasting shifts: CUSUM and EWMA.

The notebook runs an EWMA on each chosen sensor, with the settings of that lesson: λ = 0.1 and L = 2.6. Each EWMA starts at 0 at the start of phase I, and again on the first unit after it. Then the labels check the alarms: is the fail rate higher while the EWMA is in alarm?

  • EWMA of s059
  • EWMA limits
  • End of phase I
The EWMA of s059 crosses its lower limit on 1 September, the first day after phase I, and stays below it. In phase I, it is also outside its limits 38.8% of the time: phase I was not stable.
Text description of the chart

The chart shows the EWMA of sensor s059, in sigmas, for the 1,567 units. Two dashed limits are at +0.596 and −0.596. A dashed vertical line marks the end of phase I. In phase I, the EWMA goes up to 1.67 on 5 August. It comes down to −0.89 on 31 August, so it crosses both limits. After phase I, it starts again at 0 and falls below the lower limit within ten units, on 1 September. It stays below the limit for 93% of the comparison units, and its lowest value is −1.22.

sensor share in alarm, phase I first alarm after phase I share in alarm, comparison fail rate in alarm fail rate not in alarm
s059 0.388 01 Sep 09:06 0.927 0.048 0.075
s416 0.209 01 Sep 05:32 0.191 0.036 0.053
s051 0.286 01 Sep 23:05 0.108 0.038 0.051

What this tells us. Three things.

  • The EWMA found the shifts fast. All three charts alarmed on 1 September, the first day after phase I.
  • The shifts did not hurt the yield. For each sensor, the units made during an alarm failed less often than the others: 4.8% against 7.5% for s059. The process moved; the quality did not get worse.
  • Phase I was not stable. The EWMAs were in alarm in 21% to 39% of phase I itself. The I chart’s rule (1% outside) passed these sensors, but a slow drift inside phase I hides from an I chart. Check phase I with an EWMA too.

One measure: fails found in five inspections a week

Section titled “One measure: fails found in five inspections a week”

Now Beatriz’s second question: which 5 units a week should her team inspect? Each detector gives every unit a score. Each week, the team inspects the 5 units with the highest scores. Ivan compares six options in the five comparison weeks:

  • No inspection: the floor.
  • Count outside limits: how many of the 432 sensors are outside their phase I limits on this unit. This is the simple rule, the baseline.
  • Robust z count: how many sensors are more than 4 robust z (MAD) from their phase I median, as in Median, MAD and IQR.
  • I chart, 3 chosen sensors: the largest |z| of s059, s416 and s051.
  • EWMA, 3 chosen sensors: the largest |EWMA| of the same three.
  • Isolation Forest: 200 trees on all 432 sensors in z units, trained on phase I.

A perfect row, which knows the answers, shows the ceiling. Some weeks have fewer than 5 fails, so even perfect cannot reach a precision of 1.

The notebook scores every detector with the same function. The last column is the chance that random picks would find as many fails (a binomial test):

def at_budget(score, mask=compare_weeks, budget=BUDGET):
"""Each week, inspect the `budget` units with the highest scores (ties: the earlier unit first)."""
found = inspected = 0
for w in sorted(set(week[mask])):
units = np.nonzero(mask & (week == w))[0]
top = units[np.argsort(-score[units], kind="stable")][:budget]
inspected, found = inspected + len(top), found + int(fail[top].sum())
...

Predict first: which detector finds the most fails? Does any detector cost less than no inspection at all?

comparison weeks: 5 | units: 722 | fails: 36 | a random pick finds a fail 5.0% of the time
detector inspected fails found precision recall cost a week chance of this by luck
no inspection 0 0 0.00 0.000 2880 1.000
count outside limits (all sensors) 25 3 0.12 0.083 3040 0.126
count above 4 robust z (all sensors) 25 1 0.04 0.028 3200 0.722
I chart, 3 chosen sensors (largest |z|) 25 0 0.00 0.000 3280 1.000
EWMA, 3 chosen sensors (largest |EWMA|) 25 0 0.00 0.000 3280 1.000
Isolation Forest (all sensors) 25 2 0.08 0.056 3120 0.356
perfect (knows the answers) 25 23 0.92 0.639 1440 0.000
Fails found under the same budget. The simple count of sensors outside their limits finds 3, Isolation Forest 2, the charts on the chosen sensors 0. Perfect would find 23.
Text description of the chart

Seven rows, one dot each: the fails found in 25 inspections. No inspection: 0. Count outside limits: 3, highlighted. Robust z count: 1. I chart on 3 sensors: 0. EWMA on 3 sensors: 0. Isolation Forest: 2. Perfect: 23, far to the right of all the others.

  • The simplest score wins. “How many sensors are outside their limits” finds 3 fails in 25 inspections, a precision of 0.12. Random picks would find a fail 5.0% of the time. Isolation Forest, on the same 432 sensors, finds 2.
  • The charts on the chosen sensors find nothing at the unit level. An EWMA watches the process: it says “the level moved since 1 September”, and that is true for weeks. It cannot say which unit will fail.
  • But all the numbers are small. 3 fails in 25 inspections would happen by luck 12.6% of the time. So we cannot say that the count is better than random. We can say that nothing here is clearly better than it.
  • No detector pays for its inspections. With our costs, an inspection pays only if more than 80 ÷ 400 = 20% of the inspected units fail. The best precision is 12%. So inspecting costs more each week (3,040 dollars) than not inspecting (2,880 dollars).

Tomás, the course’s AI engineer, would ask for a bigger model. But more model is not more signal: Isolation Forest, with all 432 sensors, did not beat a count. When not to use machine learning showed the same on Larkfield’s payments.

Ivan writes to Beatriz in her words:

  • Do not choose units for inspection with these sensors yet. The best score found 3 fails in 25 inspections, where random picks find about 1, and that can be luck. It costs more than it saves.
  • Use the count of sensors outside their limits as the one number to watch per unit. It is simple to explain, and it was the best of the six. Check it again when there are more fails.
  • Chart the process, not the unit. EWMA charts on a few chartable sensors show fast when the process moves (here, on the first day after phase I). For each EWMA alarm, check the yield before you act.
  • Make a new phase I. The old one had a 10.5% fail rate and drifting sensors. Take a few weeks with a stable fail rate and stable EWMAs, check them, and set new limits.
  • Remove or repair the noise. 158 sensors carry no signal. 14 of them were stuck in phase I. Ask the equipment team whether they are broken, replaced, or reset.

And what not to use it for:

  • Not to stop the line. Almost every unit has a sensor outside its limits. Hold units only after the alarm steps below.
  • Not as a finding about physics. The sensors have no names.
  • Not as a fail predictor. Nothing trained on the labels, and the checks show that the readings do not separate fails well.

The last week, 13 to 17 October, was not used by any step. Ivan runs the weekly job on it. It does the drift report, the state of the EWMAs and the inspection list, with the 3 sensors that are furthest out on each unit:

13 Oct to 17 Oct: 94 units
drift against phase I: 336 sensors (BH); largest PSI: s116 7.38, s539 6.19, s247 5.98
EWMA of the chosen sensors, share of the week in alarm: {'s059': 1.0, 's416': 0.32, 's051': 0.02}
time sensors outside furthest out
unit
1489 13 Oct 20:53 49 s338, s202, s333
1482 13 Oct 15:42 32 s572, s576, s574
1525 15 Oct 07:36 30 s032, s036, s034
1533 15 Oct 14:08 30 s222, s494, s360
1539 15 Oct 19:24 29 s572, s574, s576

Then the labels of the new week, as a check:

fails among the 5 inspected: 0 | failed units this week: [1519] | their rank by the count: [8] of 94

What this tells us. The job works on new data. Its result is modest. One unit of 94 failed, and the count ranked it 8th. With a budget of 5, the team would have missed it. With a budget of 10, it would have found it, at twice the cost.

Sensors s572, s574 and s576 are furthest out on two of the five units. Only the factory’s sensor list can say whether they are on one machine.

What a quality engineer does with an alarm

Section titled “What a quality engineer does with an alarm”

An alarm is the start of the work, not the end. Beatriz’s team follows four steps:

A flow from an alarm (a unit with many sensors outside, or an EWMA alarm). Step 1: the sensor or the process? Stuck, missing, recalibrated. If the sensor: fix the data feed or the sensor, write it down, no hold. If the process, step 2: one unit or a lasting shift? Step 3: what changed at that time? Tool log, maintenance, material batch, recipe. Step 4: does it hurt the yield? If no: normal moved, write it down, new phase I limits after the change. If yes: hold and inspect the units since the change, fix the cause, then new limits.
Four steps after an alarm. Most alarms on a line like this one end at step 1 or step 4 with no hold: a sensor problem, or a change that does not hurt the yield.
Text description of the diagram

The diagram starts with an alarm: a unit with many sensors outside its limits, or an EWMA alarm. Step 1 asks: is it the sensor or the process? Look for a stuck sensor, missing readings or a recalibration. If it is the sensor, fix the data feed or the sensor and write it down, with no hold. If it is the process, step 2 asks: is it one unit or a lasting shift? The I chart shows one point, the EWMA shows a run. Step 3 asks: what changed at that time? Look at the tool log, the maintenance, the material batch and the recipe. Step 4 asks: does it hurt the yield? Look at the fail rate since the change. If not, normal moved: write it down and set new phase I limits after the change. If it does, hold and inspect the units made since the change, fix the cause, then set new limits.

On this line, the EWMA alarm of s059 on 1 September goes through all four steps. It is not a stuck sensor (step 1). It is a lasting shift, not one unit (step 2). The data cannot answer step 3: only the factory’s logs can. At step 4, the fail rate went down after the shift. So Beatriz writes it down, keeps watching, and sets new limits for s059 after a stable period. She holds no unit.

The yield check of step 4 can be a p-chart of the weekly fail rate. It comes late, at the end of the line, but it decides whether a sensor alarm matters.

Common mistake

The four sensors that were stuck

Symptom: in Ivan’s first run, the inspection list for the new week named the same sensors on every unit: s542, s543, s544 and s545. The five units had 33 to 57 sensors “outside their limits”.

Cause: these four sensors read one value on 616 of the 618 phase I units, for example 0.0026 for s544. Their moving range in phase I was close to 0 (exactly 0 for s544), so their limits were almost 0 wide. After phase I, they read real, changing values, and every value was “outside”. This was step 1 of the alarm steps: the sensor, not the process.

Fix: a rule before any limit. A sensor that reads one value on more than 95% of phase I has no usable limits. The notebook now counts 14 such sensors and removes them, so 432 sensors are left, not 440.

Consequence in AI work: a detector that adds up many features follows its worst feature. One feature with a scale near 0 can fill every alert. Check the spread of each feature in the reference window before you trust a combined score.

Common mistake

Choose the sensors with the labels

Symptom: a common way to use SECOM is to pick the sensors that differ most between fails and passes, and then to chart those. In phase I, BH keeps 2 such sensors.

Cause: in the comparison weeks, BH keeps none of the 432 sensors as related to fails. The sensors that phase I’s labels pick are not reliable later. Over all the units, 23 sensors look related, but all 23 also moved with time, together with the fail rate.

Fix: choose the sensors without the labels (chartable, and moved most in the drift report), and use the labels only to check afterwards. If you must choose with labels, choose in one period and check in a later one, as here.

Consequence in AI work: a feature chosen on the same data that you then evaluate always looks good. On a process that drifts, it often only measures the date. This is why the machine learning course keeps a separate test set, in Train, validation and test sets: keep the choice and the check apart.

  • Many sensors, little signal. 158 of 590 sensors carry none. Inside one period, no single sensor of the other 432 separated fails from passes. Over all units, the 23 that look related move with the calendar.
  • Phase I was not in control.
  • Drift is not damage. Hundreds of sensors drifted every week while the fail rate went down.
  • Small numbers. Five comparison weeks, 36 fails, 25 inspections per detector. A difference of one or two fails is noise, so the comparison cannot rank the detectors: none is clearly useful.
  • Our assumptions. The budget (5 a week) and the costs (80 and 400 dollars) are ours, not the factory’s. With a missed fail at 1,000 dollars, the break-even precision falls to 8%. Then the count would pay, if its precision held.
  • One line, 13 weeks in 2008, anonymous sensors. We cannot check the physics, the tool logs or the maintenance, which an engineer would use first.
  • The data is about a real factory. It has no personal data. The factory removed the sensor names, so do not try to find out which company or machines it describes.
  • Credit the source. CC BY 4.0 allows you to share and change the data, with the attribution text above and a note of your changes.
  • A hold affects people. Holding units slows the line, and an operator may have to explain an alarm that was a broken sensor. Check step 1 before you involve people, and say how sure the alarm is.
  • Label the assumptions. The budget, the costs and the phase I period are choices. Say so wherever a number depends on them, and report the “chance by luck” next to small counts.

This case study shows an employer that you can work with real, messy sensor data and that you report a weak result honestly. Make it yours with one change. For example, try a new phase I, a Hotelling T² chart, or another budget. Then make it public and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • sensor_triage.csv: each sensor with its rule (constant, empty, stuck, kept, chartable).
  • drift_by_week.csv and comparison.csv: the weekly drift counts, and the six detectors with fails found, precision, recall, cost and the chance by luck.
  • The data source and licence, with the attribution text above. A link and the checksum are enough.

A README template:

# Control charts and drift checks on a real semiconductor line (UCI SECOM)
## Problem
590 sensors, 1,567 units, pass/fail at the end of the line. Which sensors to chart,
and which 5 units a week to inspect? Inspection 80 dollars, a fail at the line test
400 dollars (assumptions).
## Data and licence
UCI "SECOM" (McCann, Johnston), July-October 2008, CC BY 4.0,
https://doi.org/10.24432/C54305. Labels used only as a check.
## Approach
Phase I = 19 Jul - 31 Aug. Removed 158 constant, empty or stuck sensors.
I chart limits for 432 sensors; weekly drift report (KS, PSI, BH); EWMA on 3 chartable
sensors that moved most; 6 detectors under 5 inspections a week, 5 comparison weeks.
## Results
No sensor separates fails inside one period (BH: 0 of 432). Hundreds drift every week,
but the yield improved. Best detector: count of sensors outside limits, 3 fails in
25 inspections (random: about 1; chance by luck 13%). No detector pays at these costs.
## Limits
Unstable phase I; 36 fails; anonymous sensors; assumed costs.

How to describe it in an interview, in four sentences: “I built phase I limits and a weekly drift report for the 590 sensors of a real semiconductor line. I compared six detectors under one inspection budget, with the pass/fail results only as a check. 158 sensors carried no signal, and most of the apparent link to fails was time. No detector beat random picks with confidence, and I reported that with the numbers.”

Go further:

  • Build a Hotelling T² chart: one multivariate chart on the main components of phase I. Does it beat the count?
  • Choose a new phase I, for example 1 to 28 September, and repeat the comparison on October. Do the EWMAs stay quiet?
  • Compare with the other case studies, Card fraud with a review budget and Faults in machine or server metrics: where does a simple rule win there?

Change one thing: the budget. Before you run it, predict: with 10 inspections a week, is the precision of the count higher or lower than 0.12? In the notebook, section 8, use at_budget() with budget=10 for scores["count outside limits (all sensors)"], and put the precision in precision_10.

Show the answer
precision_10 = at_budget(scores["count outside limits (all sensors)"], budget=10)["precision"]
Check passed: precision_10 is 0.12.

The precision stays at 0.12: 6 fails in 50 inspections, so the count finds twice as many fails at twice the cost. It is still far below the 20% that an inspection needs to pay. A larger budget finds more, but it does not make each inspection more useful.

Check your understanding

Over all 1,567 units, the false discovery rate keeps 23 sensors as related to the fails. Inside the comparison weeks, it keeps none. Explain why, and what it means for choosing sensors.

Show the answer

The fail rate changed with time: 10.5% in phase I and about 4% after it. At the same time, 364 of the 432 sensors moved. A sensor that only follows the calendar has different values in the period with many fails. So, over all the units, it looks related to the fails. All 23 kept sensors also moved with time. Inside one period, the calendar cannot help, and no sensor separates fails from passes. So choosing sensors by their link to fails over all the data would mostly choose clocks. Choose without the labels, and check later, in a separate period.

Check your understanding

The EWMA of sensor s059 alarms on the first day after phase I and stays in alarm for weeks. The units made during the alarm failed less often than the others. What should Beatriz do, and why?

Show the answer

She should follow the alarm steps. It is not a stuck sensor, and it is a lasting shift, not one unit, so the process really moved. She should ask what changed at that time (the tool log, maintenance, the material). Then she checks the yield: the fail rate did not go up (4.8% in alarm, 7.5% not in alarm). So the change does not hurt the quality. She writes it down, holds no units, and sets new limits for s059 after a stable period. An EWMA says that the process moved, not that it got worse; only the yield check can say that.

Check your understanding

The count of sensors outside their limits finds 3 fails in 25 inspections; random picks would find about 1. Ivan does not recommend using it to choose units yet. Give two reasons from the numbers.

Show the answer

First, the difference can be luck. With a fail rate of 5.0%, random picks would find 3 or more fails in 25 inspections about 12.6% of the time. With only 36 fails in five weeks, one or two fails more is noise. Second, it does not pay. An inspection costs 80 dollars and a fail 400 dollars, so an inspection pays only if more than 20% of the inspected units fail. The count’s precision is 12%. Inspecting with it costs 3,040 dollars a week, more than the 2,880 dollars of no inspection.

Recap

  • Of 590 sensors, 158 carry no signal (constant, mostly empty or stuck in phase I). Of the other 432, the false discovery rate keeps 23 as related to fails over all units. It keeps 0 inside the comparison weeks: most of the signal is time.
  • Phase I was not in control: 12.1 sensors outside their limits per unit, against 1.2 by chance.
  • The weekly drift report kept 246 to 336 sensors every week, and it did not follow the fails.
  • The EWMA on the chosen sensors found the shift on the first day, and the yield check showed that it did not hurt the quality.
  • Under one budget of 5 inspections a week, the simplest score (sensors outside their limits) found the most fails, 3 of 36. But it could be luck, and no detector paid for its inspections.

Next, the final assignment: Larkfield’s refunds per agent, with a new region that goes live in the middle of the month. You choose the charts and the checks, and you write the alert policy.

Next: Final assignment