Skip to content
Français

Rules for patterns, and the price of each rule

  • Lesson 9
  • 35 min
  • Module 3: Statistical process control and distribution drift

Where we are

Our control chart alarms when one day falls outside the limits. A slow pattern, such as eight days in a row above the centre, stays inside them. In this lesson, we add run rules for patterns, and we measure the price of each rule with the average run length: more rules, more false alarms.

Lesson 9 of 38 · Anomaly Detection: Find What Is UnusualRules for patterns, and the price of each rule
Course overview

What you will be able to do

Catch a pattern of many days that no single day shows, and know what each extra rule costs in false alarms. A chart with too few rules is late; a chart with too many rules cries wolf.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It runs the four run rules on Larkfield’s refunds, and a simulation of 2,000 runs that measures the price of each rule. Its setup cells install the packages and download the data (less than 1 MB). About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • The I-MR chart, its centre line, its sigma and its limits from phase I, from Control charts: is this variation normal?.
  • The planted 0.3-point rise of the refund rate from 2025-04-07, which the I-MR chart found after 98 days, from the same lesson.

What you need on your computer

Nothing. The notebook runs in Colab, Kaggle or VS Code.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1 and NumPy 2.5.4. Colab and Kaggle were not tested.

A smoke alarm must choose. If it is very sensitive, it goes off every time you make toast. After a week, you take out its battery, and then it is silent in a real fire. If it is not sensitive, it stays quiet when you cook, but it warns you late when there is a real fire.

No setting is free. Each extra way to sound the alarm finds real fires sooner, and it also sounds more often for toast. The right setting depends on two costs: the cost of a false alarm, and the cost of a late one.

A control chart has the same choice. The last lesson used one way to alarm: one day outside 3 sigmas. This lesson adds more ways, and it measures their price.

The problem: a shift that stays inside the limits

Section titled “The problem: a shift that stays inside the limits”

The planted rise of Larkfield’s refund rate started on 2025-04-07. Here is the first week after it, as z: how many sigmas each day is from the centre line. A day at the upper limit has z = 3.

2025-04-07 -0.29
2025-04-08 2.04
2025-04-09 1.34
2025-04-10 -0.03
2025-04-11 -0.87
2025-04-12 1.46
2025-04-13 -0.68

No day is near 3, so the one-point rule is silent. The shift adds about half a sigma to every day. That is too small to see in one day, and the daily noise hides it. But over many days, a shift leaves a pattern: more days above the centre than below, and more days a little high. A person who looks at the chart sees this. A rule can see it too.

In 1956, engineers at Western Electric, a telephone factory, wrote rules for these patterns. Each rule describes a pattern that is rare when nothing has changed. These are the Western Electric rules, often called run rules, because many of them look at a run of days in a row:

  1. One point beyond 3 sigma.
  2. Two of three points in a row beyond 2 sigma, on the same side.
  3. Four of five points in a row beyond 1 sigma, on the same side.
  4. Eight points in a row on the same side of the centre line.

In 1984, Lloyd Nelson added more rules of the same kind. The Nelson rules include “six points in a row, all rising or all falling”, which finds a trend. This lesson uses the four rules above. Every new rule works the same way and has the same kind of price.

Here are twelve days, in sigmas from the centre. No day is beyond 3 sigma. Before you read on, predict: which rules fire, and on which day?

Day 1 2 3 4 5 6 7 8 9 10 11 12
z 0.4 −0.6 1.2 0.5 1.4 1.1 0.3 1.6 2.3 0.8 2.6 1.2
Rules that fire 3 4 2, 4 3, 4
  • Rule 3 fires on day 9. Days 5 to 9 have the values 1.4, 1.1, 0.3, 1.6 and 2.3. Four of these five are above 1.
  • Rule 4 fires on day 10. Days 3 to 10 are all above the centre line: eight in a row. It fires again on days 11 and 12, because the run goes on.
  • Rule 2 fires on day 11. Days 9 and 11 are above 2 sigma, so two of the three days 9 to 11 are.
  • Rule 1 never fires.
  • z of the day
  • ±3 sigma: the limits
  • ±1 and ±2 sigma
Twelve days and the zones of the run rules. No point crosses 3 sigma, but rules 2, 3 and 4 fire on days 9 to 12.
Text description of the chart

The chart shows twelve dots for z from day 1 to day 12. Dashed lines are at 1, 2 and 3 sigma above and below a centre line at 0. After day 2, every dot is above the centre line. Most dots are between 1 and 2 sigma, and days 9 and 11 are just above 2 sigma, at 2.3 and 2.6. No dot reaches the 3-sigma line. Labels mark the first alarm of each rule: rule 3 on day 9, rule 4 on day 10 and rule 2 on day 11.

In the notebook, the function fired(z, rule) marks these days for one rule, on both sides of the centre:

def fired(z, rule):
"""True on each day where one Western Electric rule fires, on the high side or the low side."""
days = pd.Series(False, index=z.index)
for s in (z, -z): # the high side, then the low side
if rule == 1:
days |= s > 3
if rule == 2:
days |= (s > 2) & ((s > 2).astype(int).rolling(3, min_periods=1).sum() >= 2)
if rule == 3:
days |= (s > 1) & ((s > 1).astype(int).rolling(5, min_periods=1).sum() >= 4)
if rule == 4:
days |= (s > 0).astype(int).rolling(8).sum() == 8
return days

rolling(3) looks at each day and the two days before it, and .sum() counts how many of them are beyond the line.

The notebook runs four sets of rules on phase II. For each set, it gives the first alarm after the planted shift. It also counts the alarm days per month in the stable months (2025-01-01 to 2025-04-06), when nothing changed:

rules first alarm days to alarm false alarms per month
1 2025-07-14 98 0.00
1 + 4 2025-06-11 65 2.22
1 + 2 + 4 2025-06-11 65 2.22
1 + 2 + 3 + 4 2025-06-11 65 2.85
stable days where rule 4 fires: ['2025-02-09', '2025-02-10', '2025-02-11', '2025-02-12', '2025-02-13', '2025-04-04', '2025-04-05']

With rule 4 added, the chart found the shift 65 days after it started, a month earlier than rule 1 alone. But rule 4 also fired on 7 days of the stable months. They were two runs below the centre: 9 to 13 February and 4 to 5 April. Nothing had changed then, so these are false alarms. The table counts alarm days: a long run fires again on each new day.

One series is one story. A lucky or unlucky week changes it. To know the price of a rule, we need many series, and a measure that we can compare.

The run length is the number of days from a start until the first alarm. Its average over many runs is the average run length (ARL). Two ARLs describe a chart:

  • The in-control ARL: nothing changed, so the first alarm is a false alarm. We want it long.
  • The ARL after a shift: the first alarm finds the change. We want it short.

Here is a tiny worked example. Suppose the noise is normal and the chart uses only rule 1. A day is beyond 3 sigma by chance with a probability of 0.0027. On average, a false alarm comes every 1 / 0.0027 ≈ 370 days. That is about 30.44 / 370 = 0.08 false alarms a month. The textbook value of 370 days assumes perfect, bell-shaped noise.

Real data is not that tidy. Larkfield’s days have different numbers of orders, and some days are worse than others for reasons that nobody planted. So we measure the ARL by simulation. The notebook makes 2,000 runs of 1,500 days with the same process as the data. Each day takes its orders from a day of phase I, gets a day-to-day effect, and draws its refunds order by order. It runs each rule set on 2,000 runs with nothing new, and on 2,000 runs with the 0.3-point shift from day 1:

rules days to a false alarm days to catch the shift false alarms a month
1 209.8 60.6 0.15
1 + 4 116.1 34.1 0.26
1 + 2 + 4 99.6 28.4 0.31
1 + 2 + 3 + 4 80.2 23.2 0.38

With rule 1 alone, Larkfield’s I-MR chart gives a false alarm every 209.8 days, not 370. The real noise has heavier tails than the bell curve. It catches the 0.3-point shift after 60.6 days on average. The real data’s 98 days was an unlucky run.

The price of the run rules on Larkfield's I-MR chart, from 2,000 simulated runs. Each rule moves the chart down and to the left: faster on the shift, and sooner to a false alarm.
Text description of the chart

The chart has four dots joined by a line. The horizontal axis is the days to a false alarm, and the vertical axis is the days to catch the 0.3-point shift. Rule 1 alone is at the top right: 209.8 days to a false alarm and 60.6 days to catch the shift. Rules 1 and 4 are at 116.1 and 34.1 days. Rules 1, 2 and 4 are at 99.6 and 28.4. All four rules are at the bottom left: 80.2 and 23.2 days. No dot is both far right and low: every gain in speed costs false alarms.

This is the price of a rule. All four rules catch the shift in 23.2 days instead of 60.6, almost three times faster. And they give a false alarm every 80.2 days instead of every 209.8, almost three times as often. No rule set is “best”. The right one depends on Grace’s costs: an hour of checking for each false alarm, against a month of extra refunds for each late alarm.

The explorer shows the I-MR chart of 120 days. The first 30 days have no shift. From day 31, a shift of the size that you choose is planted. Every size uses the same random numbers, so only the shift differs. Turn the rules on and off.

Explore

Run rules on the I-MR chart

First high alarm after the shift
day 43 of the shift
False alarms before the shift
0 in the 30 days before the shift
Simulated: false alarms
one every 209.8 days (0.15 a month)
Simulated: a 0.3-point shift caught
after 60.6 days on average

I-MR chart with rule 1, a shift of 0.3 points from day 31. The first high alarm after the shift starts is on day 73 of the series: day 43 of the shift. With no shift, this chart also alarms on that day: that alarm comes from the noise of one day, not from the shift. Before the shift, 0 false alarms in 30 days. In the simulation of 2,000 runs, this chart gives a false alarm every 209.8 days and catches a 0.3-point shift after 60.6 days on average.

  • the chart’s value
  • centre line and limits
  • alarm
0246Refund rate (%)1153145607590105120Day of the series (the shift starts on day 31)shift startsthe chart’s valueDay 73: 4.56%, high alarm (rule 1)Day 79: 4.65%, high alarm (rule 1)
0246Refund rate (%)1316090120Day of the series (the shift starts on day 31)shift startsthe chart’s valueDay 73: 4.56%, high alarm (rule 1)Day 79: 4.65%, high alarm (rule 1)
Alarm days (2)
DayValueSideWhy
734.56%highrule 1
794.65%highrule 1
The I-MR chart of 120 simulated days, with limits from phase I. Turn the run rules on and off, and change the size of the shift from day 31. The readout gives the first high alarm after the shift, the false alarms before it, and the average run lengths of the simulation.
Text description of the explorer

The explorer shows the I-MR chart of 120 days of refund rates. The rate of each day is a line, the centre line is at 2.14%, and dashed limits are at 0.28% and 4.00%. A vertical dashed line marks day 31, where the planted shift starts. Alarm days are circled. Four buttons turn the run rules on and off; six buttons set the shift: none, 0.1, 0.2, 0.3, 0.5 or 1.0 points. The readout gives the day of the first high alarm after the shift and the false alarms in the 30 days before it. It also gives the simulated average run lengths of the chosen rules.

At the start, rule 1 is on and the shift is 0.3 points. The first high alarm is on day 73 of the series, day 43 of the shift. With no shift, the same chart alarms on the same day, so that alarm is noise, not the shift. With all four rules on, the chart finds the shift on day 34, day 4 of the shift. With no shift and all four rules, the chart has 9 alarm days in 120 days, against 2 with rule 1 alone. No set of rules gives a false alarm in the first 30 days of this series. For rule 1 alone, the simulation gives 209.8 days to a false alarm and 60.6 days to catch the shift. For all four rules, it gives 80.2 and 23.2 days. It did not measure other sets, such as rule 4 alone.

Two things to notice. First, in this one series, all four rules find the shift after 4 days, but the simulation says 23.2 days on average. One series can be lucky. Second, with no shift, every alarm is false. Count them with one rule and with four.

Use run rules on a chart that a person reads, where a pattern needs a name: “eight days above the centre”. They cost nothing to compute, and they find small shifts sooner. Turn on only the rules whose false alarms you can afford. Decide this with the ARLs, before you look at the data, and not after you see a pattern that you want to explain.

A run rule is not a model of the shift. It finds that something changed, not what changed. And every rule makes the chart more sensitive to everything, including a small error in the centre line, as the failure case below shows.

Guided practice: the rules in the notebook

Section titled “Guided practice: the rules in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, run the cell. You see a centre of 0.02137 and a sigma of 0.00621.
  4. In section 2, run the cell that defines fired() and alarms(). It prints nothing.
  5. In section 3, write your prediction for the twelve days. Then run the cell. The first True is on day 9.
  6. In section 4, run the cell. You see 2025-07-14 for rule 1 and 2025-06-11 with all four rules.
  7. In section 5, run the simulation cell. You see simulated days: 6000000.
  8. Write your prediction for the rule sets. Then run the next cell. You see 209.8 and 80.2 days to a false alarm.
  9. In section 6, write the three answers and run each check cell. Each prints Check passed.

Change one thing: use rules 1 and 2 only. Rule 2 looks at the outer zone, between 2 and 3 sigma. Before you run it, predict its days to a false alarm. Are they closer to rule 1 alone (209.8) or to rules 1 and 4 (116.1)? In the notebook, section 6, put the result of arl((1, 2))["days to a false alarm"] in arl_rules_1_2.

Show the answer
arl_rules_1_2 = arl((1, 2))["days to a false alarm"]
Check passed: arl_rules_1_2 is 157.3.

Rules 1 and 2 give a false alarm every 157.3 days: cheaper than rules 1 and 4 (116.1 days). Rule 2 needs two points far from the centre, and that is rarer than eight points on one side. Rule 4 is the most sensitive to a small, lasting shift, and so it is also the most expensive in false alarms.

Common mistake

Rule 4 fires in the stable months

Symptom: Grace turns on rule 4. In February 2025, when nothing changed, the chart alarms on five days in a row. She checks the agents, the carriers and the feeds, and finds nothing.

stable days where rule 4 fires: ['2025-02-09', '2025-02-10', '2025-02-11', '2025-02-12', '2025-02-13', '2025-04-04', '2025-04-05']

Cause: rule 4 reacts to any lasting offset from the centre line, even a tiny one. In the stable months, the mean rate was 2.11%, a little below the centre of phase I, 2.14%. The days were below the centre more often than above it, so runs of eight came more often. A long run also fires again on each new day.

Diagnose: look at the z values of the run. Is none of them large, and is the run on the low side? Then the cause is often the centre line or a quiet season, not the process.

Fix: decide before you start which rules you use, from the ARL table. If you use rule 4, count one alarm per run, not one per day. For small, lasting shifts, use a chart made for them. The next lessons build two.

Consequence in AI work: a monitor that is sensitive to everything gets ignored, like the smoke alarm without its battery. The team then misses the alarm that matters.

Check your understanding

Explain the average run length in your own words. Which two ARLs describe a chart, and which way do you want each one to go?

Show the answer

The run length is the number of days until the first alarm, and the ARL is its average over many runs. The in-control ARL is the time to a false alarm when nothing changed: you want it long. The ARL after a shift is the time to catch a real change: you want it short. For Larkfield’s I-MR chart with rule 1, the simulation gives 209.8 days and 60.6 days.

Check your understanding

Predict and explain: a team adds all four Western Electric rules to a chart that had only rule 1. What happens to the time to catch a small shift, and to the false alarms? Use Larkfield's numbers.

Show the answer

Both move. The chart catches the 0.3-point shift faster: 23.2 days on average instead of 60.6. And it gives false alarms more often: one every 80.2 days instead of every 209.8 days, which is 0.38 a month instead of 0.15. Each rule is one more way to alarm, for real shifts and for noise alike.

Check your understanding

Tomás says: the textbook says a 3-sigma chart gives a false alarm every 370 days, so we do not need a simulation. What do you answer?

Show the answer

The 370 days assume perfectly normal, bell-shaped noise. Larkfield’s days have different numbers of orders and a day-to-day effect, so the noise has heavier tails. The simulation of the same process gives a false alarm every 209.8 days with rule 1, almost twice as often as the textbook says. Measure the ARL on a model of your own process before you trust it.

Recap

  • Run rules find patterns that no single point shows. Examples: two of three beyond 2 sigma, four of five beyond 1 sigma, eight in a row on one side. The Nelson rules add more, such as six in a row rising.
  • The average run length gives the price of a chart. The days to a false alarm should be long, and the days to catch a shift short. Larkfield’s I-MR chart: 209.8 and 60.6 days with rule 1; 80.2 and 23.2 days with all four.
  • Every rule is faster on a real shift and sooner to a false alarm. Choose the rules from the costs, before you look at the data.

You can now measure the price of a chart. But the I-MR chart treats every day the same. Larkfield has 109 orders on a quiet day and more than 1,300 on a busy one. Next, you build charts whose limits follow the size of each day.

Next: Charts for rates and counts