Skip to content
Français

Has the distribution changed?

  • Lesson 12
  • 40 min
  • Module 3: Statistical process control and distribution drift

Where we are

Our charts watch one number per day. When a new region goes live, the whole shape of the payment amounts can change, and one number per day does not show it. In this lesson, we compare a reference window with the current one with KS, PSI and Wasserstein, and we see why a big sample always says changed.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 12 of 38 · Anomaly Detection: Find What Is UnusualHas the distribution changed?
Course overview

What you will be able to do

Find out whether the whole shape of a feature changed between two windows, and whether the change is big enough to act on. A test alone says “changed” for almost anything when you have enough data.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It does KS, Wasserstein and PSI by hand, then the new region’s payments, the chi-square test, and the same change at five sample sizes. Its setup cells install the packages and download the payments (about 7 MB). About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

Nothing. The notebook runs in Colab, Kaggle or VS Code.

Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, NumPy 2.5.4 and SciPy 1.18.1. Colab and Kaggle were not tested.

Think of the weather in your city over two weeks. Both weeks have an average temperature of 15 degrees. In the first week, every day is between 13 and 17 degrees. In the second week, the nights are near 0 and the afternoons near 30. The average is the same, but you need a winter coat and a T-shirt in the second week.

One number, the average, hides the difference. To see it, you must compare the whole spread of the values: how many cold days, how many warm days, how many in between. That spread is the distribution.

The problem: a new region changes the shape

Section titled “The problem: a new region changes the shape”

Larkfield’s new region, east, went live on 2025-09-01. This is an anomaly we planted, but a good one: a normal change that stays. Customers in the east region buy smaller baskets. Their median payment is 28.30 dollars, against about 45 dollars in the old regions. In the window that we check below (2025-09-15 to 2025-10-14), about one payment in nine comes from the east.

The control charts of this module watch one number per day, such as a rate or a mean. They cannot tell you whether the shape of the amounts changed, or where. Mei, the data engineer, needs to know: do the detectors that learned “normal” amounts in August still see the same kind of data in September? To answer, we compare a reference window (August 2025) with a current window (2025-09-15 to 2025-10-14).

For each value, take the share of each sample that is at or below it. This is the empirical cumulative distribution: a line that climbs from 0 to 1. The Kolmogorov-Smirnov (KS) test measures the largest vertical gap between the two lines. That gap is the KS statistic. The test also gives a p-value: how likely a gap this large is, if nothing changed.

Here is a tiny example with six amounts in each month. This is a constructed case. Before you read on, predict: which month has the larger amounts, and how large is the biggest gap?

1 2 3 4 5 6
Last month (dollars) 10 20 30 40 50 60
This month (dollars) 15 25 35 45 55 95
  • Last month
  • This month
The empirical cumulative distributions of the six amounts of each month. The two steps never differ by more than one value in six: the KS statistic is 0.167.
Text description of the chart

The chart shows two step lines that climb from 0% to 100%. The solid line, last month, climbs by one sixth at 10, 20, 30, 40, 50 and 60 dollars. The dashed line, this month, climbs by one sixth at 15, 25, 35, 45, 55 and 95 dollars. The two lines take turns: the gap between them is either 0 or one sixth (16.7%). From 60 to 95 dollars, last month is at 100% and this month at 83%: the dashed line waits for the last value, 95.

The notebook prints the two lines and the gap at each amount, then the test:

KS statistic 0.167 (the largest gap), p-value 1.000
Wasserstein distance: 10.0 = the mean of [5, 5, 5, 5, 5, 35]

The largest gap is one value in six, 0.167. With six values per month, a gap this small is common by chance, so the p-value is 1.000: no evidence of a change. KS looks only at the order of the values. It does not see how far the 95 is from the 60.

The Wasserstein distance answers a different question: on average, how far must each value move to turn one sample into the other? It is in the units of the data, here dollars. Sort both samples and pair them: 10 with 15, 20 with 25, and so on, and 60 with 95. Five values move 5 dollars and one moves 35. The mean is 60 / 6 = 10 dollars. The Wasserstein distance sees the large payment that KS ignored. Its other name, the “earth mover’s distance”, describes the idea: the work to move a pile of earth into the shape of another pile.

The Population Stability Index (PSI) comes from credit risk. Banks use it every month to check that new applicants still look like the applicants that their score learned from. Cut the reference into bins. Compare the share of each bin in the two windows, and add up (current − reference) × ln(current / reference). Each term is positive or zero, so changes in different bins never cancel.

A tiny example with four bins (constructed):

Bin Reference share Current share (cur − ref) × ln(cur / ref)
under 20 dollars 25% 20% 0.0112
20 to 45 dollars 25% 25% 0
45 to 100 dollars 25% 25% 0
100 dollars and more 25% 30% 0.0091
PSI 0.0203

A common rule of thumb says that a PSI below 0.1 means no real change. From 0.1 to 0.25 is a moderate change, and above 0.25 is a large change. These numbers are conventions from credit-scoring practice. No theory says that 0.1 is the right line for your data. Choose your own limits from your decision, and write down why.

The Jensen-Shannon (JS) distance is another measure on the same bins. It goes from 0 (the same shares) to 1 (nothing in common). PSI is more common in business; the JS distance has a fixed range, which makes it easy to compare.

Larkfield’s amounts after the new region

Section titled “Larkfield’s amounts after the new region”

The notebook compares August (56,553 payments) with 2025-09-15 to 2025-10-14 (53,276 payments). Before you read the output, predict: will KS say “changed”? Will PSI be above 0.1?

n ref: 56553
n cur: 53276
KS statistic: 0.0345
KS p: 7.24e-29
Wasserstein: 10.25
PSI: 0.0068
JS distance: 0.0349
median ref: 45.12
median cur: 41.9

KS says “changed”, with a p-value of 7.24e-29: 28 zeros after the decimal point. PSI is 0.0068, far below 0.1. The median fell from 45.12 to 41.90 dollars. The Wasserstein distance is 10.25 dollars. Large payments are far apart in dollars, so a small move among them adds many dollars to this distance. Which one is right? Both. The change is real, and it is small.

The PSI uses ten bins, cut at the deciles of August. Each bin has 10% of August’s payments. Here is the share of each bin in the current window:

Each column is one tenth of August's payments, from the smallest (1, under 15.95 dollars) to the largest (10, above 309.85 dollars). In the current window, the small bins have a little more than 10% and the large bins a little less.
Text description of the chart

A column chart with ten columns, one for each tenth of August’s payment amounts, ordered from the smallest amounts (1) to the largest (10). In August, every column would be exactly 10%. In the current window, the columns go down slowly from left to right. They are 11.8% in the first bin, then 10.7%, 10.6%, 10.1%, 9.9%, 9.9%, 9.5%, 9.5%, 9.0% and 9.0% in the last. The first four columns are a little above 10% and the last six a little below. No column moves by more than 1.8 points.

No bin moves by more than 1.8 points. The new region added small payments, but not many. Is the change in the old regions too? The notebook runs the same comparison without the east region:

without the east region: {'KS p': 0.0112, 'PSI': 0.0008, 'median cur': 44.36}
east region median amount: 28.295

Without the east region, PSI is 0.0008: the old customers did not change. The change is the new region, which has smaller baskets.

For a category, such as the region or the device type, the chi-square test compares the counts of each category in the two windows. PSI works on the shares of the categories, as on bins.

region: chi-square 6632.6, p 0, PSI 1.3018
coast east north south
August 0.228 0.000 0.445 0.326
after 0.202 0.111 0.400 0.287
device_type: chi-square 149.6, p 3.34e-33, PSI 0.0055
desktop mobile tablet
August 0.401 0.507 0.092
after 0.378 0.543 0.080

The region mix changed a lot: a new category appeared with 11.1% of the payments, and PSI is 1.30. That change is large, and it is expected. The device mix also gives a tiny p-value (3.34e-33), but its PSI is 0.0055: mobile went from 50.7% to 54.3%.

Why a big sample always says “changed”

Section titled “Why a big sample always says “changed””

Here is the heart of this lesson. The notebook takes the same two windows, but only n payments from each, and repeats it 20 times for each n. The change is exactly the same in every row. Only the size of the sample grows. Before you read the table, predict: what happens to the p-value, and what happens to the PSI?

Payments in each window KS p-value (median of 20 draws) KS statistic PSI
200 0.713 0.070 0.0732
1,000 0.315 0.043 0.0195
5,000 0.000149 0.044 0.0118
20,000 2.66e-12 0.037 0.0077
50,000 4.29e-26 0.034 0.0068

With 200 payments, the test sees nothing (p = 0.713). With 50,000, the same change gives p = 4.29e-26. A p-value answers one question: is there any change at all? With enough data, the answer is almost always yes, because real data almost never stays exactly the same.

The KS statistic and the PSI do something else. They settle near the true size of the change: about 0.034 and 0.007. With small samples, they are noisy and too high: 0.0732 with 200 payments is only noise. So a measure of size also needs enough data, and a few thousand values per window is a good start.

Compare a month with no planted change. July against August, at four sample sizes:

Payments in each window 200 1,000 5,000 full months (59,144 and 56,553)
KS p-value 0.394 0.723 0.851 0.654

When nothing changed, the p-value stays large at every size. So a tiny p-value with a big sample means: something changed, maybe very little. It does not mean: something important changed.

One more example. The hour of the payments, in the same two windows:

{'KS p': 6.97e-09, 'Wasserstein': 0.197, 'PSI': 0.0027}
payments came 12 minutes later on average, by the Wasserstein distance

The p-value is 6.97e-09. By the Wasserstein distance, the payments come about 12 minutes later in the day. No detector and no person needs to act on 12 minutes.

So which number answers the business question? The question is never “is there a change?” It is “is the change big enough to act on?” Mei asks two questions:

  • Do the detectors need new reference data for the amounts? PSI is 0.0068, far below any common limit, and the Wasserstein distance is 10 dollars on a median of 45. No.
  • Does the team need to know about the new region? Region PSI is 1.30. Yes: write it down as a planned change, so that nobody spends a day looking for its cause.

A measure of size, such as PSI, the KS statistic or the Wasserstein distance, answers these questions. A p-value does not. Use the p-value only to check that a size is not just noise, with small samples.

Data Test (is there a change?) Size (how big?)
A number, such as an amount KS Wasserstein (in the data’s units), PSI, KS statistic
A category, such as a region chi-square PSI, JS distance

These comparisons cost almost nothing: one line of SciPy each. Their weakness is that they look at one feature at a time. A change in how two features go together, such as large amounts at night, stays hidden. The next modules’ detectors look at many features together.

Guided practice: the tests in the notebook

Section titled “Guided practice: the tests in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with the Ready: line.
  3. In section 1, write your prediction for the six amounts. Then run the cell. You see a KS statistic of 0.167 and a Wasserstein distance of 10.0.
  4. In section 2, run the PSI cell. You see PSI: 0.0203. Then run the helpers cell. It prints nothing.
  5. In section 3, run the data cell and the amount cell. You see KS p: 7.24e-29 and PSI: 0.0068.
  6. Run the plot cell and the cell without the east region. You see a PSI of 0.0008.
  7. In section 4, run the categories cell. You see a region PSI of 1.3018.
  8. In section 5, write your prediction, then run the sample-size cell. The p-value falls from 0.713 to 4.29e-26.
  9. In section 6, run the hour cell. You see 12 minutes.
  10. In section 7, write the three answers and run each check cell. Each prints Check passed.

Change one thing: the current shares of the four-bin example. They are now 10%, 20%, 30% and 40%, against 25% each. Before you compute it, predict: is this PSI above 0.1? Above 0.25? In the notebook, compute it and put it in psi_tiny_2, rounded to 4 decimals.

Show the answer
bins2 = pd.DataFrame({"ref": [0.25, 0.25, 0.25, 0.25], "cur": [0.10, 0.20, 0.30, 0.40]})
psi_tiny_2 = round(float(((bins2.cur - bins2.ref) * np.log(bins2.cur / bins2.ref)).sum()), 4)
Check passed: psi_tiny_2 is 0.2282.

The PSI is 0.2282: a moderate change by the rule of thumb, close to the 0.25 line. The first bin alone gives (0.10 − 0.25) × ln(0.10 / 0.25) = 0.137. A bin that loses most of its share counts a lot, because of the logarithm.

Common mistake

A drift alarm on the p-value

Symptom: Tomás sets up a daily drift check: “alarm when the KS p-value is below 0.05”. After the new region, it alarms every day on the amounts, on the hour and on the device type. The team looks for causes for a week and finds nothing they can fix.

{'KS p': 6.97e-09, 'Wasserstein': 0.197, 'PSI': 0.0027}

Cause: with tens of thousands of payments, a p-value is tiny for any change, even 12 minutes in the hour of payment. The p-value measures the evidence that something changed, not the size of the change.

Fix: alarm on a measure of size, such as PSI or the Wasserstein distance, with a limit that you chose for your decision. Use the p-value only to check that a size is not just noise when samples are small.

Consequence in AI work: a drift monitor that alarms on p-values with big samples alarms all the time. Teams learn to ignore it, and then they miss the large change, such as a new category, that needs action.

Check your understanding

Explain in your own words what the KS statistic, the Wasserstein distance and PSI each measure. Which one is in the units of the data?

Show the answer

The KS statistic is the largest gap between the two cumulative distributions: the largest difference in the share of values at or below some amount. The Wasserstein distance is how far, on average, each value must move to turn one sample into the other. It is in the units of the data: dollars for Larkfield’s amounts. PSI compares the shares of bins and adds (current − reference) × ln(current / reference) over the bins, so changes never cancel. Only the Wasserstein distance is in the units of the data.

Check your understanding

Predict and explain: the same small change in the payment amounts is tested with 200 payments in each window, then with 50,000. What happens to the KS p-value and to the PSI?

Show the answer

The p-value falls toward 0 as the sample grows: from 0.713 with 200 payments to 4.29e-26 with 50,000, for the same change. A bigger sample can see a smaller change, so with enough data almost every change becomes “significant”. The PSI settles near the true size of the change, 0.0068. With small samples it is noisy and too high (0.0732 with 200 payments). So the p-value answers “is there any change?”, and the size answers “is it big enough to act on?”.

Check your understanding

After the new region went live, KS gives p = 7.24e-29 for the amounts, and the region's PSI is 1.30. Mei asks what to do about each. What do you advise?

Show the answer

The amounts changed, but very little: PSI 0.0068 and a Wasserstein distance of 10 dollars, with the median from 45.12 to 41.90 dollars. Without the east region, PSI is 0.0008, so the old customers did not change. No action is needed for the amount detectors. The region mix changed a lot (PSI 1.30), because a new category appeared. That change is planned: write it down so that nobody looks for its cause, and include the new region in the next reference window.

Recap

  • KS and chi-square test whether two windows differ; Wasserstein, PSI, the KS statistic and the Jensen-Shannon distance measure how much.
  • The new region moved the amounts: KS p = 7.24e-29, but PSI 0.0068 and a median from 45.12 to 41.90 dollars. The region mix moved a lot: PSI 1.30.
  • With more data, the p-value of the same change falls toward 0 (0.713 with 200 payments, 4.29e-26 with 50,000). Decide with the size of the change, and with a limit that you chose for your decision. The 0.1 and 0.25 limits of PSI are conventions.

You can now test one feature for a change. But Larkfield has eight payment features, checked every day, and every test has a chance of a false alarm. Next, you check many features at once, control the false alarms, and decide what to do about real drift.

Next: Drift in many features, and what to do about it