Far from the neighbours: kNN distance
- Lesson 15
- 35 min
- Module 4: Distance, density and isolation
Where we are
Our baselines and charts look at one number at a time. A fraud can look normal in each column and strange only in the combination. In this lesson, we score a payment by its distance to its nearest neighbours, and we see how scaling and the number of neighbours change the result.
Lesson 15 of 38 · Anomaly Detection: Find What Is UnusualFar from the neighbours: kNN distance
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Score a payment by how far it is from its nearest normal neighbours. Then you can find a fraud that looks normal in every single column. Also learn the two settings that decide whether the score means anything: the scaling of the features and the number of neighbours.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). The whole notebook runs in about a minute.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The comparison table, the reference months (June to August 2025) and the evaluation months (September to November 2025), from Meet the Larkfield data.
- Precision at the budget: the share of the 50 daily alerts that are planted frauds. It is explained in The payment that did not fit.
- The robust z-score per customer, from Median, MAD and IQR. This lesson explains scaling again where it needs it. Machine Learning: From Problem to Reliable Model has more.
What you need on your computer
A browser and a free Google account (Colab) or Kaggle account. Nothing to install. To work locally instead, use Python 3.12 and the setup of Meet the Larkfield data.
Tested with Python 3.12.13 in Jupyter on macOS, with scikit-learn 1.9.1 and pandas 3.1.1. Colab, Kaggle and Windows were not tested.
The guest alone in a corner
Section titled “The guest alone in a corner”Think of a party. Most guests stand in small groups and talk. One guest stands alone in a corner. You know nothing about the guests: not their names, not their jobs. Still, you notice the one in the corner. That guest is far from everybody else.
You did not compare each guest with a rule such as “taller than 2 metres”. You compared each guest with the nearest other guests. A guest in a group has neighbours close by. The guest in the corner does not.
This lesson turns that idea into a detector for payments.
The problem: normal in each column, strange together
Section titled “The problem: normal in each column, strange together”In Module 2, every detector looked at one number at a time. MAD per customer looked at the amount. The velocity rule counted payments in 10 minutes. Each one checked one column against its usual values.
Now look again at the payment that did not fit, from the first lesson of the course. Customer C-01234 makes a coffee-sized payment every Saturday morning in Canada. On 3 October 2025, at 02:14, the card pays 24.27 dollars in Portugal, on a new device. Take each column alone:
- The amount: 24.27 dollars. The usual basket is 11.15 to 17.98 dollars. A bit more than usual, but not strange.
- The hour: 2 a.m. About 3% of all payments in the reference months happen at night. Not unusual by itself.
- The country: Portugal. Customers travel: the data has 703 trip payments that are honest.
- The device: new. People buy new phones.
Each column alone has an innocent story. But all four together, for this customer, have none. MAD per customer found only 3.3% of the stolen-card payments (measured, Module 2), because it looks only at the amount.
We need a score that looks at all the columns of a payment at the same time.
The idea: distance to the nearest neighbours
Section titled “The idea: distance to the nearest neighbours”Put each payment as a point in a space. Each column is one direction of the space: amount, hour, new country, and so on. Payments that look alike are close together. A payment that looks like nothing else is far from all the others.
For each new payment, find the k nearest payments among the normal ones. Then measure the distance to the k-th of them. That distance is the score:
- A payment in a crowd: its k nearest neighbours are close. Small score.
- A payment alone: even its nearest neighbours are far. Large score.
This is the k-nearest-neighbours distance, or kNN distance for short. “k” is just the number of neighbours that you choose. The “distance” is the straight-line distance that you know from school: across a plane, it is Pythagoras.
A tiny example: seven points
Section titled “A tiny example: seven points”Take seven points on a plane. Six of them, A to F, stand close together. G stands far away. We use k = 2: the score of each point is the distance to its 2nd nearest neighbour.
Text description of the diagram
The picture shows a plane with seven points. Six grey points, A to F, are close together near the point x = 1, y = 1. One blue point, G, is at x = 3, y = 3. Two dashed lines go from G to its two nearest neighbours, D and F. The text next to the plane says that G’s 2nd nearest neighbour is 2.62 away. For A to F, the 2nd nearest neighbour is 0.22 to 0.32 away.
Here is every value. D is at (1.1, 1.2) and G is at (3.0, 3.0). The distance from G to D is:
√((3.0 − 1.1)² + (3.0 − 1.2)²) = √(3.61 + 3.24) = √6.85 = 2.62
Then F is at (1.0, 1.3), also 2.62 from G. So G’s 2nd nearest neighbour is 2.62 away. Now A, at (1.0, 1.0): its nearest neighbour is C, 0.14 away, and its 2nd nearest is B, 0.22 away.
| Point | 2nd nearest neighbour | Score (k = 2) |
|---|---|---|
| A | B | 0.22 |
| B | D | 0.32 |
| C | F | 0.22 |
| D | A | 0.22 |
| E | C | 0.32 |
| F | C | 0.22 |
| G | F | 2.62 |
G’s score is more than eight times the largest score of the group. The detector needs no labels at all. It needs only a set of points that we trust to be mostly normal.
Larkfield: the neighbours of two payments
Section titled “Larkfield: the neighbours of two payments”Priya, Larkfield’s data scientist, tries the idea on Larkfield’s payments. The detector learns “normal” from a random sample of 20,000 payments of the reference months, June to August. It stores them. Then it scores every payment of the evaluation months, September to November.
Each payment becomes a point with eight behaviour features. These features compare the payment with the customer’s own past. For example: the amount against the usual amount, a new country, a new device, and the payments in the last hour. Features decide more than the algorithm builds them one by one. For now, take them as given.
Priya compares two payments of the same customer, C-01234. The first is an ordinary Saturday basket on 27 September 2025. The second is the first payment in Portugal, on 3 October at 02:14. It is a payment that we planted: a stolen card.
Text description of the diagram
The picture has two round panels, drawn on the same scale, with dashed rings at distance 1, 2, 3, 4 and 5. On the left, the normal payment: C-01234’s Saturday basket of 27 September. Its 10 nearest neighbours are 0.005 to 0.025 away, so close that they all sit inside the centre dot. Its score ranks 1,835 of the 2,002 payments that day. On the right, the planted payment: C-01234’s first payment in Portugal, on 3 October at 02:14. Its 10 nearest neighbours are 2.02 to 4.86 away, between the 2nd and the 5th ring. Its score ranks 2 of the 1,780 payments that day. The directions of the dots are only for drawing; the distances are real.
The chart shows the same distances, neighbour by neighbour:
Text description of the chart
A line chart. The horizontal axis is the neighbour, from 1 (the nearest) to 10. The vertical axis is the distance, from 0 to 5. The normal payment’s line stays flat near 0: from 0.005 to 0.025. The planted payment’s line starts at 2.02 for the nearest neighbour and rises to 4.86 for the 10th.
The normal basket has thousands of look-alikes: a small payment, in Canada, on a known device, in the morning. The planted payment does not. Among 20,000 reference payments, not one is close to it. Its score is the 2nd highest of the 1,780 payments that day, so it goes to Sofia’s team.
One more detail is worth a look. Of the planted payment’s 10 nearest neighbours, 9 are planted payments of the reference months. The reference months are not clean: fraud happens every month. The detector still works, because the planted share is small. But it explains why the choice of k matters, as you will see below.
Run it in the notebook
Section titled “Run it in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the three setup cells.
- Run section 1, the tiny example. Check that G is 2.62 and that the others are 0.22 to 0.32.
- Run the cell that loads the payments. It prints the first three raw rows: amount (as a logarithm), hour and account type.
- Run the cell that defines
knn_scores(). It scales the features, stores the 20,000 sample and scores the evaluation months. - Check the table. The precision at the budget is 0.219 with the raw features.
The function in the notebook is short:
def knn_scores(X, k=10, sample=20_000, seed=0): """Distance of each evaluation payment to its k-th nearest neighbour among reference payments.""" scaler = StandardScaler().fit(X.loc[reference.index]) ref = scaler.transform(X.loc[reference.index]) ref = ref[np.random.default_rng(seed).choice(len(ref), sample, replace=False)] # a sample: kNN is slow nn = NearestNeighbors(n_neighbors=k).fit(ref) dist, _ = nn.kneighbors(scaler.transform(X.loc[evaluation.index])) return dist[:, -1]NearestNeighbors comes from scikit-learn. kneighbors returns the distances to the k nearest stored points, nearest first. The last column, dist[:, -1], is the distance to the k-th one: the score.
With the raw features, kNN distance reaches a precision at the budget of 0.219. With the behaviour features, it reaches 0.505. Both are measured, and both are in the comparison table. Keep the second number in mind. The last lesson of this module explains it.
Scaling decides what “far” means
Section titled “Scaling decides what “far” means”A distance adds up the differences in every column. So a column with big numbers can drown all the others.
Here is a small example. Two baskets cost 40 and 52 dollars: a difference of 12. Two payments happen at 3 a.m. and at 3 p.m.: also a difference of 12. For the distance, the two differences are equal. But a 12-dollar difference is nothing, and a 12-hour difference is large. Worse, amounts go up to several thousand dollars. So a business order of 1,500 dollars is “far” from almost everything, only because of the dollars.
Scaling puts every column on the same footing. The usual method, standard scaling, takes each value, subtracts the column’s mean and divides by its standard deviation (its typical spread). After scaling, every column has a mean of 0 and a spread of 1.
Take the column new_country in the reference months. It is 1 for a payment from a country that the customer never used before, else 0. Its mean is 0.011 (about 1 payment in 90) and its standard deviation is 0.106. So a 1 becomes:
(1 − 0.011) ÷ 0.106 ≈ 9.3
A rare “yes” becomes a big number, and a common “no” stays near 0 (−0.11). This is why the planted payment was so far away. Its new country (9.3), its night hour (5.4) and its new device (4.7) are all rare values.
The notebook removes the scaling to see what happens. The amount stays in dollars, the hour stays 0 to 23:
Text description of the chart
A column chart with three columns. Not scaled: 0.013. Scaled raw features: 0.219. Scaled behaviour features: 0.505. Without scaling, almost none of the 50 daily alerts is a planted fraud.
Without scaling, the precision falls to 0.013: less than 1 planted fraud in 50 alerts. The distance is then almost only the amount in dollars, so the alerts are the biggest business orders.
Mixed feature types need care
Section titled “Mixed feature types need care”Payments mix three kinds of column. A distance treats them all the same way:
- Amounts are skewed: most are small, a few are huge. The notebook takes the logarithm first, so 10 dollars becomes 2.3 and 1,000 dollars becomes 6.9.
- Hours go round in a circle: 23:00 and 00:00 are one hour apart, but the distance sees 23. The behaviour features use a yes/no
nightcolumn instead of the hour. - Yes/no columns (new country, business account) are 0 or 1. Scaling turns a rare “yes” into a large value, as you saw. That is often what you want, but check it.
There is no setting that fixes this for you. You choose the columns and their form with care, then you measure.
The choice of k
Section titled “The choice of k”The number of neighbours, k, is the other setting. Change it and measure again, with everything else equal:
Text description of the chart
A column chart with five columns, on an axis that starts at 0.40. k = 1: 0.450. k = 5: 0.498. k = 10: 0.505. k = 20: 0.509. k = 50: 0.515.
k = 1 is clearly worse: 0.450. Remember the 9 planted neighbours of the planted payment. A new stolen-card payment often has one old fraud close to it in the reference sample. With k = 1, that single old fraud makes the new one look normal. With k = 10 or more, the score looks past a small group of old frauds to the normal crowd behind them.
Above k = 10, the gain is small: 0.505 to 0.515. A larger k also costs more time. And it smooths the score: with a very large k, every point looks a little far. A k between 10 and 50 is a sensible start here. Measure on your own data before you choose.
When to use kNN distance, and what it costs
Section titled “When to use kNN distance, and what it costs”Use it when anomalies are points that look like nothing else. You also need a set of mostly normal data. It needs no labels. It is easy to explain to a reviewer: “this payment has no close look-alike; here are its nearest neighbours”.
Be careful in three cases:
- Speed and memory. kNN stores the normal points and compares every new point with them. Here, it stores a sample of 20,000, not all 173,271 reference payments. Scoring 169,803 evaluation payments with the behaviour features took 4.1 seconds (measured). With all reference payments, it would take far longer.
- Different densities. A business account spreads its payments more widely than a personal one. One distance cannot fit both. The next lesson deals with this.
- Bursts. Many frauds of one kind, close together, can become each other’s neighbours. A larger k helps.
Common mistake
The detector that only finds big business orders
Symptom: Tomás runs kNN on the raw payment columns. The precision at the budget is 0.013. Sofia opens the alerts: almost every one is a large business order.
Cause: he did not scale the columns. The amount goes up to thousands of dollars; the hour goes from 0 to 23; the account type is 0 or 1. The distance is almost only the amount, so “far” means “expensive”.
Fix: take the logarithm of the amount, then scale every column with StandardScaler, fitted on the reference months only. The same raw columns then give 0.219.
Consequence in AI work: any method that uses a distance (kNN, LOF, k-means, a One-Class SVM) silently follows the column with the biggest numbers. The code runs without an error, and the result is wrong. Always check the scale of each column first.
Your turn
Section titled “Your turn”In the notebook, section 6 has three tasks:
- In the tiny example, write G’s distance to its 2nd nearest neighbour as
g_distance. The check tells you if it is right. - Score the behaviour features with k = 1, 10 and 50. Put each precision at the budget in a dictionary
by_k, and the best k inbest_k. Before you run it, predict the order of the three results. This cell takes about 20 seconds. - Add your best k as a new row of the comparison table, and compare it with the rows above.
Show the expected results
g_distance = 2.62.{1: 0.45, 10: 0.505, 50: 0.515}, sobest_k = 50. k = 1 is the worst, because a single old fraud close by makes a new fraud look normal.- The new row has a precision at the budget of 0.515, a little above the k = 10 row (0.505).
Knowledge check
Section titled “Knowledge check”Check your understanding
In your own words: what is the kNN distance score of a payment, and why does a stolen card used abroad at night on a new device get a high score, when each of its columns alone looks normal?
Show the answer
The score is the distance from the payment to its k-th nearest neighbour among stored normal payments. A payment in a crowd of look-alikes has close neighbours and a small score. The stolen-card payment combines several rare values at once (a new country, a new device, the night, for this customer). Each value alone has honest look-alikes. The combination has none, so even its nearest neighbours are far, and the score is high.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you run kNN on the raw columns (amount in dollars, hour 0 to 23, account type 0 or 1) without scaling. Which payments fill the alerts, and why?
Show the answer
The largest amounts: big business orders. The amount has by far the biggest numbers, so it decides almost the whole distance. The hour and the account type barely count. Larkfield measured a precision at the budget of 0.013 without scaling, against 0.219 for the same columns with the logarithm and standard scaling.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With k = 1, kNN finds fewer planted frauds (0.450) than with k = 10 (0.505). Explain why, using what the reference months contain.
Show the answer
The reference months also contain planted frauds, so a new fraud often has one or a few old frauds close to it. With k = 1, the score is the distance to that one old fraud, which is small, so the new fraud looks normal. With k = 10, the score is the distance to the 10th neighbour, which is usually in the normal crowd and far away. The planted Portugal payment had 9 planted payments among its 10 nearest neighbours.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- kNN distance scores a point by the distance to its k-th nearest neighbour among normal points. Far from everybody means unusual. It needs no labels.
- It looks at all the columns at once, so it finds a payment that is normal in each column but strange in the combination.
- Scaling decides what “far” means. Without it, the dollars decide everything: 0.013 instead of 0.219.
- k matters. k = 1 is fooled by a single old fraud close by (0.450). k = 10 gives 0.505, and k = 50 gives 0.515.
- It is slow on many rows: it compares every new payment with the stored ones, so the course stores a sample of 20,000.
You can now score a payment by its distance to its neighbours. But one distance does not fit every area. A business account’s payments are naturally spread out. A personal account’s payments stand close together. Next, you score a payment against the density of its own area, with the Local Outlier Factor.
Next: Local density: LOF