Skip to content
Français

Isolation Forest

  • Lesson 17
  • 35 min
  • Module 4: Distance, density and isolation

Where we are

Our distance and density methods compare every payment with its neighbours, which is slow on many rows. In this lesson, we turn the question around: a strange point is easy to separate with a few random cuts. We draw one tree by hand, then run Isolation Forest on all the payments.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 17 of 38 · Anomaly Detection: Find What Is UnusualIsolation Forest
Course overview

What you will be able to do

Find unusual payments by cutting the data at random and counting how fast each payment is alone. This method is fast enough to learn from every payment. You will also see why its built-in threshold must not decide how many alerts Sofia’s team gets.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas, NumPy, SciPy, scikit-learn and matplotlib, and download the payments (about 7 MB). The whole notebook runs in about a minute.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

A browser and a free Google account (Colab) or Kaggle account. Nothing to install.

Tested with Python 3.12.13 in Jupyter on macOS, with scikit-learn 1.9.1 and pandas 3.1.1. Colab, Kaggle and Windows were not tested.

You may know the game “Guess who?”. Each player has a card with a face. You ask yes/no questions to find the other player’s face: “Does your person wear glasses?” “Is the hair red?”

Some faces are found in one question. If only one face has a red hat, “Is the hat red?” finds it at once. Other faces look like many others: brown hair, no glasses, no hat. To find one of those, you need many questions.

So the number of questions tells you how unusual a face is. Few questions: it is different from the others. Many questions: it hides in a crowd.

The problem: comparing with every neighbour is slow

Section titled “The problem: comparing with every neighbour is slow”

kNN distance and LOF compare each new payment with stored payments. To find the nearest neighbours, they search through all the stored points. That is why the course stored only 20,000 of the 173,271 reference payments. With all of them, or with millions of payments a day, the search would be too slow and use too much memory.

We want a detector that learns from every payment and still scores a new one quickly. And it should not need the distance between points at all, so that the scale of each feature matters less.

Isolation Forest plays “Guess who?” with random questions. One question is a cut:

  1. Pick one column at random, for example the amount.
  2. Pick a random value between the smallest and the largest amount of the points that are left.
  3. Split the points in two: below the value, and at or above it.
  4. Keep the side with the point that you want to isolate, and repeat until that point is alone.

The number of cuts is the score: few cuts means unusual. One sequence of cuts is called an isolation tree, because the cuts form a tree of yes/no questions. Random cuts are luck, so the method grows many trees and averages the number of cuts: a forest. The name says it all: a forest of trees that isolate points.

Take the seven points of the kNN lesson again: A to F close together, and G far away. Here is one random tree:

Three panels. Panel 1, cut 1: G is alone. The seven points with one horizontal blue line at y = 1.39. G is above the line, alone; A to F are below. Panel 2, cuts 2 to 6: A is alone. A zoom on A to F with five more blue lines, numbered: y = 0.82 cuts off E, x = 1.14 cuts off B, x = 1.08 cuts off D, y = 1.22 cuts off F, y = 1.05 cuts off C. A is left alone in a small box. Panel 3, the same tree as questions: all 7 points; y ≥ 1.39: G alone; y < 1.39: 6 left; y < 0.82: E; y ≥ 0.82: 5 left; and so on until y < 1.05: A. The text says: average over 1,000 random trees: G is alone after 1.21 cuts, A after 4.56 cuts.
One random isolation tree on the seven points (constructed). The first cut already leaves G alone. A needs six cuts. Over 1,000 random trees, G needs 1.21 cuts on average and A needs 4.56.
Text description of the diagram

The picture has three panels. The first panel, titled “Cut 1: G is alone”, shows the seven points and one horizontal blue line at y = 1.39. G is above the line and alone. A to F are below it. The second panel, titled “Cuts 2 to 6: A is alone”, zooms in on A to F. Five more blue lines are numbered 2 to 6. y = 0.82 cuts off E. x = 1.14 cuts off B. x = 1.08 cuts off D. y = 1.22 cuts off F. y = 1.05 cuts off C. A is left alone in a small box. The third panel shows the same tree as a column of questions. It starts with all 7 points. y at least 1.39 gives G alone, and y below 1.39 leaves 6 points. Then each cut sends one point to the side, until y below 1.05 leaves only A. Below, the text says that over 1,000 random trees, G is alone after 1.21 cuts on average, and A after 4.56 cuts.

Follow the cuts with the numbers:

  1. Cut 1: the column y, the value 1.39 (random, between the smallest y, 0.8, and the largest, 3.0). G (y = 3.0) is above; the six others are below. G is alone after 1 cut.
  2. To isolate A, keep cutting the side with A. Cut 2: y at 0.82 removes E (y = 0.8). Cut 3: x at 1.14 removes B (x = 1.2). Cut 4: x at 1.08 removes D (x = 1.1). Cut 5: y at 1.22 removes F (y = 1.3). Cut 6: y at 1.05 removes C (y = 1.1). A is alone after 6 cuts.

Why is G so easy? A random cut falls anywhere between the smallest and the largest value. G sits alone in a big empty range, from 1.3 to 3.0 in y. Most random values land in that range, and any of them separates G. In 81.3% of the 1,000 trees, G was alone after the very first cut.

Over 1,000 trees, the averages are: G 1.21 cuts, and 3.02 to 4.56 cuts for A to F. The forest turns this into a score between 0 and 1. About 0.5 is an average point. Close to 1 means “isolated after very few cuts”.

Priya runs scikit-learn’s IsolationForest on the behaviour features. Because it does not search for neighbours, it learns from all 173,271 reference payments. Each of its 200 trees uses a random sample of 256 payments. A tree only needs to find the few points that are easy to cut off.

  1. Open the notebook from the box at the top of this page. Run the three setup cells.
  2. Run section 1, the tiny example. Check that seed 0 gives G = 1 and A = 6, and that the average of G is 1.21.
  3. Run the cell that loads the payments. It prints (173271, 8) (169803, 8): the reference and the evaluation rows, with 8 behaviour features.
  4. Run the cell that fits the forest. Check that the precision at the budget is 0.500.

The code is three lines:

forest = IsolationForest(n_estimators=200, random_state=0, n_jobs=-1).fit(X_ref) # all 173,271 reference payments
scores = -forest.score_samples(X_ev) # about 0.5 = average; close to 1 = isolated in very few cuts
table.add(compare.score_payments(evaluation, scores, "Isolation Forest", "behaviour features, 200 trees",
explain_minutes=3))

n_estimators is the number of trees. random_state=0 fixes the random cuts, so you get the same result every time. n_jobs=-1 uses all the processor cores. score_samples returns minus the score, so the code turns it around.

The output:

detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min)
Isolation Forest behaviour features, 200 trees 50.0 0.5 0.808 3218 3
median score, honest: 0.378 | planted: 0.742

A precision at the budget of 0.500: half of the 50 daily alerts are planted frauds. That is almost the same as kNN distance (0.505), and close to the ceiling of 0.542 that even a perfect detector reaches. C-01234’s first payment in Portugal scores 0.724. The ordinary Saturday basket of the same customer scores 0.343.

The chart shows how the scores of the two groups differ:

  • Honest
  • Planted
Isolation Forest scores of the evaluation payments, in bands of 0.05, as a share of each group (measured, behaviour features, 200 trees). Most honest payments score 0.30 to 0.45; most planted payments score 0.65 to 0.80.
Text description of the chart

A line chart with two lines. The horizontal axis is the score, from 0.30 to 0.85. The vertical axis is the share of each group’s payments in each band of 0.05. The honest line peaks at about 38% in the band 0.35 to 0.40 and falls to almost zero above 0.70. The planted line is near zero below 0.55 and peaks at about 40% in the bands 0.70 to 0.75 and 0.75 to 0.80. The two humps are far apart. A thin tail of honest payments reaches above 0.65.

The median honest payment scores 0.378 and the median planted payment 0.742. The two humps are far apart. But honest payments are many, so even their thin tail is large. Over the 91 evaluation days, 2,660 honest payments score above 0.65, almost as many as the 2,624 planted ones. They are honest payments that look unusual for their customer. They compete with the frauds for the 50 daily slots, and that is why the precision stays near 0.5.

With the raw features, the same forest reaches only 0.208. Keep that gap in mind: it is the subject of the last lesson of this module.

The contamination setting: a guess, not a fact

Section titled “The contamination setting: a guess, not a fact”

The comparison table ranks the payments by score and sends the top 50 of each day. But IsolationForest can also answer “anomaly” or “normal” by itself, with predict. For that, it needs a threshold. It takes it from a setting called contamination: the share of anomalies that you expect in the data.

The default is contamination="auto". It uses a fixed threshold from the original paper on Isolation Forest. It does not look at your data or your team. Priya measures what it does on the evaluation months, where about 31 payments a day are planted:

Payments flagged a day by Isolation Forest's predict, for four contamination settings, next to Sofia's budget (50 a day) and the planted payments (31 a day). Measured on the evaluation months with the behaviour features.
Text description of the chart

A column chart with six columns. contamination “auto” flags 386.3 payments a day. 0.05 flags 111.9. The budget is 50. The planted payments are 31.0 a day. 0.01 flags 25.7. 0.001 flags 2.1.

Contamination Share flagged Flagged a day Precision
"auto" 20.7% 386.3 0.079
0.05 6.0% 111.9 0.267
0.01 1.4% 25.7 0.927
0.001 0.1% 2.1 1.000

The real share of planted payments is 1.66%. "auto" flags 20.7%: 386 payments a day, almost eight times what Sofia’s team can review, and only 8% of them planted. A setting of 0.01 looks excellent. But you know that only because you have the answer key. A real team does not know the share of fraud in its data. That is the very thing it is trying to find.

So the course never uses predict or contamination to set the threshold. It ranks the payments by score and sends the top 50 a day: the number that the team can really review. Module 7 develops this idea: choose the threshold from the budget, not from a guess.

Each tree is random, so a forest of few trees gives a noisy score. Priya measures forests of 1 to 500 trees, each with 5 different random seeds:

  • Best of 5 seeds
  • Worst of 5 seeds
Precision at the budget for forests of 1 to 500 trees, with 5 random seeds each: the best and the worst seed (measured, behaviour features). The horizontal axis is on a log scale. From about 50 trees, the seed hardly matters.
Text description of the chart

A line chart with two lines, on a log scale of the number of trees. With 1 tree, the seeds give 0.210 to 0.442. With 10 trees, 0.475 to 0.493. With 50 trees, 0.491 to 0.504. With 100 trees, 0.488 to 0.503. With 200 trees, 0.494 to 0.500. With 500 trees, 0.498 to 0.501.

One tree is a lottery: 0.210 to 0.442, depending on the seed. With 10 trees, the result is already 0.475 to 0.493. From 50 trees on, the seed hardly matters. More trees cost more time: on average 0.8 seconds to fit and score with 200 trees, and 1.9 seconds with 500 (measured). The default of 100 trees is a reasonable start; the course uses 200.

Speed is the reason to choose Isolation Forest on large data. Each tree looks at 256 payments only, and scoring a payment means following one short path in each tree. Measured on an Apple M4 with 10 cores, 200 trees:

  • All 343,074 payments, fit and score: 1.4 seconds.
  • 1,000,000 rows, fit and score: 4.0 seconds.

The computer was busy with other work during this measurement, so these are upper limits. Your time will differ. Compare it with kNN distance, which needed 4.1 seconds to score the evaluation months against only 20,000 stored payments.

When to use Isolation Forest, and what it costs

Section titled “When to use Isolation Forest, and what it costs”

Use it as a strong first detector on tables with many rows and a handful of numeric features. It is fast, it learns from all your data, and it needs little tuning. It is also less sensitive to the scale of each feature than distance methods.

Be careful in these cases:

  • The threshold. Never trust contamination to set the number of alerts. Rank and take the budget.
  • Explaining an alert. “Isolated in few random cuts” is hard to say to a reviewer. Show the features that are far from the customer’s usual values instead. Module 8 does this.
  • Anomalies that are not extreme in any column. A cut always separates the extremes of one column. A point that is strange only in a combination, in the middle of every range, needs more cuts. Good features help here.
  • Repeated attacks. A thousand identical card tests in the training data become a crowd that is no longer easy to cut off.

Common mistake

386 alerts a day from the default setting

Symptom: Tomás deploys Isolation Forest with predict, and leaves contamination="auto". On the first day, Sofia’s team receives 386 alerts. They can review 50. After a week, they stop reading the alerts.

Cause: "auto" uses a fixed threshold from the original paper. It does not know Larkfield’s data or Sofia’s team. On the evaluation months, it flags 20.7% of all payments, while 1.66% are planted. Only 7.9% of its alerts are planted frauds.

Fix: use the score, not predict. Rank each day’s payments by score and send the top 50: the number that the team can review. The same forest then has a precision of 0.500.

Consequence in AI work: a library default that sets a threshold silently decides how much work people get. Choose every threshold from the capacity of the people who act on it, and write that number down.

In the notebook, section 5 has three tasks:

  1. Write the average number of cuts for G over the 1,000 trees, as g_average.
  2. Write how many payments a day contamination="auto" flags, as auto_per_day. Compare it with the budget.
  3. Fit a forest with only 10 trees (random_state=0), and put its precision at the budget in precision_10. Predict first: how much worse than 200 trees?
Show the expected results
  1. g_average = 1.21.
  2. auto_per_day = 386.3: almost eight times the budget of 50.
  3. precision_10 = 0.49: only a little below 0.500 with 200 trees. With this seed, 10 trees are almost enough; other seeds give 0.475 to 0.493.

Check your understanding

Explain in plain words why a strange point needs fewer random cuts to be alone than a normal point.

Show the answer

A random cut falls anywhere between the smallest and the largest value of one column. A strange point sits alone in a large empty range of at least one column. So most random cuts in that range separate it from the others. A normal point sits in a crowd: each cut removes only a few neighbours, so many cuts are needed. In the tiny example, G needs 1.21 cuts on average, and A to F need 3.02 to 4.56.

Check your understanding

Predict: you leave contamination at "auto" and use predict. About how many alerts a day does Sofia's team get, and what should you do instead?

Show the answer

About 386 a day (20.7% of the payments), against a budget of 50 and about 31 planted payments a day. Only 7.9% of them are planted. Instead, rank each day’s payments by the score and send the top 50, the number that the team can review. The threshold comes from the budget, not from a guess of the share of anomalies.

Check your understanding

Why can Isolation Forest learn from all 173,271 reference payments, while kNN distance and LOF used a sample of 20,000?

Show the answer

kNN distance and LOF store the reference points and search for the nearest ones for every new payment. This is slow, and it uses more memory as the stored set grows. Isolation Forest never measures the distance between points. Each tree is built from a small random sample (256 payments), and scoring a payment only follows one short path in each tree. So it stays fast on all the payments. It needed 1.4 seconds for all 343,074, and 4.0 seconds for 1,000,000 rows (measured on a busy computer).

Recap

  • Isolation Forest cuts the data at random. A strange point is alone after few cuts. The score is the average number of cuts over many trees.
  • In the tiny example, G is alone after 1.21 cuts on average; A to F need 3.02 to 4.56.
  • On the payments, it learns from all reference rows and reaches 0.500 with the behaviour features (0.208 with the raw features).
  • contamination="auto" flags 20.7% (386 a day) against a real 1.66%. Choose the threshold from the budget: the top 50 a day.
  • From about 50 trees, the result hardly depends on the seed. It is fast: all payments in 1.4 seconds.

You now have three detectors with three different interfaces and three different ideas. But a team that wants to try many detectors, and to combine them, needs one way to run them all. Next, you draw a boundary around normal data with a One-Class SVM, and run four detectors with one interface in PyOD.

Next: One-Class SVM and the PyOD toolbox