Features decide more than the algorithm
- Lesson 19
- 40 min
- Module 4: Distance, density and isolation
Where we are
We compared four detectors on two sets of features. On the raw payment columns, none of them was very good. The raw columns do not say what makes fraud strange: a new country, or many payments in an hour. In this lesson, we build the behaviour features that we took as given, and we rerun every detector, to see what helps more.
Lesson 19 of 38 · Anomaly Detection: Find What Is UnusualFeatures decide more than the algorithm
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Build the features that make fraud visible. Then measure what really moved the comparison table in this module: the choice of algorithm, or the features. This is the most useful habit of the whole course.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson, with the instructions. Its setup cells install pandas, NumPy, SciPy, scikit-learn, matplotlib and PyOD, and download the payments (about 7 MB). The whole notebook runs in about two minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The four detectors of this module and their rows in the comparison table, from Far from the neighbours: kNN distance to One-Class SVM and the PyOD toolbox.
- Customer C-01234 and her eight payments in Portugal, from The payment that did not fit.
- The median, from Median, MAD and IQR.
What you need on your computer
A browser and a free Google account (Colab) or Kaggle account. Nothing to install.
Tested with Python 3.12.13 in Jupyter on macOS, with pandas 3.1.1, scikit-learn 1.9.1 and PyOD 3.6.7. Colab, Kaggle and Windows were not tested.
The clerk who knows her customers
Section titled “The clerk who knows her customers”Think of a clerk in a small bank. For twenty years, she has served the same customers. An old man always takes out 50 dollars on Friday morning. One day, he asks for 50 dollars on a Tuesday night, with a stranger beside him. 50 dollars is a normal amount. Tuesday is a normal day. But for him, this is strange, and she notices at once.
A new clerk, who sees only the amount, the day and the time, notices nothing. Both clerks are equally clever. The first one simply knows more: she compares each visit with the customer’s own past.
The problem: the raw columns hide the fraud
Section titled “The problem: the raw columns hide the fraud”So far in this module, every detector ran twice: on the raw columns of a payment, and on eight behaviour features that you took as given. The raw columns are the amount, the hour and the account type. Look at C-01234’s first payment in Portugal again with only those columns: 24.27 dollars, at 2 a.m., on a personal account. A small amount, a quiet hour, the most common kind of account. Nothing in these three numbers says “stolen card”.
The measurement agrees. With the raw features, kNN distance ranks C-01234’s first payment in Portugal 136th of the 1,780 payments of that day. Sofia’s team reviews the top 50, so it never sees this payment. With the behaviour features, the same detector ranks it 2nd.
The comparison table says the same thing for every detector. With the raw features, all four reach only about 0.2:
| Detector | Raw features |
|---|---|
| kNN distance | 0.219 |
| LOF | 0.168 |
| Isolation Forest | 0.208 |
| One-Class SVM | 0.195 |
Tomás’s answer is a better algorithm: “Let us try a deeper model.” Priya’s answer is a better question: what would the old clerk look at?
The idea: compare each payment with the customer’s past
Section titled “The idea: compare each payment with the customer’s past”A feature is a column that you give to a detector. Feature engineering means building new columns from the raw data, so that the pattern that you look for becomes visible. Here, the pattern is “not like this customer”. So each new feature compares the payment with the customer’s own payments before it:
Text description of the diagram
The diagram is a flow from left to right. On the left are two inputs: one payment, with its amount, country, device and time; and the customer’s past, the payments before this one. They feed four groups of features. The first is log_amount_vs_usual, the amount divided by the median of the last 10 payments. The second is new_country and new_device: is this the first time today? The third is log_minutes_since_last, payments_last_hour and device_payments_last_hour. The fourth is night and is_business, from the hour and the account type. All four groups go into any detector: kNN, LOF, Isolation Forest or the One-Class SVM.
Each feature answers one question that the clerk would ask:
log_amount_vs_usual: is the amount far from this customer’s usual amount? The usual amount is the median of the customer’s last 10 payments before this one. The feature is the logarithm of the ratio, so twice the usual amount and half of it are equally far from 0.new_country: is this the first day that the customer pays from this country? (Not counted on the customer’s very first day: everything is new then.)new_device: the same question for the device.log_minutes_since_last: how long since the customer’s last payment? (As a logarithm, because the times go from seconds to months.)payments_last_hour: how many payments has this customer made in the last hour, this one included?device_payments_last_hour: how many payments has this device made in the last hour, with any customer’s card? A card tester uses one device for many cards.night: is the hour between 0 and 5?is_business: is it a business account?
One rule matters more than any feature: each feature uses only the payments before (or at) the payment’s own time. A live detector cannot see the future, so its features must not either.
A tiny example: one customer, by hand
Section titled “A tiny example: one customer, by hand”Take customer C-01234 and compute the features of three payments, with every value visible.
The usual amount. Before the first Portugal payment, these were the customer’s last 10 payments: 14.28, 12.64, 17.49, 16.08, 13.91, 11.19, 16.02, 16.75, 17.77 and 15.16 dollars. In order, the middle two are 15.16 and 16.02, so the median is (15.16 + 16.02) ÷ 2 = 15.59 dollars. The payment is 24.27 dollars:
24.27 ÷ 15.59 = 1.557, and the logarithm of 1.557 is 0.443.
The other features come from simple counts. The table shows the last ordinary basket, and the first and the eighth payments in Portugal:
| Payment | Amount | log_amount_vs_usual |
new_country |
new_device |
Minutes since last | payments_last_hour |
night |
|---|---|---|---|---|---|---|---|
| 27 Sep, 09:17, Canada | 15.16 | −0.057 | 0 | 0 | 10,095 (7 days) | 1 | 0 |
| 3 Oct, 02:14, Portugal (1st) | 24.27 | 0.443 | 1 | 1 | 8,216 (5.7 days) | 1 | 1 |
| 3 Oct, 02:21, Portugal (8th) | 19.25 | 0.002 | 1 | 1 | 1.1 | 8 | 1 |
Now the fraud is visible in the columns themselves. There is a new country, a new device and the night. Then come eight payments in one hour, about one a minute. The amounts are not the signal: the eighth payment is almost exactly the usual amount (0.002). A detector that looks only at the amount, like MAD per customer, misses this card.
Larkfield: rerun every detector
Section titled “Larkfield: rerun every detector”The course file features.py builds the eight features for all 343,074 payments in about 10 seconds (measured). Priya reruns the four detectors with exactly the same settings, samples and scaling as before. Only the features change.
- Open the notebook from the box at the top of this page. Run the three setup cells.
- Run section 1. It builds the features of C-01234 by hand. Check the last row:
payments_last_houris 8. - Run the next cell. It compares your features with
features.behaviour(). Check that it printsTrue True True. - Before you run section 2, write down your prediction: which will give the larger gain, the detector or the features?
- Run section 2. It takes about 40 seconds. It adds eight rows to the comparison table.
The output:
raw behaviour gainkNN distance 0.219 0.505 0.286LOF 0.168 0.082 -0.086Isolation Forest 0.208 0.500 0.292One-Class SVM 0.195 0.480 0.285Each line in the chart is one detector. The left end is its precision with the raw features, the right end with the behaviour features. The dashed line at the top is the ceiling. A detector that knows every answer reaches 0.542 at 50 alerts a day, because on many days fewer than 50 payments are planted.
- kNN distance
- Isolation Forest
- One-Class SVM
- LOF
Text description of the chart
A slope chart with four solid lines and one dashed line. Each line goes from the raw features on the left to the behaviour features on the right. kNN distance rises from 0.219 to 0.505. Isolation Forest rises from 0.208 to 0.500. The One-Class SVM rises from 0.195 to 0.480. LOF falls from 0.168 to 0.082. The dashed ceiling is flat at 0.542.
Compare the two gains
Section titled “Compare the two gains”Now answer the question of this module with the table.
- The gain from the algorithm. With the raw features, the best and the worst detectors are 0.051 apart (0.219 against 0.168). With the behaviour features, the three good detectors are 0.025 apart (0.505 against 0.480).
- The gain from the features. Each good detector gains about 0.29: kNN distance +0.286, Isolation Forest +0.292, the One-Class SVM +0.285.
The features gave more than ten times the gain of choosing among the good detectors. And with the behaviour features, the good detectors reach 0.48 to 0.505, close to the ceiling of 0.542. There is little left for any algorithm to win on this table.
The recall by kind shows where the gain comes from. Isolation Forest found 19.7% of the stolen-card payments with the raw features, and 90.5% with the behaviour features (measured). A stolen card is exactly the “not like this customer” pattern that the new features describe.
LOF is the exception. It gets worse with better features, from 0.168 to 0.082. The reason is the one of the LOF lesson. With features that describe fraud well, old card tests form a tight crowd, and a new card test looks normal inside it. Better features do not rescue a method whose question does not fit the data.
Which features matter most?
Section titled “Which features matter most?”Priya asks a sharper question: which of the eight features does the work? She tries it two ways with Isolation Forest (measured).
Add one feature to the raw ones. Start from the raw features (0.208) and add one behaviour feature at a time:
Text description of the chart
A column chart with five columns. Raw features only: 0.208. Raw plus new_country: 0.437. Raw plus new_device: 0.406. All eight behaviour features: 0.500. The ceiling: 0.542.
| Feature added to the raw ones | Precision | Feature removed from the eight | Precision |
|---|---|---|---|
| none (raw only) | 0.208 | none (all eight) | 0.500 |
new_country |
0.437 | new_country |
0.462 |
new_device |
0.406 | new_device |
0.469 |
device_payments_last_hour |
0.404 | log_amount_vs_usual |
0.488 |
payments_last_hour |
0.308 | night |
0.485 |
log_minutes_since_last |
0.281 | log_minutes_since_last |
0.500 |
log_amount_vs_usual |
0.256 | device_payments_last_hour |
0.506 |
night |
0.199 | payments_last_hour |
0.511 |
Read the two halves together:
- One good feature beats every change of algorithm. Adding only
new_countrymoves Isolation Forest from 0.208 to 0.437: more than any detector swap in this module. - Features overlap. Removing
payments_last_hourordevice_payments_last_hourdoes not hurt: the other features already carry most of their information. Removing one of them even helps a little (0.511), which is within the noise of one random forest. nightalone hurts (0.199): many honest payments happen at night, so the raw hour already said as much.
This is how real work goes: add a feature, measure it in the same table, keep it or drop it, and write down the result.
When behaviour features help, and what they cost
Section titled “When behaviour features help, and what they cost”They help whenever “unusual” means “unusual for this one”: a customer, a device, an agent, a machine. The same idea works for the refunds of one agent against that agent’s past, or the latency of one server against its own week.
They cost something:
- History. A feature that needs the past fails when there is no past. A customer’s very first payment has no usual amount. So
features.pyuses the payment itself (the ratio is 1, the logarithm 0). It also never marks a new country on the first day. A thief who opens a new account is invisible to these features. Keep a separate check for new customers. - Compute and storage. The features need each customer’s recent payments at scoring time. Here, 10 seconds for all 343,074 payments. In a live system, you keep a small state per customer.
- Changes in normal behaviour. Larkfield launches its app on 15 October 2025. Then 60% of customers start to pay on a device that the shop never saw for them (planted).
new_devicethen fires for honest customers. A feature that describes “new” well also reacts strongly when everything becomes new. Later in the course, the detection job notices this and asks for retraining. - Explanation. Good features are also the best explanation for a reviewer. “New country, new device, 8 payments in an hour” says more to Sofia than any score.
Common mistake
Features that look into the future
Symptom: a new feature makes the evaluation look excellent. In production, the detector finds far fewer frauds than in the evaluation.
Cause: the feature used information that did not exist at the time of the payment. A common example: the customer’s usual amount is computed as the median of all the customer’s payments, including later ones. Another one: “number of payments today” counts the payments later that day. In the evaluation, the detector sees a little of the future; in production, it cannot.
Diagnosis: for each feature, ask: could the shop know this value at the second of the payment? Then test it. Compute the features on the data up to one day, then again on more data. The features of that day must not change.
Fix: compute every feature only from the payments before (or at) the payment’s own time. features.py does this with shift() and with rolling windows that look back. The detection job later in the course tests this with exactly that check.
Consequence in AI work: look-ahead is one of the most common reasons why a model that looked great in a notebook fails in production. It hides well, because the code runs and the numbers look better.
Your turn
Section titled “Your turn”In the notebook, section 3 has three tasks:
- Write the gain of Isolation Forest from raw to behaviour features, as
gain_forest. - Add each behaviour feature (except
is_business) to the raw features, one at a time, and run Isolation Forest. Put the results inplus_oneand the best feature inmost_useful. This takes about 30 seconds. - Write two sentences for Sofia: did the better features or the better algorithm give the larger gain on Larkfield’s payments? Use two numbers from the table.
Show the expected results
gain_forest = 0.292(0.208 to 0.500).most_useful = "new_country": 0.437, thennew_device0.406 anddevice_payments_last_hour0.404.- For example: “The better features gave the larger gain: each good detector gained about 0.29 (Isolation Forest from 0.208 to 0.500). Choosing among the good detectors changed the result by only 0.025 (0.480 to 0.505). All are close to the ceiling of 0.542.”
Knowledge check
Section titled “Knowledge check”Check your understanding
What is a behaviour feature? Give two examples from this lesson, and say why each one makes a stolen card visible.
Show the answer
A behaviour feature compares a payment with the customer’s own past (or the device’s), using only what happened before the payment. Two examples. new_country (the first day that the customer pays from this country) makes a card used abroad by a thief visible. payments_last_hour (the customer’s payments in the last hour) makes a burst of many quick payments visible. Others: new_device, the amount against the usual amount, device_payments_last_hour.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you compute the customer's usual amount as the median of all the customer's payments in the six months, including later ones. What happens in the evaluation, and in production?
Show the answer
The feature looks into the future. In the evaluation, it can use payments that happen after the one that it scores, so the detector may look better than it really is. In production, the later payments do not exist yet, so the feature has different values and the detector finds fewer frauds. Every feature must use only the payments before (or at) the payment’s own time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Use the comparison table to answer the question of this module: did the better features or the better algorithm give the larger gain? Give at least two numbers.
Show the answer
The better features. Each good detector gained about 0.29 from raw to behaviour features: kNN distance 0.219 to 0.505, Isolation Forest 0.208 to 0.500, the One-Class SVM 0.195 to 0.480. With the behaviour features, choosing among the good detectors changed the result by only 0.025 (0.480 to 0.505). With the raw features, it changed it by 0.051. With the behaviour features, the good detectors are close to the ceiling of 0.542. LOF is the exception: it fell from 0.168 to 0.082.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Behaviour features compare a payment with the customer’s past. Examples: the usual amount, a new country or device, and the payments in the last hour.
- Every feature uses only the past. A feature that sees the future makes the evaluation lie.
- With the raw features, every detector is near 0.2. With the behaviour features, three detectors reach 0.480 to 0.505. The ceiling is 0.542.
- Features decided more than the algorithm: +0.29 for each good detector, against 0.025 between the good detectors. LOF fell to 0.082.
- One feature,
new_country, moved Isolation Forest from 0.208 to 0.437 by itself.
You can now build features that describe “not like this customer”, and you have a full set of detectors for single payments. But all of them treat every payment the same, whatever the day or the hour. Daily orders and help-desk tickets rise and fall with the week and the season. Next, you see why one threshold fails on a series in time, and you build baselines that follow the hour and the season.