Skip to content
Français

When not to use machine learning

  • Lesson 28
  • 25 min
  • Module 6: Deep and embedding methods

Where we are

We have tried rules, statistics, classical models, a neural network, embeddings and an LLM on the same data. In this lesson, we read the comparison table for the cases where a rule or a MAD score won on cost, speed and explanation, and we list the signs that a deep model is not worth it.

Lesson 28 of 38 · Anomaly Detection: Find What Is UnusualWhen not to use machine learning
Course overview

What you will be able to do

Choose the simplest method that does the job, with evidence. Read the comparison table for the cases where a rule or a simple statistic won. Then learn the signs that a deep model is not worth its cost.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. It reads results that the earlier lessons and their notebooks measured.

You want to hang a picture. You can take a hammer and a nail, or you can buy a drill press. The drill press is a better machine in almost every way. It is also expensive, it takes an afternoon to set up, and it needs training to use safely. For one picture, the hammer wins.

Nobody thinks the hammer is “better” in general. It is better for this job, at this size, with this budget. A carpenter who uses the drill press for every nail is not more advanced. He is slower, and his mistakes are bigger.

Detectors are the same. The question is never “which method is the most advanced?” It is “which is the simplest method that does this job well enough, and how do I know?”

Tomás has an idea for the next year: “Let’s move every detector at Larkfield to deep learning. Payments, refunds, the service metrics, the logs. One modern stack.” It sounds tidy.

The comparison table tells a different story. Look at three measured rows on the payments:

Detector Precision at the budget Cost per week (dollars)
Autoencoder on the raw columns 0.074 8,278
Two rules, ranked (velocity and device) 0.427 6,954
Autoencoder on the behaviour features (the notebook) 0.434 3,958

The neural network on the raw columns was far worse than two lines of rules. On good features, it was only 0.007 above them in precision. It was cheaper per week, because it missed fewer planted payments. But the rules can be explained in half a minute, and the network needs five (the course’s assumptions).

So the honest answer to Tomás is not “no”. It is: measure the simple method first, on the same split and the same budget, and add complexity only when it wins. The simple method that you measure first is the baseline.

The chart puts nine rows of the payments table on one scale. The cost per week counts the missed planted payments (the amount plus a 15-dollar fee) and 3 dollars for each false alert. These costs are the course’s assumptions.

  • Rule, statistic or reference
  • Neural network
  • Classical machine learning
Cost per week of nine payments detectors, on the evaluation months at 50 alerts a day (measured; costs are the course's assumptions). The best rows are classical models on the behaviour features. The neural network on raw columns is worse than the two rules.
Text description of the chart

Nine rows, one dot each, on an axis of dollars per week from 0 to 11,000. Nothing reviewed: 10,367. LOF on the behaviour features: 9,248. MAD per customer: 8,280. The autoencoder on the raw columns: 8,278. Two rules, ranked: 6,954. The autoencoder on the behaviour features, the notebook’s run: 3,958. Isolation Forest on the behaviour features: 3,218. kNN distance on the behaviour features: 2,739. A perfect detector that knows the answers: 953.

Read the chart from the top. Doing nothing costs 10,367 dollars a week. The two rules cut that by about a third. The large step down comes from the behaviour features: the customer’s own past. With them, kNN distance and Isolation Forest are the cheapest detectors in the course. The autoencoder on the same features is more expensive than both. So the winner on payments is machine learning, but classical machine learning on good features, not the deepest model.

Payments were the hardest problem of the course, and machine learning helped there. On Larkfield’s other problems, a rule, a count or a statistic was enough. These results were all measured in earlier lessons.

In card testing, one device tries many customers’ cards with tiny payments. The rule “a device with more than 3 payments in 1 hour” finds 52.2% of the card-testing payments. A reviewer understands each alert in half a minute, for example: “this phone paid with 9 different cards in one hour”. The rule needs no training and no data. Isolation Forest on the behaviour features finds more card-testing payments, 0.718 (Autoencoders and reconstruction error). But each of its alerts takes 3 minutes to explain, not half a minute (the course’s assumptions). The rule wins on cost and on speed, not on recall.

The rule does not find everything. It finds no account takeovers at all (recall 0.0). A takeover makes only a few payments (406 in 209 takeovers), so its device never passes 3 in an hour. That is why the comparison table has more rows.

2. Account takeover: a robust score per customer

Section titled “2. Account takeover: a robust score per customer”

An account takeover is a few expensive payments from a customer’s own account. The MAD score per customer asks one question: “how far is this amount from this customer’s usual amount, in robust units?” It finds 53.4% of the account takeovers. It is one median and one MAD for each customer: a statistic, not a model.

Grace’s refunds are one rate a day. A control chart is the right tool for one number over time. On the planted 0.3-point shift, EWMA alarmed after 35 days and CUSUM after 43, with no false alarm in the 96 stable days. Per agent, a p-chart and CUSUM found agent A-17’s extra refunds in 6 days.

Agent A-29’s refunds cluster just below the 100-dollar approval limit. No model found that. A count did: 16.1% of A-29’s refunds were between 90 and 99.99 dollars, against 3.5% for all agents. A business check written in one line of code.

For ticket-api’s latency, the seasonal baseline compares each minute with the same minute one and two weeks before. It found the configuration change 3 minutes after it started. Isolation Forest on three metrics found the same change after 1,126 minutes. A plain latency threshold of 350 ms sent only 0.28 events a day, and every one was on a planted problem.

5. Logs and daily orders: templates and a calendar

Section titled “5. Logs and daily orders: templates and a calendar”

The four search-and-replace patterns of Unusual text and logs with embeddings found all three new log messages in 0.24 seconds. The embeddings ranked the harmless feature flag as the most unusual. For the daily orders, a forecast with a 99% interval alarmed on 16 days, and 15 of them were on or next to a holiday. A plain list of holidays brought the alarms down to 1.

Larkfield problem What won Measured evidence The ML alternative
Card testing Device rule recall 0.522, 0.5 min to explain Isolation Forest 0.718, 3 min to explain
Account takeover MAD per customer recall 0.534 autoencoder 0.575, 5 min to explain
Refund rate shift EWMA, CUSUM 35 and 43 days, 0 false alarms in 96 days —
One agent below the limit A count of 90–99.99 dollars 16.1% against 3.5% —
Service after a change Seasonal baseline 3 minutes to detection Isolation Forest: 1,126 minutes
New log messages Templates 3 of 3, in 0.24 s embeddings ranked the harmless one first
Payments, all kinds Isolation Forest, kNN on behaviour features 0.500, 0.505; 3,218 and 2,739 $/week autoencoder 0.434 (0.468), rules 0.427

The last row is where machine learning won. Notice what made it win: the behaviour features, which are themselves simple counts and comparisons with the customer’s past. And notice the second row: the autoencoder found more account takeovers (0.575) than the MAD score (0.534). A more complex model can win on one kind. Then the question is whether that gain pays for its cost.

The diagram turns these results into a few questions:

A decision flow. Can a person write the pattern down exactly? Yes: a rule, a limit, a count, an allow-list. No: is it one number, compared with its own past? Yes: a robust statistic, MAD, a control chart, a seasonal baseline. No: is it text, such as log lines? Yes: templates first, embeddings only to rank the few new ones. No: classical machine learning on behaviour features, Isolation Forest or kNN. Then: did a deep model beat it on the same split and the same budget? No: keep the simpler model. Yes: use the deep model, and pay to run and explain it.
Start from the simplest method that fits the problem. A deep model must win on the same split and budget before it replaces a simpler one.
Text description of the diagram

The diagram is a chain of questions. First: can a person write the pattern down exactly? If yes, use a rule: a limit, a count or an allow-list. If no, the second question: is it one number, compared with its own past? If yes, use a robust statistic: MAD, a control chart or a seasonal baseline. If no, the third question: is it text, such as log lines? If yes, use templates first, and embeddings only to rank the few new ones. If no, use classical machine learning on behaviour features, such as Isolation Forest or kNN. Then a last question: did a deep model beat it on the same split and the same budget? If no, keep the simpler model. If yes, use the deep model, and pay to run and to explain it.

The diagram is a starting point, not a law. Many real problems use several boxes at once. Larkfield’s payments use rules for card testing, a statistic for account takeovers, and Isolation Forest for the rest.

Tomás asks: “So when is deep learning worth it?” Look for these signs first. Each one says “not here, not yet”:

  • The data is a table of designed features. Larkfield’s eight behaviour features were made by hand. Forests and neighbour searches use such tables well. Neural networks shine on raw signals with many related values: images, sound, sensor waveforms, long sequences.
  • The simple row is close. A deep model that wins by less than the gap between two of its own runs has not won. The autoencoder moved from 0.434 to 0.468 with another starting point; that spread is larger than its gap to the rules.
  • The reviewer needs a reason. Sofia’s team has 50 alerts a day. “This device paid with 9 cards in one hour” takes half a minute. A score from 395 weights takes longer to explain. The course assumes 0.5, 3 and 5 minutes.
  • There are few labels to check it. A complex model can learn strange shortcuts. Without labels, you may never see them.
  • The team cannot run it. A deep model needs a framework (PyTorch is about 200 MB) and several runs. Someone must retrain and debug it.
  • The volume is small. Some problems have one number a day, such as a refund rate. A control chart has fewer choices and a known false-alarm rate.

The minutes to explain an alert look small. Multiply them. At 50 alerts a day, Sofia’s team reviews 350 alerts a week. With the course’s assumptions, that is 175 minutes a week to explain rule alerts (0.5 minutes each). Isolation Forest alerts take 1,050 minutes (3 each), and autoencoder alerts 1,750 minutes (5 each). The minutes are assumptions, and the arithmetic is constructed. But the order of the three is the point: the more complex the model, the more of the team’s time each alert takes.

Common mistake

Average every detector, to be safe

Symptom: Tomás cannot choose, so he averages four detectors on the behaviour features: kNN, LOF, Isolation Forest and One-Class SVM. “An ensemble is always safer,” he says. Its precision at the budget is 0.211 (measured). Its best member, kNN, reached 0.505.

Cause: one member was bad. LOF on the behaviour features scored 0.082. Averaging the ranks gave LOF a quarter of every decision, and it dragged the others down. Without LOF, the average of three reached 0.504, but still no better than kNN alone.

Fix: score each member alone first, in the comparison table. Remove the members that lose. Then check if the combination beats the best single member. Here it did not, so keep kNN or Isolation Forest.

Consequence in AI work: more models is not more safety. Complexity adds new ways to fail, and an average can hide the failure. Every added part must earn its place with a measurement.

Check your understanding

Explain why the autoencoder on the behaviour features (0.434) is not a reason to replace the two ranked rules (0.427), and what is.

Show the answer

The gap in precision, 0.007, is smaller than the spread between two runs of the autoencoder (0.434 and 0.468). The rules take half a minute to explain an alert, against five minutes for the autoencoder (the course’s assumptions), and they need no training. What does justify a change is the cost per week. Isolation Forest and kNN on the behaviour features cost 3,218 and 2,739 dollars a week, against 6,954 for the rules. That is a large, measured gain, and it comes from the features.

Check your understanding

Predict: Larkfield adds a new rule from its bank: “no more than 5 refunds to the same card in a day”. Which method from the diagram should check it, and why?

Show the answer

A rule. A person can write the pattern down exactly, so a count per card and day does the job. It is fast, it needs no data to train, and every alert explains itself: “card X got 7 refunds today”. A model could only learn a rough copy of this pattern, and its alerts would be harder to explain.

Check your understanding

Module check. Recommend one method for each of three Larkfield problems: (a) one agent's refunds that cluster just below the 100-dollar approval limit, (b) new kinds of message in ticket-api's logs, (c) stolen cards and account takeovers in the card payments. Give the evidence from the comparison table or the earlier lessons. At least one answer must use no machine learning.

Show the answer

(a) No machine learning: a business check that counts the share of each agent’s refunds between 90 and 99.99 dollars. A-29 had 16.1% against 3.5% for all agents, the only agent after a Bonferroni correction. (b) No machine learning: log templates, with a list of the templates seen in the normal weeks. They found all three new messages in 0.24 seconds. Embeddings can rank the new ones, but they ranked the harmless flag first. (c) Machine learning: Isolation Forest or kNN distance on the behaviour features, at 50 alerts a day. They reach a precision of 0.500 and 0.505, at 3,218 and 2,739 dollars a week. The two rules reach 0.427, at 6,954 dollars a week. The autoencoder scored 0.434 (0.468 with another start) and costs more to explain.

Recap

  • Do not ask “which method is the most advanced?” Ask “which is the simplest method that does this job?” Measure it on the same split and budget. The simple method you measure first is the baseline.
  • No machine learning won on card testing (a device rule) and on refunds (control charts and a count). It also won on the service after a change (a seasonal baseline), the logs (templates) and the daily orders (a holiday list).
  • On payments, classical models on the behaviour features won. Isolation Forest reached 0.500 and kNN 0.505, at 3,218 and 2,739 dollars a week. The autoencoder did not beat them.
  • Signs that a deep model is not worth it: a table of designed features, a small gap, and a reviewer who needs a reason. Also few labels, a team that cannot run it, and a small volume.
  • More models is not more safety: the average of four detectors scored 0.211 because one member was bad.

You can now choose a method with evidence. But all that evidence used the answer key: precision and recall need to know which payments were planted. Real work rarely has a full key, and some numbers that look good can lie. Next, you see why accuracy lies here, and which measures to trust.

Next: Why accuracy lies here