Autoencoders and reconstruction error
- Lesson 25
- 40 min
- Module 6: Deep and embedding methods
Where we are
Our classical detectors measure distance, density or isolation. Tomás asks if a neural network finds more. In this lesson, we train a tiny autoencoder on a CPU to rebuild normal payments, use the error as a score, and compare it honestly with Isolation Forest in the comparison table.
Lesson 25 of 38 · Anomaly Detection: Find What Is UnusualAutoencoders and reconstruction error
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Use a small neural network as an anomaly detector: it learns to rebuild normal payments, and its error is the score. Then decide, with the comparison table, whether it is worth its extra cost at Larkfield.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, scikit-learn and PyTorch (about 200 MB) and download the payments (about 7 MB). Training takes a few seconds on a CPU. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The behaviour features and Isolation Forest, from Isolation Forest and Features decide more than the algorithm.
- Precision at the budget, from The payment that did not fit, and the comparison table, from Meet the Larkfield data.
- It helps, but it is not required, to know what a layer and a training loop are. Neurons and layers and The training loop in Deep Learning and Transformers Explained teach them. This lesson explains the parts it uses.
What you need on your computer
A browser and a free Google or Kaggle account for the notebook, or the local setup from Meet the Larkfield data. No GPU.
Tested with Python 3.12.13 in Jupyter on macOS, with PyTorch 2.15.0 on the CPU. Colab, Kaggle and Windows were not tested.
The signature copier
Section titled “The signature copier”Think of a bank clerk who checks signatures all day. After a few months, she can redraw each regular customer’s signature from memory. She does not keep a list of forged signatures. She only knows the normal ones very well.
One day a cheque arrives with a strange signature. She tries to redraw it from memory, and her copy comes out like the customer’s usual signature. Her copy and the cheque do not match. The bigger the difference, the more suspicious the cheque.
Notice the trick. The clerk never learned what a forgery looks like. She learned to rebuild normal, and she used her failure to rebuild as the alarm.
Tomás’s question
Section titled “Tomás’s question”In Module 4, we gave every payment a score with classical methods. kNN distance asks “how far is this payment from its neighbours?” Isolation Forest asks “how quickly can random cuts isolate it?” On the behaviour features, both reached a precision at the budget of about 0.50.
Tomás, Larkfield’s AI engineer, wants more. “These are old methods,” he says. “A neural network learns its own patterns. It will find more fraud.” It is a fair question, and the answer must come from a measurement, not from a feeling about new and old methods.
So this lesson builds the bank clerk as a neural network. It scores the same payments with the same budget, and reads the result in the comparison table.
The idea: rebuild normal, and measure the failure
Section titled “The idea: rebuild normal, and measure the failure”An autoencoder is a small neural network with one job: to give back its own input. It has two halves:
- The encoder squeezes the input into fewer numbers. These few numbers are the code.
- The decoder rebuilds the input from the code alone.
The name says it: it encodes its own input, by itself (“auto”). The squeeze is the important part. The code is too small to keep every detail of every input. So, during training, the network keeps only the patterns that are common in its training data. If the training data is mostly normal, it learns the patterns of normal.
The reconstruction error is the difference between the input and the rebuilt input. We square the difference of each number and take the mean. A normal input is rebuilt well, so its error is small. An unusual input is rebuilt as if it were normal, so its error is large. The error is the anomaly score, and no labels are needed.
A tiny example with six points
Section titled “A tiny example with six points”The notebook’s first cell makes six points with two numbers each, x and y. The first five lie close to the line y = 2x. The sixth point, (2, 0.5), does not. The autoencoder squeezes 2 numbers into 1 and rebuilds 2 numbers from it. It learns from the five normal points only, for 2,000 small steps.
The output:
x y rebuilt x rebuilt y error0.0 0.0 -0.01 0.01 0.0001.0 2.0 1.00 2.00 0.0002.0 4.1 2.03 4.06 0.0013.0 5.9 2.97 5.94 0.0014.0 8.0 4.01 7.99 0.0002.0 0.5 0.98 1.96 1.591Why does this work? One number is enough to describe a point on the line: how far along the line it is. So the network learned “position along the line” as its code. It rebuilds the five normal points almost exactly, with errors of 0.001 or less.
The sixth point is not on the line. The network can only rebuild points on the line, so it rebuilds (2, 0.5) as (0.98, 1.96), a point on the line. The error is 1.591, more than a thousand times the others. This is the bank clerk’s failed copy, in numbers.
The Larkfield network works the same way, with more numbers:
Text description of the diagram
The diagram is a chain of five boxes from left to right. The first box is the input: 8 numbers, the behaviour features of one payment. The second box is the encoder, which goes from 8 numbers to 16, then to 3. The third box is the code: 3 numbers, the squeeze. The fourth box is the decoder, from 3 numbers to 16, then to 8. The fifth box is the output: 8 numbers, the rebuilt payment. Two dotted lines, from the input and from the output, meet at a last box: the reconstruction error. It is the mean of the squared differences between in and out, and it is the anomaly score.
The Larkfield autoencoder
Section titled “The Larkfield autoencoder”The network gets the same eight behaviour features as Isolation Forest in Module 4. Examples are “a new device for this customer” and “payments by this customer in the last hour”. Another is the amount compared with the customer’s usual amount. Each feature is first scaled: we subtract its mean and divide by its spread, so that every feature has a similar size. Without scaling, a feature with large numbers would decide the error alone.
It learns from the reference months, June to August 2025: 173,271 payments. It then scores the 169,803 payments of the evaluation months, September to November. This is the same split as every row of the comparison table.
Training
Section titled “Training”Training means: show the network a group of payments, measure its error, and change its weights a little to make the error smaller. A weight is one of the numbers inside the network that training changes. This network has 395 weights. One pass over all the training payments is an epoch. The notebook trains for 10 epochs, in groups of 512 payments:
def train_autoencoder(X, hidden=16, code_size=3, epochs=10, seed=0): """A small autoencoder: 8 numbers -> 16 -> 3 -> 16 -> 8. It learns from the reference payments only.""" torch.manual_seed(seed) d = X.shape[1] net = torch.nn.Sequential(torch.nn.Linear(d, hidden), torch.nn.ReLU(), torch.nn.Linear(hidden, code_size), torch.nn.ReLU(), torch.nn.Linear(code_size, hidden), torch.nn.ReLU(), torch.nn.Linear(hidden, d)) opt = torch.optim.Adam(net.parameters(), lr=1e-3) gen = torch.Generator().manual_seed(seed) for epoch in range(epochs): perm = torch.randperm(len(X), generator=gen) for i in range(0, len(X), 512): batch = X[perm[i:i + 512]] loss = ((net(batch) - batch) ** 2).mean() opt.zero_grad(); loss.backward(); opt.step() print(f"epoch {epoch + 1}: loss {loss.item():.4f}") return net
net = train_autoencoder(X_ref)with torch.no_grad(): error = ((net(X_eval) - X_eval) ** 2).mean(dim=1).numpy()The loss is the training error. The cell prints the loss of the last group of each epoch:
epoch 1: loss 0.2376epoch 2: loss 0.1660epoch 3: loss 0.0596epoch 4: loss 0.0489epoch 5: loss 0.0466epoch 6: loss 0.0526epoch 7: loss 0.0819epoch 8: loss 0.0446epoch 9: loss 0.0226epoch 10: loss 0.0452The loss goes down, with some noise from one group to the next. The training took less than a second on the test machine, with one CPU thread. The last line computes the error of every evaluation payment: this is the score.
The notebook sets torch.set_num_threads(1). With several threads, the order of the sums can change, so the numbers can change a little from run to run.
One payment, rebuilt
Section titled “One payment, rebuilt”Look at customer C-01234 from the first lesson. Her first planted payment from Portugal came on 3 October 2025 at 02:14:07. Her last usual Saturday payment came on 27 September. Here are both, feature by feature. The “error” column is the squared difference after scaling, so the features can be compared:
| Feature | 27 Sept: value | rebuilt | error | 3 Oct: value | rebuilt | error |
|---|---|---|---|---|---|---|
| amount vs usual (log) | −0.06 | −0.06 | 0.00 | 0.44 | 0.10 | 0.35 |
| new country | 0 | 0.00 | 0.00 | 1 | 1.01 | 0.01 |
| new device | 0 | 0.00 | 0.00 | 1 | 0.75 | 1.49 |
| minutes since last (log) | 9.22 | 9.26 | 0.00 | 9.01 | 5.95 | 1.40 |
| payments in the last hour | 1 | 0.99 | 0.00 | 1 | 1.28 | 0.08 |
| device payments in the last hour | 1 | 0.97 | 0.00 | 1 | −0.61 | 0.48 |
| night | 0 | 0.00 | 0.00 | 1 | 0.94 | 0.11 |
| business account | 0 | 0.01 | 0.00 | 0 | −0.24 | 0.32 |
| Reconstruction error (mean) | 0.000 | 0.530 |
The usual payment is rebuilt almost exactly. Its error is the 1,757th largest of its day, so nobody reviews it.
The planted payment is harder. Its error is 0.530, the 43rd largest of 3 October, so it is in Sofia’s 50 slots. The largest errors are on the new device and on the time since the last payment. The network rebuilds “new country” and “night” well: the reference months also have payments like that, from trips and from the summer’s planted fraud. Nobody removed them before training. An autoencoder learns what is common, not what is honest.
The errors of all payments
Section titled “The errors of all payments”Now look at all the evaluation payments. The errors cover a huge range, from under 0.0001 to about 10, so the charts show the logarithm of the error. On this scale, −3 means 0.001, −2 means 0.01, and 0 means 1. The first chart shows 4,000 honest payments chosen at random (of 166,986). The second shows all 2,817 planted payments.
Text description of the chart
The x-axis is the base-10 logarithm of the reconstruction error, from −4.25 to 1.25. The y-axis counts honest payments. Most payments are on the left, between −4 and −3, which means errors between 0.0001 and 0.001. A smaller group lies between −3 and −1. Very few are to the right of the dashed line at −0.24. That line is the error of the 50th alert on a median day. Of the 4,000 payments, 59 are above it.
Text description of the chart
The x-axis is the same as in the chart above, from −4.25 to 1.25. The y-axis counts planted payments. Most planted payments lie between −0.5 and 1, which means errors between about 0.3 and 10. The tallest bars are just right of 0. A tail goes left, down to about −4: some planted payments are rebuilt as well as normal ones. Of the 2,817, 2,243 are above the dashed line at −0.24.
The two shapes are very different: the median error is 0.0004 for honest payments and 1.14 for planted ones. So the autoencoder learned something real. But the honest payments are many. Above the median day’s cut, there are 2,243 planted payments and 2,244 honest ones. A small share of a big group is still a big number, and those honest payments take Sofia’s slots.
The honest comparison
Section titled “The honest comparison”The notebook scores the autoencoder and Isolation Forest on the same features, the same split and the same budget of 50 alerts a day:
detector setting alerts per day precision at budget recall cost per week ($) explain one alert (min) autoencoder behaviour features; 8-16-3-16-8, 10 epochs 50.0 0.434 0.701 3958 5Isolation Forest behaviour features; 200 trees 50.0 0.500 0.808 3218 3Tomás’s network lost. Its precision at the budget is 0.434 against 0.500. It finds 70.1% of the planted payments against 80.8%, and it costs 740 dollars more a week. The chart puts it next to the other rows of the comparison table:
- Rules, or the ceiling
- Autoencoder
- Classical machine learning
Text description of the chart
Seven rows, one dot each, on an axis of precision at 50 alerts a day from 0.40 to 0.60. Two rules, ranked: 0.427. The autoencoder of the notebook: 0.434. The same autoencoder with another starting point of its weights: 0.468. One-Class SVM: 0.480. Isolation Forest: 0.500. kNN distance: 0.505. A perfect detector that knows the answers: 0.542, the ceiling. Both autoencoder dots are left of Isolation Forest and kNN, and only a little right of the two rules.
Two runs, two answers
Section titled “Two runs, two answers”The comparison table has a second autoencoder row: 0.468. It is the same network on the same data, with another initialisation: the random numbers that the weights start from. Here they come from another point of the same random sequence. One change of starting point moved precision by 0.034. Isolation Forest, with its random seed fixed, gives 0.500 on every run.
So, for a neural network, one run is not a result. Report the spread of several runs, and compare it with the simpler model. Here even the better run (0.468) stays below Isolation Forest (0.500) and kNN (0.505).
Better on one kind of fraud
Section titled “Better on one kind of fraud”A single row hides detail. The notebook also prints the recall for each kind of planted fraud:
| Kind of fraud | Autoencoder | Isolation Forest |
|---|---|---|
| Account takeover | 0.575 | 0.443 |
| Card testing | 0.503 | 0.718 |
| Stolen card | 0.825 | 0.905 |
The autoencoder finds more account takeovers: a few expensive payments from the customer’s own account. It finds fewer card tests and stolen cards. If account takeovers were Larkfield’s main problem, this could matter. Look at the kinds before you choose a model.
When to use an autoencoder, and what it costs
Section titled “When to use an autoencoder, and what it costs”An autoencoder is a good choice when:
- the data has many related numbers, so that “normal” is a shape in many dimensions: images, sound, sensor readings of a machine, or long sequences;
- you have a lot of normal data to learn from;
- the simpler models were measured first, on the same split, and lost.
At Larkfield, none of these is strong. The eight features were designed by hand, so a forest or a neighbour search already uses them well.
And the network costs more, in four ways:
- More choices. Layer sizes, code size, epochs, learning rate, scaling. Each choice can change the result, as the second run showed.
- More variation. Each initialisation gives another precision, so you must run it several times.
- Harder to explain. The comparison table assumes 5 minutes to explain one alert, against 3 for Isolation Forest. These minutes are the course’s assumption. The per-feature errors in the table above help, but a reviewer still sees a score from 395 weights.
- More to install and run. PyTorch is about 200 MB. Isolation Forest is part of scikit-learn.
The squeeze also has a trap. If the code is as wide as the input, the network can learn to copy every input. Then every error is small. Your second task in the notebook measures whether that happens here after 10 epochs.
Common mistake
Give the network the raw columns
Symptom: Tomás trains the autoencoder on the three raw columns: the amount, the hour and the account type. “A neural network finds its own features,” he says. Its precision at the budget is 0.074 (comparison table, measured). Isolation Forest on the same raw columns reaches 0.208.
Cause: the raw columns do not carry the fraud. A thief keeps the amounts small, as customer C-01234’s eight payments in Portugal showed. On the planted payments, 49% of the error came from the amount and 50% from the account type. Business customers make large payments, so the network flags them. The signal is in the customer’s own past: a new device, many payments in one hour. No column of the raw table holds that past.
Fix: give the network the behaviour features. Then it reaches 0.434 (0.468 with another start). Features decide more than the algorithm, for a network too.
Consequence in AI work: a deep model does not discover a customer’s history if you do not give it the history. A bigger network on the wrong inputs only fits the wrong inputs better.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain in your own words why the reconstruction error of an autoencoder can be an anomaly score, and why it needs no labels.
Show the answer
The network learns to rebuild its training data through a small code. It keeps only the patterns that are common in that data, which is mostly normal. So it rebuilds normal inputs well and unusual inputs badly. The size of the failure, the reconstruction error, says how unusual an input is. Training only asks the network to give back its own input, so it never needs to know which rows are fraud.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: an honest customer pays from Portugal on a holiday, at night, on a new phone. Will the autoencoder give this payment a small or a large error? What does that tell you about the score?
Show the answer
A large error, probably. New country, new device and night together are not common in the reference months, so the network cannot rebuild them well. The error measures how unusual a payment is, not whether it is fraud. That is why Sofia’s team still reviews each alert, and why honest trips and new phones take some of the 50 slots.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: “The comparison table shows 0.468 for the autoencoder. With a few more runs, I will find a starting point that beats Isolation Forest.” What is wrong with this plan?
Show the answer
Choosing the best of many runs on the evaluation months tunes the model to the answer key. That best run’s precision is too optimistic for new months. Report the spread of several runs instead: 0.434 and 0.468 here, both below Isolation Forest (0.500) and kNN (0.505). Also count the cost: more choices, more to install, and 5 minutes to explain an alert instead of 3 (the course’s assumption).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An autoencoder squeezes its input into a small code and rebuilds it. It learns what is common, so normal data is rebuilt well.
- The reconstruction error is the anomaly score. In the tiny example, the five normal points had errors of 0.001 or less, and the point off the line had 1.591.
- On Larkfield’s payments, the honest median error was 0.0004 and the planted median 1.14. But above the cut of a median day, there were as many honest payments as planted ones.
- On the same features, split and budget, the autoencoder scored 0.434 (0.468 with another start). Isolation Forest scored 0.500 and kNN 0.505. It found more account takeovers, but fewer card tests and stolen cards.
- A neural network costs more choices, more runs, more to explain and more to install. Measure the simpler rows first.
You can now build a detector from a network that rebuilds normal numbers. But some of Larkfield’s unusual events are not numbers at all: the help desk’s service writes sentences in its logs. Next, you find new kinds of log message, first with a cheap template and then with embeddings.