Where an LLM helps, and where it does not
- Lesson 27
- 30 min
- Module 6: Deep and embedding methods
Where we are
We can now find unusual events in numbers and in text. Tomás asks why we do not give all the events to an LLM. In this lesson, we see why an LLM is a poor detector, and use it after detection to summarise an alert group for Sofia, checking every summary against the data.
Lesson 27 of 38 · Anomaly Detection: Find What Is UnusualWhere an LLM helps, and where it does not
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Use a language model where it helps: after the detector, to turn the facts of one alert into a short summary for the reviewer. Check every summary against its facts with code, and know what that check cannot see.
NotebookDo this lesson in a notebook
The notebook replays the 20 recorded summaries and runs the check on each one. It needs pandas and NumPy, and a 22 kB file. No account, no key and no model call. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The alert budget of 50 alerts a day, from The payment that did not fit.
- The behaviour features and Isolation Forest, from Isolation Forest.
- It helps, but it is not required, to know how an application calls a language model. What an LLM API is in Building Reliable Applications with LLM APIs teaches it.
What you need on your computer
A browser and a free Google or Kaggle account for the notebook, or the local setup from Meet the Larkfield data. The model’s answers were recorded once, so you need no key and no paid service.
Tested with Python 3.12.13 in Jupyter on macOS. Colab, Kaggle and Windows were not tested.
The laboratory and the note
Section titled “The laboratory and the note”Think of a hospital laboratory. A machine tests thousands of blood samples a day. It is fast, cheap per sample and always the same. It marks the few samples with values outside the normal range.
Then an assistant writes a short note for the doctor about each marked sample: “Potassium 6.1, above the normal range. Last month: 4.2. Check the kidney results first.” The doctor reads the note, looks at the numbers and decides.
Nobody asks the assistant to read every sample in the laboratory. The assistant is slow and expensive, and that is not his job. His job is to make the marked samples easy to understand. And the doctor trusts the note only because the numbers are written next to it.
Tomás’s question
Section titled “Tomás’s question”A large language model, or LLM, is a model trained on a great deal of text. Given some text, it writes more text: an answer, a summary, a list. It writes fluent, confident sentences about almost anything.
Tomás has an idea: “Why build detectors at all? Send every payment to an LLM and ask: is this unusual?” It sounds modern. Look at what it would mean for Larkfield.
The problem: an LLM as a detector
Section titled “The problem: an LLM as a detector”The 20 calls recorded for this lesson give real averages per call. The mean was 324.8 tokens in, 166.4 tokens out and US$0.000116, and the median time was 3.08 seconds. A token is a piece of a word, the unit in which models read text and are paid.
Now multiply by the 169,803 payments of the evaluation months. This is a constructed estimate, not a measurement:
| One call (measured) | Every payment, Sept to Nov (constructed) | |
|---|---|---|
| Tokens | about 491 | about 83 million |
| Cost | US$0.000116 | about US$19.65 |
| Time, one call at a time | 3.08 s (median) | about 145 hours, 1.6 hours for each day |
With a small model, the money is not the problem. Prices change: check the provider’s current page. Many calls can also run at the same time. The real problems are elsewhere:
- One payment says nothing alone. “24.27 dollars in Portugal” is normal for one customer and strange for another. The model needs the customer’s past. To give it the past, we must compute the behaviour features first. Once they exist, Isolation Forest learns and scores 343,074 payments in 1.4 seconds on the test machine.
- There is no measurement. Every detector in this course has a row in the comparison table: the same split, the same budget, precision and cost. An LLM detector has no row. Nobody measured whether it would choose better alerts. A fluent “this looks unusual” is not a precision.
- It may answer differently. Ask the same question twice, and the words, and sometimes the answer, can change.
- The data leaves the shop. Every payment of every customer would go to an outside service. That needs a reason and an agreement, not only an API key.
So the LLM is a poor detector here. The detector finds the alert. The question is whether the LLM can help after that.
The idea: the detector finds, the LLM explains
Section titled “The idea: the detector finds, the LLM explains”Sofia’s team reviews 50 alerts a day. Each alert must be understood quickly. The comparison table assumes 3 minutes to explain one Isolation Forest alert (an assumption of the course). A short, correct summary could save part of those minutes. That is the job of the hospital assistant: make the marked cases easy to read.
Text description of the diagram
The diagram is a flow from left to right. It starts with all payments, about 1,866 a day in the evaluation months. The detector, Isolation Forest, finds the 50 most unusual of each day. The alerts are put in alert groups, one for each customer and day. Code computes a facts table for each group. The LLM gets only that table and writes a summary and a first check. Then code checks the summary: is every number and every country in the facts? If yes, Sofia reads the summary with the facts and decides. If no, she gets the facts only, and the problem is logged.
Three choices in this design matter:
- An alert group, not a payment. The alerts of one customer on one day are one thing to look at. Ten payments of a stolen card are one story, not ten.
- Facts, not the raw table. Code computes about 20 facts, such as the amounts, the times, the countries and the customer’s usual amount. The model only puts them into words. It gets no names and no card numbers.
- The model does not decide. Its instructions say: use only the facts given, and do not invent numbers or reasons. Do not say that it is fraud. Sofia decides.
Twenty recorded summaries
Section titled “Twenty recorded summaries”For this lesson, we chose 20 alert groups with a fixed random seed. They come from Isolation Forest’s 50 alerts a day in the evaluation months. Each group went once to a small hosted model, through an Azure deployment named chat-small. The model reported its name as gpt-6-luna-2026-09-22. The 20 calls used 6,496 tokens in and 3,328 out, and cost US$0.0023 in total. The notebook replays these recorded answers; if you call a model yourself, your answers will differ.
Here is one group, a planted stolen card. Part of its facts table:
| Fact | Value |
|---|---|
| customer, day | C-02349, 2025-10-18 |
| payments in the group | 10 |
| first and last payment | 02:25 and 02:46 |
| amounts (dollars) | 15.68, 26.10, 25.50, 8.45, 14.15, 23.81, 52.71, 8.10, 26.17, 12.30 |
| total | 212.97 |
| countries, currencies | DE, EUR |
| customer’s usual amount | 44.35 |
| new country, new device | yes, yes |
And the model’s summary, as recorded:
Customer C-02349 made 10 payments totaling CAD 212.97 between 02:25 and 02:46 on 2025-10-18. They were in EUR to Germany, a new country for the customer, and used a new device. The payments covered gift cards, home, and kitchen.
First check: Review whether the customer recognizes the new device and the payments to Germany.It is short, correct and useful. Sofia sees the story in ten seconds: night, a new country, a new device, ten payments in about 20 minutes.
What the 20 groups really were
Section titled “What the 20 groups really were”The model never saw the answer key. We did, after the recording:
| What the group really was | Groups |
|---|---|
| Planted card testing | 9 |
| Planted stolen card | 2 |
| Planted account takeover | 1 |
| Honest bulk order of a business customer | 5 |
| Honest: a new device from the app launch | 1 |
| Honest: a customer of the new region | 1 |
| Honest: an ordinary large payment | 1 |
Compare two summaries. A planted card test:
One payment of CAD 1.84 for gift cards was made in JPY in Japan. This is a new country and device for the customer; the device had 35 payments in one hour.
First check: Review the device’s activity in that hour.An honest bulk order:
Customer C-03875 made 7 payments totaling CAD 2,980.34 between 19:30 and 19:38 on 2025-09-23. The payments were in CAD, in Canada, and covered garden and tools; each was above the customer’s usual amount of CAD 190.13.
First check: Review the seven payment amounts and timing against the customer’s usual activity.Both are calm and factual. The model did not call the honest order “fraud” or the card test “safe”. That is what we asked for. A summary that sounds sure about fraud would push Sofia to block honest business customers.
The calls were not all equally fast:
Text description of the chart
One row of 20 dots on an axis of seconds from 0 to 13. Most dots are between 1.7 and 4.5 seconds. Two are slower: 7.52 and 12.19 seconds. A tick marks the median, 3.08 seconds.
Check every summary against its facts
Section titled “Check every summary against its facts”A model can write a number that is not in its facts. It can name the wrong country, or forget the first check. Sofia must not have to verify each summary by hand, so code checks it before she reads it. The notebook’s check has four rules:
- Every number in the summary must be a number of the facts, or the count of a list in the facts.
- Every two-letter country code in the summary must be in the facts.
- The summary must have a line that starts with “First check:”.
- The summary must not say that the payment is fraud.
Here is rule 1 on the stolen-card summary, number by number:
| Number in the summary | Where it is in the facts |
|---|---|
| 10 | payments in the group: 10 |
| 212.97 | total: 212.97 |
| 02:25 | first payment: 02:25 |
| 02:46 | last payment: 02:46 |
| 2025-10-18 | day: 2025-10-18 |
Every number has a source, so the summary passes. “Germany” is a word, not a code. The facts say DE, and the check does not test words. On all 20 summaries, the notebook prints:
Summaries with a problem: 0 of 20Test the check
Section titled “Test the check”Zero problems is good news only if the check can find a problem. So the notebook plants one: it changes “1.03” into “10.30” in the first summary, and runs the check again:
Customer C-06825 made one gift-card payment of CAD 10.30 in EUR in the Netherlands on 2025-09-03, below their usual amount of CAD 54.15. The country and device are both new for this customer.['number 10.3 is not in the facts']The check found the planted mistake. This is the same idea as the planted anomalies of this course. Trust a detector, or a check, only after you have seen it catch a mistake that you planted.
A check that is too strict
Section titled “A check that is too strict”The first version of the check had no idea that a summary may write the date in words. It flagged 3 of the 20 summaries:
C-04445 2025-09-25 ['number 25 is not in the facts'] | Customer C-04445 made one CAD 3.56 gift-card payment in EUR from Italy on SeptemC-07286 2025-09-30 ['number 30 is not in the facts'] | Customer C-07286 made one CAD 1.39 gift-card payment in USD in the US on SeptembC-12874 2025-09-30 ['number 30 is not in the facts'] | On September 30, customer C-12874 made one US gift-card payment in USD, totaling“September 25” is the day of the facts, so all three were false alarms about the summaries. The fixed check also accepts each part of the date (2025, 9 and 25 in 2025-09-25) and each part of a time (02 and 25 in 02:25). A check that is too strict has the same problem as a detector with too many alerts: people stop reading its warnings.
What the check cannot see
Section titled “What the check cannot see”The number check tests numbers. It cannot test a comparison. Look at this recorded summary, of an honest bulk order:
Customer C-13297 made six payments totaling CAD 4,658.72 between 10:43 and 10:49 on 2025-09-20. The payments were all in CAD and from Canada, across furniture, garden, and tools; the amounts were above the customer’s usual CAD 441.75. The country and device were not new for this customer.
First check: Review the six payment amounts and their timing against the customer’s usual payment pattern.Every number is in the facts, so the check passed it. But the amounts were 725.43, 972.08, 798.71, 383.17, 987.16 and 792.17. One of the six is below the usual 441.75. “The amounts were above” is not true for all of them.
It is a small error, and it would not change Sofia’s decision. But it shows the limit: code can check what the summary cites, not every claim it makes. Two protections help:
- Show the facts table next to the summary. Sofia can see the amounts herself in a second.
- Add checks for the claims that matter. For example: if a summary says the amounts were above the usual one, test each amount. Your first task in the notebook writes this check.
Common mistake
Trust a summary because the check passed
Symptom: Omar, Larkfield’s product manager, likes the summaries. “The check found 0 problems in 20,” he says. “Let’s show only the summary, and hide the facts table. The screen is cleaner.”
Cause: the check proves that every number in a summary comes from the facts. It does not prove that every sentence is true. One of the 20 recorded summaries said the amounts of customer C-13297 were above the usual amount, but one amount was below it. Without the facts on the screen, nobody can see that.
Fix: keep the facts next to the summary. Test the check with planted mistakes. Add a check for each kind of claim that a reviewer acts on, such as “above the usual amount” or “a new country”.
Consequence in AI work: an automatic check makes a fluent text look safe. Know exactly what your check tests, and say it in the screen’s help text, so nobody reads “0 problems” as “0 errors”.
When to use an LLM here, and what it costs
Section titled “When to use an LLM here, and what it costs”Use it when:
- the detector has already found a few alerts a day, and people must read each one;
- the facts can be computed by code, and the model only puts them into words;
- you can check its output, and show the facts next to it.
Do not use it:
- as the detector over every event: it has no row in the comparison table, and it needs the features that the detector already uses;
- when the reviewer must trust it blindly, without the facts;
- to decide: blocking a card is Sofia’s decision, with its costs.
The costs in this lesson are small: US$0.0023 for 20 calls, and a median of 3 seconds each. For 50 alerts a day, that is about 50 calls, less than one cent a day with this small model. The larger costs are the check you must write and keep up to date, and the time to decide what the model may see.
Your turn in the notebook
Section titled “Your turn in the notebook”The notebook ends with two tasks, with checks:
- Find the summary that says all the amounts were above the customer’s usual amount when one was not. Write the check yourself, with
amounts_cadandcustomer_usual_amount_cad. - Estimate the cost and the time of Tomás’s plan, the LLM on every payment, from the measured averages per call.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain why an LLM that reads every payment is a poor detector for Larkfield, even if each call is cheap.
Show the answer
One payment alone says little. The model needs the customer’s past, so the behaviour features must be computed first. With them, Isolation Forest scores all the payments in about a second. An LLM detector has no row in the comparison table, so nobody knows its precision at the budget. Its answers can change between runs. And every customer’s payment would go to an outside service. The estimate for the evaluation months was about 145 hours one call at a time; the money (about 20 dollars) was the smallest problem.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you change one amount in a summary from 26.17 to 26.71, and run the check. What happens? And what happens if you change “a new device” to “a known device”?
Show the answer
The first change is caught. The check reports “number 26.71 is not in the facts”. The second change is not caught. The check tests numbers and country codes, not the meaning of words. So the facts table stays next to the summary. And claims that matter, such as “a new device”, need their own check.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Sofia's team has a **runbook**: its written list of steps for each kind of alert. Sofia asks: “Can the model's first check replace my team's runbook?” What do you answer, with what you saw in the 20 summaries?
Show the answer
No. The first checks were short and sensible, for example “Review the new device” or “Review the device’s activity in that hour”. But they only repeat the facts in other words; the model knows nothing about Larkfield’s procedures. A runbook is the team’s agreed first step, owned and kept up to date by people. The summary can point to the right facts; the team’s runbook says what to do.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- As a detector, an LLM needs the features that a cheaper detector already uses. It has no measured precision, and its answers can change. An estimate for every payment of the evaluation months: about 145 hours one call at a time, about 20 dollars.
- After detection, it helps: it turns about 20 facts of one alert group into a short summary and a first check. The detector finds; the LLM explains; Sofia decides.
- 20 recorded summaries cost US$0.0023, with a median of 3.08 seconds a call. Code checked every number and country: 0 problems. A planted mistake was caught. The first, too strict check flagged 3 dates written in words.
- The check cannot test a comparison: one summary said all amounts were above the usual one, and one was not. Keep the facts next to the summary.
You have now tried rules, statistics, classical models, a neural network, embeddings and an LLM on the same data. But the most advanced method was never the winner on its own. Next, you read the comparison table for the cases where a rule or a simple statistic was the better answer.