Skip to content
Français

Where an LLM helps, and where it does not

  • Lesson 27
  • 30 min
  • Module 6: Deep and embedding methods

Where we are

We can now find unusual events in numbers and in text. Tomás asks why we do not give all the events to an LLM. In this lesson, we see why an LLM is a poor detector, and use it after detection to summarise an alert group for Sofia, checking every summary against the data.

Lesson 27 of 38 · Anomaly Detection: Find What Is UnusualWhere an LLM helps, and where it does not
Course overview

What you will be able to do

Use a language model where it helps: after the detector, to turn the facts of one alert into a short summary for the reviewer. Check every summary against its facts with code, and know what that check cannot see.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook replays the 20 recorded summaries and runs the check on each one. It needs pandas and NumPy, and a 22 kB file. No account, no key and no model call. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

A browser and a free Google or Kaggle account for the notebook, or the local setup from Meet the Larkfield data. The model’s answers were recorded once, so you need no key and no paid service.

Tested with Python 3.12.13 in Jupyter on macOS. Colab, Kaggle and Windows were not tested.

Think of a hospital laboratory. A machine tests thousands of blood samples a day. It is fast, cheap per sample and always the same. It marks the few samples with values outside the normal range.

Then an assistant writes a short note for the doctor about each marked sample: “Potassium 6.1, above the normal range. Last month: 4.2. Check the kidney results first.” The doctor reads the note, looks at the numbers and decides.

Nobody asks the assistant to read every sample in the laboratory. The assistant is slow and expensive, and that is not his job. His job is to make the marked samples easy to understand. And the doctor trusts the note only because the numbers are written next to it.

A large language model, or LLM, is a model trained on a great deal of text. Given some text, it writes more text: an answer, a summary, a list. It writes fluent, confident sentences about almost anything.

Tomás has an idea: “Why build detectors at all? Send every payment to an LLM and ask: is this unusual?” It sounds modern. Look at what it would mean for Larkfield.

The 20 calls recorded for this lesson give real averages per call. The mean was 324.8 tokens in, 166.4 tokens out and US$0.000116, and the median time was 3.08 seconds. A token is a piece of a word, the unit in which models read text and are paid.

Now multiply by the 169,803 payments of the evaluation months. This is a constructed estimate, not a measurement:

One call (measured) Every payment, Sept to Nov (constructed)
Tokens about 491 about 83 million
Cost US$0.000116 about US$19.65
Time, one call at a time 3.08 s (median) about 145 hours, 1.6 hours for each day

With a small model, the money is not the problem. Prices change: check the provider’s current page. Many calls can also run at the same time. The real problems are elsewhere:

  1. One payment says nothing alone. “24.27 dollars in Portugal” is normal for one customer and strange for another. The model needs the customer’s past. To give it the past, we must compute the behaviour features first. Once they exist, Isolation Forest learns and scores 343,074 payments in 1.4 seconds on the test machine.
  2. There is no measurement. Every detector in this course has a row in the comparison table: the same split, the same budget, precision and cost. An LLM detector has no row. Nobody measured whether it would choose better alerts. A fluent “this looks unusual” is not a precision.
  3. It may answer differently. Ask the same question twice, and the words, and sometimes the answer, can change.
  4. The data leaves the shop. Every payment of every customer would go to an outside service. That needs a reason and an agreement, not only an API key.

So the LLM is a poor detector here. The detector finds the alert. The question is whether the LLM can help after that.

The idea: the detector finds, the LLM explains

Section titled “The idea: the detector finds, the LLM explains”

Sofia’s team reviews 50 alerts a day. Each alert must be understood quickly. The comparison table assumes 3 minutes to explain one Isolation Forest alert (an assumption of the course). A short, correct summary could save part of those minutes. That is the job of the hospital assistant: make the marked cases easy to read.

A flow from left to right. All payments, about 1,866 a day. Then the detector: Isolation Forest finds the 50 most unusual. Then alert groups, one per customer and day. Then a facts table, computed by code. Then the LLM: a summary and a first check. Then a check by code: is every number and country in the facts? Yes leads to Sofia decides, with the summary and the facts. No leads to show the facts only, and log the problem, then also to Sofia.
The LLM comes after detection, and code checks its words before Sofia reads them. Sofia always sees the facts too.
Text description of the diagram

The diagram is a flow from left to right. It starts with all payments, about 1,866 a day in the evaluation months. The detector, Isolation Forest, finds the 50 most unusual of each day. The alerts are put in alert groups, one for each customer and day. Code computes a facts table for each group. The LLM gets only that table and writes a summary and a first check. Then code checks the summary: is every number and every country in the facts? If yes, Sofia reads the summary with the facts and decides. If no, she gets the facts only, and the problem is logged.

Three choices in this design matter:

  • An alert group, not a payment. The alerts of one customer on one day are one thing to look at. Ten payments of a stolen card are one story, not ten.
  • Facts, not the raw table. Code computes about 20 facts, such as the amounts, the times, the countries and the customer’s usual amount. The model only puts them into words. It gets no names and no card numbers.
  • The model does not decide. Its instructions say: use only the facts given, and do not invent numbers or reasons. Do not say that it is fraud. Sofia decides.

For this lesson, we chose 20 alert groups with a fixed random seed. They come from Isolation Forest’s 50 alerts a day in the evaluation months. Each group went once to a small hosted model, through an Azure deployment named chat-small. The model reported its name as gpt-6-luna-2026-09-22. The 20 calls used 6,496 tokens in and 3,328 out, and cost US$0.0023 in total. The notebook replays these recorded answers; if you call a model yourself, your answers will differ.

Here is one group, a planted stolen card. Part of its facts table:

Fact Value
customer, day C-02349, 2025-10-18
payments in the group 10
first and last payment 02:25 and 02:46
amounts (dollars) 15.68, 26.10, 25.50, 8.45, 14.15, 23.81, 52.71, 8.10, 26.17, 12.30
total 212.97
countries, currencies DE, EUR
customer’s usual amount 44.35
new country, new device yes, yes

And the model’s summary, as recorded:

Customer C-02349 made 10 payments totaling CAD 212.97 between 02:25 and 02:46 on 2025-10-18. They were in EUR to Germany, a new country for the customer, and used a new device. The payments covered gift cards, home, and kitchen.
First check: Review whether the customer recognizes the new device and the payments to Germany.

It is short, correct and useful. Sofia sees the story in ten seconds: night, a new country, a new device, ten payments in about 20 minutes.

The model never saw the answer key. We did, after the recording:

What the group really was Groups
Planted card testing 9
Planted stolen card 2
Planted account takeover 1
Honest bulk order of a business customer 5
Honest: a new device from the app launch 1
Honest: a customer of the new region 1
Honest: an ordinary large payment 1

Compare two summaries. A planted card test:

One payment of CAD 1.84 for gift cards was made in JPY in Japan. This is a new country and device for the customer; the device had 35 payments in one hour.
First check: Review the device’s activity in that hour.

An honest bulk order:

Customer C-03875 made 7 payments totaling CAD 2,980.34 between 19:30 and 19:38 on 2025-09-23. The payments were in CAD, in Canada, and covered garden and tools; each was above the customer’s usual amount of CAD 190.13.
First check: Review the seven payment amounts and timing against the customer’s usual activity.

Both are calm and factual. The model did not call the honest order “fraud” or the card test “safe”. That is what we asked for. A summary that sounds sure about fraud would push Sofia to block honest business customers.

The calls were not all equally fast:

The time of each of the 20 recorded calls (measured). The median was 3.08 seconds, the slowest 12.19 seconds. A detector scores a whole day of payments in less time than one call.
Text description of the chart

One row of 20 dots on an axis of seconds from 0 to 13. Most dots are between 1.7 and 4.5 seconds. Two are slower: 7.52 and 12.19 seconds. A tick marks the median, 3.08 seconds.

A model can write a number that is not in its facts. It can name the wrong country, or forget the first check. Sofia must not have to verify each summary by hand, so code checks it before she reads it. The notebook’s check has four rules:

  1. Every number in the summary must be a number of the facts, or the count of a list in the facts.
  2. Every two-letter country code in the summary must be in the facts.
  3. The summary must have a line that starts with “First check:”.
  4. The summary must not say that the payment is fraud.

Here is rule 1 on the stolen-card summary, number by number:

Number in the summary Where it is in the facts
10 payments in the group: 10
212.97 total: 212.97
02:25 first payment: 02:25
02:46 last payment: 02:46
2025-10-18 day: 2025-10-18

Every number has a source, so the summary passes. “Germany” is a word, not a code. The facts say DE, and the check does not test words. On all 20 summaries, the notebook prints:

Summaries with a problem: 0 of 20

Zero problems is good news only if the check can find a problem. So the notebook plants one: it changes “1.03” into “10.30” in the first summary, and runs the check again:

Customer C-06825 made one gift-card payment of CAD 10.30 in EUR in the Netherlands on 2025-09-03, below their usual amount of CAD 54.15. The country and device are both new for this customer.
['number 10.3 is not in the facts']

The check found the planted mistake. This is the same idea as the planted anomalies of this course. Trust a detector, or a check, only after you have seen it catch a mistake that you planted.

The first version of the check had no idea that a summary may write the date in words. It flagged 3 of the 20 summaries:

C-04445 2025-09-25 ['number 25 is not in the facts'] | Customer C-04445 made one CAD 3.56 gift-card payment in EUR from Italy on Septem
C-07286 2025-09-30 ['number 30 is not in the facts'] | Customer C-07286 made one CAD 1.39 gift-card payment in USD in the US on Septemb
C-12874 2025-09-30 ['number 30 is not in the facts'] | On September 30, customer C-12874 made one US gift-card payment in USD, totaling

“September 25” is the day of the facts, so all three were false alarms about the summaries. The fixed check also accepts each part of the date (2025, 9 and 25 in 2025-09-25) and each part of a time (02 and 25 in 02:25). A check that is too strict has the same problem as a detector with too many alerts: people stop reading its warnings.

The number check tests numbers. It cannot test a comparison. Look at this recorded summary, of an honest bulk order:

Customer C-13297 made six payments totaling CAD 4,658.72 between 10:43 and 10:49 on 2025-09-20. The payments were all in CAD and from Canada, across furniture, garden, and tools; the amounts were above the customer’s usual CAD 441.75. The country and device were not new for this customer.
First check: Review the six payment amounts and their timing against the customer’s usual payment pattern.

Every number is in the facts, so the check passed it. But the amounts were 725.43, 972.08, 798.71, 383.17, 987.16 and 792.17. One of the six is below the usual 441.75. “The amounts were above” is not true for all of them.

It is a small error, and it would not change Sofia’s decision. But it shows the limit: code can check what the summary cites, not every claim it makes. Two protections help:

  • Show the facts table next to the summary. Sofia can see the amounts herself in a second.
  • Add checks for the claims that matter. For example: if a summary says the amounts were above the usual one, test each amount. Your first task in the notebook writes this check.

Common mistake

Trust a summary because the check passed

Symptom: Omar, Larkfield’s product manager, likes the summaries. “The check found 0 problems in 20,” he says. “Let’s show only the summary, and hide the facts table. The screen is cleaner.”

Cause: the check proves that every number in a summary comes from the facts. It does not prove that every sentence is true. One of the 20 recorded summaries said the amounts of customer C-13297 were above the usual amount, but one amount was below it. Without the facts on the screen, nobody can see that.

Fix: keep the facts next to the summary. Test the check with planted mistakes. Add a check for each kind of claim that a reviewer acts on, such as “above the usual amount” or “a new country”.

Consequence in AI work: an automatic check makes a fluent text look safe. Know exactly what your check tests, and say it in the screen’s help text, so nobody reads “0 problems” as “0 errors”.

When to use an LLM here, and what it costs

Section titled “When to use an LLM here, and what it costs”

Use it when:

  • the detector has already found a few alerts a day, and people must read each one;
  • the facts can be computed by code, and the model only puts them into words;
  • you can check its output, and show the facts next to it.

Do not use it:

  • as the detector over every event: it has no row in the comparison table, and it needs the features that the detector already uses;
  • when the reviewer must trust it blindly, without the facts;
  • to decide: blocking a card is Sofia’s decision, with its costs.

The costs in this lesson are small: US$0.0023 for 20 calls, and a median of 3 seconds each. For 50 alerts a day, that is about 50 calls, less than one cent a day with this small model. The larger costs are the check you must write and keep up to date, and the time to decide what the model may see.

The notebook ends with two tasks, with checks:

  1. Find the summary that says all the amounts were above the customer’s usual amount when one was not. Write the check yourself, with amounts_cad and customer_usual_amount_cad.
  2. Estimate the cost and the time of Tomás’s plan, the LLM on every payment, from the measured averages per call.

Check your understanding

Explain why an LLM that reads every payment is a poor detector for Larkfield, even if each call is cheap.

Show the answer

One payment alone says little. The model needs the customer’s past, so the behaviour features must be computed first. With them, Isolation Forest scores all the payments in about a second. An LLM detector has no row in the comparison table, so nobody knows its precision at the budget. Its answers can change between runs. And every customer’s payment would go to an outside service. The estimate for the evaluation months was about 145 hours one call at a time; the money (about 20 dollars) was the smallest problem.

Check your understanding

Predict: you change one amount in a summary from 26.17 to 26.71, and run the check. What happens? And what happens if you change “a new device” to “a known device”?

Show the answer

The first change is caught. The check reports “number 26.71 is not in the facts”. The second change is not caught. The check tests numbers and country codes, not the meaning of words. So the facts table stays next to the summary. And claims that matter, such as “a new device”, need their own check.

Check your understanding

Sofia's team has a **runbook**: its written list of steps for each kind of alert. Sofia asks: “Can the model's first check replace my team's runbook?” What do you answer, with what you saw in the 20 summaries?

Show the answer

No. The first checks were short and sensible, for example “Review the new device” or “Review the device’s activity in that hour”. But they only repeat the facts in other words; the model knows nothing about Larkfield’s procedures. A runbook is the team’s agreed first step, owned and kept up to date by people. The summary can point to the right facts; the team’s runbook says what to do.

Recap

  • As a detector, an LLM needs the features that a cheaper detector already uses. It has no measured precision, and its answers can change. An estimate for every payment of the evaluation months: about 145 hours one call at a time, about 20 dollars.
  • After detection, it helps: it turns about 20 facts of one alert group into a short summary and a first check. The detector finds; the LLM explains; Sofia decides.
  • 20 recorded summaries cost US$0.0023, with a median of 3.08 seconds a call. Code checked every number and country: 0 problems. A planted mistake was caught. The first, too strict check flagged 3 dates written in words.
  • The check cannot test a comparison: one summary said all amounts were above the usual one, and one was not. Keep the facts next to the summary.

You have now tried rules, statistics, classical models, a neural network, embeddings and an LLM on the same data. But the most advanced method was never the winner on its own. Next, you read the comparison table for the cases where a rule or a simple statistic was the better answer.

Next: When not to use machine learning