Unusual text and logs with embeddings
- Lesson 26
- 40 min
- Module 6: Deep and embedding methods
Where we are
Our detectors work on numbers. Log lines are text, and a new kind of error is a new kind of message. In this lesson, we first turn log lines into templates, which is cheap and often enough, and then use embeddings to find messages far from every known one.
Lesson 26 of 38 · Anomaly Detection: Find What Is UnusualUnusual text and logs with embeddings
1. What "unusual" means
2. Simple baselines that are hard to beat
3. Statistical process control and distribution drift
4. Distance, density and isolation
5. Anomalies in time
6. Deep and embedding methods
7. Evaluate with few labels
8. Learn from the reviewers
9. Run a detector in production
What you will be able to do
Find a new kind of message in a service’s logs. Start with templates, which are cheap and often enough. Then use embeddings to rank the few new messages by how different they are, and learn why “different” is not the same as “dangerous”.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, PyTorch and sentence-transformers, and download the logs (about 2 MB) and a small embedding model (about 91 MB). About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- How to run the course notebooks, from Meet the Larkfield data.
- What “known” and “new” mean for a baseline period, from Normal changes with the hour and the season.
- It helps, but it is not required, to know embeddings. Tokens and embeddings in Deep Learning and Transformers Explained teaches them. So does Embeddings for search in RAG: Building AI That Uses Your Data. This lesson explains the parts it uses.
What you need on your computer
A browser and a free Google or Kaggle account for the notebook, or the local setup from Meet the Larkfield data. The embedding model runs on a CPU. No account and no paid service.
Tested with Python 3.12.13 in Jupyter on macOS, with sentence-transformers 6.2.0 and the model all-MiniLM-L6-v2. Colab, Kaggle and Windows were not tested.
A new sound in the car
Section titled “A new sound in the car”You drive the same car every day. You know its sounds: the engine, the indicator, the hum of the tyres on the motorway. You do not think about them. Then one morning you hear a click when you turn left. You do not know what it means. But you know at once that it is new.
Some new sounds are harmless. A new set of tyres hums in a different way. Some are serious: a click in the steering. Your ear tells you “new”. It does not tell you “dangerous”. To know that, you must look.
A service’s logs work the same way. Most lines are the usual sounds. A new kind of message is the click.
Amira’s logs
Section titled “Amira’s logs”Amira is the developer of the help desk’s ticket-api service. The service writes a log line each time something happens: a request, a slow database query, an email sent, an error. Each line has a time, a level (INFO for normal events, WARN for a warning, ERROR for a failure) and a message. Here are six lines from the first morning of the data, 1 September 2025:
| Time | Level | Message |
|---|---|---|
| 00:01:02 | INFO | GET /tickets/T-472971 200 in 39 ms |
| 00:05:50 | INFO | GET /tickets/T-421326 200 in 58 ms |
| 00:08:43 | WARN | slow query on tickets table took 1434 ms |
| 00:14:16 | INFO | worker 4 picked job J-1945010 (queue depth 9) |
| 00:26:45 | ERROR | upstream timeout calling email-service after 1207 ms |
| 00:28:24 | INFO | sent reply email for ticket T-410889 (2579 bytes) |
The data has 137,013 lines over 60 days, about 2,500 a day. Nobody reads them all. Amira wants one thing from them: “tell me when the service says something it has never said before”.
The problem: almost every line is new
Section titled “The problem: almost every line is new”The simplest idea is to keep every message of the first two weeks, 1 to 14 September. Then flag every later message that is not in that list. Try it on the data. The first two weeks had 24,693 different messages. After them came 108,393 lines, and 91,023 of them had a message never seen before. That is 84%, about 2,126 “new” lines a day.
This detector is useless. The two GET lines above are the same event with another ticket number and another time in milliseconds. The data has 108,502 different messages, but only a few dozen different sentences. The IDs and the numbers make almost every line unique.
Templates: the sentence without its numbers
Section titled “Templates: the sentence without its numbers”The fix is the idea that the car driver uses without thinking. “Click when I turn left” is one sound, on Monday and on Tuesday. A template is a log message with the parts that change replaced by a mask. The mask <id> replaces a ticket, job, request or agent ID, and <num> replaces a number. The notebook does it with four search-and-replace patterns:
MASKS = [(re.compile(r"\b[TJ]-\d+\b"), "<id>"), (re.compile(r"\bq-[0-9a-f]+\b"), "<id>"), (re.compile(r"\b[AK]-\d+\b"), "<id>"), (re.compile(r"\d+(\.\d+)?"), "<num>")]
def template(message): """Replace the parts that change (IDs, numbers) with <id> and <num>.""" for pattern, mask in MASKS: message = pattern.sub(mask, message) return messageHere are the six lines again, as templates:
| Message | Template |
|---|---|
GET /tickets/T-472971 200 in 39 ms |
GET /tickets/<id> <num> in <num> ms |
GET /tickets/T-421326 200 in 58 ms |
GET /tickets/<id> <num> in <num> ms |
slow query on tickets table took 1434 ms |
slow query on tickets table took <num> ms |
worker 4 picked job J-1945010 (queue depth 9) |
worker <num> picked job <id> (queue depth <num>) |
upstream timeout calling email-service after 1207 ms |
upstream timeout calling email-service after <num> ms |
sent reply email for ticket T-410889 (2579 bytes) |
sent reply email for ticket <id> (<num> bytes) |
The two GET lines now have the same template. On all the lines, the notebook prints:
137,013 lines, 108,502 different messages, 36 templates, in 0.24 sThe 108,502 messages are only 36 templates. Templating every line took 0.24 seconds on the test machine, about 2 microseconds a line. Real logging systems often use a log parser, such as Drain, that learns the masks by itself. The idea is the same.
Three new templates
Section titled “Three new templates”Now “new” means “a template that never appeared in the first two weeks”. The notebook finds 33 templates in those weeks and 3 new ones after them:
33 templates in the first two weeks; 3 new ones after them: first_seen linestemplatefeature flag new-reply-editor is on for <num>% of agents 2025-09-16 19:25:26.448 878unhandled exception in update_ticket: KeyError: 'priority' 2025-10-08 14:05:16.494 59cache pool exhausted, request <id> waited <num> ms for a connection 2025-10-20 10:02:19.778 4757From 91,023 “new” lines to 3 new templates. Each one has a story, and the service’s list of releases and changes tells it:
- The feature flag (INFO, from 16 September). The team turned on a new reply editor for some agents. A feature flag is a switch that turns a new feature on for some users. It is harmless: we planted it as a normal change.
- The
KeyError(ERROR, 59 lines on 8 October). Release 4.12.0 went out at 14:05, and one function failed with a missing field,priority. The team rolled back at 14:45. This is a real problem. - The cache pool (WARN, 4,757 lines from 20 October). At 10:00 that day, a configuration change made the cache pool smaller: 64 connections became 16. Two minutes later, requests began to wait for a connection. This is a real problem, and it lasts.
The whole method fits in one picture:
Text description of the diagram
The diagram is a flow from left to right. It starts with one log line: “cache pool exhausted, request q-3f0097 waited 877 ms”. The next box makes its template by masking the IDs and the numbers. Then comes a question: was this template seen in the first two weeks? If yes, the template is known, and we only count it. If no, it is a new template; there were 3 in 60 days. Each new template goes to the embedding step: 384 numbers, and the cosine similarity to the nearest known template. The last box says that Amira reads the new templates, the most different first.
Templates found every new kind of message, with a few lines of code and no model. For Amira’s service, this is often enough. But templates can only say “new” or “known”. They cannot say “a little new” or “very new”. And for text that has no fixed sentences, such as the text of a customer’s ticket, there is no template at all.
Embeddings: a map of meaning
Section titled “Embeddings: a map of meaning”Think of a map of a city. Two cafés on the same street are close on the map. A café and a car park at the other end of town are far apart. An embedding is a map like that for sentences. A small neural network, an embedding model, turns each sentence into a list of numbers. It was trained so that sentences with a similar meaning get similar lists.
The course uses the model all-MiniLM-L6-v2 (90.9 MB, Apache-2.0 licence, runs on a CPU). It turns each sentence into 384 numbers. The notebook pins its exact version, so everyone gets the same numbers.
How close are two lists of numbers?
Section titled “How close are two lists of numbers?”The cosine similarity compares the direction of two lists. It is 1 when they point the same way, 0 when they have nothing in common, and in between otherwise. The length of a list is the square root of the sum of its squared numbers. When both lists have length 1, the cosine is the sum of the products of their numbers.
A tiny example with 2 numbers instead of 384 (constructed):
| Sentence | Embedding | Cosine with A |
|---|---|---|
| A | (1.0, 0.0) | 1.0 × 1.0 + 0.0 × 0.0 = 1.0 |
| B, a similar sentence | (0.8, 0.6) | 1.0 × 0.8 + 0.0 × 0.6 = 0.8 |
| C, another subject | (0.0, 1.0) | 1.0 × 0.0 + 0.0 × 1.0 = 0.0 |
All three lists have length 1: for B, 0.8 × 0.8 + 0.6 × 0.6 = 1. So the cosine is just the sum of the products.
The nearest known template
Section titled “The nearest known template”For each new template, the notebook finds the most similar of the 33 known ones. A low cosine means that the new message is far from everything the service said before:
new template nearest known template cosine feature flag new-reply-editor is on for <num>% of agents agent <id> signed in 0.340 unhandled exception in update_ticket: KeyError: 'priority' unhandled exception in update_ticket: TimeoutError 0.801cache pool exhausted, request <id> waited <num> ms for a connection cache hit ratio <num> over the last <num> requests 0.491Known templates: median cosine to their nearest known neighbour: 0.793The last line is the baseline. A known template is usually close to another known one: the median cosine is 0.793. The chart puts the three new templates next to it:
Text description of the chart
Four rows, one dot each, on an axis of cosine similarity from 0 to 1. The feature flag, new and harmless: 0.34, the lowest. The cache pool message, new and a real problem: 0.491. The median of the known templates: 0.793. The KeyError, new and a real problem: 0.801, as high as a usual known template.
Read the dots from left to right. The most different message is the harmless feature flag. The KeyError is the least different. It is almost the same sentence as a known error of the same function, with KeyError: 'priority' instead of TimeoutError. By distance alone, it looks normal.
The map
Section titled “The map”The 384 numbers of each template can be squeezed into 2 numbers with PCA (principal component analysis). PCA keeps the two directions in which the data varies most. Then we can draw the templates on a map:
- Known template (first two weeks)
- New: a real problem (KeyError, cache pool)
- New: harmless (feature flag)
Text description of the chart
The map has 36 dots, one for each template. The 33 known templates are in one colour. The two new templates that are real problems are in a second colour, and the harmless new feature flag is in a third. At the top left is a group of exception messages, such as “unhandled exception in update_ticket: TimeoutError”. The new KeyError message sits inside this group. At the bottom are the request and classification messages. On the right are messages about timeouts, connection pools and health checks; the new cache pool message sits among them. The new feature flag message sits in the middle right, near the rate-limit and health-check messages.
The map shows the same story with a warning. The KeyError sits inside the group of known exceptions. The cache pool message sits next to the other messages about pools and timeouts. The feature flag seems close to other messages, but that comes from the squeeze to 2 numbers. In 384 numbers, the cosine says it is the farthest. Never measure distances on a 2-D map. Use it to see groups.
Different is not dangerous
Section titled “Different is not dangerous”Embeddings measure how different a message is. They do not measure how bad it is. On this service:
| Template | New? | Cosine (lower = more different) | Real problem? |
|---|---|---|---|
| feature flag | yes | 0.34 | no: a planned change |
| cache pool | yes | 0.49 | yes: the configuration change |
KeyError |
yes | 0.80 | yes: the release, rolled back |
If Amira alerted on “cosine below the known median”, she would get the harmless flag and the cache pool, and miss the KeyError. Templates found all three. The level helps too: the KeyError is an ERROR, the cache pool a WARN, the flag an INFO. A simple, useful rule is “every new template, with its level and its first line, goes to a person”.
Where do embeddings help, then?
- To rank the new templates when there are many of them, for example after a large release.
- To group new messages that say the same thing in different words. A log parser makes a new template for each wording. Embeddings put them together.
- For free text, such as the text of a customer’s ticket. A complaint about a new problem, for example “the app charged me twice”, has no template. The distance from all known complaints can find it. The course data has no ticket text, so we do not measure this here.
What each method costs
Section titled “What each method costs”The notebook times both methods on the same lines:
templates: 1.8 microseconds a lineembeddings: 654 microseconds a line, about 1.5 minutes for all 137,013 linesThese times are from the test machine; yours will differ. Embedding a line costs about 360 times more than templating it. At Larkfield’s size, both are cheap. Take a constructed example of a million lines a day. The templates take about 2 seconds, and the embeddings about 11 minutes of CPU. The embeddings also need a model of 91 MB to install and keep up to date.
So the order is: templates first for every line, embeddings only for the few new templates. Three new templates take well under a second to embed.
Common mistake
Flag every message never seen before
Symptom: Amira, the developer of ticket-api, writes a quick check: “send an alert for every log message not seen in the first two weeks”. On the first evening, the help desk channel gets thousands of alerts. After the first two weeks, 91,023 of 108,393 lines are “new”, about 2,126 a day.
Cause: the check compares whole messages. Every message holds a ticket number, a job number or a time in milliseconds, so almost every line is unique. The 108,502 different messages are only 36 sentences.
Fix: compare templates, not messages. Mask the IDs and the numbers first. The same data then gives 3 new templates in 60 days, and each one is a real change to look at.
Consequence in AI work: the unit of a detector decides everything. Here the right unit is “a kind of sentence”, not “a line”. A check with the wrong unit floods the team, and the team learns to mute it, so it also hides the real error.
Your turn in the notebook
Section titled “Your turn in the notebook”The notebook ends with two tasks, with checks:
- The configuration change was on 2025-10-20 at 10:00. Find the template first seen after that time, from the table of new templates.
- Find the new template with the lowest cosine to its nearest known template, and decide if it is a problem.
Then compare the two methods for Amira, in two sentences: what each found, and what each cost.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain why flagging every log message never seen before fails on ticket-api, and what a template changes.
Show the answer
Almost every message holds a ticket ID, a job ID or a number of milliseconds, so almost every line is unique. After the first two weeks, 84% of lines were “new”, about 2,126 a day: the detector flags almost everything. A template masks the parts that change, so lines that are the same sentence get the same template. The 108,502 messages become 36 templates, and only 3 are new in 60 days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a new release adds the message “unhandled exception in search: KeyError: 'query'”. The service already logs “unhandled exception in search: TimeoutError”. Will its cosine to the nearest known template be high or low? Should it reach a person?
Show the answer
High, probably: it is almost the same sentence as a known exception of the same function, just like the KeyError of 8 October (cosine 0.80). By distance, it looks normal. But it is a new template and an ERROR, so it must reach a person. A rule “every new template goes to a person, with its level” catches it. A cosine threshold alone would miss it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Amira asks: “Should we embed every log line every day, to be safe?” Answer with the costs from this lesson.
Show the answer
No. Embeddings cost about 0.65 milliseconds a line on the test machine, against about 2 microseconds for a template: about 360 times more. They also need a 91 MB model to install and to keep the same version. On this data they ranked the harmless flag first and the real KeyError last. Template every line, and embed only the few new templates, to group and rank them for a person.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Comparing whole log messages fails: after two weeks, 84% of lines were “new”, because of IDs and numbers.
- A template masks the parts that change. The 108,502 messages of
ticket-apiare 36 templates, and 3 are new in 60 days. They are the harmless feature flag, theKeyErrorafter a release, and the cache pool after the configuration change. - An embedding turns a sentence into numbers so that similar meanings are close. The cosine similarity to the nearest known template says how different a new message is.
- Different is not dangerous: the harmless flag was the most different (0.34) and the real
KeyErrorthe least (0.80). - Templates cost about 2 microseconds a line; embeddings about 360 times more. Template every line; embed only the new templates.
You can now find new kinds of message in logs, and you know that “new” still needs a person to judge it. But that person must read each alert and understand it quickly. Next, you see whether an LLM can write that first summary for Sofia, and how to check what it writes.