Skip to content
Français

Unusual text and logs with embeddings

  • Lesson 26
  • 40 min
  • Module 6: Deep and embedding methods

Where we are

Our detectors work on numbers. Log lines are text, and a new kind of error is a new kind of message. In this lesson, we first turn log lines into templates, which is cheap and often enough, and then use embeddings to find messages far from every known one.

Lesson 26 of 38 · Anomaly Detection: Find What Is UnusualUnusual text and logs with embeddings
Course overview

What you will be able to do

Find a new kind of message in a service’s logs. Start with templates, which are cheap and often enough. Then use embeddings to rank the few new messages by how different they are, and learn why “different” is not the same as “dangerous”.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. Its setup cells install pandas, PyTorch and sentence-transformers, and download the logs (about 2 MB) and a small embedding model (about 91 MB). About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

A browser and a free Google or Kaggle account for the notebook, or the local setup from Meet the Larkfield data. The embedding model runs on a CPU. No account and no paid service.

Tested with Python 3.12.13 in Jupyter on macOS, with sentence-transformers 6.2.0 and the model all-MiniLM-L6-v2. Colab, Kaggle and Windows were not tested.

You drive the same car every day. You know its sounds: the engine, the indicator, the hum of the tyres on the motorway. You do not think about them. Then one morning you hear a click when you turn left. You do not know what it means. But you know at once that it is new.

Some new sounds are harmless. A new set of tyres hums in a different way. Some are serious: a click in the steering. Your ear tells you “new”. It does not tell you “dangerous”. To know that, you must look.

A service’s logs work the same way. Most lines are the usual sounds. A new kind of message is the click.

Amira is the developer of the help desk’s ticket-api service. The service writes a log line each time something happens: a request, a slow database query, an email sent, an error. Each line has a time, a level (INFO for normal events, WARN for a warning, ERROR for a failure) and a message. Here are six lines from the first morning of the data, 1 September 2025:

Time Level Message
00:01:02 INFO GET /tickets/T-472971 200 in 39 ms
00:05:50 INFO GET /tickets/T-421326 200 in 58 ms
00:08:43 WARN slow query on tickets table took 1434 ms
00:14:16 INFO worker 4 picked job J-1945010 (queue depth 9)
00:26:45 ERROR upstream timeout calling email-service after 1207 ms
00:28:24 INFO sent reply email for ticket T-410889 (2579 bytes)

The data has 137,013 lines over 60 days, about 2,500 a day. Nobody reads them all. Amira wants one thing from them: “tell me when the service says something it has never said before”.

The simplest idea is to keep every message of the first two weeks, 1 to 14 September. Then flag every later message that is not in that list. Try it on the data. The first two weeks had 24,693 different messages. After them came 108,393 lines, and 91,023 of them had a message never seen before. That is 84%, about 2,126 “new” lines a day.

This detector is useless. The two GET lines above are the same event with another ticket number and another time in milliseconds. The data has 108,502 different messages, but only a few dozen different sentences. The IDs and the numbers make almost every line unique.

Templates: the sentence without its numbers

Section titled “Templates: the sentence without its numbers”

The fix is the idea that the car driver uses without thinking. “Click when I turn left” is one sound, on Monday and on Tuesday. A template is a log message with the parts that change replaced by a mask. The mask <id> replaces a ticket, job, request or agent ID, and <num> replaces a number. The notebook does it with four search-and-replace patterns:

MASKS = [(re.compile(r"\b[TJ]-\d+\b"), "<id>"), (re.compile(r"\bq-[0-9a-f]+\b"), "<id>"),
(re.compile(r"\b[AK]-\d+\b"), "<id>"), (re.compile(r"\d+(\.\d+)?"), "<num>")]
def template(message):
"""Replace the parts that change (IDs, numbers) with <id> and <num>."""
for pattern, mask in MASKS:
message = pattern.sub(mask, message)
return message

Here are the six lines again, as templates:

Message Template
GET /tickets/T-472971 200 in 39 ms GET /tickets/<id> <num> in <num> ms
GET /tickets/T-421326 200 in 58 ms GET /tickets/<id> <num> in <num> ms
slow query on tickets table took 1434 ms slow query on tickets table took <num> ms
worker 4 picked job J-1945010 (queue depth 9) worker <num> picked job <id> (queue depth <num>)
upstream timeout calling email-service after 1207 ms upstream timeout calling email-service after <num> ms
sent reply email for ticket T-410889 (2579 bytes) sent reply email for ticket <id> (<num> bytes)

The two GET lines now have the same template. On all the lines, the notebook prints:

137,013 lines, 108,502 different messages, 36 templates, in 0.24 s

The 108,502 messages are only 36 templates. Templating every line took 0.24 seconds on the test machine, about 2 microseconds a line. Real logging systems often use a log parser, such as Drain, that learns the masks by itself. The idea is the same.

Now “new” means “a template that never appeared in the first two weeks”. The notebook finds 33 templates in those weeks and 3 new ones after them:

33 templates in the first two weeks; 3 new ones after them:
first_seen lines
template
feature flag new-reply-editor is on for <num>% of agents 2025-09-16 19:25:26.448 878
unhandled exception in update_ticket: KeyError: 'priority' 2025-10-08 14:05:16.494 59
cache pool exhausted, request <id> waited <num> ms for a connection 2025-10-20 10:02:19.778 4757

From 91,023 “new” lines to 3 new templates. Each one has a story, and the service’s list of releases and changes tells it:

  • The feature flag (INFO, from 16 September). The team turned on a new reply editor for some agents. A feature flag is a switch that turns a new feature on for some users. It is harmless: we planted it as a normal change.
  • The KeyError (ERROR, 59 lines on 8 October). Release 4.12.0 went out at 14:05, and one function failed with a missing field, priority. The team rolled back at 14:45. This is a real problem.
  • The cache pool (WARN, 4,757 lines from 20 October). At 10:00 that day, a configuration change made the cache pool smaller: 64 connections became 16. Two minutes later, requests began to wait for a connection. This is a real problem, and it lasts.

The whole method fits in one picture:

A flow from left to right. A log line, for example cache pool exhausted, request q-3f0097 waited 877 ms. Then the template: mask the IDs and the numbers. Then a question: seen in the first two weeks? Yes leads to known: count it. No leads to new template, 3 in 60 days, then to embedding: 384 numbers, cosine to the nearest known template, then to Amira reads the new ones, most different first.
Templates first, embeddings only for the few new templates. The cheap step does most of the work.
Text description of the diagram

The diagram is a flow from left to right. It starts with one log line: “cache pool exhausted, request q-3f0097 waited 877 ms”. The next box makes its template by masking the IDs and the numbers. Then comes a question: was this template seen in the first two weeks? If yes, the template is known, and we only count it. If no, it is a new template; there were 3 in 60 days. Each new template goes to the embedding step: 384 numbers, and the cosine similarity to the nearest known template. The last box says that Amira reads the new templates, the most different first.

Templates found every new kind of message, with a few lines of code and no model. For Amira’s service, this is often enough. But templates can only say “new” or “known”. They cannot say “a little new” or “very new”. And for text that has no fixed sentences, such as the text of a customer’s ticket, there is no template at all.

Think of a map of a city. Two cafés on the same street are close on the map. A café and a car park at the other end of town are far apart. An embedding is a map like that for sentences. A small neural network, an embedding model, turns each sentence into a list of numbers. It was trained so that sentences with a similar meaning get similar lists.

The course uses the model all-MiniLM-L6-v2 (90.9 MB, Apache-2.0 licence, runs on a CPU). It turns each sentence into 384 numbers. The notebook pins its exact version, so everyone gets the same numbers.

The cosine similarity compares the direction of two lists. It is 1 when they point the same way, 0 when they have nothing in common, and in between otherwise. The length of a list is the square root of the sum of its squared numbers. When both lists have length 1, the cosine is the sum of the products of their numbers.

A tiny example with 2 numbers instead of 384 (constructed):

Sentence Embedding Cosine with A
A (1.0, 0.0) 1.0 × 1.0 + 0.0 × 0.0 = 1.0
B, a similar sentence (0.8, 0.6) 1.0 × 0.8 + 0.0 × 0.6 = 0.8
C, another subject (0.0, 1.0) 1.0 × 0.0 + 0.0 × 1.0 = 0.0

All three lists have length 1: for B, 0.8 × 0.8 + 0.6 × 0.6 = 1. So the cosine is just the sum of the products.

For each new template, the notebook finds the most similar of the 33 known ones. A low cosine means that the new message is far from everything the service said before:

new template nearest known template cosine
feature flag new-reply-editor is on for <num>% of agents agent <id> signed in 0.340
unhandled exception in update_ticket: KeyError: 'priority' unhandled exception in update_ticket: TimeoutError 0.801
cache pool exhausted, request <id> waited <num> ms for a connection cache hit ratio <num> over the last <num> requests 0.491
Known templates: median cosine to their nearest known neighbour: 0.793

The last line is the baseline. A known template is usually close to another known one: the median cosine is 0.793. The chart puts the three new templates next to it:

How similar each new template is to its nearest known template (measured). The harmless feature flag is the most different. The real error, the KeyError, is as close to a known template as a usual known one is.
Text description of the chart

Four rows, one dot each, on an axis of cosine similarity from 0 to 1. The feature flag, new and harmless: 0.34, the lowest. The cache pool message, new and a real problem: 0.491. The median of the known templates: 0.793. The KeyError, new and a real problem: 0.801, as high as a usual known template.

Read the dots from left to right. The most different message is the harmless feature flag. The KeyError is the least different. It is almost the same sentence as a known error of the same function, with KeyError: 'priority' instead of TimeoutError. By distance alone, it looks normal.

The 384 numbers of each template can be squeezed into 2 numbers with PCA (principal component analysis). PCA keeps the two directions in which the data varies most. Then we can draw the templates on a map:

  • Known template (first two weeks)
  • New: a real problem (KeyError, cache pool)
  • New: harmless (feature flag)
The 36 log templates of ticket-api, from 384 numbers to 2 with PCA (measured). Similar sentences are close: the exceptions are at the top left, with the new KeyError among them; the new cache pool message is on the right, among pools and timeouts. The map is only a rough guide: the feature flag looks close to others here, but in 384 numbers it is the farthest from every known template.
Text description of the chart

The map has 36 dots, one for each template. The 33 known templates are in one colour. The two new templates that are real problems are in a second colour, and the harmless new feature flag is in a third. At the top left is a group of exception messages, such as “unhandled exception in update_ticket: TimeoutError”. The new KeyError message sits inside this group. At the bottom are the request and classification messages. On the right are messages about timeouts, connection pools and health checks; the new cache pool message sits among them. The new feature flag message sits in the middle right, near the rate-limit and health-check messages.

The map shows the same story with a warning. The KeyError sits inside the group of known exceptions. The cache pool message sits next to the other messages about pools and timeouts. The feature flag seems close to other messages, but that comes from the squeeze to 2 numbers. In 384 numbers, the cosine says it is the farthest. Never measure distances on a 2-D map. Use it to see groups.

Embeddings measure how different a message is. They do not measure how bad it is. On this service:

Template New? Cosine (lower = more different) Real problem?
feature flag yes 0.34 no: a planned change
cache pool yes 0.49 yes: the configuration change
KeyError yes 0.80 yes: the release, rolled back

If Amira alerted on “cosine below the known median”, she would get the harmless flag and the cache pool, and miss the KeyError. Templates found all three. The level helps too: the KeyError is an ERROR, the cache pool a WARN, the flag an INFO. A simple, useful rule is “every new template, with its level and its first line, goes to a person”.

Where do embeddings help, then?

  • To rank the new templates when there are many of them, for example after a large release.
  • To group new messages that say the same thing in different words. A log parser makes a new template for each wording. Embeddings put them together.
  • For free text, such as the text of a customer’s ticket. A complaint about a new problem, for example “the app charged me twice”, has no template. The distance from all known complaints can find it. The course data has no ticket text, so we do not measure this here.

The notebook times both methods on the same lines:

templates: 1.8 microseconds a line
embeddings: 654 microseconds a line, about 1.5 minutes for all 137,013 lines

These times are from the test machine; yours will differ. Embedding a line costs about 360 times more than templating it. At Larkfield’s size, both are cheap. Take a constructed example of a million lines a day. The templates take about 2 seconds, and the embeddings about 11 minutes of CPU. The embeddings also need a model of 91 MB to install and keep up to date.

So the order is: templates first for every line, embeddings only for the few new templates. Three new templates take well under a second to embed.

Common mistake

Flag every message never seen before

Symptom: Amira, the developer of ticket-api, writes a quick check: “send an alert for every log message not seen in the first two weeks”. On the first evening, the help desk channel gets thousands of alerts. After the first two weeks, 91,023 of 108,393 lines are “new”, about 2,126 a day.

Cause: the check compares whole messages. Every message holds a ticket number, a job number or a time in milliseconds, so almost every line is unique. The 108,502 different messages are only 36 sentences.

Fix: compare templates, not messages. Mask the IDs and the numbers first. The same data then gives 3 new templates in 60 days, and each one is a real change to look at.

Consequence in AI work: the unit of a detector decides everything. Here the right unit is “a kind of sentence”, not “a line”. A check with the wrong unit floods the team, and the team learns to mute it, so it also hides the real error.

The notebook ends with two tasks, with checks:

  1. The configuration change was on 2025-10-20 at 10:00. Find the template first seen after that time, from the table of new templates.
  2. Find the new template with the lowest cosine to its nearest known template, and decide if it is a problem.

Then compare the two methods for Amira, in two sentences: what each found, and what each cost.

Check your understanding

Explain why flagging every log message never seen before fails on ticket-api, and what a template changes.

Show the answer

Almost every message holds a ticket ID, a job ID or a number of milliseconds, so almost every line is unique. After the first two weeks, 84% of lines were “new”, about 2,126 a day: the detector flags almost everything. A template masks the parts that change, so lines that are the same sentence get the same template. The 108,502 messages become 36 templates, and only 3 are new in 60 days.

Check your understanding

Predict: a new release adds the message “unhandled exception in search: KeyError: 'query'”. The service already logs “unhandled exception in search: TimeoutError”. Will its cosine to the nearest known template be high or low? Should it reach a person?

Show the answer

High, probably: it is almost the same sentence as a known exception of the same function, just like the KeyError of 8 October (cosine 0.80). By distance, it looks normal. But it is a new template and an ERROR, so it must reach a person. A rule “every new template goes to a person, with its level” catches it. A cosine threshold alone would miss it.

Check your understanding

Amira asks: “Should we embed every log line every day, to be safe?” Answer with the costs from this lesson.

Show the answer

No. Embeddings cost about 0.65 milliseconds a line on the test machine, against about 2 microseconds for a template: about 360 times more. They also need a 91 MB model to install and to keep the same version. On this data they ranked the harmless flag first and the real KeyError last. Template every line, and embed only the few new templates, to group and rank them for a person.

Recap

  • Comparing whole log messages fails: after two weeks, 84% of lines were “new”, because of IDs and numbers.
  • A template masks the parts that change. The 108,502 messages of ticket-api are 36 templates, and 3 are new in 60 days. They are the harmless feature flag, the KeyError after a release, and the cache pool after the configuration change.
  • An embedding turns a sentence into numbers so that similar meanings are close. The cosine similarity to the nearest known template says how different a new message is.
  • Different is not dangerous: the harmless flag was the most different (0.34) and the real KeyError the least (0.80).
  • Templates cost about 2 microseconds a line; embeddings about 360 times more. Template every line; embed only the new templates.

You can now find new kinds of message in logs, and you know that “new” still needs a person to judge it. But that person must read each alert and understand it quickly. Next, you see whether an LLM can write that first summary for Sofia, and how to check what it writes.

Next: Where an LLM helps, and where it does not