Skip to content
Français

The Collector

  • Lesson 6
  • 45 min
  • Module 2: OpenTelemetry for agents

Where we are

In the last two lessons, we made one run into one trace and gave its spans the names that tools understand. But the spans still go to files on one computer. Where should they go, and who removes the data that must not leave? In this lesson, we put the OpenTelemetry Collector between the application and the tools, and we let it redact, sample and route in one place. We start in the mail room of an office building.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Send the workflow’s telemetry to one place outside the application. That place removes private text, keeps the traces that matter and routes each signal to its tool. The application never needs to know where its data goes.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • Your project as the last lesson left it (79 tests pass).
  • Optional, for the last part: about 470 MB of free disk space for the Collector’s program. It is a 97 MB download that unpacks to a 363 MB file. Docker is not needed.

Tested on macOS (Apple silicon) with Python 3.12.13 and otelcol-contrib 0.162.0. The numbers about the whole week come from the course team’s measured run of it. The Windows and Linux commands are written, not run.

An office building has fifty companies and one mail room. Every company puts its outgoing letters in the same tray, and it knows only that tray. The mail room does the rest. It checks that no letter shows a bank card number on the envelope. It sends ordinary letters by the cheap courier, and registered letters by the courier that signs for them. Some letters go to two places: the client and the archive.

When the building changes courier, no company changes anything. The companies did not even know the old courier’s address.

The problem: an application that knows too much

Section titled “The problem: an application that knows too much”

So far, the workflow writes its telemetry to files. In production, the traces must go to Tempo, the metrics to Prometheus and the events to Loki, the Grafana tools of Module 1. In Module 8, the traces also go to Langfuse, and in Module 6, the audit events go to an audit store. Suppose the application sends to each of them itself. Four problems follow.

Every application holds every address and every key. The Langfuse key would be in the workflow’s settings. It would also be in the settings of every other app that sends traces.

Every application must remove private text itself, and one will forget. The course team ran the replayed week with one setting changed, as a developer might do to debug a prompt: OBS_CAPTURE_CONTENT=on. The application’s own export then had 718 model spans with prompt text. The text was the customers’ tickets, sent to every place the application knew.

The application cannot sample well. To keep every trace with an error and only some ordinary ones, you must see the whole trace first. The application decides when a span starts, before it knows how the run ends.

Some data must never be sampled or mixed. Audit events must all arrive, and only at the audit store. A logging tool is the wrong place for them.

The idea: one Collector between the application and the tools

Section titled “The idea: one Collector between the application and the tools”

The OpenTelemetry Collector is a separate program that receives telemetry, changes it, and sends it on. It is the mail room. The application sends everything to one address, OTEL_EXPORTER_OTLP_ENDPOINT, over OTLP. The Collector knows the rest.

A Collector’s configuration has three kinds of parts, joined in pipelines, one per signal:

  • A receiver takes telemetry in. Ours is otlp: gRPC on port 4317 and HTTP on port 4318.
  • A processor changes the data on its way. Ours limit memory, remove text (redaction), keep some traces (sampling) and group data in batches.
  • An exporter sends the data out: to Tempo, Prometheus, Loki, Langfuse or a file.

A connector joins two pipelines: it is an exporter for one and a receiver for the next. Ours, routing, sends audit events to their own pipeline. And when one pipeline has two exporters, the same data goes to both: this is fan-out.

The workflow sends OTLP to one address, 127.0.0.1:4318. Inside the box 'The OpenTelemetry Collector', the receiver otlp (gRPC 4317, HTTP 4318) feeds three pipelines. Traces: memory_limiter, transform/redact, tail_sampling, batch, then to Tempo, and with a dashed arrow, fan-out to Langfuse through a second exporter on the same pipeline (Module 8). Metrics: memory_limiter, batch, then Prometheus. Events: memory_limiter, then a routing connector that asks: has audit.type? No: other events, transform/redact, batch, then Loki. Yes: never sampled, never to Loki, to the audit store (Module 6).
The course's Collector (collector/otelcol-config.yaml): one receiver, one pipeline per signal, redaction and sampling for traces, a route for audit events, and fan-out to a second destination.
Text description of the diagram

Read it from left to right. The workflow knows one address. The receiver takes every signal in. Traces pass through redaction and tail sampling, then go to Tempo, and in Module 8 also to Langfuse. Metrics go to Prometheus. Events go to a routing connector. An event with an audit.type attribute goes only to the audit store, and it is never sampled. Every other event is redacted and goes to Loki.

The processor transform/redact runs statements on every span. Two kinds of statements do the work:

- delete_key(attributes, "gen_ai.input.messages")
- replace_all_patterns(attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "[email]")
- replace_all_patterns(attributes, "value", "\\b(?:\\d[ -]?){13,19}\\b", "[card number]")

The first deletes an attribute that holds text by the GenAI conventions. Seven such keys are deleted: the prompt, the answer, the system instructions, a tool’s arguments and result, and two older names. The next two look inside every other value. The first pattern matches an email address. The second matches 13 to 19 digits, with optional spaces or dashes: the shape of a card number.

The course team sent one made-up span through the Collector to check it. Its attribute larkfield.note was:

Write to sam.example@example.com, card 4111 1111 1111 1111

The address is invented, and 4111 1111 1111 1111 is a well-known test card number. The Collector let out:

Write to [email], card [card number]

So the second pattern matched the 16 digits with their spaces, and the first matched the address. On the whole week with content capture on, the application’s export had 718 problems, and the Collector’s output had 0.

This is defence in depth. The workflow records no text by default, and the Collector removes it again, in case a library or a debug setting adds it. Neither layer trusts the other.

Storing every trace costs money, and most runs are ordinary. Sampling keeps only some of them. Head sampling decides when the first span starts, by chance. Tail sampling waits until the trace is complete, then decides from its spans. The course’s Collector waits 10 seconds (decision_wait: 10s), then keeps a trace if any policy says so:

Policy Keeps a trace when…
errors a span has the status ERROR
writes-and-approvals a span has gen_ai.operation.name = approve
write-calls a span has larkfield.tool.kind = write
injected-input a span has larkfield.input.injection = true
slow-runs the trace lasts more than 30 s
ten-percent-of-the-rest a random 10% of the others

Apply it by hand to the double refund’s two traces. Trace 1 has the agent, two model calls, one read and the approval request. No error, no approval, no write: only the 10% chance keeps it. Trace 2 has approve support-resolution and the refund writes: it is kept twice over. So the Collector usually keeps the half of the story with the refunds, and drops the half with the model’s decision.

That is what happened on the whole week:

Measured: the replayed week (434 traces) through the standalone Collector with the course's tail sampling. Each trace counted once, under the first reason that applies; the 10% sample varies from run to run. One of the write traces is the constructed redaction probe.
Text description of the chart

Six bars. Kept because of an approval: 100 traces. Because of a write: 95. Because of an error: 53. Because of an injected input: 13. By the 10% sample: 15. Dropped: 159 of 434 traces. Spans went from 2027 to 1287.

Every trace with an error, an approval or a write was kept. But 99 runs lost one of their two traces, as the double refund’s first trace would. The policies decide trace by trace, and a run with an approval has two traces. Module 3 explains why a run has two traces.

Tomás says: “Sampling is fine for everything.” It is fine for ordinary traces. It is not fine for audit events: an audit trail with 10% of its records is not an audit trail. So the tail_sampling processor is in the traces pipeline only. In the same run, all 1331 audit events reached the audit store, and none reached the Loki output. Module 6 builds that store.

  1. Copy the Collector’s configurations, the week driver, the new test, Grafana’s compose file and the README from the snapshot:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m02
    cp -R $S/collector $S/scenarios .
    cp $S/tests/test_collector_files.py tests/
    cp $S/grafana/compose.yaml grafana/
    cp $S/README.md .
  2. Run the tests: python -m pytest -q. The last line says 81 passed, 1 deselected. Your project is now the same as the snapshot end-of-m02. The two new tests read the configuration. They check that redaction is there, that tail sampling is in the traces pipeline only, and that audit events have their own route.

  3. Open collector/otelcol-config.yaml. Find the four parts of the diagram: receivers, processors, connectors and exporters. Then find service.pipelines at the end: each pipeline lists its receivers, processors and exporters, in order.

  4. Open grafana/compose.yaml. It now mounts collector/otelcol-config.yaml over the Collector configuration inside the Grafana container. So when you use Grafana, its Collector applies the same redaction, sampling and routing. Module 4 shows how to start it and send it a week. If Grafana runs, restart it to load the file: docker compose -f grafana/compose.yaml down, then up -d.

  5. Find the exporters. Their addresses are Tempo, Prometheus, Loki and the audit store. Then search the workflow’s code for them: grep -rn "4418\|9090\|3100" resolver/ (PowerShell: Select-String -Path resolver\*.py -Pattern "4418|9090|3100"). It finds nothing. The workflow knows only OTEL_EXPORTER_OTLP_ENDPOINT.

  6. Check your own telemetry: python -m obs check-redaction work/telemetry. It reads every attribute, as the redaction does.

    Checked 4 span(s), 0 event(s), 4 metric line(s) in work/telemetry.
    No text attribute, no e-mail address and no card number found.

    Your counts depend on how many runs you made.

Now try the debug setting that caused the 718 problems:

Terminal window
rm -rf work
OBS_CAPTURE_CONTENT=on python -m resolver run T-80008
python -m obs check-redaction work/telemetry

In PowerShell, use Remove-Item -Recurse work and $env:OBS_CAPTURE_CONTENT = "on", then remove the variable afterwards. The check still says No text attribute, no e-mail address and no card number found. Why? At this point, your workflow records only the run span, and the text would be on the model spans. Module 3 adds them. The course team’s run was made at the course end, with every model span.

The snapshot also has the week driver, which makes the replayed week again with your own code:

Terminal window
python -m scenarios week --out work/week
week: 316 runs in 1 s -> work/week
resolved 203, right by the course's labels 273
scenario runs: double_refund 2, hidden_tool_error 1, injection_seen 22, local_model 16, loop 3, model_switch 40, no_action 4, non_english 14, overnight_approval 19

The same 316 runs as the saved week, with today’s instrumentation: python -m obs trace R-1014-040 --data work/week shows one span, the run span, and the outcome waiting_approval. The saved week was made at the course end, so it has every span.

Why the application should not know where its data goes

Section titled “Why the application should not know where its data goes”

Look at what each side knows now. The workflow knows one address and what it records. The Collector knows the destinations, their keys, what to remove and what to keep. Each change happens on one side only:

  • Kwame moves traces to another backend: he changes an exporter. The workflow does not change.
  • Mei shortens how long traces are kept, or Priya wants more ordinary traces: they change the sampling policy.
  • Larkfield adds Langfuse in Module 8: one more exporter on the traces pipeline. That is the fan-out.
  • A new app joins: it sends to the same address and gets the same redaction.

This is also safer. A key in a Collector’s configuration is in one place, read by one process. A key in every app is in every app’s settings, logs and crash reports.

The Collector is one more program to run, update and watch. If it stops, telemetry stops: the application’s exporter tries again for a while, then drops data. Tail sampling holds every trace in memory for 10 seconds, up to num_traces: 20000. The memory_limiter refuses data above 400 MiB, so the Collector protects itself instead of crashing. Sampling loses data on purpose: a dropped trace is gone. And a Collector can only remove what its rules describe: a new kind of private data needs a new rule. The check-redaction command checks the output, so a test can catch a missing rule.

For one developer on a laptop, files are enough, and the course’s lessons use them. For a team, the Collector is where the rules live.

Optional: run the Collector without Docker

Section titled “Optional: run the Collector without Docker”

The course’s Collector is the contrib distribution of OpenTelemetry Collector, otelcol-contrib 0.162.0. Inside Docker, it runs in the grafana/otel-lgtm image of Module 1. Without Docker, collector/standalone.yaml runs the same processors and writes files instead of sending to Grafana.

Component names change between versions: this version names the HTTP exporter otlp_http. Older examples on the web write otlphttp. Use the names of the version that you run.

  1. Download the release for your computer from the project’s release page, into a folder next to your project. Check its SHA-256 against the .sha256 file of the same release.

    Terminal window
    mkdir -p ../otelcol && cd ../otelcol
    curl -LO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.162.0/otelcol-contrib_0.162.0_darwin_arm64.tar.gz
    shasum -a 256 otelcol-contrib_0.162.0_darwin_arm64.tar.gz
    tar -xzf otelcol-contrib_0.162.0_darwin_arm64.tar.gz otelcol-contrib
    cd ../observable-workflow

    The checksum must be d5e11974d2e4adac3cc001a25137aad52f6e77ec3034ba7735cac7c219a5f97a.

  2. In a first terminal, in the project folder, start the Collector. It writes its output to work/collector-out. AUDIT_STORE_URL names the audit store of Module 6; nothing listens there yet, and no audit event exists yet, so nothing is sent.

    Terminal window
    mkdir -p work/collector-out
    OUT_DIR=work/collector-out AUDIT_STORE_URL=http://127.0.0.1:8313 ../otelcol/otelcol-contrib --config collector/standalone.yaml
  3. In a second terminal, with the virtual environment active, send two runs to the Collector instead of the files. T-80002 is the recorded content-filter refusal, so its run span has the status ERROR. T-80008 is an ordinary run.

    Terminal window
    OBS_EXPORT=otlp python -m resolver run T-80002
    OBS_EXPORT=otlp python -m resolver run T-80008

    In PowerShell, set $env:OBS_EXPORT = "otlp" first.

  4. Run T-80008 nine more times, the same way. Then wait about 15 seconds: 10 for the sampling decision, then the batch. List what the Collector let out, and check it:

    Terminal window
    python -m obs runs --data work/collector-out
    python -m obs check-redaction work/collector-out
  5. Stop the Collector with Ctrl+C in the first terminal.

The course’s run sent eleven runs and got two back:

run ticket started outcome model tools appr cost US$ secs model flags
T-80002-agent-chat-small-1 T-80002 … handed_to_person 0 0 0 0.00000 0.0 chat-small -
T-80008-agent-chat-small-10 T-80008 … resolved 0 0 0 0.00000 0.0 chat-small -
2 run(s), 0 flagged
Checked 2 span(s), 0 event(s), 11 metric line(s) in work/collector-out.
No text attribute, no e-mail address and no card number found.

T-80002’s run is there because the errors policy keeps it. Of the ten ordinary runs of T-80008, the 10% sample kept one, here the tenth. Your count will differ: the sample is random. The metrics all arrived: sampling is for traces only. The Collector logged no error, although no audit store listens yet.

Common mistake

My run is not in the Collector's output

Symptom: after OBS_EXPORT=otlp python -m resolver run T-80008, python -m obs runs --data work/collector-out does not list the run, but obs lists T-80002.

Cause: this is tail sampling working as configured. T-80008’s trace has no error, no approval, no write and no injected input, so only the 10% policy can keep it. It can also be too early: the Collector waits 10 seconds before it decides, then sends a batch.

Diagnose: wait 15 seconds and list again. Check which policies the trace could match: open its spans in work/telemetry (with OBS_EXPORT=file,otlp, the workflow writes both).

Recover: nothing is broken. For practice, use OBS_EXPORT=file,otlp to keep your own copy of every run. In production, decide on purpose which traces must always be kept, and write a policy for them.

Common mistake

The Collector stops at once

Symptom: the Collector prints an error and stops right after it starts. Without AUDIT_STORE_URL, the course’s run printed a warning, then:

Error: invalid configuration: exporters::otlp_http/audit: at least one endpoint must be specified

Cause: usually one of three things. OUT_DIR or AUDIT_STORE_URL is not set in that terminal, so the configuration has an empty value, as above. Another program uses port 4317, 4318 or 13133. Or the configuration has a component name from another version.

Diagnose: read the first error line. It names the component or the port.

Recover: set both variables in the same terminal. Stop the other program, for example a Grafana container from Module 1: docker compose -f grafana/compose.yaml down. Or use the names of version 0.162.0.

Completion check: draw a tree from a span list

Section titled “Completion check: draw a tree from a span list”

Here are the eight spans of one run, as a tool lists them. The run is the two workers on T-90103, replayed with Module 3’s project. Each row gives the first four characters of the span ID and of the parent’s ID. Then come the start, in seconds from the start of the trace, and the duration.

Span Parent Start (s) Duration (s) Name
4f2b – 0.00 4.97 invoke_workflow support-resolution
af66 4f2b 0.00 3.47 invoke_agent investigator
f727 af66 0.00 1.26 chat chat-small
45d7 af66 1.26 0.01 execute_tool get_order
35b4 af66 1.27 2.20 chat chat-small
c4c7 0b3d 3.47 1.49 chat chat-small
0b3d 4f2b 3.47 1.49 invoke_agent resolver
0fc9 4f2b 4.95 0.02 execute_tool send_reply

Check your understanding

Tomás says: 'We already record no text, so a redaction processor in the Collector is wasted work.' Use the course's measurement to answer him.

Show the answer

The workflow records no text by default, but one debug setting changes that. With content capture on, the application’s own export had 718 model spans with prompt text. A new library could do the same without anyone noticing. The Collector removed every one of them. It also turned a probe with an email address and a test card number into Write to [email], card [card number]. Two layers that do not trust each other are defence in depth. The cost is a few statements in one configuration file.

Check your understanding

Why is tail sampling in the traces pipeline only, and what did it cost the double refund's kind of run?

Show the answer

Audit events must all arrive, and only at the audit store, so they are never sampled. The routing connector sends them on their own pipeline, and all 1,331 arrived. Metrics are already small counts. For traces, the policies keep every trace with an error, an approval or a write, and 10% of the rest. But they decide trace by trace: 99 runs with two traces lost one of them. A double refund’s first trace, the model’s decision, has no error or write. So it is usually dropped, while its approval trace is kept.

Check your understanding

Completion check for this module: draw the trace tree of the eight spans in the table above, and name the step that took longest. Say why the root span is not the answer.

Show the answer

The root is 4f2b, invoke_workflow support-resolution, with no parent: 4.97 s, the whole run. It has three children: af66 invoke_agent investigator (3.47 s), 0b3d invoke_agent resolver (1.49 s) and 0fc9 execute_tool send_reply. The investigator has three children: a model call (1.26 s), execute_tool get_order, and a second model call (2.20 s). The resolver has one child, c4c7, a model call of 1.49 s. Its parent ID says resolver, although the list shows it before the resolver: both start at the same moment. The longest step under the root is the investigator, 3.47 s. The longest single call is the investigator’s second model call, 2.20 s. The root lasts longer, but it is the whole run, not a step.

Recap

  • The Collector receives, processes and exports, in one pipeline per signal. The application knows one address.
  • Redaction deletes the GenAI text attributes and masks email addresses and card numbers. On the week with content capture on: 718 problems in the application’s export, 0 after the Collector.
  • Tail sampling decides per trace after 10 s: 434 traces became 275, every error, approval and write was kept, and 99 runs lost one of their two traces.
  • Audit events have their own route: never sampled, never in Loki. Fan-out sends the same traces to a second tool.
  • The Collector costs a program to run and memory to hold traces; it is where a team’s rules live.
  • Your project is now the snapshot end-of-m02: 81 tests pass.

You can now send a run’s telemetry through one place that cleans, samples and routes it. But a run of our workflow is still one span. The model calls, the tool calls and the 15-hour wait for Sam are not in it yet. Next, in Module 3, you record every model call with its model, tokens and prompt version, without its text.

Next: Model calls and tokens