What to record, and what not to
- Lesson 2
- 35 min
- Module 1: Why agents need their own observability
Where we are
We have seen that one log line per ticket cannot explain the double refund. The answer is not to log everything: some data must never leave the application. In this lesson, we sort the signals an agent needs (traces, metrics, events and audit records), what each one answers, how long we keep it and who may read it. We start in a small shop.
Lesson 2 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsWhat to record, and what not to
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Choose which records an agent needs, and which data must stay out of them. You see one event through four kinds of record, learn what each answers, how long it is kept and who may read it. Then you decide where prompt text, personal data and secrets belong: by default, nowhere.
Before you startWhat to know and to have ready
What you need to know
- The double refund of run R-1014-040 and its log line, from A refund nobody can explain.
- Your list of five questions about it, from the same lesson.
What you need on your computer
- Nothing new. You open the saved records yourself in the next lesson.
A small shop’s records
Section titled “A small shop’s records”A small shop keeps four kinds of records, and each one has its own job.
The till receipt says what one customer bought and paid. The shop keeps a copy for a few months. The camera over the door records the path of each customer through the shop: when they came in, where they stopped, when they left. Its video is deleted after 30 days, and only the owner may watch it. The daily total says how much the shop sold today, and how many customers came. It has no names, so everyone in the shop may see it, and the owner keeps it for years. The safe’s ledger is a signed book: who opened the safe, when, how much they took out, and who allowed it. Nobody may tear out a page. The accountant reads it once a year, and the law says how long to keep it.
Notice two more things. The camera does not film the keypad where customers type their card number. And nobody copies the ledger into the daily total. Each record holds only what its job needs.
The problem: “log everything”
Section titled “The problem: “log everything””Tomás, an AI engineer at Larkfield, has an answer to Grace’s questions: “Log every prompt and every answer. Then we can explain anything.”
Try it on the double refund. The model’s last answer in run R-1014-040 was the reply draft, and it says:
We'll submit a refund of your $9.95 CAD shipping fee for the team's approval.That text is true, and it explains nothing. The model asked for one refund. The second refund came from the workflow’s retry code, after a timeout, on another day. No prompt and no answer contains it, because no model was involved. A record of the model’s words cannot explain an action that the model did not take.
“Log everything” also has costs that Tomás does not see:
- Personal data spreads. A prompt holds the customer’s ticket and the order details. Real customers write their names, addresses, and sometimes their card numbers in a ticket. Every copy of the prompt is one more place that holds them.
- Telemetry lives long and travels far. Logs go to several tools, are kept for months, and many engineers can search them. A secret in a prompt, such as an API key that a customer pasted, is then readable by all of them.
- It costs. The course measured the workflow with prompt text off and on. A run made 9.3 KB of trace data without the text, and 26.7 KB with it: nearly three times more. When the course team turned it on for the replayed week, 718 model spans carried prompt text.
So the question is not “how much can we record?” It is “which record answers which question, and what must stay out of all of them?”
The idea: four signals, each with its own job
Section titled “The idea: four signals, each with its own job”Telemetry is the data that a running system sends about itself. Each kind of telemetry is called a signal. An agent needs four, as the shop needs four records:
- A trace is the path of one run through the system, like the shop’s camera. It is made of spans: one span per step, with a start, an end, a status and named values called attributes. A span can have child spans, so a trace is a tree of steps. In this course, one run is one trace, or two when a person approves later.
- A metric is a number that the system adds up over time, for all runs, like the daily total. Examples: runs by outcome, writes by outcome, tokens, cost. A metric has no run IDs, so it is small and cheap, and it never skips a run.
- A log or event is a record of one moment, with details. Amira’s line is a log. An event is a log record with a fixed name, such as
larkfield.run.closed, so that a program can count it. - An audit record says who decided what, on whose authority, and what changed, like the safe’s ledger. It is written for people outside engineering, such as Grace or an auditor. It must be complete, and nobody may change it without detection. The list of all audit records is the audit trail.
The first three come from OpenTelemetry, the open standard that this course uses to record telemetry. The audit record is Larkfield’s own format, because no telemetry standard promises that nothing is lost.
A worked example: one event, four views
Section titled “A worked example: one event, four views”Here is the same moment, the second refund of run R-1014-040, in each of the four records. Every value is pasted from Larkfield’s saved week, which you open in the next lesson.
Text description of the diagram
The log line is the one from the last lesson: resolved, one write, reply sent. The trace shows the four steps after the approval. First Sam’s approval, then the refund that timed out with key 797fbb5bbb9f8a7e. Then the same refund, done, with key 797fbb5bbb9f8a7e-retry1, and the reply. The metric counts refund calls by outcome for all runs. Between 09:00 and 10:00 on Thursday, 2 timed out and 2 were done, after 0 from 05:00 to 09:00. The audit record is number 673 of 1,331. It is a write, attempt 2, done: the refund of 9.95 with the second key. The approval b5188674 by usr-sam allowed it. It holds the hash of the record before it, so a changed record is found. The run ID joins the log line, the trace and the audit record. The trace ID joins the trace and the audit record. The metric has no IDs.
Read the four views against Grace’s questions:
- Which step paid the second refund? The trace: the second
request_refundspan, with the key797fbb5bbb9f8a7e-retry1. Its parent span is Sam’s approval. - Did it happen to other customers? The metric says that something happened twice in that hour: 2 timeouts, 2 refunds done. It cannot say who. The trace of each run can.
- Who allowed it? The audit record: the write names the approval that authorised it,
b5188674, and that approval names Sam, their role and their limit. - Did the first call pay? No record of the workflow can say for sure. The first call timed out: the workflow never heard the answer. Only the payment service’s own records, or a check that asks it, can answer. Module 3 adds that check.
The metric is worth a closer look. Here are the refund calls of Thursday, hour by hour:
- done
- timeout
- unavailable
Text description of the chart
Three lines, one point per hour from 05:00 to 17:00. Refund calls that were done: 05:00-06:00 0, 06:00-07:00 0, 07:00-08:00 0, 08:00-09:00 0, 09:00-10:00 2, 10:00-11:00 2, 11:00-12:00 4, 12:00-13:00 0, 13:00-14:00 2, 14:00-15:00 1, 15:00-16:00 0, 16:00-17:00 1, 17:00-18:00 0. Timeouts: 09:00-10:00 2, and 0 in every other hour. Calls refused because the service was unavailable: 09:00-10:00 1, and 0 in every other hour.
The timeouts appear in one hour only, 09:00 to 10:00, when Larkfield’s people started approving the night’s proposals. A metric is the signal that would show this on a dashboard while it happens, and an alert can fire on it. But a timeout is not yet a double refund. To know which runs paid twice, you go from the metric to the traces of that hour, and from a trace to its audit records. The IDs make that path possible.
How long each signal is kept, and who reads it
Section titled “How long each signal is kept, and who reads it”Each signal answers other questions, so each one has its own retention: how long it is kept before it is deleted. Each also has its own readers. This map shows the defaults of the tools that the course runs, and Larkfield’s choice for the audit trail:
Text description of the diagram
Four rows. Traces answer the path of one run. They may be sampled, Tempo keeps them 14 days by default, and the engineers on call read them. Metrics answer how often and how much, and count every run. Prometheus keeps them 15 days by default. Everyone with a dashboard reads them: on call, Omar for the product, Kwame for the cost. Logs and events answer what happened, with details. They may be filtered, Loki sets no retention limit by default, and engineers read them, or support for one ticket. Audit records answer who decided what and what changed. They are never sampled. The audit store keeps them 7 years, an assumption for Larkfield. Grace, the auditors and Mei read them, with read-only access.
The tools here are the ones in the course’s Grafana container, grafana/otel-lgtm 0.36.0. It holds Tempo 3.1.0 for traces, Prometheus 3.15.0 for metrics and Loki 3.7.8 for logs. The container sets no retention of its own, so each product’s default applies:
- Tempo deletes trace data after 14 days (
block_retention, default 336 hours). - Prometheus deletes samples after 15 days (
--storage.tsdb.retention.time, default15d). - Loki deletes nothing: retention is off by default, so logs “live forever”, in the words of Loki’s documentation, until the disk is full.
These defaults suit a practice container, not Larkfield. Three lessons follow from them:
- Retention is a decision, not a default. Loki’s default keeps every log line forever. If a log holds personal data, “forever” is a promise that Larkfield cannot keep to a customer who asks to be forgotten.
- The audit trail outlives the telemetry. A trace from Thursday is gone two weeks later. An auditor may ask about the refund in a year. That is one reason why the audit record is a separate signal in a separate store, not a trace. Larkfield keeps it 7 years in this course: an assumption, because the real period depends on the law and the contracts that apply. Ask your own legal team.
- Another tool has another limit. In Module 8, the same traces also go to Langfuse. Its free Hobby plan gives 30 days of data access. Plans change: check the current pricing page.
Who may read what
Section titled “Who may read what”Readers matter as much as time. A trace holds order IDs, tool arguments and error messages. The engineers who run the workflow need them, so traces are for them. A metric holds only counts and labels, so anyone may see a dashboard. An audit record names the person who approved, and it points to a customer. Larkfield keeps it in a store that only Grace, the auditors and Mei can read, and nobody can change.
In the audit record of the worked example, the customer is not C-61975. It is cust:6899a74add4687df, a pseudonym: a code computed from the customer ID with a secret key. Module 6 shows how deleting that key “forgets” a customer while the trail stays intact.
What stays out: off by default
Section titled “What stays out: off by default”Three kinds of data stay out of every signal, unless a person turns them on for a reason:
| Data | Why it stays out | What the signals record instead |
|---|---|---|
| Prompt and answer text | It holds the customer’s words, and it does not explain actions | the model, the tokens, the finish reason, the step number, the policy rule |
| Personal data (names, email addresses, card numbers, the ticket’s text) | each copy is one more place to protect, delete and explain | the ticket ID, the order ID, a pseudonym of the customer, the language |
| Secrets (API keys, tokens, passwords) | anyone who can read the telemetry could use them | nothing; at most the ID of the key |
“Off by default” means that the workflow’s code never puts these values into a span, a metric or a log record. The course’s project has one switch, OBS_CAPTURE_CONTENT, and its default is off. You turn it on only on your own computer, with practice data, for a short time, to debug one prompt.
The code is the first line of defence, and not the only one. In Module 2, the OpenTelemetry Collector, a program between the workflow and the tools, removes such values again before anything leaves the computer. This is called redaction. The course tested it with a made-up probe:
before: Write to sam.example@example.com, card 4111 1111 1111 1111after: Write to [email], card [card number]The AI security course teaches how to handle personal data in AI systems in full. This course applies its rules to telemetry.
Sampling: fine for some signals, never for the audit trail
Section titled “Sampling: fine for some signals, never for the audit trail”A busy system cannot keep every trace, so tools keep only some of them. This is sampling. It is fine for traces: you need enough examples of each kind of run, not all of them. It is never fine for the audit trail.
The course measured this. It sent the replayed week through the Collector of Module 2. Its rules keep every trace with an approval, a write, an error or an injected instruction, and 10% of the rest. Of 434 traces, 275 were kept. 99 runs lost one of their two traces, and 60 runs lost both. In the same run, all 1,331 audit events reached the audit store, and none went to the log store. A trace is evidence of how a run behaved. It is not a record that nothing was lost.
Your turn: mark the signal
Section titled “Your turn: mark the signal”Take the five questions that you wrote in the last lesson.
- For each question, write the signal that answers it: trace, metric, log or event, or audit record. If two signals are needed, write both, in the order that you would use them.
- If no signal of the workflow can answer it, write who can. For example, the payment service’s own records.
- For each question, write whether its answer contains personal data. If it does, write how the signal avoids holding it: an ID, a pseudonym or a count.
- Compare with the answers below.
The five questions from the last lesson, marked
- “Which call to the payment service made the second refund, and with which key?” Trace: the second
request_refundspan and its key. Then the audit record of that write, for the authority. - “Did the first call make a refund, even though it timed out?” None of the workflow’s signals. The payment service’s records, or a check that asks the service (Module 3).
- “Who approved the refund, and what exactly did that person see?” Audit record: the approval, with the person, role, limit and the proposal’s ID. The person is a staff ID,
usr-sam, not a customer. - “How many refunds timed out this week, and how many were sent again?” Metric:
larkfield.writesby outcome counts every run, with no personal data. Then the traces of those runs to see which were sent again. - “When did the ticket arrive, and how long did it wait for a person?” Trace: trace 1 starts when the run starts, and trace 2 records how long the approval waited. An event also marks the moment that the run started to wait.
Common mistake
The trace is there, but half of the run is missing
Symptom: in Grafana, or later with obs, a run has its first trace, with the proposal and approval.request, but no second trace. You cannot see the approval or the writes.
Cause: sampling. A Collector that keeps only some traces decides for each trace on its own. In the course’s measured run, 99 of 316 runs lost one of their two traces this way.
Diagnose: look for the run in the audit trail. If its approval and write records are there, the run happened and the trace was dropped. If they are not, the run stopped before the approval.
Recover: for the question “what happened in this run?”, use the audit trail; it is never sampled. For “how do runs like this behave?”, sampling must keep both traces of an approved run. Module 2 shows why a rule that decides trace by trace loses one of them. Module 3 links the two traces and says how to keep both.
When to record more, and what it costs
Section titled “When to record more, and what it costs”Sam, a support agent who is new to all this, asks: “If metrics are so cheap, why not use only metrics?”
Amira answers: “Because a count tells you that something went wrong, not where. On Thursday, the metric said two timeouts. Only the traces said which runs, and only the audit trail said who approved them.”
Record each signal for the question that it answers, and no more. The course measured the cost of the workflow’s full telemetry, as it is at the end of the course. A replayed run’s own code took a median of 0.53 ms with telemetry off and 2.42 ms with it on: 1.9 ms more per run. A real run spends seconds waiting for the model, so 2 ms is small. Storage and care cost more than time: someone must keep the names stable, delete data on time and keep private text out.
Do not add a signal that nobody will read. Every record is a cost, and a risk if it holds the wrong data.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'We log every prompt and answer, so we can explain anything.' Give one reason from the double refund why this is wrong, and one cost of doing it.
Show the answer
The second refund was not in any prompt or answer. The model asked for one refund, and the workflow’s retry code sent a second write after a timeout. A record of the model’s words cannot explain an action that no model took. The cost: prompts hold the customer’s ticket text and order details, so personal data spreads to every tool that keeps the logs. The trace data was also nearly three times larger per run, 26.7 KB against 9.3 KB.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why may traces be sampled, while audit events are never sampled? Use the course's measurement.
Show the answer
A trace answers how runs behave, and enough examples of each kind of run answer that. An audit record answers who decided what and what changed, for one specific run, often months later. A missing record is a gap that nobody can fill. In the measured run, sampling kept 275 of 434 traces, and 99 runs lost one of their two traces. All 1,331 audit events reached the audit store.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Loki's default keeps logs forever, and Tempo's keeps traces 14 days. Why is neither of these right for Larkfield's audit trail?
Show the answer
Fourteen days is too short: an auditor or a customer may ask about a refund months later. Forever is too long for data that points to customers, because Larkfield must be able to delete or forget it. Logs and traces can also be sampled, filtered or changed. The audit trail needs its own store, with its own retention (7 years here, an assumption), read-only access and a way to detect changes.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An agent needs four signals. Traces show the path of one run, metrics count all runs, and logs and events record moments with details. Audit records say who decided what, on whose authority, and what changed.
- One event, four views. The trace has the second refund’s span and key, the metric its count, and the audit record its authority. The log line has none of them.
- No record of the workflow can say whether a timed-out write happened. Only the system that was written to, or a check that asks it, can.
- Retention and readers differ. By default, Tempo keeps 14 days, Prometheus 15 days and Loki forever. The audit store keeps 7 years (an assumption), read-only for a few people.
- Prompt and answer text, personal data and secrets are off by default. The Collector redacts them again before export. Traces may be sampled; the audit trail never is.
You can now say which record answers which question, and what stays out of all of them. But so far you have only read about these records. Next, you set up the workflow on your computer, run recorded tickets and read the saved telemetry of Larkfield’s week.
Next: Meet the live workflow