An audit trail is not a log
- Lesson 19
- 30 min
- Module 6: Audit trails
Where we are
In Module 5, we put a price on every run and found the expensive ones. Our traces now explain failures and costs. But Grace asks another kind of question: who approved the second refund of T-90104, and on whose authority? Traces are sampled, kept for two weeks and read by engineers. In this lesson, we see why an audit trail is a separate record, with its own rules. We go back to the shop of Module 1, and to its safe.
Lesson 19 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsAn audit trail is not a log
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Decide what belongs in an audit trail and what belongs in traces. See with the replayed week’s numbers why a trace store cannot answer an audit question. Then read the first audit records of the double refund.
Before you startWhat to know and to have ready
What you need to know
- The signals and their readers, from What to record and what not to.
- Tail sampling in the Collector, from The Collector.
- The approval as two linked traces, from Approvals, people and waiting time.
What you need on your computer
- The project as Module 5 left it:
python -m pytest -qsays96 passed, 1 skipped(or97 passedwith the MCP package installed). - A terminal in
observable-workflow, with the virtual environment active.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
The camera and the safe’s ledger
Section titled “The camera and the safe’s ledger”In Module 1, we met a small shop with four records. Two of them matter now: the camera over the door, and the ledger of the safe.
The camera films everybody, all day. Its video shows how each customer moved through the shop. The owner watches it when something strange happens, and the video is deleted after 30 days. If the camera skips a few minutes, nobody minds much.
The ledger is different. It has one line each time somebody opens the safe: who, when, how much, and who allowed it. Its pages are numbered, and nobody may tear one out. The accountant reads it once a year, and the law says how long the shop keeps it.
Now a customer says, a year later: “You paid me back twice.” The video is gone. The ledger answers, if every opening is in it, and if nobody changed a line.
The problem: Grace’s question, and the trace’s answer
Section titled “The problem: Grace’s question, and the trace’s answer”Grace leads support at Larkfield. She asks the question of every audit. “Who approved the second refund of T-90104? On whose authority? And why did we pay twice?” An auditor may ask her the same question in a year.
Tomás answers at once: “Traces are the audit trail. Tempo has everything.” In this course, traces already explained the double refund. So why not? The replayed week gives four reasons, with numbers.
- Traces are sampled. In the course team’s run of the week through the Collector, tail sampling kept 275 of 434 traces. Of the 118 runs with an approval, 99 lost one of their two traces. The trace that holds the model’s decision has no error and no write, so it is usually the one that goes.
- Traces are kept for days, not years. Tempo deletes trace data after 14 days by default. The second refund happened on Thursday 15 October. Two weeks later, its trace is gone.
- Traces are written for engineers.
execute_tool request_refund ERROR timeoutmeans something to Amira. Grace and an auditor need a plain record of a decision. And a span carries the customer ID in clear text, which nobody may keep for years. - Nothing protects a trace against change. A trace store may drop spans when it is full. Anybody with write access may change or delete them. Nobody would know.
Traces are like the camera. They are the right tool for “how did this run go?”. They are the wrong tool for “who allowed this, and can you prove it?”.
The idea: an audit trail
Section titled “The idea: an audit trail”An audit trail is a record of who did what, when, on whose authority and with what evidence. Each line of it is an audit record: one decision or one change. The word comes from accounting: an auditor checks the accounts, and the trail leads from each payment back to the decision that allowed it.
Four rules make an audit trail different from every other signal:
- Complete. Every decision and every change is in it. It is never sampled, and a lost record is a failure, not noise.
- Kept longer. For as long as somebody may ask: in this course, 7 years. That is an assumption for Larkfield. The real period comes from the law and the contracts, so ask your legal team.
- Protected against change. Nobody may change or remove a record without detection.
- Read by other people. Support leads, auditors and data owners read it, not only engineers. So it holds facts in plain fields, not text, and no secrets.
A log is different. A log line is a message from a program to its developers: free text, often filtered, often deleted after days. Amira’s log said status=resolved writes=1 about the double refund, and nothing about a person or an authority.
Text description of the diagram
Two columns. The trace is for engineers and answers how one run went. It holds every step, is sampled and kept 14 days, has no protection against change, and shows the customer ID. The audit trail is for the people who must answer for a decision. It holds decisions and changes as events with fixed fields. It is never sampled and is kept 7 years (an assumption). Grace, the auditors and Mei read it, read-only. A hash chain protects it, and the customer is a pseudonym.
The last three rows of the table are the next lessons. This lesson is about the first four rows: what goes in, and why it must all arrive.
Worked example: the same refund, two records
Section titled “Worked example: the same refund, two records”The course project already holds the audit trail of the replayed week. In Module 1, you checked it with obs audit verify. Now you read it.
The obs tool of Module 5 can only verify the trail. Try to show the records of the double refund:
python -m obs audit show R-1014-040 --data data/telemetry/weekusage: python -m obs audit [-h] [--data DATA] [--audit AUDIT] [--markdown] {verify} [target]python -m obs audit: error: argument action: invalid choice: 'show' (choose from verify)So you copy the new version of obs first.
-
Copy four files from the snapshot
end-of-m06: the newobscommand, the audit story, Larkfield’s list of staff, and the new test.Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m06cp $S/obs/__main__.py $S/obs/story.py obs/cp $S/data/people.json data/cp $S/tests/test_obs_cli.py tests/Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m06"Copy-Item $S\obs\__main__.py, $S\obs\story.py obs\Copy-Item $S\data\people.json data\Copy-Item $S\tests\test_obs_cli.py tests\Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m06cp $S/obs/__main__.py $S/obs/story.py obs/cp $S/data/people.json data/cp $S/tests/test_obs_cli.py tests/ -
Run the tests:
python -m pytest -q. The last line says97 passed, 1 skipped, 1 deselected. The new test reads the audit story of the double refund. -
Show the audit records of run R-1014-040:
Terminal window python -m obs audit show R-1014-040 --data data/telemetry/week610 2026-10-14T21:52:02+00:00 data_access agent:support-workflow/agent_structured {"orders": ["LK-793996"], "status": "ok", "tool": "get_order"}611 2026-10-14T21:52:06+00:00 proposal agent:support-workflow/agent_structured {"actions": [{"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}], "design": "agent_structured", "outcome": "resolve", "rule": "W2"}671 2026-10-15T13:17:12+00:00 approval person:usr-sam/agent {"decision": "approved", "proposal_digest": "f28887b94cb6", "proposal_event_id": "2f48cd67-4d3a-428e-b4f2-b9ed3f266100", "wait_s": 55505.4}672 2026-10-15T13:17:22+00:00 write system:support-workflow/executor {"attempt": 1, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e", "operation_id": "797fbb5bbb9f8a7e", "outcome": "timeout", "tool": "request_refund"}673 2026-10-15T13:17:22+00:00 write system:support-workflow/executor {"attempt": 2, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e-retry1", "operation_id": "797fbb5bbb9f8a7e", "outcome": "done", "tool": "request_refund"}674 2026-10-15T13:17:22+00:00 run_closed system:support-workflow/executor {"helpdesk_status": "resolved", "outcome": "resolved", "reply_status": "sent", "writes_done": 1} -
Show the trace of the same run, as in Module 3:
python -m obs trace R-1014-040 --data data/telemetry/week. Here is the second trace:Trace 2 15ba0d9388d1b9e001af373f21c4b955 Thu 15 Oct 09:17:12 10.0 s0.0 s |========================| 10.01 s approve support-resolution approved by usr-sam (agent, limit 100)0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e10.0 s | =| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e-retry110.0 s | =| 0.00 s execute_tool send_reply sent
Read the two outputs side by side. Each line of audit show is one audit record: its number in the store, its time, its type, the actor who acted, and what was done.
- Who: record 671 is a person,
usr-sam, with the roleagent. Records 672 and 673 are the workflow itself, theexecutor. The proposal, 611, is the AI agent with its design. - What: a read of the order, a proposal of one refund of 9.95, an approval, two write calls, and the close.
- When: in UTC.
13:17:12+00:00is 09:17:12 in Toronto, the time that the trace shows. - On whose authority: each write names the approval that allowed it, and the approval names the proposal. You see these links in the next lesson.
The trace has things that the trail does not: the two model calls with their tokens and cost, the 10-second timeout, the reply. The trail has things that the trace may not have tomorrow: all six records, for 7 years.
There is no record for the model calls, and none for the reply. That is on purpose. The trail records decisions and changes, not every step. The model’s work ends in one record, the proposal, with the evidence it read.
How much of the week arrives
Section titled “How much of the week arrives”Rule one is “complete”. Here is what the course team’s Collector run kept of the replayed week, signal by signal:
Text description of the chart
Four bars. Spans: 63.5% kept (1287 of 2027). Traces: 63.4% (275 of 434). Whole runs, with every trace kept: 49.7% (157 of 316). Audit records: 100% (1331 of 1331), and none of them went to Loki.
Half of the runs lost part of their story in the trace store. That is fine for traces: the sampler kept every trace with an error, an approval or a write, so the engineers still see the problems. The audit records took another route, with no sampler on it, and all 1331 arrived.
Your own workflow records no audit trail yet
Section titled “Your own workflow records no audit trail yet”The saved week has an audit trail because the course team’s version of the workflow wrote it. Yours does not, yet. Check it:
-
Delete the
workfolder, so that you start clean. -
Run one ticket:
python -m resolver run T-80008. The last line saysReply sent. Ticket: resolved. -
Verify your own trail:
python -m obs audit verify.No audit trail at …/observable-workflow/work/audit/audit.jsonl.
Your workflow writes traces, metrics and events, but no audit record. The next lesson adds them.
Guided practice: a run with no write
Section titled “Guided practice: a run with no write”Not every run changes something. Run R-1012-034 is the same ticket, T-90104, on Monday, with chat-small. The model asked the customer a question, and nothing was paid.
-
Show its audit records:
Terminal window python -m obs audit show R-1012-034 --data data/telemetry/week137 2026-10-12T19:31:38+00:00 data_access agent:support-workflow/agent {"orders": ["LK-793996"], "status": "ok", "tool": "get_order"}138 2026-10-12T19:31:40+00:00 proposal agent:support-workflow/agent {"actions": [], "design": "agent", "outcome": "ask_customer", "rule": "W3"}139 2026-10-12T19:31:40+00:00 run_closed system:support-workflow/executor {"helpdesk_status": "waiting_customer", "outcome": "asked_customer", "reply_status": "sent", "writes_done": 0} -
Show its trace:
python -m obs trace R-1012-034 --data data/telemetry/week. It has five spans: the run, two model calls, the order lookup and the reply. -
Count: three audit records against five spans. There is no approval and no write, because nothing needed a person.
The run still has a proposal record with "actions": []. That matters. “The agent decided to change nothing” is a decision too, and an auditor may ask about it.
Your turn: which record answers which question?
Section titled “Your turn: which record answers which question?”Change the question, not the run. For each question below, predict which record answers it: the audit trail, the trace, or both. Then check with the two commands of R-1014-040.
- How long did the second model call take?
- Which rule did the agent use?
- What was Sam’s approval limit?
- Was the reply sent?
- How many input tokens did the run use?
The answers
- The trace only: 4.17 seconds, the second
chat chat-strongspan of trace 1. - Both:
rule W2in record 611, and on theinvoke_agentspan. - Both:
limit 100on theapprovespan; the trail records it in record 671’s authority, which the next lesson shows in full. - Both:
reply_status: sentin record 674, and thesend_replyspan. - The trace only: 4,586 tokens in. Tokens are cost, not a decision.
Where both answer, the trail is the one that is still there in a year.
Common mistake
The approval seems to come four hours after the write
Symptom: Sam compares the two outputs and says: “The trail says the approval was at 13:17. The trace says 09:17. Which one is wrong?”
Cause: neither. obs audit show prints each record’s at field as it is stored, in UTC: the suffix +00:00 says so. obs trace prints Larkfield’s local time, Toronto, which is 4 hours behind UTC in October.
Diagnose: look for the offset at the end of the time. +00:00 is UTC. A time with no offset, such as Thu 15 Oct 09:17:12, is local time.
Recover: compare times in one zone. Store every audit record in UTC, as the project does, and convert when you show it to a person. The audit story of the last lesson of this module prints Toronto time.
When to keep an audit trail, and what it costs
Section titled “When to keep an audit trail, and what it costs”Keep an audit trail when somebody may have to answer for a decision later. Money, a customer’s data, a change in another system and a person’s approval are such decisions. An AI agent that only drafts text for a person to edit may need little more than a log. An agent that pays refunds needs a trail.
It costs something. The replayed week’s trail is 955 KB for 1,331 records: about 0.7 KB a record, 3 KB a run. At that rate, 7 years of Larkfield’s traffic is about 350 MB. That is small. The real costs are elsewhere: a second route for the records, a fixed schema to keep up, access rights, and a process for deletion requests. The next three lessons build each of them.
Do not put everything in it. A trail with every model call and every token is a trace store again, and it is harder to read. Record each decision and each change once, and link it to the trace for the details.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'Traces are the audit trail. Tempo has everything.' Give three reasons from the replayed week why Tempo cannot answer Grace's question in a year.
Show the answer
Traces are sampled: the Collector kept 275 of 434 traces, and 99 of the 118 runs with an approval lost one of their two traces. Tempo keeps traces 14 days by default, so the trace of the Thursday refund is gone two weeks later. Traces are written for engineers and carry the customer ID in clear text, so they cannot be given to auditors or kept for years. Also, nothing protects a trace against a change or a lost span.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A run proposed no action and asked the customer a question. Should it have audit records at all? Which ones, and why?
Show the answer
Yes. R-1012-034 has three: a data access (order LK-793996), a proposal with no action (ask_customer, rule W3), and the close (writes_done 0). Reading a customer’s order and deciding to change nothing are both facts that someone may have to explain later. There is no approval and no write, because nothing needed a person or changed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical: run python -m obs audit show R-1014-040 --data data/telemetry/week. Name the record that shows who approved the refund, the records of the two write calls, and the time of the approval in Toronto time.
Show the answer
Record 671 is the approval: person usr-sam with the role agent. Records 672 (attempt 1, key 797fbb5bbb9f8a7e, timeout) and 673 (attempt 2, key 797fbb5bbb9f8a7e-retry1, done) are the two write calls. The approval’s time is 2026-10-15T13:17:12+00:00 in UTC, that is Thursday 15 October at 09:17:12 in Toronto.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An audit trail records who did what, when, on whose authority and with what evidence. A trace records how one run went.
- The trail is complete, kept longer, protected against change and read by other people. In the replayed week, the trace store kept 63% of the traces and half of the whole runs. The audit route kept all 1,331 records.
- It records decisions and changes, not every step: the double refund is six records, against nine spans.
obs audit showreads the trail. Its times are in UTC.
Your project now: Module 5’s files plus the new obs, obs/story.py, data/people.json and the new test, with 97 tests that pass. You can read the week’s trail. But your own workflow still writes no audit record, and you do not know yet what exactly a record must hold. Next, you design the record, and the workflow writes one for every decision and every change.