Skip to content
Français

An audit trail is not a log

  • Lesson 19
  • 30 min
  • Module 6: Audit trails

Where we are

In Module 5, we put a price on every run and found the expensive ones. Our traces now explain failures and costs. But Grace asks another kind of question: who approved the second refund of T-90104, and on whose authority? Traces are sampled, kept for two weeks and read by engineers. In this lesson, we see why an audit trail is a separate record, with its own rules. We go back to the shop of Module 1, and to its safe.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Decide what belongs in an audit trail and what belongs in traces. See with the replayed week’s numbers why a trace store cannot answer an audit question. Then read the first audit records of the double refund.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • The project as Module 5 left it: python -m pytest -q says 96 passed, 1 skipped (or 97 passed with the MCP package installed).
  • A terminal in observable-workflow, with the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

In Module 1, we met a small shop with four records. Two of them matter now: the camera over the door, and the ledger of the safe.

The camera films everybody, all day. Its video shows how each customer moved through the shop. The owner watches it when something strange happens, and the video is deleted after 30 days. If the camera skips a few minutes, nobody minds much.

The ledger is different. It has one line each time somebody opens the safe: who, when, how much, and who allowed it. Its pages are numbered, and nobody may tear one out. The accountant reads it once a year, and the law says how long the shop keeps it.

Now a customer says, a year later: “You paid me back twice.” The video is gone. The ledger answers, if every opening is in it, and if nobody changed a line.

The problem: Grace’s question, and the trace’s answer

Section titled “The problem: Grace’s question, and the trace’s answer”

Grace leads support at Larkfield. She asks the question of every audit. “Who approved the second refund of T-90104? On whose authority? And why did we pay twice?” An auditor may ask her the same question in a year.

Tomás answers at once: “Traces are the audit trail. Tempo has everything.” In this course, traces already explained the double refund. So why not? The replayed week gives four reasons, with numbers.

  1. Traces are sampled. In the course team’s run of the week through the Collector, tail sampling kept 275 of 434 traces. Of the 118 runs with an approval, 99 lost one of their two traces. The trace that holds the model’s decision has no error and no write, so it is usually the one that goes.
  2. Traces are kept for days, not years. Tempo deletes trace data after 14 days by default. The second refund happened on Thursday 15 October. Two weeks later, its trace is gone.
  3. Traces are written for engineers. execute_tool request_refund ERROR timeout means something to Amira. Grace and an auditor need a plain record of a decision. And a span carries the customer ID in clear text, which nobody may keep for years.
  4. Nothing protects a trace against change. A trace store may drop spans when it is full. Anybody with write access may change or delete them. Nobody would know.

Traces are like the camera. They are the right tool for “how did this run go?”. They are the wrong tool for “who allowed this, and can you prove it?”.

An audit trail is a record of who did what, when, on whose authority and with what evidence. Each line of it is an audit record: one decision or one change. The word comes from accounting: an auditor checks the accounts, and the trail leads from each payment back to the decision that allowed it.

Four rules make an audit trail different from every other signal:

  • Complete. Every decision and every change is in it. It is never sampled, and a lost record is a failure, not noise.
  • Kept longer. For as long as somebody may ask: in this course, 7 years. That is an assumption for Larkfield. The real period comes from the law and the contracts, so ask your legal team.
  • Protected against change. Nobody may change or remove a record without detection.
  • Read by other people. Support leads, auditors and data owners read it, not only engineers. So it holds facts in plain fields, not text, and no secrets.

A log is different. A log line is a message from a program to its developers: free text, often filtered, often deleted after days. Amira’s log said status=resolved writes=1 about the double refund, and nothing about a person or an authority.

A table that compares a trace and an audit trail. What it records: a trace records every step of one run, with model calls, tool calls, times and errors; the audit trail records decisions and changes: proposals, approvals, refusals, writes, data access and closes. One record is: a span with a name, a start, an end and a parent; or an event with fixed fields: who, what, when, on whose authority, with what evidence. Sampled: yes, the Collector kept 275 of 434 traces of the week; never, 1,331 of 1,331 audit records arrived. Kept for: 14 days, Tempo's default; 7 years, an assumption for Larkfield. Read by: engineers on call, Amira and Tomás; Grace, the auditors and Mei, read-only. Protection against change: none, a store can drop or change spans; append-only, a hash chain and an anchor. The customer: the customer ID in clear text; a pseudonym such as cust:6899a74a.
A trace and an audit record side by side: what each keeps, for how long, and for whom. The sampling numbers come from the course team's measured run of the replayed week through the Collector.
Text description of the diagram

Two columns. The trace is for engineers and answers how one run went. It holds every step, is sampled and kept 14 days, has no protection against change, and shows the customer ID. The audit trail is for the people who must answer for a decision. It holds decisions and changes as events with fixed fields. It is never sampled and is kept 7 years (an assumption). Grace, the auditors and Mei read it, read-only. A hash chain protects it, and the customer is a pseudonym.

The last three rows of the table are the next lessons. This lesson is about the first four rows: what goes in, and why it must all arrive.

Worked example: the same refund, two records

Section titled “Worked example: the same refund, two records”

The course project already holds the audit trail of the replayed week. In Module 1, you checked it with obs audit verify. Now you read it.

The obs tool of Module 5 can only verify the trail. Try to show the records of the double refund:

Terminal window
python -m obs audit show R-1014-040 --data data/telemetry/week
usage: python -m obs audit [-h] [--data DATA] [--audit AUDIT] [--markdown]
{verify} [target]
python -m obs audit: error: argument action: invalid choice: 'show' (choose from verify)

So you copy the new version of obs first.

  1. Copy four files from the snapshot end-of-m06: the new obs command, the audit story, Larkfield’s list of staff, and the new test.

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m06
    cp $S/obs/__main__.py $S/obs/story.py obs/
    cp $S/data/people.json data/
    cp $S/tests/test_obs_cli.py tests/
  2. Run the tests: python -m pytest -q. The last line says 97 passed, 1 skipped, 1 deselected. The new test reads the audit story of the double refund.

  3. Show the audit records of run R-1014-040:

    Terminal window
    python -m obs audit show R-1014-040 --data data/telemetry/week
    610 2026-10-14T21:52:02+00:00 data_access agent:support-workflow/agent_structured {"orders": ["LK-793996"], "status": "ok", "tool": "get_order"}
    611 2026-10-14T21:52:06+00:00 proposal agent:support-workflow/agent_structured {"actions": [{"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}], "design": "agent_structured", "outcome": "resolve", "rule": "W2"}
    671 2026-10-15T13:17:12+00:00 approval person:usr-sam/agent {"decision": "approved", "proposal_digest": "f28887b94cb6", "proposal_event_id": "2f48cd67-4d3a-428e-b4f2-b9ed3f266100", "wait_s": 55505.4}
    672 2026-10-15T13:17:22+00:00 write system:support-workflow/executor {"attempt": 1, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e", "operation_id": "797fbb5bbb9f8a7e", "outcome": "timeout", "tool": "request_refund"}
    673 2026-10-15T13:17:22+00:00 write system:support-workflow/executor {"attempt": 2, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e-retry1", "operation_id": "797fbb5bbb9f8a7e", "outcome": "done", "tool": "request_refund"}
    674 2026-10-15T13:17:22+00:00 run_closed system:support-workflow/executor {"helpdesk_status": "resolved", "outcome": "resolved", "reply_status": "sent", "writes_done": 1}
  4. Show the trace of the same run, as in Module 3: python -m obs trace R-1014-040 --data data/telemetry/week. Here is the second trace:

    Trace 2 15ba0d9388d1b9e001af373f21c4b955 Thu 15 Oct 09:17:12 10.0 s
    0.0 s |========================| 10.01 s approve support-resolution approved by usr-sam (agent, limit 100)
    0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e
    10.0 s | =| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e-retry1
    10.0 s | =| 0.00 s execute_tool send_reply sent

Read the two outputs side by side. Each line of audit show is one audit record: its number in the store, its time, its type, the actor who acted, and what was done.

  • Who: record 671 is a person, usr-sam, with the role agent. Records 672 and 673 are the workflow itself, the executor. The proposal, 611, is the AI agent with its design.
  • What: a read of the order, a proposal of one refund of 9.95, an approval, two write calls, and the close.
  • When: in UTC. 13:17:12+00:00 is 09:17:12 in Toronto, the time that the trace shows.
  • On whose authority: each write names the approval that allowed it, and the approval names the proposal. You see these links in the next lesson.

The trace has things that the trail does not: the two model calls with their tokens and cost, the 10-second timeout, the reply. The trail has things that the trace may not have tomorrow: all six records, for 7 years.

There is no record for the model calls, and none for the reply. That is on purpose. The trail records decisions and changes, not every step. The model’s work ends in one record, the proposal, with the evidence it read.

Rule one is “complete”. Here is what the course team’s Collector run kept of the replayed week, signal by signal:

Measured: the replayed week sent through the course's Collector (otelcol-contrib 0.162.0, standalone), from runs/collector.json. A whole run is a run with all of its traces kept.
Text description of the chart

Four bars. Spans: 63.5% kept (1287 of 2027). Traces: 63.4% (275 of 434). Whole runs, with every trace kept: 49.7% (157 of 316). Audit records: 100% (1331 of 1331), and none of them went to Loki.

Half of the runs lost part of their story in the trace store. That is fine for traces: the sampler kept every trace with an error, an approval or a write, so the engineers still see the problems. The audit records took another route, with no sampler on it, and all 1331 arrived.

Your own workflow records no audit trail yet

Section titled “Your own workflow records no audit trail yet”

The saved week has an audit trail because the course team’s version of the workflow wrote it. Yours does not, yet. Check it:

  1. Delete the work folder, so that you start clean.

  2. Run one ticket: python -m resolver run T-80008. The last line says Reply sent. Ticket: resolved.

  3. Verify your own trail: python -m obs audit verify.

    No audit trail at …/observable-workflow/work/audit/audit.jsonl.

Your workflow writes traces, metrics and events, but no audit record. The next lesson adds them.

Not every run changes something. Run R-1012-034 is the same ticket, T-90104, on Monday, with chat-small. The model asked the customer a question, and nothing was paid.

  1. Show its audit records:

    Terminal window
    python -m obs audit show R-1012-034 --data data/telemetry/week
    137 2026-10-12T19:31:38+00:00 data_access agent:support-workflow/agent {"orders": ["LK-793996"], "status": "ok", "tool": "get_order"}
    138 2026-10-12T19:31:40+00:00 proposal agent:support-workflow/agent {"actions": [], "design": "agent", "outcome": "ask_customer", "rule": "W3"}
    139 2026-10-12T19:31:40+00:00 run_closed system:support-workflow/executor {"helpdesk_status": "waiting_customer", "outcome": "asked_customer", "reply_status": "sent", "writes_done": 0}
  2. Show its trace: python -m obs trace R-1012-034 --data data/telemetry/week. It has five spans: the run, two model calls, the order lookup and the reply.

  3. Count: three audit records against five spans. There is no approval and no write, because nothing needed a person.

The run still has a proposal record with "actions": []. That matters. “The agent decided to change nothing” is a decision too, and an auditor may ask about it.

Your turn: which record answers which question?

Section titled “Your turn: which record answers which question?”

Change the question, not the run. For each question below, predict which record answers it: the audit trail, the trace, or both. Then check with the two commands of R-1014-040.

  1. How long did the second model call take?
  2. Which rule did the agent use?
  3. What was Sam’s approval limit?
  4. Was the reply sent?
  5. How many input tokens did the run use?
The answers
  1. The trace only: 4.17 seconds, the second chat chat-strong span of trace 1.
  2. Both: rule W2 in record 611, and on the invoke_agent span.
  3. Both: limit 100 on the approve span; the trail records it in record 671’s authority, which the next lesson shows in full.
  4. Both: reply_status: sent in record 674, and the send_reply span.
  5. The trace only: 4,586 tokens in. Tokens are cost, not a decision.

Where both answer, the trail is the one that is still there in a year.

Common mistake

The approval seems to come four hours after the write

Symptom: Sam compares the two outputs and says: “The trail says the approval was at 13:17. The trace says 09:17. Which one is wrong?”

Cause: neither. obs audit show prints each record’s at field as it is stored, in UTC: the suffix +00:00 says so. obs trace prints Larkfield’s local time, Toronto, which is 4 hours behind UTC in October.

Diagnose: look for the offset at the end of the time. +00:00 is UTC. A time with no offset, such as Thu 15 Oct 09:17:12, is local time.

Recover: compare times in one zone. Store every audit record in UTC, as the project does, and convert when you show it to a person. The audit story of the last lesson of this module prints Toronto time.

When to keep an audit trail, and what it costs

Section titled “When to keep an audit trail, and what it costs”

Keep an audit trail when somebody may have to answer for a decision later. Money, a customer’s data, a change in another system and a person’s approval are such decisions. An AI agent that only drafts text for a person to edit may need little more than a log. An agent that pays refunds needs a trail.

It costs something. The replayed week’s trail is 955 KB for 1,331 records: about 0.7 KB a record, 3 KB a run. At that rate, 7 years of Larkfield’s traffic is about 350 MB. That is small. The real costs are elsewhere: a second route for the records, a fixed schema to keep up, access rights, and a process for deletion requests. The next three lessons build each of them.

Do not put everything in it. A trail with every model call and every token is a trace store again, and it is harder to read. Record each decision and each change once, and link it to the trace for the details.

Check your understanding

Tomás says: 'Traces are the audit trail. Tempo has everything.' Give three reasons from the replayed week why Tempo cannot answer Grace's question in a year.

Show the answer

Traces are sampled: the Collector kept 275 of 434 traces, and 99 of the 118 runs with an approval lost one of their two traces. Tempo keeps traces 14 days by default, so the trace of the Thursday refund is gone two weeks later. Traces are written for engineers and carry the customer ID in clear text, so they cannot be given to auditors or kept for years. Also, nothing protects a trace against a change or a lost span.

Check your understanding

A run proposed no action and asked the customer a question. Should it have audit records at all? Which ones, and why?

Show the answer

Yes. R-1012-034 has three: a data access (order LK-793996), a proposal with no action (ask_customer, rule W3), and the close (writes_done 0). Reading a customer’s order and deciding to change nothing are both facts that someone may have to explain later. There is no approval and no write, because nothing needed a person or changed.

Check your understanding

Practical: run python -m obs audit show R-1014-040 --data data/telemetry/week. Name the record that shows who approved the refund, the records of the two write calls, and the time of the approval in Toronto time.

Show the answer

Record 671 is the approval: person usr-sam with the role agent. Records 672 (attempt 1, key 797fbb5bbb9f8a7e, timeout) and 673 (attempt 2, key 797fbb5bbb9f8a7e-retry1, done) are the two write calls. The approval’s time is 2026-10-15T13:17:12+00:00 in UTC, that is Thursday 15 October at 09:17:12 in Toronto.

Recap

  • An audit trail records who did what, when, on whose authority and with what evidence. A trace records how one run went.
  • The trail is complete, kept longer, protected against change and read by other people. In the replayed week, the trace store kept 63% of the traces and half of the whole runs. The audit route kept all 1,331 records.
  • It records decisions and changes, not every step: the double refund is six records, against nine spans.
  • obs audit show reads the trail. Its times are in UTC.

Your project now: Module 5’s files plus the new obs, obs/story.py, data/people.json and the new test, with 97 tests that pass. You can read the week’s trail. But your own workflow still writes no audit record, and you do not know yet what exactly a record must hold. Next, you design the record, and the workflow writes one for every decision and every change.

Next: Record decisions, approvals and changes