Answer an audit question
- Lesson 22
- 40 min
- Module 6: Audit trails
Where we are
In the last three lessons, we built an audit trail: one event for every decision and change, chained, verified and sent through its own route. A trail is only worth its answers. In this lesson, we take the question that opened the course, why ticket T-90104 was refunded twice, and answer it from the trail, the traces and the payment service, for Grace and for an auditor. We start with a disputed card payment.
Lesson 22 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsAnswer an audit question
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Answer an audit question from the records. Check the trail, find the right run, read its story, and name the source of each part of the answer. Then write it twice, for a support lead and for an auditor, with every gap said out loud.
Before you startWhat to know and to have ready
What you need to know
- The audit event, its links, the chain and the anchor, from the last three lessons.
- How to read the double refund’s two traces, from Approvals, people and waiting time.
What you need on your computer
- The project as the last lesson left it, the snapshot
end-of-m06: 104 tests pass. - A terminal in
observable-workflow, with the virtual environment active. No Collector is needed: you read the saved week.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A disputed card payment
Section titled “A disputed card payment”A customer calls her bank: “The shop charged me twice.” The bank does not ask the shop’s cashier what she remembers. It opens its own records: two payments, their times, the card, the terminal, and the code that authorised each one.
Then it writes two answers. The customer gets a short letter in plain words: what happened, and what the bank does now. The file for the bank’s auditor has the record numbers, the times and the evidence, and it says what the bank could not check.
One story, from records, told twice. That is an audit answer.
The problem: the easy answers are wrong
Section titled “The problem: the easy answers are wrong”Grace asks: “Why did we refund ticket T-90104 twice?” The team has three quick answers, and each one fails.
- Amira’s log says
status=resolved writes=1 reply=sent. That is wrong: two refunds were paid. - Tomás’s waterfall shows two
request_refundspans. Grace cannot read it, nothing proves that it is complete, and in two weeks it is gone. - The ticket’s story:
python -m obs audit story T-90104 --data data/telemetry/weektells a clean story with one refund. It is the wrong run, as the failure case below shows.
A good answer has four properties. It comes from records that are complete and unchanged. It says which source each fact comes from. It is written for its reader. And it names what the records cannot say.
The method in five steps
Section titled “The method in five steps”- Check the record. Verify the trail, and write down the result.
- Find the run. A ticket can have several runs. Find the one with the writes.
- Read the story of that run, from the trail alone.
- Name a source for each part of the answer, and mark the gaps: the trail, the traces, the payment service, the release log.
- Write two answers: one for the person who asked, one for an auditor.
Steps 1 and 2: check the record, find the run
Section titled “Steps 1 and 2: check the record, find the run”python -m obs audit verify --data data/telemetry/weekThe audit trail is intact: 1331 records, the chain and the anchor match (head e378b7d00139f1c3).Write the head down. If anybody changes a record later, the head changes, and your answer still names the trail it came from.
Then find the runs of T-90104 that wrote something. The trail has one record per write, with its run ID:
grep '"ticket_id":"T-90104"' data/telemetry/week/audit.jsonl | grep '"type":"write"' | grep -o '"run_id":"[^"]*"' | sort | uniq -cIn PowerShell:
Select-String -Path data\telemetry\week\audit.jsonl -Pattern '"ticket_id":"T-90104"' | Where-Object { $_.Line -match '"type":"write"' } | ForEach-Object { [regex]::Match($_.Line, '"run_id":"[^"]*"').Value } | Group-Object | Select-Object Count, Name 2 "run_id":"R-1014-040" 1 "run_id":"R-1015-020" 1 "run_id":"R-1016-016"Three runs of the ticket paid something. One of them has two write calls: R-1014-040.
Step 3: the story from the trail
Section titled “Step 3: the story from the trail”python -m obs audit story R-1014-040 --data data/telemetry/weekAudit story of ticket T-90104, run R-1014-040: 6 records (the trail verifies).
610. Wed 14 Oct 17:52:02 The AI agent support-workflow (chat-strong, agent_structured) read get_order (LK-793996): ok. 611. Wed 14 Oct 17:52:06 The AI agent support-workflow (chat-strong, agent_structured) proposed: resolve (rule W2): refund 9.95 (RFD-LATE) on LK-793996. It needs a support agent or above to approve. Evidence it read: order:LK-793996. 671. Thu 15 Oct 09:17:12 Sam (usr-sam, support agent) approved the proposal of record 611 after 15 h 25 min (limit 100). 672. Thu 15 Oct 09:17:22 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-793996: TIMEOUT (attempt 1, key 797fbb5bbb9f8a7e). Authority: the approval of record 671. 673. Thu 15 Oct 09:17:22 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-793996: done (attempt 2, key 797fbb5bbb9f8a7e-retry1). Authority: the approval of record 671. 674. Thu 15 Oct 09:17:22 The ticket was closed: resolved (writes done: 1, reply: sent).
Changes asked of the systems: 2 write call(s) after 1 approval(s). refund 9.95 (RFD-LATE) on LK-793996: 2 call(s) that may have changed it (timeout, done), 2 different key(s)The story uses the trail only, with the names from data/people.json and Toronto time. Read what it proves:
- The agent proposed one refund of 9.95, under rule W2, after it read one order.
- Sam approved one proposal, within his limit of 100, 15 h 25 min later.
- The workflow sent two calls for that one approval. The first timed out. The second had a different key.
- The close counted one write done. That is where
writes=1in Amira’s log comes from.
And what it does not prove: whether the first call, the one that timed out, made a refund. “TIMEOUT” means the workflow stopped waiting, not that nothing happened. The story says it carefully: “2 call(s) that may have changed it”.
Step 4: a source for each part
Section titled “Step 4: a source for each part”The trail answers who and what. For the rest, you go to the other sources while they still exist.
The traces give the details and the release. From python -m obs trace R-1014-040 --data data/telemetry/week:
Run R-1014-040 | ticket T-90104 | chat-strong (agent_structured) | release 2.1.0 | language en…Trace 2 15ba0d9388d1b9e001af373f21c4b955 Thu 15 Oct 09:17:12 10.0 s 0.0 s |========================| 10.01 s approve support-resolution approved by usr-sam (agent, limit 100) 0.0 s |========================| 10.00 s execute_tool request_refund ERROR timeout LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e 10.0 s | =| 0.00 s execute_tool request_refund done LK-793996 9.95 RFD-LATE key 797fbb5bbb9f8a7e-retry1 10.0 s | =| 0.00 s execute_tool send_reply sentThe first call waited the full 10 seconds. And the run is on release 2.1.0.
The release log says what 2.1.0 changed. In the practice data, it is the releases list of data/telemetry/week/plan.json:
{'from': '2026-10-12T00:00:00-04:00', 'version': '2.0.0', 'retry': 'safe', 'reply_failure': 'person', 'hard_model': ''}{'from': '2026-10-14T09:00:00-04:00', 'version': '2.1.0', 'retry': 'naive', 'reply_failure': 'close', 'hard_model': 'chat-strong'}{'from': '2026-10-16T11:30:00-04:00', 'version': '2.1.1', 'retry': 'safe', 'reply_failure': 'person', 'hard_model': 'chat-strong'}Release 2.1.0, from Wednesday 09:00, has Tomás’s naive retry: after a timeout, it sends the call again with a new key. The safe retry of 2.0.0 checks first, with the same key. Release 2.1.1 brought the safe retry back on Friday at 11:30.
The payment service knows what it paid. That is the system of record: the system whose data is the truth for money. In the practice data, the payment service’s records of the week are not kept. The scenario truth stands in for them. The plan’s incident list says why the call was slow on Thursday from 08:30 to 09:30. It says: “the first call of each refund answers after the 10-second timeout, but the refund is made”. In data/telemetry/week/truth.json, R-1014-040 has duplicate_refunds 1. In your own runs, python -m resolver changes shows the payment service’s records in the practice database: two changes for the double refund.
Text description of the diagram
Read each lane from left to right. The trail has the proposal on Wednesday, and the approval, the two write calls and the close on Thursday. The traces have the same steps with their times, the 10-second timeout and the release, 2.1.0. The release log says that 2.1.0 retries with a new key. The payment service made both refunds, one per key. No single source tells the whole story.
Now put each question of an audit against each source:
Text description of the diagram
Seven rows. The trail answers who proposed, who approved with which authority, and what was sent. It shows that the record is complete and unchanged, and it can still answer in a year. It cannot say why the second call had a new key, or whether the call that timed out paid. The traces add the release, but only for 14 days and only if sampling kept them. The release log answers why. The payment service answers what was paid.
So the full answer needs three sources. That is not a weakness of this trail; it is the shape of every audit answer. The trail says who decided what. The system of record says what happened to the money. Neither can do the other’s job.
Step 5: two answers
Section titled “Step 5: two answers”For Grace, in plain words:
On Wednesday 14 October at 17:52, the AI agent proposed one refund of 9.95 for order LK-793996, which was late. It used rule W2. Sam approved it on Thursday at 09:17, within his limit of 100. The workflow then sent the refund to the payment service. The service was slow that morning and did not answer within 10 seconds, so the workflow tried again.
The release that ran that week, 2.1.0, retried with a new key. So the payment service took the second call as a new refund and paid both. No person and no model decided the second refund: the retry code did. Release 2.1.1 removed that retry on Friday at 11:30. One other customer was paid twice the same way, on ticket T-90606. We recommend that both refunds of 9.95 are recovered or accepted as goodwill, and that every retry of a write keeps its key.
For an auditor, the same story with its evidence. The trail exports it as Markdown:
python -m obs audit story R-1014-040 --data data/telemetry/week --markdown# Audit story: ticket T-90104
Run `R-1014-040`. 6 records (the trail verifies).
| # | Time (Toronto) | What happened ||---|---|---|| 610 | Wed 14 Oct 17:52:02 | The AI agent support-workflow (chat-strong, agent_structured) read get_order (LK-793996): ok. || 611 | Wed 14 Oct 17:52:06 | The AI agent support-workflow (chat-strong, agent_structured) proposed: resolve (rule W2): refund 9.95 (RFD-LATE) on LK-793996. It needs a support agent or above to approve. Evidence it read: order:LK-793996. || 671 | Thu 15 Oct 09:17:12 | Sam (usr-sam, support agent) approved the proposal of record 611 after 15 h 25 min (limit 100). || 672 | Thu 15 Oct 09:17:22 | The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-793996: TIMEOUT (attempt 1, key 797fbb5bbb9f8a7e). Authority: the approval of record 671. || 673 | Thu 15 Oct 09:17:22 | The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-793996: done (attempt 2, key 797fbb5bbb9f8a7e-retry1). Authority: the approval of record 671. || 674 | Thu 15 Oct 09:17:22 | The ticket was closed: resolved (writes done: 1, reply: sent). |
Changes asked of the systems: 2 write call(s) after 1 approval(s).- refund 9.95 (RFD-LATE) on LK-793996: 2 call(s) that may have changed it (timeout, done), 2 different key(s)
Source: the audit store (append-only, hash-chained). The trace IDs of each record are in the store.Add four short sections below the table, in your own words:
- The record checked: “
obs audit verify: 1,331 records, the chain and the anchor match, head e378b7d00139f1c3.” - Other sources: trace 1
f0433c4403e2767e1d69ece18a947526and trace 215ba0d9388d1b9e001af373f21c4b955, release 2.1.0 in the release log, and the payment service’s two refunds, with their keys. - Gaps: the trail records the outcome of each call as the workflow saw it, not what the payment service did. It does not record the release or the retry rule. The traces were kept here, but they are sampled and kept 14 days.
- Limits: the trail proves that the records did not change after they arrived. It cannot prove that every event arrived. The counts agree: 316 runs and 316 close records.
The run inspector, the download that this course uses later, exports exactly this Markdown from its audit screen.
Guided practice: the other double refund
Section titled “Guided practice: the other double refund”T-90606 was paid twice too, in run R-1014-051. Read its story:
python -m obs audit story R-1014-051 --data data/telemetry/weekAudit story of ticket T-90606, run R-1014-051: 7 records (the trail verifies).
642. Wed 14 Oct 19:31:58 The AI agent support-workflow (chat-strong, agent_structured) read get_order (LK-645556): ok. 643. Wed 14 Oct 19:32:02 The AI agent support-workflow (chat-strong, agent_structured) proposed: resolve (rule W2): refund 9.95 (RFD-LATE) on LK-645556. It needs a support agent or above to approve. Evidence it read: order:LK-645556. 666. Thu 15 Oct 09:16:27 Sam (usr-sam, support agent) approved the proposal of record 643 after 13 h 44 min (limit 100). 667. Thu 15 Oct 09:16:27 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-645556: UNAVAILABLE (attempt 1, key c18d8c3f31b823c4). Authority: the approval of record 666. 668. Thu 15 Oct 09:16:37 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-645556: TIMEOUT (attempt 2, key c18d8c3f31b823c4). Authority: the approval of record 666. 669. Thu 15 Oct 09:16:37 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-645556: done (attempt 3, key c18d8c3f31b823c4-retry2). Authority: the approval of record 666. 670. Thu 15 Oct 09:16:37 The ticket was closed: resolved (writes done: 1, reply: sent).
Changes asked of the systems: 3 write call(s) after 1 approval(s). refund 9.95 (RFD-LATE) on LK-645556: 2 call(s) that may have changed it (unavailable, timeout, done), 2 different key(s)Three calls this time. The first found the service unavailable: it changed nothing, so the story does not count it. The second kept the key and timed out. The third had a new key, -retry2. Write the two sentences for Grace that differ from the answer above. The scenario truth gives duplicate_refunds 1 for this run too.
Common mistake
The story of the ticket shows the wrong run
Symptom: you ask by ticket, and the story looks clean:
python -m obs audit story T-90104 --data data/telemetry/weekAudit story of ticket T-90104, run R-1016-016: 5 records (the trail verifies).
940. Fri 16 Oct 10:17:30 The AI agent support-workflow (chat-strong, agent_structured) read get_order (LK-793996): ok. 941. Fri 16 Oct 10:17:34 The AI agent support-workflow (chat-strong, agent_structured) proposed: resolve (rule W2): refund 9.95 (RFD-LATE) on LK-793996. It needs a support agent or above to approve. Evidence it read: order:LK-793996. 942. Fri 16 Oct 10:25:56 Sam (usr-sam, support agent) approved the proposal of record 941 after 8 min (limit 100). 943. Fri 16 Oct 10:25:56 The workflow (support-workflow) sent refund 9.95 (RFD-LATE) on LK-793996: done (attempt 1, key 797fbb5bbb9f8a7e). Authority: the approval of record 942. 944. Fri 16 Oct 10:25:56 The ticket was closed: resolved (writes done: 1, reply: sent).Cause: T-90104 has five runs in the replayed week, from Monday to Friday. When a ticket has several runs, story tells the last one. Friday’s run paid once, on release 2.1.0, and the payment service was fast again.
Diagnose: read the run ID in the first line. Compare the dates with the question: Grace asked about Thursday’s refunds.
Recover: always ask by run ID. Find the run from the trail, as in step 2, by its writes or by its date. Then write the run ID into the answer, so that anybody can check it.
Completion check: an answer from the trail alone
Section titled “Completion check: an answer from the trail alone”Here is a new question. Kwame asks: “On ticket T-90604, did we send the customer two return labels?” Answer it from the audit trail alone. Then name the gap that stops a full answer.
python -m obs audit story R-1014-024 --data data/telemetry/weekPredict first: does this story look different from the double refund’s?
The course team’s answer
Audit story of ticket T-90604, run R-1014-024: 6 records (the trail verifies).
543. Wed 14 Oct 13:24:10 The AI agent support-workflow (chat-small, agent) read get_order (LK-645334): ok. 544. Wed 14 Oct 13:24:13 The AI agent support-workflow (chat-small, agent) proposed: resolve (rule W6): return label for LADDER-3 (change_of_mind) on LK-645334. It needs a support agent or above to approve. Evidence it read: order:LK-645334. 552. Wed 14 Oct 13:36:16 Sam (usr-sam, support agent) approved the proposal of record 544 after 12 min (limit 100). 553. Wed 14 Oct 13:36:26 The workflow (support-workflow) sent return label for LADDER-3 (change_of_mind) on LK-645334: TIMEOUT (attempt 1, key 2ed3b2a93ab0f494). Authority: the approval of record 552. 554. Wed 14 Oct 13:36:26 The workflow (support-workflow) sent return label for LADDER-3 (change_of_mind) on LK-645334: done (attempt 2, key 2ed3b2a93ab0f494-retry1). Authority: the approval of record 552. 555. Wed 14 Oct 13:36:26 The ticket was closed: resolved (writes done: 1, reply: sent).
Changes asked of the systems: 2 write call(s) after 1 approval(s). return label for LADDER-3 (change_of_mind) on LK-645334: 2 call(s) that may have changed it (timeout, done), 2 different key(s)From the trail alone: the agent proposed one return label, and Sam approved it. Under that one approval, the workflow sent two calls with two keys. The first timed out, the second was done. So one label was certainly made, and maybe two.
The gap: the trail records that the first call timed out, not whether the order service made a label before the timeout. It looks exactly like the double refund. The traces cannot tell either: their duplicate_write flag fires on this run, and it is a false alarm. Only the order service, the system of record for labels, can say. In the practice data, the task file says it: "Simulated: the first call times out before the label is written." One label.
This is why the trail and the system of record decide, and the trace’s flag only warns.
When to write an audit answer, and what it costs
Section titled “When to write an audit answer, and what it costs”Write one for every question from outside engineering: a complaint, a manager’s question, an auditor’s sample, a regulator’s letter. Write the run ID, the trail’s head and the sources into it, so that anybody can check it later.
It costs a person’s time: the course team’s answer took six commands and two short texts. The cost grows when the trail is missing a field. Here, a release field in each record would answer why without the release log. And with the safe retry, a record of its reconcile check, “found: it happened”, would answer whether a timed-out call paid. Today that result is only in the traces. Adding a field is cheap now and impossible for the records that are already written.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why is 'writes done: 1' in the close record not a lie, and why is it still not the answer to 'how many refunds did the customer get'?
Show the answer
The workflow counted one write as done. The first call timed out, so for the workflow it had not succeeded; the second call was done. That is what it saw. But the payment service made a refund for each key, so the customer got two. The close record says what the workflow believed; the system of record, the payment service, says what happened to the money.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name the sources of the full answer to 'why did we refund T-90104 twice?', and say which part each one gives.
Show the answer
The audit trail gives who proposed (the agent, rule W2, evidence order LK-793996) and who approved (Sam, agent, limit 100). It gives what was sent (two calls, two keys, one approval), and it is verified. The traces give the 10-second timeout and the release, 2.1.0. The release log says that 2.1.0 retries with a new key. The payment service, here the practice data’s truth, says that both calls paid.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Completion check: answer 'On ticket T-90604, did we send two return labels?' from the audit trail alone, and name the gap that stops a full answer.
Show the answer
From the trail (run R-1014-024): one proposal of one return label, and one approval by Sam. Under it, two write calls with two keys: the first timed out, the second was done. So at least one label, perhaps two. The gap: the trail records the timeout, not whether the order service made the label before it; the trace’s duplicate_write flag cannot tell either. Only the order service’s records can. In the practice data, the task says that the first call times out before the label is written, so there was one label.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An audit answer has five steps: verify, find the run, read the story, name each source, and write for each reader.
- Ask by run ID. A ticket can have several runs, and the story of a ticket is its last run.
- The trail answers who decided what, for years. The system of record answers what happened. The release log answers why the code did it.
- Write every gap into the answer. For T-90104 and T-90604, the trail looks the same; only the payment and order services can tell them apart.
In this module, you built an audit trail that answers Grace’s question, and you saw where it needs other records. Your telemetry can now explain any single run, after the fact. But nobody wants to learn about a double refund from an auditor’s letter. Next, in Module 7, you decide what “working well” means for the help desk. Then you set alerts that fire while a problem is still small.