Skip to content
Français

Record decisions, approvals and changes

  • Lesson 20
  • 40 min
  • Module 6: Audit trails

Where we are

In the last lesson, we saw that an audit trail is not a log: it is complete, kept for years and read by other people. We read the trail of the replayed week. But our own workflow writes no audit record yet, and we have not said what a record must hold. In this lesson, we fix the fields of an audit event, link each event to its trace, keep text and secrets out, and make the workflow write one for every decision and every change. We start at a pharmacy counter.

What you will be able to do

Give every audit event the same fixed fields: who, what, when, on whose authority, with what evidence, and where to look in the traces. Make the workflow write one event for each proposal, approval, refusal, write, data access, handoff and close, and keep text and secrets out of it.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project as the last lesson left it: 97 tests pass, and show_spans.py (from Module 3) is in the project folder.
  • A terminal in observable-workflow, with the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

Some medicines are controlled: a pharmacy must record each one that it gives out. It keeps a register with printed columns. Each line has the date, the patient’s number, the medicine, the amount, the doctor who prescribed it, and the pharmacist who checked it.

There is no column for “how the patient seemed”. That is on purpose. Every line can be checked against a prescription, and an inspector can count lines. A line without the doctor’s name is useless. A long note in the margin is a risk: it is private, and nobody can check it.

The register works because its columns are fixed. Every pharmacist writes the same facts in the same places.

The problem: “just log the prompt and the answer”

Section titled “The problem: “just log the prompt and the answer””

Tomás has a simple plan: “We log every prompt and every answer as the audit record. Then we can explain anything.” Look at what that record would hold for the double refund. The agent’s proposal of T-90104 explains itself in a paragraph:

Why: The order is 12 business days late, with a $9.95 shipping fee and no prior refunds. W2 allows a
shipping-fee refund for approval by any agent; W3's lost-delivery threshold has not been reached.

And the reply draft tells the customer about her watering can and her dates. Three things go wrong when text like this is the audit record:

  • The facts are hidden in sentences. “Who approved, with which limit?” is not in any field. Nobody can count approvals by role, or check that every write had one.
  • The text is the model’s words, not a fact. The “Why” may be wrong. An audit record must say what happened, not what a model wrote about it.
  • The text holds personal data, for 7 years. A trail that keeps the customer’s words must be protected and deleted like the customer’s data. Then it is no longer a safe, long record.

And every developer would write a different message. In a year, nobody knows which messages mean an approval.

The fix is the pharmacy’s: fixed columns. A schema is a fixed list of fields, each with one meaning, that every record must have. A program checks each record against it before the record is stored.

The project’s schema is larkfield.audit/1 in auditstore/schema.py. Every audit event has the same 13 top-level fields, and one of 7 types:

Type Who acts What it answers
data_access the AI agent which customer data did the agent read?
proposal the AI agent what did the agent propose, under which rule, from which evidence?
approval a person who allowed it, with which role and limit, after how long?
refusal a person, or the rules who said no, or who tried to approve and was not allowed?
write the workflow what change was sent, which attempt, which key, with what result, on whose approval?
handoff the AI agent when did the ticket go to a person, and why (a code)?
run_closed the workflow how did the ticket end?

The actor is the one who acted. It has a kind, person, agent or system, and an ID. A person is a staff ID such as usr-sam, never a name. The names are in data/people.json, which the last lesson of this module uses for its story.

Worked example: Sam’s approval, field by field

Section titled “Worked example: Sam’s approval, field by field”

Print record 671 of the replayed week in full:

Terminal window
sed -n 671p data/telemetry/week/audit.jsonl | python -m json.tool
{
"event": {
"action": {
"decision": "approved",
"proposal_digest": "f28887b94cb6",
"proposal_event_id": "2f48cd67-4d3a-428e-b4f2-b9ed3f266100",
"wait_s": 55505.4
},
"actor": {
"id": "usr-sam",
"kind": "person",
"role": "agent"
},
"at": "2026-10-15T13:17:12+00:00",
"authority": {
"limit": 100.0,
"needs_role": "agent",
"role": "agent",
"rule": "refund approval limits"
},
"event_id": "b5188674-35aa-4e51-b9f5-71e2d624d19f",
"evidence": [],
"run_id": "R-1014-040",
"schema": "larkfield.audit/1",
"span_id": "cc0db5a2598f6be9",
"subject": {
"customer": "cust:6899a74add4687df"
},
"ticket_id": "T-90104",
"trace_id": "15ba0d9388d1b9e001af373f21c4b955",
"type": "approval"
},
"hash": "ac63a5ff0037f764c1686e17c8916cfdb860973a0c72cac21756ddf8ebe3a171",
"prev": "8dd09d48a0736258704950715fb90485225a3604ec76a3713c8114b2c52e6b77",
"seq": 671
}

The same record, grouped by the question that each field answers:

Record 671 of the replayed week, grouped by question. Who: actor, kind person, id usr-sam, role agent. When: at 2026-10-15T13:17:12+00:00, in UTC. What: type approval; action with decision approved, wait_s 55505.4, proposal_event_id 2f48cd67 and proposal_digest f28887b94cb6. On whose authority: role agent, limit 100.0, needs_role agent, rule refund approval limits. With what evidence: an empty list; the proposal holds it. About whom: subject customer cust:6899a74add4687df, a pseudonym. Where to look next: run_id R-1014-040, ticket_id T-90104, trace_id 15ba0d93 and span_id cc0db5a2598f6be9. The store adds: seq 671, prev 8dd09d48 and hash ac63a5ff.
One audit event, Sam's approval of the double refund (record 671 of the replayed week), with its fields grouped by the question they answer.
Text description of the diagram

Each group of fields answers one question. The actor says who: a person, usr-sam, with the role agent. The time says when, in UTC. The type and the action say what: approved, after 55,505 seconds, for the proposal with that event ID and that digest. The authority says on whose authority: the role agent, its limit of 100 and the rule. The evidence is empty for an approval, because the proposal holds it. The subject is a pseudonym. The run, ticket, trace and span IDs say where to look next. The store adds the sequence number and two hashes, for the next lesson.

Some fields need a word more:

  • authority holds what made the decision allowed: Sam’s role, its limit, the role the proposal needed, and the rule’s name. An auditor can check 9.95 ≤ 100 from this record alone.
  • proposal_digest is a fingerprint of the proposal that Sam saw, from the agents course. If the proposal changes after the approval, the workflow refuses to write.
  • proposal_event_id names the proposal’s own audit event. So the approval points to the proposal, and each write points to the approval, with approval_event_id.
  • trace_id and span_id name the span that was open when the event was written. For an approval, that is the approve support-resolution span in trace 2. This is the link to the details, for as long as the trace exists.

Here are the six records of the run and their links:

The six audit records of run R-1014-040, with the 59 records of other runs between 611 and 671. 610 data_access by the agent: read get_order LK-793996, ok; trace 1, span 8027f3b3. 611 proposal by the agent: refund 9.95 RFD-LATE, rule W2, needs agent; trace 1, span 508ef7aa. 671 approval by person usr-sam: approved, role agent, limit 100; its proposal_event_id points to record 611; trace 2, span cc0db5a2. 672 write by the workflow: refund 9.95, attempt 1, key ending 5bbb9f8a7e, timeout; its approval_event_id points to record 671; trace 2, span c4e1313c. 673 write: attempt 2, key ending a7e-retry1, done; it also points to record 671; trace 2, span c3295385. 674 run_closed: resolved, writes_done 1, reply sent; trace 2, span cc0db5a2.
Run R-1014-040 in the audit trail: each write names the approval that allowed it, the approval names the proposal, and every record names its span in the traces.
Text description of the diagram

Six records, from the read of the order to the close. The approval points back to the proposal with proposal_event_id. Both writes point back to the same approval with approval_event_id: one approval allowed two calls. Every record also names a trace and a span, so a reader can open the details while the traces exist.

Read the two writes again. Both name record 671. One approval of one refund led to two calls with two keys. The trail does not hide it: an auditor finds it by following the links.

The schema refuses two things, in _walk():

  • Fields that hold text: text, reply, reason, message, prompt, answer, note, body, content and goal.
  • Any string longer than 80 characters. IDs, codes and names of tools and rules are short. A sentence is usually longer.

So the prompt, the answer, the reply draft and the model’s “Why” never enter the trail. The rule’s name, W2, does. Secrets stay out too: no API key and no token is ever a field. And the customer is a pseudonym, cust:6899a74a…, made by the store from the customer ID. The next lesson shows how, and why that helps with a deletion request.

The project also keeps the agents course’s approval note out. When Sam types --note, the note stays in the run’s saved state, not in the trail.

The workflow writes each audit event with one function, observe.audit(). It fills the fields that every event has: the time from the run clock, the run and ticket IDs and the customer. It also adds the trace and span IDs of the open span. The caller gives the rest.

Event Where the workflow writes it
data_access observe.py, after a tool span of get_order, get_customer_orders or get_payments
proposal observe.py, at the end of the run span
approval, refusal live.py, in _decide(), inside the approve span
write observe.py, in write_done(), once for each call
handoff, run_closed live.py, in close()

audit() sends the event as an OpenTelemetry event named larkfield.audit.<type>, with the record as its body. Then a new class in telemetry.py, AuditRouter, takes every event with that name and appends it to the audit store, work/audit/audit.jsonl. It never samples, and if the store refuses an event, it raises an error.

  1. Copy the three changed files of the workflow and the two tests:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m06
    cp $S/resolver/observe.py $S/resolver/live.py $S/resolver/telemetry.py resolver/
    cp $S/tests/test_approval_traces.py $S/tests/test_audit.py tests/
  2. Open resolver/observe.py and find def audit(. Read how it fills trace_id and span_id. Then find the calls to audit( in observe.py and live.py, and match them to the table above.

  3. Run the tests: python -m pytest -q. The last line says 103 passed, 1 skipped, 1 deselected. The new tests check the chain, the anchor, the refusal of text, a deletion request, a refused approval, and that audit events are never sampled.

  4. Delete work/. Then run one ticket: python -m resolver run T-80008. It ends with Reply sent. Ticket: resolved.

  5. Show your trail:

    Terminal window
    python -m obs audit show
    1 … data_access agent:support-workflow/agent {"orders": ["LK-581106"], "status": "ok", "tool": "get_order"}
    2 … proposal agent:support-workflow/agent {"actions": [], "design": "agent", "outcome": "reply_only", "rule": "W1"}
    3 … run_closed system:support-workflow/executor {"helpdesk_status": "resolved", "outcome": "resolved", "reply_status": "sent", "writes_done": 0}

    The times are your own, in UTC.

  6. Verify it: python -m obs audit verify. It says The audit trail is intact: 3 records, the chain and the anchor match (head …). Your head differs.

Three records for a reply with no change: the read, the decision, the close.

Guided practice: the double refund in your own trail

Section titled “Guided practice: the double refund in your own trail”
  1. Delete work/, so that the record numbers match the ones below.

  2. Replay the double refund, as in Module 3:

    Terminal window
    python -m resolver run T-90104 --variant agent_structured --model chat-strong
    python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --fault request_refund:timeout_after_write --naive-retry

    The approval ends with Reply sent. Ticket: resolved.

  3. Show the trail of the ticket: python -m obs audit show T-90104.

    1 … data_access agent:support-workflow/agent_structured {"orders": ["LK-793996"], "status": "ok", "tool": "get_order"}
    2 … proposal agent:support-workflow/agent_structured {"actions": [{"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}], "design": "agent_structured", "outcome": "resolve", "rule": "W2"}
    3 … approval person:usr-sam/agent {"decision": "approved", "proposal_digest": "f28887b94cb6", "proposal_event_id": "55f75b4b-88f5-4ef7-9aee-925ddaf85bd0", "wait_s": 0.0}
    4 … write system:support-workflow/executor {"attempt": 1, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e", "operation_id": "797fbb5bbb9f8a7e", "outcome": "timeout", "tool": "request_refund"}
    5 … write system:support-workflow/executor {"attempt": 2, "effect": {"amount": 9.95, "order_id": "LK-793996", "reason_code": "RFD-LATE", "tool": "request_refund"}, "idempotency_key": "797fbb5bbb9f8a7e-retry1", "operation_id": "797fbb5bbb9f8a7e", "outcome": "done", "tool": "request_refund"}
    6 … run_closed system:support-workflow/executor {"helpdesk_status": "resolved", "outcome": "resolved", "reply_status": "sent", "writes_done": 1}

    The same six records as R-1014-040 in the week. The digest of the proposal, f28887b94cb6, is the same too: it is the same proposal. The event IDs are new.

  4. Find the approval’s span in the traces. Print the approve span, then the approval’s span_id:

    Terminal window
    python show_spans.py approve | head -1
    grep '"type":"approval"' work/audit/audit.jsonl | grep -o '"span_id":"[^"]*"'
    approve support-resolution trace 826f3506 span c07282c8 parent none
    "span_id":"c07282c8e932199d"

    The span ID begins the same: c07282c8. Your IDs differ, but they must match each other.

  5. Do the same for the writes: python show_spans.py request_refund, and "type":"write" in the grep. The course team’s run gave spans a52157aa and 32df5564, and the two write records named the same two spans.

Each write record leads to its own span: the one with ERROR timeout, and the one with the new key.

One test proves the first rule. Run it on its own:

Terminal window
python -m pytest -q tests/test_audit.py -k sampled -v
======================= 1 passed, 4 deselected in 0.35s ========================

Open tests/test_audit.py and read test_audit_events_are_never_sampled. It sets the trace sampler to ALWAYS_OFF, so no trace is kept at all. Then it runs a ticket. The traces file does not exist, and the audit store still has the data access and the proposal. Sampling decides about spans, and audit events are not spans.

The replayed week has 1331 audit records for 316 runs. Here they are by type:

The 1,331 audit records of the replayed week (316 runs), by type, from data/telemetry/week/audit.jsonl. Replayed decisions, simulated traffic.
Text description of the chart

Seven types. Data access: 347. Closed: 316, one per run. Proposal: 300. Write: 156. Approval: 118. Handoff: 94. Refusal: 0.

Every run has one close record. Only 300 of 316 runs have a proposal. The other 16 stopped before they could propose. The model provider refused 3 calls (the content filter of T-80002). The loop’s limits stopped the local model 13 times: 10 for too many errors, 3 for a repeated call. Each of them has a handoff record with the stop’s code. Of the 156 writes, 140 are done. The others are 10 timeouts, 4 calls to a service that was down, and 2 reshipments that the order service refused. There is no refusal of an approval in the week: nobody tried to approve beyond their limit. Your next task makes one.

Change one condition: the approver’s limit. T-65273 proposes a refund of 119.00, and an agent may approve up to 100. Predict before you run it: which records does the trail get, and which one is missing?

Terminal window
python -m resolver run T-65273
python -m resolver approve T-65273-agent-chat-small-1 --as agent
python -m obs audit show T-65273
The course team’s result
Not approved: An agent may not approve this: it needs a team lead (refund approval limits).
7 … data_access agent:support-workflow/agent {"orders": ["LK-507862"], "status": "ok", "tool": "get_order"}
8 … data_access agent:support-workflow/agent {"orders": ["LK-507862"], "status": "ok", "tool": "get_payments"}
9 … proposal agent:support-workflow/agent {"actions": [{"amount": 119.0, "order_id": "LK-507862", "payment_id": "PAY-507862-2", "reason_code": "RFD-DOUBLE", "tool": "request_refund"}], "design": "agent", "outcome": "resolve", "rule": "W7"}
10 … refusal person:usr-sam/agent {"code": "role_below_limit", "decision": "refused_by_rules", "proposal_event_id": "d2d50c4b-ef6c-4b49-b4df-75b275e60d7d"}

Two reads this time: the agent read the order and the payments. The refusal is Sam’s attempt, refused by the rules: role_below_limit. Print record 10 in full (sed -n 10p work/audit/audit.jsonl | python -m json.tool): its authority says "role": "agent", "limit": 100.0 and "needs_role": "team_lead". There is no write and no close: the run still waits for a team lead. An auditor can see that the limits worked, and who tried.

Common mistake

A developer puts the approval note into the event

Symptom: Amira wants the trail to keep Sam’s note. She adds "note": note to the approval’s action in live.py, line 173. Then the approval of T-90104 fails:

exit status 1
File "…/observable-workflow/auditstore/schema.py", line 31, in _walk
raise AuditEventError(f"{path}.{k}: text fields are not allowed in an audit event")
auditstore.schema.AuditEventError: event.action.note: text fields are not allowed in an audit event

python -m resolver changes says 0 change(s) in the practice database, and the trail has only the read and the proposal. The course team made this change on purpose, then undid it.

Cause: the schema refuses the field note. In file mode, AuditRouter appends each event at once and lets the error stop the process. The approval event comes before the writes, so the refund is never sent.

Diagnose: the last line names the field: event.action.note. Search the code for the audit( call that sets it.

Recover: keep text out of the event. If the reason matters, record a code from a fixed list, such as "code": "customer_called". Keep the note where personal data is handled, with its own retention. Stopping here is the right behaviour: a refund with no approval record would be worse than no refund.

The schema is a net, not a proof. Create try_text.py in the project folder. It checks three approval events with text in them:

"""Three approval events with text in them: which ones does the schema refuse?"""
from auditstore.schema import AuditEventError, check
tries = {
"a note field": {"decision": "approved", "note": "Customer was upset, refund ok"},
"a long code": {"decision": "approved",
"code": "Approved by Sam because the customer phoned twice and was very upset about the delay"},
"a short code": {"decision": "approved", "code": "customer phoned twice, upset"},
}
for name, action in tries.items():
event = {"schema": "larkfield.audit/1", "event_id": "e1", "type": "approval", "at": "2026-10-15T13:17:12+00:00",
"actor": {"kind": "person", "id": "usr-sam", "role": "agent"}, "run_id": "R-1014-040",
"ticket_id": "T-90104", "trace_id": "", "span_id": "", "subject": {"order_id": "LK-793996"},
"action": action, "authority": {}, "evidence": []}
try:
check(event)
print(f"{name}: accepted")
except AuditEventError as e:
print(f"{name}: refused: {e}")

Run it with python try_text.py:

a note field: refused: event.action.note: text fields are not allowed in an audit event
a long code: refused: event.action.code: a string of 84 characters looks like text, not an ID
a short code: accepted

A short sentence in a field with an allowed name passes: customer phoned twice, upset. So the check catches the common mistakes, and a code review catches the rest. A list of allowed codes per field would catch more, at the cost of more upkeep.

When to record an event, and what it costs

Section titled “When to record an event, and what it costs”

Record an audit event for each decision and each change that somebody may have to explain. That is a proposal, an approval or a refusal, a write, a read of customer data, a handoff and the end of the ticket. Do not record each model call or each retry of a read: the traces hold those.

The costs are small in time and real in upkeep. The course team measured 2.1 audit events per proposal-only run, about 1.5 KB, written and synced to disk one by one. The workflow’s own code took 1.9 ms more per run with all its telemetry on, against seconds of model time. The upkeep is the schema: when the workflow gains a new kind of decision, the schema and its tests change first.

Write each event where the decision happens, inside the span that does it. Then trace_id and span_id are right without any extra work.

Check your understanding

Tomás wants to log every prompt and answer as the audit record. Give two reasons against it, and say what the project records instead for the proposal of T-90104.

Show the answer

The facts are hidden in sentences: nobody can count approvals by role or check that each write had one. The text is the model’s words, which may be wrong, not the facts of what happened. And it holds personal data that would be kept for 7 years. Instead, the proposal is one event with fixed fields. The actor is the agent, with its design and model. The action is outcome resolve, rule W2, one refund of 9.95 RFD-LATE on LK-793996. The authority is needs_role agent, and the evidence is order:LK-793996. The event also has the run, ticket, trace and span IDs, and a pseudonym for the customer.

Check your understanding

Predict: you run T-90103 with --variant agent_structured --model chat-strong (it proposes a reship and a return label) and approve it as an agent, with no fault. Which audit records does the trail get, and which approval does each write name?

Show the answer

Six records. A data_access for get_order. A proposal with two actions, reship_item and create_return_label. An approval by usr-sam with the role agent. Two write records, one per action, each attempt 1 with its own key and outcome done. A run_closed with writes_done 2. Both writes name the same approval in authority.approval_event_id, and the approval names the proposal in action.proposal_event_id.

Check your understanding

Practical: in your own trail of the double refund, show that the approval record and the approve span are the same moment of the same run. Which fields and which commands prove it?

Show the answer

The approval record has span_id and trace_id. python show_spans.py approve prints the approve support-resolution span with its IDs. grep ‘“type”:“approval”’ work/audit/audit.jsonl shows the record’s span_id. The course team’s run gave span c07282c8 in both. The record’s run_id is the same as larkfield.run.id on the span. The write records name the two request_refund spans the same way.

Recap

  • Every audit event has the same fields: who, what, when (UTC), on whose authority and with what evidence. It also says about whom (a pseudonym) and where to look (run, ticket, trace and span IDs).
  • Seven types: data access, proposal, approval, refusal, write, handoff, close. The approval names the proposal; each write names the approval.
  • Text fields and strings over 80 characters are refused. That catches common mistakes, not every sentence.
  • Audit events are not spans: a sampler cannot drop them. In file mode, a refused event stops the run before it writes.

Your project now: the files of the last lesson plus the new observe.py, live.py, telemetry.py and two tests, with 103 tests that pass. Your workflow writes a complete trail. But it is a plain text file: anybody who can edit it can change 9.95 into 0.95, and nobody would know. Next, you make every change visible, and you decide who may read, write and delete.

Next: Make it tamper-evident and controlled