A refund nobody can explain
- Lesson 1
- 30 min
- Module 1: Why agents need their own observability
Where we are
In the agents course, we built a support workflow that proposes an action, waits for a person's approval and then writes. It now runs every day at Larkfield. Last week a customer got the same refund twice, and the dashboard stayed green. In this lesson, we look at what our logs can answer about it, and what they cannot. We start with a note on a fridge.
Lesson 1 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsA refund nobody can explain
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
See why a log that was enough for a simple service cannot explain what an agent did. You read Larkfield’s only record of a refund that was paid twice, and you list the questions that it cannot answer. At the end, you download the course project.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s support workflow from AI Agents and Workflow Orchestration. It reads a ticket, looks up the order and the policy, and proposes an action. It waits for a person’s approval, and then writes the change.
- What an operation ID and a retry are, from Side effects and retries. This lesson explains them again briefly.
What you need on your computer
- Nothing for the first part of the lesson. At the end, you download the course project. You need Git, or a browser to download a ZIP file.
You need no account, no key and no payment in this course. Every model decision is a real recording of the agents course, replayed, as the last lesson of this module shows.
A note on the fridge
Section titled “A note on the fridge”A family pays its bills online. When someone pays a bill, they write it on a note on the fridge: “Electricity: paid.” One evening, Dad pays the electricity bill. The bank’s page freezes, so he presses “Pay” again. The second time, the page says “Thank you”. He writes “Electricity: paid” on the note.
Next week, the bank statement shows two payments to the electricity company. The family looks at the note. It says “paid”, once. It does not say that the page froze, that Dad pressed the button twice, or that the first payment worked after all. The note records the result that Dad believed, not the steps that happened.
Larkfield has a note on the fridge too. It is a log line.
The problem: a refund paid twice
Section titled “The problem: a refund paid twice”This is a constructed practice case: Larkfield, its people and its customers are made up for the course. The model decisions in it are real recordings, replayed.
Larkfield sells home and garden products, and its support workflow now runs live. It reads each ticket, proposes a resolution, and a support agent approves the changes. Then the workflow makes them, sends the reply and closes the ticket. Amira, the developer who owns the workflow, made it write one log line per closed ticket. The help desk’s dashboard counts these lines.
This morning, Grace, the support lead, finds a problem. A customer’s order, LK-793996, a watering can, has two refunds of 9.95 on the payment service. The customer asked for one: the delivery was late. Grace asks Amira three questions:
- Which step paid the second refund?
- Did the model decide it, did a person approve it, or did something else do it?
- Did it happen to other customers?
Amira opens the log.
What the log says
Section titled “What the log says”The workflow handled this ticket in one run, with the run ID R-1014-040. Its log line:
2026-10-15T09:17:22-04:00 INFO ticket=T-90104 run=R-1014-040 model=chat-strong status=resolved writes=1 reply=sentRead it field by field:
2026-10-15T09:17:22-04:00: the run closed on Thursday 15 October at 09:17, Toronto time. That is when the line was written. It does not say when the ticket arrived.ticket=T-90104,run=R-1014-040: the ticket and the run. One ticket can have several runs.model=chat-strong: the larger of Larkfield’s two models answered.status=resolved: the ticket is closed, and the customer’s problem is solved.writes=1: the workflow made one change in another system.reply=sent: the customer got the reply.
Every field is true from the workflow’s point of view, and the line is green. But writes=1 is the opposite of what Grace sees: the payment service has two refunds. So Amira can answer only a few of Grace’s questions:
| Question | Does the log answer it? |
|---|---|
| Did the ticket close, and how? | Yes: resolved, Thursday 09:17. |
| Which model was used? | Yes: chat-strong. |
| How many refunds were paid? | No, and worse: it says one write. |
| Which step paid the second refund? | No: the line has no steps. |
| What did the model propose, and why? | No: no proposal, no policy rule. |
| Who approved it, and when? | No: no person, no time of approval. |
| When did the ticket arrive, and how long did it wait? | No: only the time of the close. |
| What did the run cost? | No. |
| Did it happen to other customers? | No: a run like this one looks like every other resolved refund. |
The dashboard is green
Section titled “The dashboard is green”The dashboard counts the log lines of each day. Here is what it showed for the week.
Text description of the chart
Seven bars, one per day from Monday 12 to Sunday 18 October. The share of closed tickets that the log marks resolved: Mon 12 57.8% (26 of 45), Tue 13 73.2% (41 of 56), Wed 14 60% (30 of 50), Thu 15 75.4% (43 of 57), Fri 16 66.7% (34 of 51), Sat 17 42.9% (12 of 28), Sun 18 55.6% (15 of 27). Thursday, the day the two refunds were paid, has the highest share of the week: 43 of 57 tickets.
Thursday was the best day of the week on the dashboard: 43 of 57 closed tickets resolved. The double refund is one of them, counted as a success. A dashboard can only count what the log line says, and this line said “resolved”.
What really happened
Section titled “What really happened”The course team can show you what happened, because it has better records than Larkfield has today. You make those records yourself in the next modules. For now, here is the story they tell:
Text description of the diagram
The ticket arrived on Wednesday at 17:52. The workflow used chat-strong. Its first model call asked to look up order LK-793996. Its second model call proposed a refund of the 9.95 shipping fee, under rule W2. Then the run waited 15 hours 25 minutes for a person, overnight. On Thursday at 09:17, Sam approved. The workflow sent the refund to the payment service, with the key 797fbb5bbb9f8a7e. The service made the refund, but its answer came after the workflow’s 10-second timeout. The workflow sent the same refund again with a new key, 797fbb5bbb9f8a7e-retry1, and the service made a second refund. Then the reply was sent, the ticket was closed, and the log got one line: status resolved, one write, reply sent.
So who caused the second refund? Look at each actor in turn:
- The model proposed one refund of 9.95, under rule W2. That was the right decision: the delivery was 12 business days late. The model never saw the payment service.
- Sam, a support agent, approved one refund of 9.95, within their limit of 100.
- The payment service was slow that morning. This is a practice incident we caused on purpose. Between 08:30 and 09:30 on Thursday, it answered the first call of each refund too late, after the workflow’s timeout. The refund was already made.
- The workflow’s code decided to try again. Release 2.1.0, from Wednesday, had a new retry. When a write timed out, it sent the write again with a new key.
The key is the important part. In the agents course, each write gets an idempotency key: an ID that tells the payment service that it has already done this request. If the same key arrives twice, the service makes the refund only once. A safe retry sends the same key, or first asks the service whether the write happened. Tomás’s retry sent a new key, so the service saw a new refund. This is the agents course’s --naive-retry, shipped by mistake.
Neither the model nor the person caused the double refund: the code between them did. And the log line has no place for any of this. It counts one write, because the workflow counts one operation, the refund that it planned, which finally said “done”.
Why an agent needs more than one log line
Section titled “Why an agent needs more than one log line”One log line per request works for many services. A web page loads or fails; a log line with the status and the time says most of what matters. The MLOps and LLMOps course shows how far logs, metrics and traces take you for such a service.
An agent’s run is different in four ways, and each one appeared in this ticket:
- A run has many steps, and their number changes. This run made two model calls, one read and two write calls. Another run of the same ticket may take other steps, because the model chooses them.
- A run can stop and wait for a person. This one waited 15 hours 25 minutes. The approval happened on another day, in another process, after a person read the proposal.
- The actions change other systems. The refund is a write in the payment service. A write can be uncertain: a timeout does not say whether it happened.
- The cost depends on the path. A run on chat-strong with two calls cost US$0.01182. Another path costs more or less. This run spent its model time on Wednesday evening and its write time on Thursday morning.
So the questions about an agent are about steps, people and writes, not only results. A log line of the result cannot answer them. Logging more lines of the same kind does not help either: the next lesson shows why “log everything” is a trap.
A small worked example: count what the log can count
Section titled “A small worked example: count what the log can count”Can the log at least find the other customers? Try the simplest search: every run that ended like this one.
grep -c "status=resolved writes=1 reply=sent" data/logs/week.log83The week has 316 log lines. 83 of them look exactly like the double refund: resolved, one write, reply sent. Most of these are correct refunds, returns and reships. The log cannot tell which of the 83 paid twice, because the line is the same for both.
Now search for the ticket itself:
grep "ticket=T-90104" data/logs/week.log2026-10-12T15:31:40-04:00 INFO ticket=T-90104 run=R-1012-034 model=chat-small status=asked_customer writes=0 reply=sent2026-10-13T18:16:55-04:00 INFO ticket=T-90104 run=R-1013-049 model=chat-small status=asked_customer writes=0 reply=sent2026-10-15T09:17:22-04:00 INFO ticket=T-90104 run=R-1014-040 model=chat-strong status=resolved writes=1 reply=sent2026-10-15T13:33:46-04:00 INFO ticket=T-90104 run=R-1015-020 model=chat-strong status=resolved writes=1 reply=failed2026-10-16T10:25:56-04:00 INFO ticket=T-90104 run=R-1016-016 model=chat-strong status=resolved writes=1 reply=sentFive runs for one ticket number. The practice week replays the agents course’s 70 recorded tickets, so a ticket number comes back several times. That is simulated traffic. Each run has its own run ID, so always search by run.
Two of these lines say something new. On Monday and Tuesday, chat-small asked the customer a question instead of refunding: asked_customer. From Wednesday, chat-strong handled this ticket. And on Thursday afternoon, one run says status=resolved with reply=failed. A ticket closed as resolved while the customer never got the reply. The log shows the contradiction, but not why. Later modules come back to both.
Get the project
Section titled “Get the project”You read the same log on your computer now. You set up Python in the last lesson of this module.
-
Open a terminal and move to your projects folder. If it does not exist, make it first with
mkdir ~/projects.Terminal window cd ~/projects -
If you have Git, clone the labs repository. If you cloned it for an earlier course, run
git -C nextia-learning-labs pullinstead.Terminal window git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.gitWithout Git, download the ZIP file of the repository. Unzip it in
~/projects. Its folder isnextia-learning-labs-main: use that name in the next steps. -
Copy the snapshot
startinto a new folder,observable-workflow, and move into it:Terminal window cp -R nextia-learning-labs/C31/snapshots/start observable-workflowcd observable-workflowTerminal window Copy-Item -Recurse nextia-learning-labs\C31\snapshots\start observable-workflowcd observable-workflowTerminal window cp -R nextia-learning-labs/C31/snapshots/start observable-workflowcd observable-workflow -
Find the double refund’s line:
Terminal window grep "run=R-1014-040" data/logs/week.logTerminal window Select-String -Path data\logs\week.log -Pattern "run=R-1014-040"Terminal window grep "run=R-1014-040" data/logs/week.logYou see the line from this lesson. PowerShell adds the file name and the line number, 157, in front of it (not tested).
-
Open the folder
observable-workflowin VS Code, or in another editor. Opendata/logs/week.logand scroll through Thursday’s lines, from line 152.
The project is Larkfield’s workflow from the agents course, with two new steps: it sends the reply and closes the ticket. Its README.md says what each folder holds. The snapshot start writes only this log line. Each later snapshot, from end-of-m01 to end-of-m08, is the project at the end of one module. If your project breaks, the snapshots README shows how to start again from a snapshot.
Common mistake
The course folder is missing
Symptom: the copy stops with a message like cp: nextia-learning-labs/C31/snapshots/start: No such file or directory. On Windows, Copy-Item says that it cannot find the path.
Cause: your clone of the labs repository is older than this course, or you are in the wrong folder.
Diagnose: run ls nextia-learning-labs (Windows: dir nextia-learning-labs). If there is no C31, the clone is old. If the command fails, you are not in ~/projects.
Recover: run git -C nextia-learning-labs pull, or download the ZIP file again. Then do step 3 again. If observable-workflow already exists from a failed try, delete it first.
Common mistake
grep finds nothing
Symptom: grep "run=R-1014-040" data/logs/week.log prints nothing, or says No such file or directory.
Cause: you are not in the project folder, or you typed the run ID with another character, for example R-1014-40.
Recover: run pwd (Windows: Get-Location). The path must end with observable-workflow. Then copy the command from this page again.
Your turn: Grace’s five questions
Section titled “Your turn: Grace’s five questions”Grace must answer the customer, and she must say whether it can happen again. Write down five questions that she needs answered about the double refund. Use the diagram and the table above for ideas, but write them in your own words.
- Write five questions, each one about something that a person could check: a step, a time, a person, an amount, a count.
- For each question, write yes or no: can
data/logs/week.loganswer it? - For each no, write one sentence: what record would answer it?
- Keep your list. In the next lesson, you mark which signal answers each question.
Five questions that other learners wrote
- “Which call to the payment service made the second refund, and with which key?” No. It needs a record of each step, with the key that it sent.
- “Did the first call make a refund, even though it timed out?” No. Only the payment service’s own records can say for sure.
- “Who approved the refund, and what exactly did that person see?” No. It needs a record of the approval: the person, their role, the proposal they approved.
- “How many refunds timed out this week, and how many were sent again?” No. It needs a count of writes by outcome, for every run.
- “When did the ticket arrive, and how long did it wait for a person?” No. The line has only the time of the close.
What this course builds
Section titled “What this course builds”Larkfield needs records of the steps, the people and the writes. In this course, you add them to the workflow, one kind at a time, and you use them to answer Grace:
- Module 2 records each run as a trace with OpenTelemetry, the open standard for telemetry, and routes it through a Collector that removes private text.
- Module 3 records each model call, tool call and write, the approval that pauses a run overnight, and the handoffs between workers.
- Module 4 finds the failures that only agents have, the double refund among them, in Grafana.
- Module 5 splits cost by run, step and model.
- Module 6 keeps an audit trail that says who decided what, and that nobody can change without detection.
- Module 7 writes alerts that fire on such a week and stay quiet on a healthy one.
- Module 8 sends the same data to Langfuse, compares tools, and tours a view built for agent runs.
Each module works on Larkfield’s replayed week. The course ends with case studies on other public data. You need to complete one of them for the certificate:
Case studies in this course
At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.
- Find failures in public agent runsConvert an open set of recorded customer-service agent runs to OpenTelemetry, load them into Grafana, and find and count failure patterns with evidence.
- The cost of a coding agentMeasure cost per task, per step and per outcome on open coding-agent trajectories, find the runs that loop or over-spend, and propose a budget and a step limit.
- Could the records have answered?Take a public, documented case of an AI assistant's answer that a company had to defend, list the questions it raised, and design the audit trail and traces that would have answered them.
Knowledge check
Section titled “Knowledge check”Check your understanding
The log line of run R-1014-040 says writes=1, but the payment service made two refunds. Why does the log say one?
Show the answer
The workflow counts operations, not calls. It planned one operation, the refund of 9.95, and that operation finally ended “done”. The first call timed out, so the workflow did not count it, and sent the refund again with a new key. The payment service had made the first refund anyway, and it made the second one too. The log records what the workflow believed, not what each call did.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace asks: 'Did the model pay twice?' What is the honest answer, and why can the log not give it?
Show the answer
No. The model proposed one refund of 9.95 under rule W2, which was right, and Sam approved one refund. The second refund came from the workflow’s own retry code in release 2.1.0. After a timeout, it sent the write again with a new idempotency key. The log line has no proposal, no approval and no write calls. So it cannot separate the model’s decision, the person’s decision and the code’s action.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Thursday was the best day of the week on the dashboard. Why is that not evidence that the workflow worked well on Thursday?
Show the answer
The dashboard counts log lines with status=resolved. A double refund closes as resolved, and so does a ticket whose reply failed. So the dashboard counts them as successes. It measures what the workflow reported, not what happened to the customer or to the money.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Larkfield’s only record of a ticket is one log line, written when the ticket closes. The dashboard counts these lines.
- Run R-1014-040 paid two refunds of 9.95, and its line says
status=resolved writes=1 reply=sent. Thursday was the dashboard’s best day. - The model and Sam each decided one refund. The second came from a retry with a new idempotency key, after a timeout that we caused on purpose.
- An agent’s run has steps whose number changes, waits for people, writes to other systems and costs what its path costs. A log line of the result cannot explain any of these.
- 83 lines of the week look exactly like the double refund. The log cannot say which of them paid twice.
Your project now: ~/projects/observable-workflow, a copy of the snapshot start. You can now say what the log line does not know. But adding more log lines is not the answer: some data must never leave the workflow. Next, you sort the kinds of records that an agent needs, how long each one is kept and who may read it.