Tool calls, errors and retries
- Lesson 8
- 35 min
- Module 3: Instrument the workflow
Where we are
In the last lesson, we gave every model call its own span, with its tokens and the model that answered. But the model only proposes: the tools read the orders and write the refunds, and they are still invisible. In this lesson, we record each tool call with the shape of its arguments, its status, its errors and its retries, and the idempotency key that makes a second refund visible. We start with a bank transfer that seems to freeze.
Lesson 8 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsTool calls, errors and retries
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Give every tool call its own span. Mark it as a read or a write, and record the names of its arguments, its status and its errors. On every write, record the operation and the idempotency key. Then a second refund is visible in the trace, with the key that caused it.
Before you startWhat to know and to have ready
What you need to know
- The
chatspans and your helpershow_spans.py, from Model calls and tokens. - That the workflow’s tools are the agents course’s: four reads (
get_order,get_customer_orders,get_payments,search_policy) and three writes (request_refund,reship_item,create_return_label). A write happens only after a person’s approval. - The double refund of A refund nobody can explain: a timeout, then Tomás’s naive retry.
What you need on your computer
- The project as the last lesson left it: 81 tests pass, and
show_spans.pyis in the project folder. - A terminal in
observable-workflow, with the virtual environment active.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A bank transfer that seems to freeze
Section titled “A bank transfer that seems to freeze”You send money to a friend in your bank’s app. You press Send, and the app shows a spinning wheel for a long time. Then it says “No connection”. Did the money go?
If you press Send again, your friend may get the money twice. A careful bank prevents this. Each transfer gets a reference number when you start it. If the same reference arrives twice, the bank pays once and answers “already done”. Before you try again, you can also look at your account: is the transfer there?
Two habits make this safe: a reference that stays the same when you try again, and a check before you try again. Checking your balance twice is harmless. Sending money twice is not. So the bank treats reads and writes differently.
The problem: the trace cannot see the tools
Section titled “The problem: the trace cannot see the tools”Your trace of T-80008 has two model calls. Between them, the model asked for get_order, and the workflow read the order. The trace shows a model call that ends at 0.96 seconds and the next one that starts there. The read is not in it.
Now replay the double refund. The first command proposes a refund of 9.95, the second approves it as an agent, and the third lists the changes. The approval simulates the payment service’s timeout after the refund was made, and Tomás’s naive retry:
python -m resolver run T-90104 --variant agent_structured --model chat-strongpython -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --name Sam --fault request_refund:timeout_after_write --naive-retrypython -m resolver changesrequest_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 2 call(s))Status: doneReply sent. Ticket: resolved.… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e)… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e-retry1)2 change(s) in the practice databaseThe workflow’s own summary says one operation, done, in two calls. The practice database has two refunds of 9.95. Your times differ. The decision is replayed: chat-strong’s recorded proposal from the agents course. The timeout and the retry are simulated: a practice incident we caused on purpose.
The first command also printed the proposal, which ends with the approve command to type. The commands are the same in PowerShell. Nothing in your telemetry shows the two calls to the payment service. If a customer complains, the trace says “two model calls, a proposal”, and nothing about money.
The idea: one execute_tool span per tool call
Section titled “The idea: one execute_tool span per tool call”Give each tool call its own span, as a child of the step that made it. The GenAI conventions name it execute_tool {tool}, for example execute_tool get_order, with the operation name execute_tool and gen_ai.tool.name. The workflow adds its own attributes, with the larkfield. prefix:
| Attribute | Example | Why |
|---|---|---|
larkfield.tool.kind |
read or write |
A read can be repeated. A write changes money or stock. |
larkfield.tool.arguments |
order_id |
The names of the arguments, sorted. Never their values. |
larkfield.order.id |
LK-793996 |
One ID that an investigation needs. It is an ID, not free text. |
larkfield.tool.status |
ok, not_found, service_timeout |
The result as a code, never the result itself. |
error.type |
service_timeout |
The stable name for an error, with the span’s status set to ERROR. |
Why the names of the arguments and not the values? The search_policy tool takes a query, which is free text from the model. It can contain the customer’s words. The names say which tool was called correctly; the values stay out. The order ID is the exception that we chose: it is an ID, and every question about a refund starts with it.
A write span also records what an auditor needs about the change, as numbers and codes: the amount, the reason code (RFD-LATE), the SKU and the quantity. And it records three things that make retries visible:
- The operation ID (
larkfield.operation.id): one ID for “refund 9.95 on LK-793996 in this run”. It stays the same for every try. - The idempotency key (
larkfield.idempotency_key): the reference that the workflow sends to the payment service with this call. An operation is idempotent when doing it twice has the same effect as doing it once. The key gives the service a way to make the refund idempotent: the same key twice, one refund. - The attempt (
larkfield.attempt): 1 for the first try, 2 for the next one.
Then a fourth span kind, reconcile {tool}, records the check before a retry: “is this operation already there?”, with larkfield.reconcile.found.
Worked example: two retries, two endings
Section titled “Worked example: two retries, two endings”The payment service makes the refund, and its answer does not arrive within 10 seconds. The workflow cannot know whether the refund happened. Here are the two ways to go on:
Text description of the diagram
The workflow sends the refund with its key. The service makes it, but the answer is late: a timeout. With the naive retry, the workflow sends the refund again with a new key, ending in -retry1. The service sees a new refund: the customer is paid twice. With the safe retry, the workflow first asks the service whether the operation exists. The service answers yes, so nothing more is written.
On the spans, the two endings look like this:
| Naive retry | Safe retry | |
|---|---|---|
| Write spans | 2: attempt 1 timeout (ERROR), attempt 2 done |
1: attempt 1 timeout (ERROR) |
| Operation ID | the same on both: 797fbb5bbb9f8a7e |
797fbb5bbb9f8a7e |
| Idempotency key | two different keys: …8a7e and …8a7e-retry1 |
one key |
| Reconcile span | none | reconcile request_refund, found: true |
| Refunds in the database | 2 | 1 |
The rule to read it: one operation with two different keys and two done or timeout calls is a possible double write. One key cannot pay twice, because the service refuses a key that it already knows. A new key is a new request for the service. Module 4 turns this rule into a search.
Add the tool spans to your project
Section titled “Add the tool spans to your project”-
Copy the files of this lesson: the read path (
tools.py), the write path (execute.py), the practice data’s injection markers, and the snapshot’s telemetry test.Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/tools.py $S/resolver/execute.py resolver/cp $S/data/injection_markers.json data/cp $S/tests/test_telemetry.py tests/Terminal window $S = "..\nextia-learning-labs\C31\snapshots\end-of-m03"Copy-Item $S\resolver\tools.py, $S\resolver\execute.py resolver\Copy-Item $S\data\injection_markers.json data\Copy-Item $S\tests\test_telemetry.py tests\Terminal window S=../nextia-learning-labs/C31/snapshots/end-of-m03cp $S/resolver/tools.py $S/resolver/execute.py resolver/cp $S/data/injection_markers.json data/cp $S/tests/test_telemetry.py tests/ -
Open
resolver/tools.pyand findrun_read. The agents course’s function is now_run_read, unchanged. The newrun_readopens a span withobserve.tool_span(), calls it, and records the result withobserve.read_result(). -
Open
resolver/execute.pyand find_attempt. Each call of a write is in its own span, with the operation ID, the key (call_id) and the attempt number. Findnaive:inexecute(): the naive retry makescall_id = f"{op_id}-retry{op.attempts}", a new key. -
Run the tests:
python -m pytest -q. The last line says81 passed, 1 deselected. The new test file checks that the first tool span isget_orderwith the argumentsorder_id. -
Delete your old runs, so that the next outputs are easy to read: delete the folder
work/(PowerShell:Remove-Item -Recurse work). -
Run T-80008 and read its trace:
Terminal window python -m resolver run T-80008python -m obs trace T-80008Trace 1 01e7e7965e75f288a478bfc56e5fed11 … 2.1 s0.0 s |========================| 2.14 s invoke_agent support-resolver reply_only rule W10.0 s |=========== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls1.0 s | = | 0.00 s execute_tool get_order ok LK-5811061.0 s | ==============| 1.18 s chat chat-small 2411 in / 96 out - tool_callsThe read is between the two calls: the first call asked for it, and the second call used its result.
Guided practice: the double refund in the trace
Section titled “Guided practice: the double refund in the trace”-
Run the double refund again, with the two commands of the problem above:
run, thenapprove … --naive-retry. -
Read the trace:
python -m obs trace T-90104.Run T-90104-agent_structured-chat-strong-1 | ticket T-90104 | chat-strong (agent_structured) | release 2.0.0 | language en2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.00000Trace 1 e7b26f791153ccc4f5f9d6ba36cb140a … 6.6 s0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W20.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out - stop2.4 s | = | 0.00 s execute_tool get_order ok LK-7939962.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out - stopOutcome: waiting_approvalRead the second line:
0 write call(s). And the outcome is stillwaiting_approval, although the refund was paid twice. -
Find the write spans in the file:
python show_spans.py request_refund.execute_tool request_refund trace 7162b31d span fcd69358 parent nonegen_ai.operation.name = execute_toolgen_ai.tool.name = request_refundgen_ai.tool.type = functionlarkfield.tool.kind = writelarkfield.tool.arguments = amount,order_id,reason_code,toollarkfield.order.id = LK-793996larkfield.operation.id = 797fbb5bbb9f8a7elarkfield.idempotency_key = 797fbb5bbb9f8a7elarkfield.attempt = 1larkfield.refund.amount = 9.95larkfield.refund.reason = RFD-LATElarkfield.write.outcome = timeouterror.type = timeoutexecute_tool request_refund trace f23daaac span 81210460 parent none…larkfield.operation.id = 797fbb5bbb9f8a7elarkfield.idempotency_key = 797fbb5bbb9f8a7e-retry1larkfield.attempt = 2larkfield.refund.amount = 9.95larkfield.refund.reason = RFD-LATElarkfield.write.outcome = doneThe
…stands for the same first seven lines. Your trace and span IDs differ. -
Apply the rule of the worked example. One operation,
797fbb5bbb9f8a7e. Two keys. Attempt 1 timed out, attempt 2 is done. This is the double refund, and the keys name the call that caused it: the retry. -
Now read the first line of each span:
parent none, and two different traces.
The spans hold every fact. But they are not where they belong:
Text description of the diagram
Trace 1 is the run, with its two model calls and its read. The two refund calls are two more traces, each with one span and no parent. They have the operation ID and the keys, but no run ID and no approver. obs trace groups spans by run, so it cannot show them, and nothing says which person allowed them.
Why? The writes happen in python -m resolver approve, a second command, maybe hours later. No span of the run is open in that process. So each write span starts a new trace. The next lesson fixes this.
Errors on reads
Section titled “Errors on reads”A read can fail too, and the model then decides what to do with the error. Delete work/ again, so that obs trace finds this run only. Then make the order service time out once:
python -m resolver run T-80008 --fault get_order:timeoutpython -m obs trace T-80008Note: Your request at step 2 differs from the recorded one (evidence 931a981a here, 783b17ea when recorded). The recorded decision is replayed and checked against your state.… 0.0 s |========================| 12.14 s invoke_agent support-resolver reply_only rule W1 0.0 s |== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls 1.0 s | ===================== | 10.00 s execute_tool get_order ERROR service_timeout LK-581106 11.0 s | ===| 1.18 s chat chat-small 2411 in / 96 out - tool_calls
Outcome: resolvedThe read span is an error, service_timeout. It adds the simulated 10 seconds of waiting. Then the run still ends resolved, with a reply about the order’s dates. Read the note above it. The second request was not the recorded one, because the model saw an error instead of the order. So the replay used the recorded decision, which was made with the order. The chat span says so: larkfield.replay = inexact (check it with python show_spans.py chat).
So this run is not proof of what a model does with a timeout. But it shows the pattern that Module 4 calls a hidden tool error: an error span, then a confident reply. The trace makes it visible; the reply alone never would.
Your turn: the safe retry
Section titled “Your turn: the safe retry”Change one condition: the same timeout after the write, without --naive-retry. Before you run it, predict:
- How many
execute_tool request_refundspans, and with which keys? - Which other span appears?
- How many refunds does
python -m resolver changeslist?
Then delete work/ and run it:
python -m resolver run T-90104 --variant agent_structured --model chat-strongpython -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --fault request_refund:timeout_after_writepython -m resolver changespython show_spans.py request_refundThe course team’s result
request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 1 call(s))Status: doneReply sent. Ticket: resolved.… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e)1 change(s) in the practice databaseexecute_tool request_refund trace 8e052077 span 272c5858 parent none … larkfield.operation.id = 797fbb5bbb9f8a7e larkfield.idempotency_key = 797fbb5bbb9f8a7e larkfield.attempt = 1 larkfield.write.outcome = timeout error.type = timeoutreconcile request_refund trace bb7ef0fc span 09aa3f15 parent none larkfield.operation.id = 797fbb5bbb9f8a7e larkfield.order.id = LK-793996 larkfield.reconcile.found = TrueOne write span with the timeout, then a reconcile span: the workflow asked the service about operation 797fbb5bbb9f8a7e, and it was there. One refund. The reconcile span is alone in its own trace too, for the same reason as the writes.
Common mistake
Two keys, but only one change
Symptom: Tomás uses the rule of this lesson on the replayed week. Run R-1014-024 (ticket T-90604) shows one operation with two keys: a return label, timeout then done. He reports a second double write. But the practice database of that run has one label, not two.
Cause: the trace records what the workflow sent, not what the other service did. A timeout before the write and a timeout after the write give the same span: write.outcome = timeout, because the workflow never got an answer. On R-1014-024 the first call timed out before the write, so only the retry made a label.
Diagnose: two keys for one operation means “maybe twice”. Only the system that made the change knows. Ask it: the safe retry’s reconcile span records its answer, and python -m resolver changes lists the changes in your practice database.
Recover: keep the key on every write span, because it names the call. But decide from the system of record, through a reconcile or the audit trail of Module 6. A trace that says “possible double write” is a reason to check, not a proof.
When to record tool calls, and what it costs
Section titled “When to record tool calls, and what it costs”Record every tool call: reads and writes, successes and errors. A tool span is small. In the course’s measurement, a run’s whole telemetry was 9.3 KB, and its instrumentation cost 1.9 milliseconds. Errors are the reason to keep all of them: a sampled trace that drops the failed read loses the evidence.
Record the idempotency key and the operation ID on every write, always. They cost two short strings, and they are the only way to connect a payment in another system to one call of one run. Record the arguments’ names, not their values, unless the value is an ID that you chose to keep, as the order ID here. A new ID on a span is a decision: write down why.
The workflow also marks a read whose result carries an injected instruction (larkfield.input.injection). The practice data knows these results, and real systems do not. Module 4 uses the marker to find runs that followed one.
Knowledge check
Section titled “Knowledge check”Check your understanding
Two write spans of one run both have larkfield.operation.id = 797fbb5bbb9f8a7e. One has the key 797fbb5bbb9f8a7e and outcome timeout; the other has the key 797fbb5bbb9f8a7e-retry1 and outcome done. What happened, what can you not know from the spans, and how do you find out?
Show the answer
The workflow tried one refund twice. The first call timed out, so the workflow never learned whether it happened. The retry used a new key, so the payment service treated it as a new refund, and it was done. From the spans alone, you cannot know whether the first call made a refund: a timeout before the write and a timeout after it look the same. Ask the system of record: a reconcile on the operation, the practice database’s changes, or later the audit trail. For T-90104 with the timeout after the write, the database lists two refunds.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the search_policy span record larkfield.tool.arguments = query and not the query itself, when the get_order span keeps the order ID?
Show the answer
The query is free text that the model wrote, and it can contain the customer’s words or personal data. The name of the argument is enough to see that the tool was called correctly. The order ID is an ID that Larkfield chose to keep: it has no personal text in it, and every question about a refund starts with the order. Keeping a value on a span is a decision for each attribute.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: in your own telemetry of the naive retry, show the two write spans and say which one paid the second refund and why it was not refused.
Show the answer
Run python show_spans.py request_refund after the naive-retry approval. The span with larkfield.attempt = 2, larkfield.idempotency_key = 797fbb5bbb9f8a7e-retry1 and larkfield.write.outcome = done paid the second refund. The first call (attempt 1, the key 797fbb5bbb9f8a7e) had already paid, and its timeout came after the write. The service did not refuse the retry, because a new key is a new request for it: only the same key twice is recognised as a repeat.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- One
execute_tool {tool}span per tool call, markedreadorwrite, with the arguments’ names, a status code anderror.typeon errors. - On every write: the operation ID (the same for every try), the idempotency key (the reference sent with this call) and the attempt. One operation with two keys is a possible double write.
- A
reconcilespan records the check before a retry, and its answer. - A trace records what the workflow sent, not what the other system did: a timeout before and after the write look the same.
Your project now: the files of the last lesson plus tools.py, execute.py, data/injection_markers.json and the snapshot’s tests/test_telemetry.py, with 81 tests that pass. Every tool call has a span. But each write is a trace of its own, with no run ID and no person: nothing says who approved it, or how long the run waited. Next, you join the approval and the writes to the run.