Skip to content
Français

Tool calls, errors and retries

  • Lesson 8
  • 35 min
  • Module 3: Instrument the workflow

Where we are

In the last lesson, we gave every model call its own span, with its tokens and the model that answered. But the model only proposes: the tools read the orders and write the refunds, and they are still invisible. In this lesson, we record each tool call with the shape of its arguments, its status, its errors and its retries, and the idempotency key that makes a second refund visible. We start with a bank transfer that seems to freeze.

What you will be able to do

Give every tool call its own span. Mark it as a read or a write, and record the names of its arguments, its status and its errors. On every write, record the operation and the idempotency key. Then a second refund is visible in the trace, with the key that caused it.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • The chat spans and your helper show_spans.py, from Model calls and tokens.
  • That the workflow’s tools are the agents course’s: four reads (get_order, get_customer_orders, get_payments, search_policy) and three writes (request_refund, reship_item, create_return_label). A write happens only after a person’s approval.
  • The double refund of A refund nobody can explain: a timeout, then Tomás’s naive retry.

What you need on your computer

  • The project as the last lesson left it: 81 tests pass, and show_spans.py is in the project folder.
  • A terminal in observable-workflow, with the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

You send money to a friend in your bank’s app. You press Send, and the app shows a spinning wheel for a long time. Then it says “No connection”. Did the money go?

If you press Send again, your friend may get the money twice. A careful bank prevents this. Each transfer gets a reference number when you start it. If the same reference arrives twice, the bank pays once and answers “already done”. Before you try again, you can also look at your account: is the transfer there?

Two habits make this safe: a reference that stays the same when you try again, and a check before you try again. Checking your balance twice is harmless. Sending money twice is not. So the bank treats reads and writes differently.

The problem: the trace cannot see the tools

Section titled “The problem: the trace cannot see the tools”

Your trace of T-80008 has two model calls. Between them, the model asked for get_order, and the workflow read the order. The trace shows a model call that ends at 0.96 seconds and the next one that starts there. The read is not in it.

Now replay the double refund. The first command proposes a refund of 9.95, the second approves it as an agent, and the third lists the changes. The approval simulates the payment service’s timeout after the refund was made, and Tomás’s naive retry:

Terminal window
python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --name Sam --fault request_refund:timeout_after_write --naive-retry
python -m resolver changes
request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 2 call(s))
Status: done
Reply sent. Ticket: resolved.
… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e)
… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e-retry1)
2 change(s) in the practice database

The workflow’s own summary says one operation, done, in two calls. The practice database has two refunds of 9.95. Your times differ. The decision is replayed: chat-strong’s recorded proposal from the agents course. The timeout and the retry are simulated: a practice incident we caused on purpose.

The first command also printed the proposal, which ends with the approve command to type. The commands are the same in PowerShell. Nothing in your telemetry shows the two calls to the payment service. If a customer complains, the trace says “two model calls, a proposal”, and nothing about money.

The idea: one execute_tool span per tool call

Section titled “The idea: one execute_tool span per tool call”

Give each tool call its own span, as a child of the step that made it. The GenAI conventions name it execute_tool {tool}, for example execute_tool get_order, with the operation name execute_tool and gen_ai.tool.name. The workflow adds its own attributes, with the larkfield. prefix:

Attribute Example Why
larkfield.tool.kind read or write A read can be repeated. A write changes money or stock.
larkfield.tool.arguments order_id The names of the arguments, sorted. Never their values.
larkfield.order.id LK-793996 One ID that an investigation needs. It is an ID, not free text.
larkfield.tool.status ok, not_found, service_timeout The result as a code, never the result itself.
error.type service_timeout The stable name for an error, with the span’s status set to ERROR.

Why the names of the arguments and not the values? The search_policy tool takes a query, which is free text from the model. It can contain the customer’s words. The names say which tool was called correctly; the values stay out. The order ID is the exception that we chose: it is an ID, and every question about a refund starts with it.

A write span also records what an auditor needs about the change, as numbers and codes: the amount, the reason code (RFD-LATE), the SKU and the quantity. And it records three things that make retries visible:

  • The operation ID (larkfield.operation.id): one ID for “refund 9.95 on LK-793996 in this run”. It stays the same for every try.
  • The idempotency key (larkfield.idempotency_key): the reference that the workflow sends to the payment service with this call. An operation is idempotent when doing it twice has the same effect as doing it once. The key gives the service a way to make the refund idempotent: the same key twice, one refund.
  • The attempt (larkfield.attempt): 1 for the first try, 2 for the next one.

Then a fourth span kind, reconcile {tool}, records the check before a retry: “is this operation already there?”, with larkfield.reconcile.found.

The payment service makes the refund, and its answer does not arrive within 10 seconds. The workflow cannot know whether the refund happened. Here are the two ways to go on:

A sequence between the workflow's request_refund and the payment service. The workflow sends refund 9.95 on LK-793996 with idempotency key 797fbb5bbb9f8a7e. The service makes the refund, but no answer arrives within 10 seconds: a timeout. The workflow cannot know if it happened. First case, the naive retry, Tomás's shortcut: the workflow sends the refund again with a new key, 797fbb5bbb9f8a7e-retry1. A new key is a new refund: the customer is paid twice. Second case, the safe retry: the workflow asks, 'is operation 797fbb5bbb9f8a7e there?', and the service answers 'found: it happened'. Nothing more is written.
One timeout, two retries: a new key pays twice; a check with the same operation pays once.
Text description of the diagram

The workflow sends the refund with its key. The service makes it, but the answer is late: a timeout. With the naive retry, the workflow sends the refund again with a new key, ending in -retry1. The service sees a new refund: the customer is paid twice. With the safe retry, the workflow first asks the service whether the operation exists. The service answers yes, so nothing more is written.

On the spans, the two endings look like this:

Naive retry Safe retry
Write spans 2: attempt 1 timeout (ERROR), attempt 2 done 1: attempt 1 timeout (ERROR)
Operation ID the same on both: 797fbb5bbb9f8a7e 797fbb5bbb9f8a7e
Idempotency key two different keys: …8a7e and …8a7e-retry1 one key
Reconcile span none reconcile request_refund, found: true
Refunds in the database 2 1

The rule to read it: one operation with two different keys and two done or timeout calls is a possible double write. One key cannot pay twice, because the service refuses a key that it already knows. A new key is a new request for the service. Module 4 turns this rule into a search.

  1. Copy the files of this lesson: the read path (tools.py), the write path (execute.py), the practice data’s injection markers, and the snapshot’s telemetry test.

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m03
    cp $S/resolver/tools.py $S/resolver/execute.py resolver/
    cp $S/data/injection_markers.json data/
    cp $S/tests/test_telemetry.py tests/
  2. Open resolver/tools.py and find run_read. The agents course’s function is now _run_read, unchanged. The new run_read opens a span with observe.tool_span(), calls it, and records the result with observe.read_result().

  3. Open resolver/execute.py and find _attempt. Each call of a write is in its own span, with the operation ID, the key (call_id) and the attempt number. Find naive: in execute(): the naive retry makes call_id = f"{op_id}-retry{op.attempts}", a new key.

  4. Run the tests: python -m pytest -q. The last line says 81 passed, 1 deselected. The new test file checks that the first tool span is get_order with the arguments order_id.

  5. Delete your old runs, so that the next outputs are easy to read: delete the folder work/ (PowerShell: Remove-Item -Recurse work).

  6. Run T-80008 and read its trace:

    Terminal window
    python -m resolver run T-80008
    python -m obs trace T-80008
    Trace 1 01e7e7965e75f288a478bfc56e5fed11 … 2.1 s
    0.0 s |========================| 2.14 s invoke_agent support-resolver reply_only rule W1
    0.0 s |=========== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls
    1.0 s | = | 0.00 s execute_tool get_order ok LK-581106
    1.0 s | ==============| 1.18 s chat chat-small 2411 in / 96 out - tool_calls

    The read is between the two calls: the first call asked for it, and the second call used its result.

Guided practice: the double refund in the trace

Section titled “Guided practice: the double refund in the trace”
  1. Run the double refund again, with the two commands of the problem above: run, then approve … --naive-retry.

  2. Read the trace: python -m obs trace T-90104.

    Run T-90104-agent_structured-chat-strong-1 | ticket T-90104 | chat-strong (agent_structured) | release 2.0.0 | language en
    2 model call(s), 1 read(s), 0 write call(s), 4586 tokens in, 265 out, US$0.00000
    Trace 1 e7b26f791153ccc4f5f9d6ba36cb140a … 6.6 s
    0.0 s |========================| 6.56 s invoke_agent support-resolver resolve rule W2
    0.0 s |========= | 2.39 s chat chat-strong 2160 in / 47 out - stop
    2.4 s | = | 0.00 s execute_tool get_order ok LK-793996
    2.4 s | ================| 4.17 s chat chat-strong 2426 in / 218 out - stop
    Outcome: waiting_approval

    Read the second line: 0 write call(s). And the outcome is still waiting_approval, although the refund was paid twice.

  3. Find the write spans in the file: python show_spans.py request_refund.

    execute_tool request_refund trace 7162b31d span fcd69358 parent none
    gen_ai.operation.name = execute_tool
    gen_ai.tool.name = request_refund
    gen_ai.tool.type = function
    larkfield.tool.kind = write
    larkfield.tool.arguments = amount,order_id,reason_code,tool
    larkfield.order.id = LK-793996
    larkfield.operation.id = 797fbb5bbb9f8a7e
    larkfield.idempotency_key = 797fbb5bbb9f8a7e
    larkfield.attempt = 1
    larkfield.refund.amount = 9.95
    larkfield.refund.reason = RFD-LATE
    larkfield.write.outcome = timeout
    error.type = timeout
    execute_tool request_refund trace f23daaac span 81210460 parent none
    …
    larkfield.operation.id = 797fbb5bbb9f8a7e
    larkfield.idempotency_key = 797fbb5bbb9f8a7e-retry1
    larkfield.attempt = 2
    larkfield.refund.amount = 9.95
    larkfield.refund.reason = RFD-LATE
    larkfield.write.outcome = done

    The … stands for the same first seven lines. Your trace and span IDs differ.

  4. Apply the rule of the worked example. One operation, 797fbb5bbb9f8a7e. Two keys. Attempt 1 timed out, attempt 2 is done. This is the double refund, and the keys name the call that caused it: the retry.

  5. Now read the first line of each span: parent none, and two different traces.

The spans hold every fact. But they are not where they belong:

Three waterfalls on a scale of 0 to 10 seconds. Trace 1, the run: invoke_agent support-resolver for 6.6 s, with chat chat-strong (2160 in, 47 out, stop), execute_tool get_order (ok) and chat chat-strong (2426 in, 218 out, stop). Trace 2, one write alone, with no parent span, no run ID and no person: execute_tool request_refund, ERROR timeout, key ending 5bbb9f8a7e, 10 seconds long. Trace 3, also one write alone: execute_tool request_refund, done, key ending a7e-retry1.
The double refund after this lesson's change: each write is a trace of its own, with no parent, no run ID and no approver.
Text description of the diagram

Trace 1 is the run, with its two model calls and its read. The two refund calls are two more traces, each with one span and no parent. They have the operation ID and the keys, but no run ID and no approver. obs trace groups spans by run, so it cannot show them, and nothing says which person allowed them.

Why? The writes happen in python -m resolver approve, a second command, maybe hours later. No span of the run is open in that process. So each write span starts a new trace. The next lesson fixes this.

A read can fail too, and the model then decides what to do with the error. Delete work/ again, so that obs trace finds this run only. Then make the order service time out once:

Terminal window
python -m resolver run T-80008 --fault get_order:timeout
python -m obs trace T-80008
Note: Your request at step 2 differs from the recorded one (evidence 931a981a here, 783b17ea when recorded). The recorded decision is replayed and checked against your state.
…
0.0 s |========================| 12.14 s invoke_agent support-resolver reply_only rule W1
0.0 s |== | 0.96 s chat chat-small 2169 in / 21 out - tool_calls
1.0 s | ===================== | 10.00 s execute_tool get_order ERROR service_timeout LK-581106
11.0 s | ===| 1.18 s chat chat-small 2411 in / 96 out - tool_calls
Outcome: resolved

The read span is an error, service_timeout. It adds the simulated 10 seconds of waiting. Then the run still ends resolved, with a reply about the order’s dates. Read the note above it. The second request was not the recorded one, because the model saw an error instead of the order. So the replay used the recorded decision, which was made with the order. The chat span says so: larkfield.replay = inexact (check it with python show_spans.py chat).

So this run is not proof of what a model does with a timeout. But it shows the pattern that Module 4 calls a hidden tool error: an error span, then a confident reply. The trace makes it visible; the reply alone never would.

Change one condition: the same timeout after the write, without --naive-retry. Before you run it, predict:

  • How many execute_tool request_refund spans, and with which keys?
  • Which other span appears?
  • How many refunds does python -m resolver changes list?

Then delete work/ and run it:

Terminal window
python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --fault request_refund:timeout_after_write
python -m resolver changes
python show_spans.py request_refund
The course team’s result
request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 1 call(s))
Status: done
Reply sent. Ticket: resolved.
… T-90104 request_refund LK-793996 9.95 RFD-LATE (operation 797fbb5bbb9f8a7e)
1 change(s) in the practice database
execute_tool request_refund trace 8e052077 span 272c5858 parent none
…
larkfield.operation.id = 797fbb5bbb9f8a7e
larkfield.idempotency_key = 797fbb5bbb9f8a7e
larkfield.attempt = 1
larkfield.write.outcome = timeout
error.type = timeout
reconcile request_refund trace bb7ef0fc span 09aa3f15 parent none
larkfield.operation.id = 797fbb5bbb9f8a7e
larkfield.order.id = LK-793996
larkfield.reconcile.found = True

One write span with the timeout, then a reconcile span: the workflow asked the service about operation 797fbb5bbb9f8a7e, and it was there. One refund. The reconcile span is alone in its own trace too, for the same reason as the writes.

Common mistake

Two keys, but only one change

Symptom: Tomás uses the rule of this lesson on the replayed week. Run R-1014-024 (ticket T-90604) shows one operation with two keys: a return label, timeout then done. He reports a second double write. But the practice database of that run has one label, not two.

Cause: the trace records what the workflow sent, not what the other service did. A timeout before the write and a timeout after the write give the same span: write.outcome = timeout, because the workflow never got an answer. On R-1014-024 the first call timed out before the write, so only the retry made a label.

Diagnose: two keys for one operation means “maybe twice”. Only the system that made the change knows. Ask it: the safe retry’s reconcile span records its answer, and python -m resolver changes lists the changes in your practice database.

Recover: keep the key on every write span, because it names the call. But decide from the system of record, through a reconcile or the audit trail of Module 6. A trace that says “possible double write” is a reason to check, not a proof.

When to record tool calls, and what it costs

Section titled “When to record tool calls, and what it costs”

Record every tool call: reads and writes, successes and errors. A tool span is small. In the course’s measurement, a run’s whole telemetry was 9.3 KB, and its instrumentation cost 1.9 milliseconds. Errors are the reason to keep all of them: a sampled trace that drops the failed read loses the evidence.

Record the idempotency key and the operation ID on every write, always. They cost two short strings, and they are the only way to connect a payment in another system to one call of one run. Record the arguments’ names, not their values, unless the value is an ID that you chose to keep, as the order ID here. A new ID on a span is a decision: write down why.

The workflow also marks a read whose result carries an injected instruction (larkfield.input.injection). The practice data knows these results, and real systems do not. Module 4 uses the marker to find runs that followed one.

Check your understanding

Two write spans of one run both have larkfield.operation.id = 797fbb5bbb9f8a7e. One has the key 797fbb5bbb9f8a7e and outcome timeout; the other has the key 797fbb5bbb9f8a7e-retry1 and outcome done. What happened, what can you not know from the spans, and how do you find out?

Show the answer

The workflow tried one refund twice. The first call timed out, so the workflow never learned whether it happened. The retry used a new key, so the payment service treated it as a new refund, and it was done. From the spans alone, you cannot know whether the first call made a refund: a timeout before the write and a timeout after it look the same. Ask the system of record: a reconcile on the operation, the practice database’s changes, or later the audit trail. For T-90104 with the timeout after the write, the database lists two refunds.

Check your understanding

Why does the search_policy span record larkfield.tool.arguments = query and not the query itself, when the get_order span keeps the order ID?

Show the answer

The query is free text that the model wrote, and it can contain the customer’s words or personal data. The name of the argument is enough to see that the tool was called correctly. The order ID is an ID that Larkfield chose to keep: it has no personal text in it, and every question about a refund starts with the order. Keeping a value on a span is a decision for each attribute.

Check your understanding

Practical check: in your own telemetry of the naive retry, show the two write spans and say which one paid the second refund and why it was not refused.

Show the answer

Run python show_spans.py request_refund after the naive-retry approval. The span with larkfield.attempt = 2, larkfield.idempotency_key = 797fbb5bbb9f8a7e-retry1 and larkfield.write.outcome = done paid the second refund. The first call (attempt 1, the key 797fbb5bbb9f8a7e) had already paid, and its timeout came after the write. The service did not refuse the retry, because a new key is a new request for it: only the same key twice is recognised as a repeat.

Recap

  • One execute_tool {tool} span per tool call, marked read or write, with the arguments’ names, a status code and error.type on errors.
  • On every write: the operation ID (the same for every try), the idempotency key (the reference sent with this call) and the attempt. One operation with two keys is a possible double write.
  • A reconcile span records the check before a retry, and its answer.
  • A trace records what the workflow sent, not what the other system did: a timeout before and after the write look the same.

Your project now: the files of the last lesson plus tools.py, execute.py, data/injection_markers.json and the snapshot’s tests/test_telemetry.py, with 81 tests that pass. Every tool call has a span. But each write is a trace of its own, with no run ID and no person: nothing says who approved it, or how long the run waited. Next, you join the approval and the writes to the run.

Next: Approvals, people and waiting time