Side effects and retries

What you will be able to do

Make every approved write happen exactly once, even when a call times out or the process crashes in the middle of it.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The approval and its digest, from Approval boundaries.
• The checkpoint, resume, and a crash just before a write, from Checkpoint and resume.
• A side effect: a change that a call makes outside the program, such as a refund, from Tool contracts.

What you need on your computer

• The project after Checkpoint and resume. python -m pytest prints 47 passed.
• A fresh practice database: run python -m resolver reset --all once before you start.

This lesson adds two test files, and then your project matches the snapshot end-of-m04. Every timeout and crash in it is simulated by the mock services. No real payment is made.

Tap the card again?

You pay for groceries with your card. You tap it, and the card machine freezes. After ten seconds, it says “No response”. The queue is long, and the cashier says: “Just tap it again.”

Did the first payment go through? Perhaps the bank took the money and only the answer was lost. Perhaps the request never left the shop. The screen cannot tell you. If you tap again and the first payment did go through, you pay twice.

A careful cashier does something else first. She checks the till’s record or your banking app. Did a payment of this amount arrive a minute ago? If yes, you are done. If no, you tap again.

Now think of a lift button. You press it, nothing seems to happen, and you press it five more times. One lift comes. The button is safe to press again, because pressing it twice means the same as pressing it once. The card machine is not.

The problem: “If the refund call times out, just call it again”

Tomás reviews the executor. “Payment services time out sometimes,” he says. “If the refund call times out, just call it again. A retry is free.”

Let us try it on T-90104: a watering can, 12 business days late, with a shipping fee of 9.95 dollars. The model chat-strong (gpt-6.1-sol on Azure) proposes the right refund under W2. The project can simulate a timeout after the write: the payment service makes the refund, and then the answer never comes back. The --naive-retry option does what Tomás says: call it again as a new request, without checking.

python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --fault request_refund:timeout_after_write --naive-retry

request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 2 call(s))
Status: done

The output of run is not shown: it is the approval request for request_refund LK-793996 9.95 CAD RFD-LATE. “Done” looks fine. Now look in the practice database:

python -m resolver changes

2026-10-09T17:00:53  T-90104  request_refund       LK-793996  9.95 RFD-LATE  (operation 797fbb5bbb9f8a7e)
2026-10-09T17:00:53  T-90104  request_refund       LK-793996  9.95 RFD-LATE  (operation 797fbb5bbb9f8a7e-retry1)
2 change(s) in the practice database

The customer got the fee back twice. Grace’s policy says “never refund twice for the same reason on the same order”, and the workflow broke it with one approval. The trace shows how:

  2  approval  approved by agent (agent)
  2  write     request_refund op 797fbb5bbb9f8a7e -> timeout
  2  write     request_refund op 797fbb5bbb9f8a7e-retry1 -> done
  2  stop      done

Only the end of python -m resolver trace T-90104 is shown. The first call timed out, but the refund had happened. The retry was a new request, so the payment service treated it as a new refund. The service checks only that the total is not more than the customer paid, and 19.90 is less than that. It cannot know that the two refunds mean the same thing.

The idea: name each change, and ask before you repeat it

A timeout is an uncertain outcome: the change may or may not have happened. Two ideas make a write safe under that uncertainty.

The first is an operation ID: a name for one intended change, the same on every attempt. The service keeps a register of the IDs that it has done. If the same ID comes again, the service returns the first result and changes nothing. A write that gives the same result when you repeat it is idempotent, like the lift button. The word comes from Latin: “the same power”. Many real payment services accept such an ID; they often call it an idempotency key.

The second is reconciliation: after an uncertain outcome, ask the service what happened before you act again. It is the cashier who checks the till. In this project, world.operation(op_id) returns the stored result of that operation, or nothing.

Where does the ID come from? Not from a random number, because a new process would make a new one. The project computes it from the ticket and the exact action:

resolver/execute.py (part)

def operation_id(task_id: str, action) -> str:
    text = json.dumps({"ticket": task_id, **action.model_dump()}, sort_keys=True)
    return hashlib.sha256(text.encode()).hexdigest()[:16]

SHA-256 turns any text into a fixed fingerprint, and the same text always gives the same fingerprint. The project keeps the first 16 characters. Try it in Python, in the project folder, after the run above:

from resolver.checkpoint import CheckpointStore
from resolver.execute import operation_id
state = CheckpointStore().load("T-90104-agent_structured-chat-strong-1")
action = state.proposal.actions[0]
print(action.model_dump())
print(operation_id("T-90104", action))
print(operation_id("T-90103", action))
print(operation_id("T-90104", action.model_copy(update={"amount": 19.90})))

{'tool': 'request_refund', 'order_id': 'LK-793996', 'amount': 9.95, 'reason_code': 'RFD-LATE', 'payment_id': None}
797fbb5bbb9f8a7e
1f9d2528fdce4b31
af291d28b738e056

The same ticket and action always give 797fbb5bbb9f8a7e: in this process, after a crash, in a second run, on another computer. A different ticket or a different amount is a different change, so it gets a different ID. That is why the operation IDs in Approval boundaries were the same on your computer as in the lesson.

The same timeout, with reconciliation

Reset the practice database and the saved runs with python -m resolver reset --all. Then run the same ticket and the same simulated timeout, without --naive-retry:

python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --fault request_refund:timeout_after_write
python -m resolver changes

request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 1 call(s))
Status: done
2026-10-09T17:00:54  T-90104  request_refund       LK-793996  9.95 RFD-LATE  (operation 797fbb5bbb9f8a7e)
1 change(s) in the practice database

One refund, one call. The end of the trace shows why:

  2  write     request_refund op 797fbb5bbb9f8a7e -> timeout
  2  reconcile request_refund op 797fbb5bbb9f8a7e -> found: it happened
  2  stop      done

After the timeout, the executor asked the service about 797fbb5bbb9f8a7e. The service found it, so the executor recorded the result and did not call again. Ticket T-90603 has the same simulated failure built in, on a double charge of 45.95 dollars. Every design that runs on an Azure model handled it with one refund, in every recorded run.

A timeout can also happen before the write reaches the service. T-90604, a step ladder to return, has that simulated failure. Run it and approve it:

python -m resolver run T-90604
python -m resolver approve T-90604-agent-chat-small-1 --as agent

create_return_label LK-645334: done (operation 2ed3b2a93ab0f494, 2 call(s))
Status: done

  2  write     create_return_label op 2ed3b2a93ab0f494 -> timeout
  2  reconcile create_return_label op 2ed3b2a93ab0f494 -> not found
  2  write     create_return_label op 2ed3b2a93ab0f494 -> done

This time the service had no record of the ID, so the executor called again, with the same ID. Two calls, one label. If the first call had arrived late after all, the service would have answered the second call with the first result.

The rule in one picture

The executor follows the same rule for every approved action. Read it from the top:

What execute() does for each approved action: save first, call with the same ID, and after an uncertain answer ask the service before calling again.

Text description of the diagram

1. Compute the operation ID from the ticket and the action.
2. If the checkpoint says the operation is done, skip it. If it is pending and a call is on record, ask the service whether the ID happened. If yes, record it as done.
3. Otherwise, save the operation as pending, before the call. This is a write-ahead record.
4. Call the service with the same ID, at most 3 times.
5. If the answer is done, record it. If the service is unavailable (503), call again. If it refuses, stop: a person takes over, and nothing changed.
6. If the call times out, ask the service whether the ID happened. If yes, record it as done. If no, call again with the same ID.

Step 3 is the write-ahead record. The executor saves “pending, attempt 1” in the checkpoint before the call, so a crash during the call still leaves a trace of the attempt. Then a resume knows that it must ask first. Without that record, a resume would think that nothing was tried.

Not every error deserves a retry. A timeout and a 503 are about the connection, so the same call may work later. A refusal is the world’s answer: T-90605’s reship is refused by the warehouse (simulated), and calling again cannot change that. The run goes to a person with no change. After 3 calls without success, a person takes over too.

Crash before, during or after the write

Now combine this with the checkpoint from the last lesson. A crash can come before the write, during the call or after the write. The explorer replays all three on T-90104, with a resume that keeps the operation ID and a resume that does not.

Explore

Stop the process around one refund, then resume it

Refunds in the database

2

The customer gets

19.90 CAD

Calls to the service

2

2 refunds of 9.95 CAD: the customer is paid 2 times. The refund happened, then the process stopped. The record still says "not done", and with no ID there is nothing to ask about. So the resume sends it again.

• a call that wrote to the practice database
• a simulated crash or timeout
• reconcile: ask the service by operation ID

Stop: after the write. Resume: without an ID (a new request).

#	Part	What happens
1	write	request_refund 9.95, request request-1: the payment service makes the refund (RF-19F1064B)
2	stop	the process stops after the write, before it was recorded (simulated crash)
3	checkpoint	the record says: refund not done (no ID to ask about)
4	resume	resume: the refund is not done, so send it
5	write	request_refund 9.95, request request-2: the payment service makes the refund (RF-6FC63565)
6	end	status: done

Refunds for LK-793996 in the practice database at the end:

• RF-19F1064B 9.95 CAD RFD-LATE, request request-1
• RF-6FC63565 9.95 CAD RFD-LATE, request request-2

Ticket T-90104: the proposal is the recorded decision of chat-strong (gpt-6.1-sol on Azure), agent_structured, repeat 1, approved by Amira (agent). Every crash and timeout is simulated on the practice database. "Without an ID" is a constructed client; for a timeout it is the project's own --naive-retry.

T-90104: one approved refund of 9.95, a simulated crash or timeout, then the resume. Every row comes from the project's own execute() on a fresh practice database; the client without an ID is constructed.

Text description of the explorer

The explorer has two controls. The first chooses where the process stops. The choices are before the write, during the call (a timeout after the service made the refund), and after the write, before it was recorded. The second chooses how the resume calls the payment service: with the operation ID, or without an ID, as a new request. It shows the number of refunds in the database, the amount the customer gets, the number of calls, and a table of the events.

Before the write: with or without an ID, one refund of 9.95, because nothing was written before the crash.

During the call: with the operation ID, the executor asks the service, finds the refund and records it: one refund, one call. Without an ID, the retry is a new request: two refunds, 19.90.

After the write: with the operation ID, the checkpoint holds the operation as pending with 1 call. So the resume asks the service first and finds the refund: one refund. Without an ID, the record says “not done” and there is nothing to ask about, so the resume sends it again: two refunds, 19.90.

Two of the six cases pay twice, and both are “without an ID” after the change happened. Before the write, a crash is harmless either way. The danger is not the crash itself. It is not knowing whether the change happened.

Try the “after the write” case on your own project. The --interrupt after-write option simulates a crash just after the service made the refund:

python -m resolver reset --all
python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --interrupt after-write
python -m resolver resume T-90104-agent_structured-chat-strong-1
python -m resolver changes

The process stopped: simulated crash after the write of 797fbb5bbb9f8a7e. Resume with: python -m resolver resume T-90104-agent_structured-chat-strong-1
Resuming T-90104-agent_structured-chat-strong-1 (status executing)
request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 1 call(s))
Status: done
2026-10-09T17:00:55  T-90104  request_refund       LK-793996  9.95 RFD-LATE  (operation 797fbb5bbb9f8a7e)
1 change(s) in the practice database

The output of run is not shown. The trace ends with note interrupted, then reconcile … found: it happened. The resume did not call the service again. It did not ask for a new approval either, because the approval is in the checkpoint with its digest.

Common mistake

'If the refund call times out, just call it again'

Symptom: after one approval of a 9.95 fee refund for T-90104, changes shows two refunds of 9.95: operations 797fbb5bbb9f8a7e and 797fbb5bbb9f8a7e-retry1. The run says done (… 2 call(s)). The timeout is simulated (--fault request_refund:timeout_after_write), and the retry is Tomás’s shortcut (--naive-retry).

Cause: the first call made the refund, and only the answer was lost. The retry did not check that, and it was a new request with a new ID, so the service made a second refund.

Diagnose: in the trace, look for a write … -> timeout followed by a second write with no reconcile between them. In the database, look for two changes with the same order, amount and reason, from one approval.

Fix: give every write an operation ID that stays the same on every attempt, and make the service keep a register of the IDs. After a timeout, reconcile before you call again. tests/test_execute.py keeps Tomás’s shortcut as a test, test_just_calling_it_again_refunds_twice, so that nobody brings it back by accident.

Optional: the same pause in LangGraph

Frameworks offer the ideas that you built in plain Python. LangGraph, a well-known library for agent orchestration, has a state graph, a SQLite checkpointer, and interrupt() to wait for a person. We ran the course’s own nodes inside a LangGraph 1.2.14 graph, with the recorded chat-small decisions. It proposed the same as the plain loop on 70 of 70 tickets. On T-90103 it paused at the approval, and the two writes happened only after the resume.

LangGraph’s documentation states one rule that matters here: on resume, an interrupted node runs again from its start. So a write placed before interrupt() in the same node runs twice. We tested a node that makes a 9.95 refund and then calls interrupt():

The write in the node	Node runs	Refunds
a new ID on every call	2	2
the course’s operation ID (ticket and action)	2	1

The framework gives you checkpoints and pauses. It does not make a write safe to repeat; the operation ID does. The framework also costs something: 43 packages and 44 MB in a fresh environment, against 19 packages and 31 MB for the project (measured with uv; plain pip makes both larger). The comparison script, graph.py, is in the labs repository, in C10/optional/langgraph. It is optional and is not part of the project. It runs in its own virtual environment, on the snapshot end-of-m04, with no key. The README of the optional examples shows how to set it up. Agent frameworks change fast, so check the current documentation of any framework that you choose.

When to use operation IDs, and what they cost

Use them for every write that has a side effect: money, goods, messages, records in another system. A read needs none: reading twice changes nothing.

They have costs:

• The service must help. It needs a register of IDs and a way to ask about one. Many payment services offer an idempotency key. If a service offers neither, put the register in your own database, and reconcile against the service’s list of recent changes.
• The ID must name the right thing. It is the ticket plus the exact action. If two different refunds on one ticket are both correct, they must differ in the action, for example in the reason code. If the amount changes, the ID changes, and so does the approval digest: that is a new change, and it needs a new approval.
• Reconciliation takes time. Each uncertain outcome costs one more call to the service. Here, it is a lookup in SQLite.
• The register grows. Keep the IDs at least as long as a retry or a resume can come. In this project, they stay with the practice database.

Guided practice: finish the module

1. Check your starting point:

   python -m pytest

   It prints 47 passed.

2. Copy the last two test files from the snapshot end-of-m04:

   cp ../nextia-learning-labs/C10/snapshots/end-of-m04/tests/{test_systems,test_execute}.py tests/

   On Windows, in PowerShell: foreach ($f in 'test_systems','test_execute') { Copy-Item "..\nextia-learning-labs\C10\snapshots\end-of-m04\tests\$f.py" tests\ }.

   tests/test_systems.py checks the mock services. The same operation ID writes once. A timeout after the write hides a change, and operation() finds it. The services refuse a refund above what was paid.

   tests/test_execute.py is the completion check of this module. Here are its test names:

   tests/test_execute.py (the test names)

   def test_interrupt_before_the_write_then_resume(world): ...
   def test_interrupt_after_the_write_then_resume_reconciles(world): ...
   def test_a_timeout_after_the_write_is_reconciled_not_repeated(world): ...
   def test_just_calling_it_again_refunds_twice(world): ...
   def test_a_timeout_before_the_write_is_retried_with_the_same_id(world): ...
   def test_a_service_that_is_down_once_is_retried(world): ...
   def test_a_refused_write_goes_to_a_person_with_no_change(world): ...
   def test_running_the_same_approved_proposal_twice_changes_nothing_more(world): ...

3. Run the tests:

   python -m pytest

   It prints 60 passed. Your project now matches the snapshot end-of-m04.

4. Run the examples of this lesson again with and without --naive-retry, and compare changes each time.

Module practice: interrupt a write, then resume it

Write tests/test_module4_practice.py for T-90604, the step ladder return. The approved action is one return label: create_return_label LK-645334 LADDER-3 change_of_mind. Write three checks:

1. Interrupt the approved execution before the write, then resume it twice. There must be exactly one label.
2. Do the same with an interrupt after the write.
3. Try to execute the proposal without an approval, and after a rejection. Both must be refused, with no change.

Use proposed() from tests/test_approval.py, the world fixture, decide(), and execute() with interrupt=.

Show an example answer

tests/test_module4_practice.py

"""Module 4 practice: interrupt a simulated task before and after a mock write, then resume.
The crashes are SIMULATED (`interrupt=`); every write goes to a fresh practice database."""

import pytest

from resolver.approval import decide
from resolver.execute import ExecutionRefused, Interrupted, execute

from .test_approval import proposed

LABEL = {"tool": "create_return_label", "order_id": "LK-645334", "sku": "LADDER-3", "reason": "change_of_mind"}


@pytest.mark.parametrize("where", ["before-write", "after-write"])
def test_resume_after_a_crash_writes_the_label_once(world, where):
    s, w = proposed("T-90604", [LABEL]), world("T-90604")
    decide(s, True, "Amira", "agent")
    with pytest.raises(Interrupted):
        execute(s, w, interrupt=where)
    assert len(w.changes()) == (0 if where == "before-write" else 1)
    execute(s, w)                                   # the resume
    execute(s, w)                                   # a second resume changes nothing
    assert len(w.changes()) == 1 and s.status == "done"


def test_a_resume_cannot_skip_the_approval(world):
    s, w = proposed("T-90604", [LABEL]), world("T-90604")   # waiting_approval: nobody approved it
    with pytest.raises(ExecutionRefused):
        execute(s, w)
    decide(s, False, "Grace", "grace", "the customer already sent it back")
    with pytest.raises(ExecutionRefused):
        execute(s, w)
    assert w.changes() == []

python -m pytest tests/test_module4_practice.py prints 3 passed, and python -m pytest prints 63 passed. The world fixture does not add the task’s own simulated timeout, so these tests check the crash alone. Remove the operation ID, for example by making operation_id() return a new random value on each call. Then the “after-write” case finds two labels.

Module check

You have finished the module when you can show these, on your own project:

• python -m pytest prints 60 passed (or more, with your practice tests). tests/test_execute.py passes: a resume after a crash before or after the write makes the change once, and nothing runs without an approval.
• You can run a ticket and stop at the approval request. You can explain each part of it to Grace: the exact action, who may approve it, and the evidence.
• You can explain why a resume uses the saved run and never a new run of the ticket.
• You can explain, with the trace, why --naive-retry gave two refunds, and why the same timeout gave one refund without it.

Reset. If your project is broken, rename it and copy the snapshot end-of-m04:

mv resolution-workflow resolution-workflow-old
cp -R nextia-learning-labs/C10/snapshots/end-of-m04 resolution-workflow

On Windows, in PowerShell: Rename-Item resolution-workflow resolution-workflow-old, then Copy-Item -Recurse nextia-learning-labs\C10\snapshots\end-of-m04 resolution-workflow. Make the virtual environment again, as the snapshots README shows. python -m pytest then prints 60 passed. These steps were tested on macOS; the Windows commands are not tested.

Knowledge check

Check your understanding

A refund call times out. Name the two things that could have happened at the payment service, and say what the executor does next in each case.

Show the answer

Either the service made the refund and only the answer was lost (a timeout after the write). Or the request never made the change (a timeout before the write). The executor cannot tell from the timeout, so it reconciles: it asks the service about the operation ID. If the service found it (T-90104 with timeout_after_write), the executor records it as done and does not call again. If not (T-90604), it calls again with the same ID: two calls, one label.

Check your understanding

Why does the project compute the operation ID from the ticket and the action, and not take a random number for each call?

Show the answer

The same intended change must have the same ID on every attempt. That includes a retry, a resume in a new process after a crash, and a second run of the same proposal. Only then can the service recognise a repeat and return the first result. A random ID is new each time, so every attempt looks like a new refund. That is exactly what --naive-retry did with 797fbb5bbb9f8a7e-retry1.

Check your understanding

In the explorer, a crash before the write gives one refund even without an operation ID. Does that mean the ID is only needed for timeouts? Explain with the other cases.

Show the answer

No. Before the write, nothing happened, so any resume is safe. But the executor does not know where the crash came. After the write, the record says “not done”, and without an ID the resume sends it again: two refunds. During the call, the same happens. The ID and reconciliation are needed whenever the change may have happened and the record does not say so.

Recap

• A timeout is an uncertain outcome: the change may have happened. “Just call it again” turned one approved fee refund into two (simulated timeout, --naive-retry).
• An operation ID names one intended change. The project computes it from the ticket and the exact action, so every attempt and every resume uses the same ID. The service keeps a register, so the write is idempotent.
• Reconciliation asks the service what happened before acting again. The executor saves each operation as pending before the call (write-ahead), so a resume after a crash knows to ask.
• Retry only connection errors (timeout, 503), at most 3 calls. A refusal goes to a person.
• Frameworks such as LangGraph give checkpoints and pauses, but a resumed node runs again from its start: the operation ID is still your job.
• tests/test_execute.py and tests/test_systems.py: 60 passed. The project matches end-of-m04.

You have finished Module 4. Next, the workflow does several reads at once, and merges their results with clear rules.

Next: Parallel work