Checkpoint and resume

What you will be able to do

Save the whole state of a task after every step. Then a new process can continue it after a crash, or after an approval that takes three days.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The approval request, approve and reject, from Approval boundaries.
• The fields of TaskState: status, evidence, messages, proposal, from Explicit task state and Manage working context.

What you need on your computer

• The project after Approval boundaries: the six project files of end-of-m04 and two new test files. python -m pytest prints 45 passed.
• A fresh practice database: run python -m resolver reset --all once before you start.

This lesson adds one test file. Every crash in it is simulated, and every write goes to the practice database.

A note on the bakery wall

A bakery opens at seven. The night baker starts the dough at four, then her shift ends. She does not take the work home in her head. She writes on the board by the oven: “Rye dough mixed at 4:10. Salt added. First rise until 6:00.” The morning baker reads the board and continues from there.

Two things make the board work. It holds everything the next baker needs, not only the last step. And it is written as the work happens, not at the end of the shift. If the night baker falls ill at 5:00, the board still says what is done.

The board has one weak spot. Suppose the last line says “adding salt”, and nothing after it. Did the salt go in? The morning baker cannot know from the board. She must taste the dough. The last lesson of this module is about that taste.

The problem: the state lives in the process

Until Module 4, the project kept a run only in memory. run printed the proposal and ended, and the state was gone. That was fine while the workflow only proposed. Now a proposal must wait for a person, and people are not always there.

T-65273 is a double charge of 119 dollars, so it needs a team lead or Grace. Grace will approve it, and she is away until Monday. On Friday, Amira’s terminal shows the proposal. What happens to it over the weekend?

If the state lives only in the process, there are two bad choices. Amira can keep the process running for three days. A laptop update, a closed lid or a crash ends it. Or she can run the ticket again on Monday. That costs a second set of model calls, and it may not give the same proposal.

The second point is real, not a guess. The chat-small agent has three recorded runs of every ticket, and 11 tickets got a different proposal in at least one run. Here is T-90103, the cracked planter, in its first two recordings:

python -m resolver run T-90103 --repeat 1
python -m resolver run T-90103 --repeat 2

Actions (nothing has happened yet):
  1. reship_item          LK-640436  PLANTER x 1
  2. create_return_label  LK-640436  PLANTER  reason damaged (return shipping: free)

Actions (nothing has happened yet):
  1. reship_item          LK-640436  PLANTER x 1

Only the action lines are shown. The second run forgot the free return label that W5 asks for. If Grace approved the first proposal on Friday, a new run on Monday is not the thing she approved.

The idea: save the state, not the process

A checkpoint is a saved copy of the whole task state at a known point, in storage that outlives the process. Resume means: start a new process, load the last checkpoint, and continue from that point. Like the bakery board, the checkpoint is written as the work happens: after every step, not at the end.

The name comes from long races and video games. At a checkpoint, your progress is recorded. If something goes wrong later, you start again from there, not from the start line.

What must the checkpoint hold? Everything that the next step needs, and nothing that only the process has. In this project, that is the whole TaskState:

• the status: where the task is (running, waiting_approval, executing, done …);
• the evidence and the messages: what the tools returned, and the model’s working context, so that the loop can make its next call;
• the proposal and the approval, with its digest;
• the operations: each approved write, its ID, its status and how many calls it took;
• the faults already used, so that a simulated failure does not happen twice.

The open loop, the Python variables and a call that is waiting for an answer are not in the checkpoint. They die with the process, and resume does not need them.

The project’s checkpoint store is resolver/checkpoint.py, 52 lines. It keeps one row per run in work/runs.sqlite, with the whole state as JSON. save() replaces the row, and load() turns it back into a TaskState:

resolver/checkpoint.py (part)

    def save(self, state: TaskState) -> None:
        state.updated_at = now()
        with closing(sqlite3.connect(self.path)) as db:
            db.execute("INSERT OR REPLACE INTO runs VALUES (?,?,?,?,?,?,?)",
                       (state.run_id, state.task_id, state.variant, state.model, state.status, state.updated_at,
                        state.model_dump_json()))
            db.commit()

    def load(self, run_id: str) -> TaskState:
        with closing(sqlite3.connect(self.path)) as db:
            row = db.execute("SELECT state FROM runs WHERE run_id = ?", (run_id,)).fetchone()
        if row is None:
            raise KeyError(f"No run {run_id}. Run `python -m resolver runs` to see the saved runs.")
        return TaskState.model_validate_json(row[0])

Pydantic does the hard part. model_dump_json() writes every field, and model_validate_json() checks every field on the way back. A checkpoint with a missing or wrong field fails loudly when it loads.

run saves after every step of the loop (on_step), and again at the end. approve, reject and the executor save after every change of status and before every write. So the state on disk is never more than one step behind.

The picture shows the statuses of a run, and what resume does from each one.

The statuses of a run. The checkpoint is saved at every step and every change of status; the dashed notes say what resume does from each status that is not final.

Text description of the diagram

running: the loop makes one model call per step. With a proposal that has actions, it goes to waiting_approval. A reply only ends in done. A hand to a person, or a limit of the loop, ends in handed_to_person or stopped. Resume from running runs the loop again from the saved messages.

waiting_approval: a proposal with actions, and nothing written. Approve goes to executing; reject goes to handed_to_person. Resume shows the approval request again.

executing: the approved writes, one operation at a time. When every operation is done, the run is done. A write that is refused or keeps failing goes to handed_to_person. Resume runs execute() again and reconciles the pending operations.

A worked example: an approval that waits for Monday

On Friday, Amira runs T-65273:

python -m resolver run T-65273

The output ends with Next: python -m resolver approve T-65273-agent-chat-small-1 --as team_lead. The run is saved, so Amira can close the terminal. runs lists every saved run, its status and the time of its last checkpoint:

python -m resolver runs

T-65273-agent-chat-small-1               waiting_approval  2026-10-09T17:00:50-04:00

On Monday, Grace opens a new terminal. She does not need Amira’s process. resume loads the checkpoint and, for a run that waits for approval, shows the request again:

python -m resolver resume T-65273-agent-chat-small-1

Resuming T-65273-agent-chat-small-1 (status waiting_approval)
Approval needed for run T-65273-agent-chat-small-1 (ticket T-65273)
Proposed by: agent, chat-small. Rule: W7.

Actions (nothing has happened yet):
  1. request_refund       LK-507862  119.00 CAD  RFD-DOUBLE  payment PAY-507862-2

Who may approve: team lead or above

The output continues with the reason, the evidence and the reply draft, as on Friday. It is the same proposal, because it is the same saved state: no model call, no new cost. Before Grace approves it, think about what else could change in three days.

You cannot wait three days in a lesson. The time does not matter to the checkpoint: one minute or one week, the process that approves reads the same row.

The world can change while the run waits

A checkpoint saves the task, not the world. Over the weekend, the customer phoned the help desk, and a colleague refunded the second charge by hand. Simulate that call in the practice database. Paste this in Python, in the project folder:

from resolver.systems import World
World(ticket_id="phone-call").request_refund("manual-1", "LK-507862", 119.00, "RFD-DOUBLE", "PAY-507862-2")

Python prints the new refund, RF-4CFEAE8E. Type exit() to leave Python. Now Grace approves the saved proposal:

python -m resolver approve T-65273-agent-chat-small-1 --as grace --name Grace

request_refund LK-507862: failed (operation b029a4020ded5cae, 1 call(s)) PAY-507862-2 is not a captured charge of LK-507862
Status: handed_to_person

The payment service checks its own rule: it refunds only a captured charge. The charge was already refunded, so it refuses, and the run goes to a person with no second refund. This is why the project checks again at the moment of the write. execute() calls authorize_write() again, and the services check their own invariants. A check made on Friday says nothing about Monday.

Run python -m resolver reset --all before the next example.

A crash in the middle of the loop

Now a crash while the agent is still working. The project has no command for this, so we simulate it in Python. The run saves its state after each step, then “crashes” after step 1. Paste this in Python, in the project folder:

from resolver.checkpoint import CheckpointStore
from resolver.data import load_task
from resolver.providers import MockProvider
from resolver.runner import new_state, run_task
from resolver.systems import PRACTICE_DB, Faults, World

class Crash(Exception):
    pass

def save_then_crash(state):
    store.save(state)
    if state.step == 1:
        raise Crash("simulated crash after step 1")

store = CheckpointStore()
task = load_task("T-65273")
state = new_state(task, "agent", "chat-small", "T-65273-crash-1")
world = World(PRACTICE_DB, task.task_id, Faults(task.faults))
try:
    run_task(task, "agent", "chat-small", MockProvider().complete, world, state=state, on_step=save_then_crash)
except Crash as e:
    print(e)

simulated crash after step 1

Type exit() to leave Python. The checkpoint holds the state after step 1, with status running:

python -m resolver runs
python -m resolver resume T-65273-crash-1

T-65273-crash-1                          running           2026-10-09T17:00:51-04:00
Resuming T-65273-crash-1 (status running)
Outcome: resolve (rule W7) | 3 model call(s), 2 tool call(s) | 7155 tokens in, 192 out | US$0.00081

Then the approval request follows. Look at the counts: 3 model calls and 7,155 tokens in, the same as a run without a crash. The resume did not repeat step 1. It loaded the saved messages and the evidence from get_order, and asked the model for step 2. python -m resolver trace T-65273-crash-1 shows the three steps, numbered 1, 2 and 3.

Why does this matter? A model call costs money and time, and it may give a different answer. A resume that starts from the beginning pays again, and it may end with a different proposal, as T-90103 showed. A resume from the checkpoint pays only for the steps that are left.

What survives a crash

The picture shows where each thing lives when the process dies.

What a crash takes away, and the two records that a resume uses: the task's checkpoint and the service's own register of operations.

Text description of the diagram

The Python process holds the variables, the open loop and any call that is waiting for an answer. A crash loses all of them.

The checkpoint store, work/runs.sqlite, holds the TaskState, saved after every step. It has the status, evidence, messages, proposal and approval, and the operations with their ID, status and number of calls.

The practice database, work/larkfield.sqlite, holds the refunds, labels and reshipments, and the operations register: for each operation ID, its result.

A resume starts a new process. It loads the state from the checkpoint store. For a pending write, it asks the practice database whether that operation ID happened.

There are two records, and they are owned by different systems. The checkpoint says what the workflow meant to do and how far it got. The service’s register says what really happened. A write sits between them, and a crash can come at the worst moment.

Try it. A crash just before a write is the easy case:

python -m resolver run T-90104 --variant agent_structured --model chat-strong
python -m resolver approve T-90104-agent_structured-chat-strong-1 --as agent --interrupt before-write

The process stopped: simulated crash before the write of 797fbb5bbb9f8a7e. Resume with: python -m resolver resume T-90104-agent_structured-chat-strong-1

The output of run is not shown here: it is the approval request for a refund of 9.95. changes prints 0 change(s) in the practice database, and runs shows the run as executing. Resume it:

python -m resolver resume T-90104-agent_structured-chat-strong-1

Resuming T-90104-agent_structured-chat-strong-1 (status executing)
request_refund LK-793996: done (operation 797fbb5bbb9f8a7e, 1 call(s))
Status: done

One refund, one call. The approval was in the checkpoint, so the resume did not ask for it again, and it did not skip it either. A second resume prints Nothing to resume: the run is done.

A crash just after the write is the hard case. The refund happened, but the checkpoint still says “pending”. That is the bakery board with “adding salt” as its last line. The next lesson shows what the executor does then.

Common mistake

'Just run the ticket again on Monday'

Symptom: on Friday, Grace reads the proposal for T-90103 (reship the planter, free return label) and says yes. On Monday, Tomás runs the ticket again, sees a proposal for the same order, and approves it. The customer gets a new planter and no return label. python -m resolver runs shows two runs of T-90103.

Cause: the run was repeated instead of resumed. A model can give a different proposal for the same ticket: in the recordings, repeat 2 of T-90103 proposed the reship without the label. The second run also cost a second set of model calls.

Diagnose: compare the actions of the two runs, line by line. Check whose approval is on which run: Grace’s approval belongs to the first run and its digest, not to the ticket.

Fix: save the state at every step, and continue a waiting run with resume or approve, never with a new run. If a ticket really needs a fresh run, treat it as a new proposal that needs a new approval.

When to checkpoint, and what it costs

Checkpoint any task that can wait for a person, or that writes to another system. Both are true for Larkfield. A task that only reads and answers in two seconds may not need it: if it fails, you run it again, and nothing has changed.

The costs are small but real:

• A write per step. One SQLite row per run, replaced at every step. Here, a few milliseconds. A busy system needs a real database and a plan for many writers.
• Stored data. The messages hold the ticket text and order details. A checkpoint is personal data, like the memory in Memory design. Decide how long you keep it, and delete old runs.
• Changes to the state. If a later version adds a required field to TaskState, old checkpoints fail to load. Give new fields a default, as operations and faults_seen have, or migrate the old rows.
• Stale facts. A checkpoint can be days old. Check again at the moment of the write, as the payment service did above.

Guided practice: test the checkpoint

1. Check your starting point:

   python -m pytest

   It prints 45 passed.

2. Make the file tests/test_checkpoint.py. Copy it from the snapshot end-of-m04, or type it:

   tests/test_checkpoint.py

   """Checkpoint and resume: the whole state survives the process."""
   
   from resolver.checkpoint import CheckpointStore
   from resolver.data import load_task
   from resolver.runner import new_state
   
   
   def test_save_and_load_the_whole_state(tmp_path):
       store = CheckpointStore(tmp_path / "runs.sqlite")
       s = new_state(load_task("T-90103"), "agent", "chat-small", "r-1")
       s.plan = ["look up the order", "propose"]
       s.log("note", "hello", x=1)
       store.save(s)
       back = CheckpointStore(tmp_path / "runs.sqlite").load("r-1")
       assert back.model_dump() == s.model_dump()
       assert store.latest("T-90103").run_id == "r-1"
   
   
   def test_a_crashed_cli_run_resumes(capsys):
       from resolver.__main__ import main
       main(["run", "T-90104", "--variant", "agent", "--repeat", "1"])
       main(["runs"])
       assert "T-90104-agent-chat-small-1" in capsys.readouterr().out

   The first test loads the state with a new store object, as a new process would. The second runs the CLI and checks that the run is in the saved list. The tests use a temporary folder, not your work/ folder.

3. Run the tests:

   python -m pytest

   It prints 47 passed.

4. Run python -m resolver reset --all. Then run the crash before the write and the resume from this lesson again, and python -m resolver changes. It shows one refund of 9.95.

Your turn: crash at a different step

Change the Python example so that it crashes after step 2 of T-65273, with a new run ID. Before you resume, predict: how many model calls will the resumed run report, and what will trace show at step 2? Then resume it and check.

Show an example answer

Change two lines: if state.step == 2: in save_then_crash, and the run ID "T-65273-crash-2". Run the code, then:

python -m resolver resume T-65273-crash-2
python -m resolver trace T-65273-crash-2

The outcome line is the same as before: 3 model call(s), 2 tool call(s) | 7155 tokens in, 192 out. The calls are counted in the saved state, so the resumed run reports all three. Only one of them, step 3, happened after the resume. The trace shows get_payments at step 2, from before the crash, then finish at step 3. The saved evidence and messages let the model continue, and nothing was asked twice.

Knowledge check

Check your understanding

What must a checkpoint of this workflow hold so that a new process can continue, and what does it not need to hold?

Show the answer

It must hold the whole task state. That is the status, the evidence and the messages for the next model call, and the proposal and the approval with its digest. It also holds the operations with their IDs and statuses, and the simulated faults already used. It does not need the process: the Python variables, the open loop or a call that is waiting for an answer. Those die with the process, and resume rebuilds what it needs from the saved state.

Check your understanding

T-65273 waited for Grace from Friday to Monday. Why is 'resume the saved run' better than 'run the ticket again'? Give two reasons with numbers from this lesson.

Show the answer

First, the same proposal. In the recordings, 11 tickets got a different proposal in at least one of three runs, and T-90103’s second run dropped the free return label. The approval belongs to the proposal that the approver read. Second, no new cost. The resume made no model call, but a new run of T-65273 makes 3 calls with 7,155 tokens in again.

Check your understanding

Over the weekend, a colleague refunded the second charge of T-65273 by hand. On Monday, Grace approves the saved proposal. What happens, and which check catches it?

Show the answer

The executor calls the payment service, which refuses: PAY-507862-2 is not a captured charge of LK-507862. The run goes to a person with no second refund. The check is the service’s own invariant, made at the moment of the write. The checkpoint saved the task, not the world, so the workflow must check the world again before it changes it.

Recap

• A checkpoint is a saved copy of the whole task state at a known point, in storage that outlives the process. Resume loads it in a new process and continues from there.
• The project saves TaskState in work/runs.sqlite after every step and every change of status. runs lists the saved runs; resume continues one.
• An approval can wait for days: resume and approve read the same saved proposal, with no new model call. Running the ticket again can give a different proposal.
• A crash in the loop resumes at the next step: T-65273 still made 3 model calls in total.
• The checkpoint saves the task, not the world: check again at the moment of the write. A crash just after a write is the hard case, for the next lesson.
• tests/test_checkpoint.py: 47 passed.

Next: Side effects and retries