Plan and revise

What you will be able to do

Let the agent write down the steps it still has to do, so that you can check each one. Then watch it change those steps when a tool result shows that the first idea was wrong.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The TaskState of a run and its events, from Explicit task state.
• The read tools and their refusals, from Tool contracts.
• The loop and its stop reasons (max_steps, repeated_call, too_many_errors …), from Control the execution loop.

What you need on your computer

• The project resolution-workflow at the end of Module 2, after Control the execution loop: Make the loop end in the project, with its virtual environment active. python -m pytest prints 35 passed.
• If you skipped Module 2, copy the snapshot end-of-m02 from the labs repository (nextia-learning-labs/C10/snapshots/), and make the virtual environment as its README shows.
• If you kept tests/test_my_loop.py from the module practice of Module 2, every test count in this module is 4 higher: 39 passed here.

This lesson adds one test file. You need no account and no key: every model decision below is a recording.

A list for a Saturday of errands

On Saturday morning you write a short list: post the parcel, collect the prescription, buy food for the week. Each line has a clear end. You hold a receipt, you hold a paper bag, the fridge is full. So at any moment you know which lines are done.

At the post office, a sign says “Closed today”. You do not go home. You change the list: “post the parcel at the shop on Elm Road”, and you put it last, because that shop is near the supermarket. The goal is the same. The list changed, because the world told you something new.

Two things made that list useful. Each line could be checked, and you changed it when a fact surprised you. A list that says “be useful today” has neither. Nobody can say when it is done, and no closed door can change it.

The problem: steps that nobody can check

The agent of this course decides its own reads. With chat-strong and the local gemma3:4b, the project uses the agent_structured design: at each step, the model answers with one JSON object. One of its fields is plan, “the steps left, short”. You saw it in the trace in Control the execution loop.

Here is the ticket for this lesson, T-80009, from the LLM applications course:

My order LK-55190 never arrived. It was a bag of potting soil and two trowels. Can you check?

The order ID has five digits, not six. Larkfield’s IDs are LK- and six digits, so get_order will refuse it. The customer’s real order is LK-551906. Here is the recorded run of gemma3:4b, a small model that runs on your own computer:

python -m resolver run T-80009 --variant agent_structured --model gemma3:4b --trace

Run T-80009-agent_structured-gemma3:4b | single agent with one JSON step per call (for models without tool calling) | gemma3:4b
Stopped: repeated_call. No proposal: the ticket goes to a person.

Run T-80009-agent_structured-gemma3:4b: agent_structured, gemma3:4b, ticket T-80009, status stopped
6 model call(s), 13940 tokens in, 244 out, US$0.00000, 10.484 s waiting for the model

  1  plan      ["check order status"]
  1  model     gemma3:4b: get_order({"order_id": "LK-55190"})  (1899 in, 39 out, 1.594 s)
  1  tool      get_order({"order_id": "LK-55190"}) -> REFUSED invalid_arguments: Invalid arguments for get_order: order_id: String should ma…
  2  model     gemma3:4b: search_policy({"query": "invalid order id"})  (1982 in, 37 out, 1.436 s)
  2  tool      search_policy({"query": "invalid order id"}) -> ok
  3  plan      ["w3 lost"]
  3  model     gemma3:4b: get_order({"order_id": "LK-55190"})  (2294 in, 41 out, 2.065 s)
  3  tool      get_order({"order_id": "LK-55190"}) -> REFUSED invalid_arguments: Invalid arguments for get_order: order_id: String should ma…
  4  plan      ["ask_customer"]
  4  model     gemma3:4b: get_customer_orders({})  (2383 in, 43 out, 1.523 s)
  4  tool      get_customer_orders({}) -> ok
  5  plan      ["w3 lost"]
  5  model     gemma3:4b: get_order({"order_id": "LK-551906"})  (2507 in, 42 out, 1.705 s)
  5  tool      get_order({"order_id": "LK-551906"}) -> ok
  6  plan      ["w2 late"]
  6  model     gemma3:4b: get_order({"order_id": "LK-551906"})  (2875 in, 42 out, 2.161 s)
  6  stop      repeated_call
  6  stop      repeated_call

This is the project’s recording of T-80009, the one that the command replays. It is a different run from the one in the explorer of Control the execution loop, so its calls differ. A trace shows a plan line only when the plan changes, so step 2 still has the plan of step 1.

Read the plan lines from top to bottom. “w3 lost” is a guess at a rule, not a step: no tool result can complete it. At step 4 the plan says “ask_customer”, but the call is get_customer_orders. At step 3 the model sends the refused ID again. The plan and the actions do not follow each other. The loop’s repeated-call rule stopped the run after six calls, and a person gets the ticket. The loop did its job; the plan did not help.

The idea: steps that can be checked, revised by evidence

A plan is the list of steps that the agent still intends to do, written as text that you can read. Planning means writing that list before acting. Revising (or replanning) means changing the list when a result shows that an assumption was wrong.

A step is useful when it can be checked: you can name the tool result that completes it. “Check the order and delivery details” ends when get_order returns the order with its expected_by date. “w3 lost” names no evidence, so nothing can complete it.

The plan is a field of the model’s answer, like the tool name. It is not the model’s hidden reasoning. Some models think in tokens that you never see; this course does not use them and does not need them. Everything in this lesson is observable: the plan text, the tool call, the result.

Models change fast, and a newer small model may plan better than gemma3:4b. The method of this lesson lasts: read the plan next to the evidence, step by step.

A tiny worked example: T-80009 with chat-strong

Here is the same ticket with chat-strong (gpt-6.1-sol on Azure), repeat 1 of its recordings. Run it:

python -m resolver run T-80009 --variant agent_structured --model chat-strong --trace

Run T-80009-agent_structured-chat-strong | single agent with one JSON step per call (for models without tool calling) | chat-strong
Outcome: reply_only (rule W1) | 4 model call(s), 3 tool call(s) | 9464 tokens in, 326 out | US$0.02219
Reply draft: I found your order as LK-551906, containing one 40 L bag of potting soil and two trowels. It shipped on September 29 and is still marked in transit. Its expected delivery date was October 6, so it is now 3 business days late. The latest tracking scan was September 29. Please allow a little more time; if it still hasn’t arrived after it is more than 5 business days late, contact us again so we can review the next steps.

Run T-80009-agent_structured-chat-strong: agent_structured, chat-strong, ticket T-80009, status done
4 model call(s), 9464 tokens in, 326 out, US$0.02219, 9.497 s waiting for the model

  1  plan      ["Check the order and delivery details.", "Apply the delivery policy and draft a reply."]
  1  model     chat-strong: get_order({"order_id": "LK-55190"})  (2163 in, 48 out, 2.352 s)
  1  tool      get_order({"order_id": "LK-55190"}) -> REFUSED invalid_arguments: Invalid arguments for get_order: order_id: String should ma…
  2  plan      ["Find the customer's order using the products mentioned.", "Check delivery details and apply the delivery po…
  2  model     chat-strong: get_customer_orders({})  (2253 in, 48 out, 1.662 s)
  2  tool      get_customer_orders({}) -> ok
  3  plan      ["Check delivery details for the matching order.", "Apply the delivery policy and draft a reply."]
  3  model     chat-strong: get_order({"order_id": "LK-551906"})  (2367 in, 49 out, 1.589 s)
  3  tool      get_order({"order_id": "LK-551906"}) -> ok
  4  plan      []
  4  model     chat-strong: finish({"outcome": "reply_only", "rule": "W1",…)  (2681 in, 181 out, 3.894 s)
  4  proposal  reply_only (W1): no change
  4  stop      finished

The picture follows the run from left to right. Each box is one step: the plan the model wrote, the call it made, and the result the application returned.

T-80009 with chat-strong (gpt-6.1-sol on Azure, recorded): the plan changes twice, each time after a tool result. The dashed box is the refused call.

Text description of the diagram

Step 1. The plan has two steps: check the order and delivery details; apply the delivery policy and draft a reply. The call get_order with LK-55190 is refused, because the ID is not LK- and 6 digits.

Step 2. A new plan: find the customer’s order using the products mentioned; check delivery details and apply the delivery policy. The call get_customer_orders returns one order, LK-551906, with potting soil and a trowel.

Step 3. The plan: check delivery details for the matching order; apply the delivery policy and draft a reply. The call get_order with LK-551906 returns the order: expected by 6 October, 3 business days late.

Step 4. The model finishes with reply_only, rule W1. The plan is now empty.

Follow the evidence that changed each plan:

1. Step 1. The plan assumes that LK-55190 is a real order. The application refuses the ID before any lookup.
2. Step 2. That refusal breaks the assumption, so the first step changes. “Find the customer’s order using the products mentioned” can be checked: get_customer_orders returns the list, and one order has potting soil and a trowel.
3. Step 3. Now the order is known, so the plan names “the matching order”. get_order completes the step with business_days_late: 3.
4. Step 4. Nothing is left, so the plan is empty and the model finishes. Rule W2 needs more than 5 business days late, so 3 days gives a status reply (W1).

The trace cuts long plans at 110 characters. To read every plan in full, ask the run’s TaskState. Start Python in the project folder with python, then paste these lines. Press Enter until the >>> prompt comes back:

from resolver.data import load_task
from resolver.evaluate import run_and_score
from resolver.providers import MockProvider
task = load_task("T-80009")
score, state = run_and_score(task, "agent_structured", "chat-strong", MockProvider().complete)
for event in state.events:
    if event.kind == "plan":
        print(event.step, event.detail["plan"])

print("Current plan:", state.plan)

1 ['Check the order and delivery details.', 'Apply the delivery policy and draft a reply.']
2 ["Find the customer's order using the products mentioned.", 'Check delivery details and apply the delivery policy.']
3 ['Check delivery details for the matching order.', 'Apply the delivery policy and draft a reply.']
4 []
Current plan: []

Type exit() to leave Python. run_and_score runs the task on a fresh copy of the practice database, as eval does. The loop records a plan event only when the plan changes. So the four lines are the four versions of the plan, and state.plan holds the last one.

Here is the same idea as a table. Each step of a good plan names the evidence that ends it:

Plan step (chat-strong)	Ends when	Done in this run?
Check the order and delivery details	get_order returns the order	no: the ID was refused
Find the customer’s order using the products mentioned	get_customer_orders lists an order with those products	yes, LK-551906
Check delivery details for the matching order	get_order returns expected_by and business_days_late	yes, 3 days late
Apply the delivery policy and draft a reply	finish with a resolution that the application accepts	yes, W1

A plan is a guess

The recordings have three repeats of chat-strong on every ticket. Repeat 3 of T-80009 did not try the short ID at all:

python -m resolver run T-80009 --variant agent_structured --model chat-strong --repeat 3 --trace

The trace part of the output:

  1  plan      ["Find the customer's order because the supplied ID is incomplete.", "Check delivery evidence and apply the a…
  1  model     chat-strong: get_customer_orders({})  (2163 in, 50 out, 1.716 s)
  1  tool      get_customer_orders({}) -> ok
  2  plan      ["Check the order's tracking and delivery dates.", "Determine the appropriate delivery resolution."]
  2  model     chat-strong: get_order({"order_id": "LK-551906"})  (2279 in, 47 out, 1.698 s)
  2  tool      get_order({"order_id": "LK-551906"}) -> ok
  3  plan      []
  3  model     chat-strong: finish({"outcome": "reply_only", "rule": "W1",…)  (2591 in, 183 out, 3.716 s)
  3  proposal  reply_only (W1): no change
  3  stop      finished

This run noticed the short ID at once and saved one call: 7,033 tokens in instead of 9,464. Both runs reached the same, correct reply. A plan is the model’s best guess at the start. A good agent does not need a perfect first guess. It needs steps that a result can confirm or break.

How often plans change

Plans are not always busy. Over all 210 recorded runs of chat-strong (70 tickets, three repeats), most runs wrote one plan and never changed it:

Non-empty plans per run, chat-strong as a structured agent, 210 recorded runs. No plan: the run finished or stopped at its first step.

Text description of the chart

A column chart. No plan: 46 runs. One plan: 124 runs. Two plans: 36 runs. Three plans: 4 runs.

The 46 runs with no plan ended at step 1. In 42, the model handed the ticket to a person at once. Rules H1 to H5 applied, or the request was outside the workflow’s six kinds. One replied, and the provider’s content filter stopped three. Then 124 runs kept their first plan to the end.

In 40 runs, the model revised a non-empty plan into another one. Some runs revised it twice, so there were 44 revisions. 11 of them came right after a failed or refused tool result, like T-80009. The other 33 came after a result that worked but changed the picture.

T-90201 is one of those 33. The customer asks about two orders. After the first get_order, the plan gains a new step: “Propose the eligible late-delivery refund and draft a reply covering both orders.” The first order was 13 business days late. That is more than 5, so rule W2 applies and the fee comes back.

Who checks the plan

The plan is text that a model wrote. The application never treats it as a fact. Three things in the project do the checking instead:

• The tool contracts check every call, whatever the plan says (Tool contracts).
• The evidence in TaskState holds only results that the application really received. The model cannot add evidence by writing “done” in a plan.
• The resolution check refuses an action on an order that no tool returned in this task.

So what is the plan for? First, it is for the people who read the trace. Grace, who approves Larkfield’s changes, wants to know why the agent looked at LK-551906 when the customer wrote LK-55190. The plan answers in one line: “Find the customer’s order using the products mentioned”. Second, it helps the model, because its own earlier answers stay in front of it at the next call. The next lesson looks at that context.

When to plan, when not to, and what it costs

The fixed workflow and the router from Common patterns have no plan. Code chooses their steps, so the steps are always the same and always checkable. Choose them when the path does not depend on what you find.

A plan earns its place when the next read depends on the last result: a wrong ID, two orders, a service that fails. T-90602 shows the last case. The payment service answers HTTP 503 (a simulated failure). chat-strong’s new plan has two steps. The first is “Retry the payment check once”. The second is “Apply the double-charge policy if evidence is available; otherwise hand to a person”. It retries once, fails again, and hands the ticket to a person. That plan names its own way out.

A plan costs output tokens. In T-80009, the plan is 88 of the 169 characters of the model’s first answer. More importantly, more planning is not better work. gemma3:4b changed its plan in 69 of its 70 runs and got 28 of 70 tickets right. chat-strong changed it in 40 of 210 runs and got 70, 69 and 69. Count the evidence that a plan uses, not the number of plans.

Guided practice: read plans and keep them honest

1. Check your starting point. In the project folder, with the virtual environment active, run:

   python -m pytest

   It prints 35 passed.

2. Run the three traces of T-80009 from this lesson, if you have not yet. They are chat-strong repeat 1, chat-strong repeat 3 and gemma3:4b. Compare the plan lines with the tool lines.

3. Make the file tests/test_context.py with this test. It checks a rule of TaskState: the state holds only the current plan, and the trace keeps the old ones.

   tests/test_context.py

   """Working context: the ticket is marked as data, and the state keeps only the current plan."""
   
   from resolver.data import load_task
   from resolver.evaluate import run_and_score
   from resolver.providers import MockProvider
   
   
   def test_the_state_keeps_the_current_plan_and_the_trace_keeps_the_old_ones():
       """chat-strong writes a plan at every step (recorded); the state holds only the last one."""
       task = load_task("T-90201")
       _, state = run_and_score(task, "agent_structured", "chat-strong", MockProvider().complete)
       plans = [e for e in state.events if e.kind == "plan"]
       assert len(plans) >= 2
       assert state.plan == plans[-1].detail["plan"]

4. Run the tests again:

   python -m pytest

   It prints 36 passed.

The next lesson adds a second test to the same file, and explains why the state keeps only one plan.

Your turn: put each plan next to its action

Write a small function plan_and_actions(task_id, model, repeat=1). It prints one line per tool call: the first step of the plan at that moment, the tool, and ok or the refusal code. Run it in Python for T-80009 with chat-strong and with gemma3:4b. Before you run it, predict: at which steps will gemma3:4b’s plan and action disagree?

Show an example answer

Paste this in Python, in the project folder:

from resolver.data import load_task
from resolver.evaluate import run_and_score
from resolver.providers import MockProvider
def plan_and_actions(task_id, model, repeat=1):
    task = load_task(task_id)
    _, state = run_and_score(task, "agent_structured", model, MockProvider().complete, repeat=repeat)
    plan = []
    for event in state.events:
        if event.kind == "plan":
            plan = event.detail["plan"]
        elif event.kind == "tool":
            first = plan[0] if plan else "(no plan)"
            result = "ok" if event.detail["ok"] else event.detail["code"]
            print(f"step {event.step}: {first!r:<62} -> {event.name} {result}")
    print("Stopped:" if state.stop_reason != "finished" else "Finished:", state.stop_reason, state.status)

plan_and_actions("T-80009", "chat-strong")
plan_and_actions("T-80009", "gemma3:4b")

step 1: 'Check the order and delivery details.'                        -> get_order invalid_arguments
step 2: "Find the customer's order using the products mentioned."      -> get_customer_orders ok
step 3: 'Check delivery details for the matching order.'               -> get_order ok
Finished: finished done
step 1: 'check order status'                                           -> get_order invalid_arguments
step 2: 'check order status'                                           -> search_policy ok
step 3: 'w3 lost'                                                      -> get_order invalid_arguments
step 4: 'ask_customer'                                                 -> get_customer_orders ok
step 5: 'w3 lost'                                                      -> get_order ok
Stopped: repeated_call stopped

chat-strong’s plan and actions agree at every step. gemma3:4b disagrees at steps 2, 3 and 4. At step 2 the plan did not change after the refusal, but the model searched the policy. At step 3 the plan names a rule, and the call repeats the refused ID. At step 4 the plan says “ask_customer”, and the call looks up orders. Try T-90602 and T-90201 with chat-strong too: each plan changes after a result, and the new first step matches the next call.

Common mistake

'The plan says the order was checked'

Symptom: Tomás wants to show the plan to the approver as a checklist: “The agent wrote ‘Check delivery details’, then removed that step. So the delivery details were checked.” But in gemma3:4b’s run, step 4 says “ask_customer” while the call is get_customer_orders. The plan says one thing, and the agent does another.

Cause: the plan is model text. Nothing forces the model to do what it wrote, or to remove a step only when it is done. A step can disappear because the model forgot it.

Diagnose: put each plan next to the next tool call and its result, as in the practice above. Then look in state.evidence for the result that should complete each step. If no result is there, the step was not done.

Fix: show the approver the evidence, not the plan: the order that the tools returned, its dates, the policy rule. Keep the plan as a label in the trace that explains why the agent made each call. Module 4 builds the approver’s view from the evidence and the proposal.

Knowledge check

Check your understanding

In T-80009 with chat-strong, which step of the first plan could not be done, what evidence showed it, and how did the plan change?

Show the answer

“Check the order and delivery details” could not be done for LK-55190. The evidence was the refusal of get_order: the ID does not match LK- and 6 digits (invalid_arguments). The next plan began with “Find the customer’s order using the products mentioned”. That step could be checked: get_customer_orders returned one order with potting soil and a trowel, LK-551906.

Check your understanding

Which of these plan steps can be checked, and by which tool result? (a) 'w3 lost' (b) 'Retry the payment check once' (c) 'Be helpful to the customer'

Show the answer

Only (b). It ends when get_payments returns a result: the payments, or a second failure. (a) names a rule, not an action, so no result completes it. (c) names no evidence at all. A checkable step names a read or a decision whose result you can see in the trace.

Check your understanding

gemma3:4b changed its plan in 69 of 70 runs; chat-strong in 40 of 210. Does more replanning mean better work? Use the recorded scores.

Show the answer

No. gemma3:4b got 28 of 70 tickets right, and chat-strong got 70, 69 and 69 in its three repeats. Frequent changes can mean that the plan follows nothing. What matters is whether each change follows new evidence, and whether the next action matches the new plan.

Recap

• A plan is the list of steps that the agent still intends to do, as text in its answer. It is observable; it is not hidden model reasoning.
• A good step can be checked: you can name the tool result that ends it. “w3 lost” cannot be checked; “Find the customer’s order using the products mentioned” can.
• Revise the plan when evidence breaks an assumption. In T-80009, a refused ID changed the plan, and chat-strong found LK-551906. In 40 of 210 chat-strong runs the plan changed, 44 times in all: 11 times after a failed result, 33 after a surprising one.
• The application never trusts the plan. Contracts check the calls, the evidence holds what really came back, and the resolution check refuses unseen orders.
• tests/test_context.py checks that the state keeps only the current plan: 36 passed.

Next: Manage working context