Control the execution loop

What you will be able to do

Make sure that every run of the agent ends, for one named reason, whatever the model does. Prove it with tests, including a model that calls tools that do not exist and a model that fails again and again.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The step limit, the time limit and the repeated-call check of the ticket assistant, from Bound the loop. This lesson builds the same ideas into the agent’s loop, and adds two more.
• The task state and its stop_reason, from Explicit task state, and the error results of the tools, from Tool contracts.

What you need on your computer

• The project after Tool contracts: Add the contracts to the project: python -m pytest shows 24 passed.
• The snapshot end-of-m02. You copy resolver/loop.py and tests/test_loop.py from it.

No account and no key: every model decision is a recording. For this lesson, gemma3:4b in Ollama ran five tickets with the stop rules switched off, on a computer with 16 GB of memory. Tested with Python 3.12.13 on macOS. The Windows commands are not tested.

A card that keeps failing

A customer pays at a shop with a card. The reader says “declined”. The cashier tries again, because readers sometimes fail once, and this time it works. Good.

Another customer’s card is declined three times in a row. The cashier stops and asks for another way to pay. She does not swipe all afternoon. Sometimes a card goes through and the customer hands it over again for the same thing. She does not charge it twice: something is wrong, and she stops. And if the queue behind the customer grows too long, the manager steps in.

Three rules: try again after one failure, stop after three failures in a row, never repeat something that already worked. Plus a limit on time. The cashier does not need to understand why the card fails. She needs the rules.

The problem: the model decides whether to stop

In the agent’s loop, the model chooses the next step: a read, or finish. Nothing forces it to finish. The loop at this point has one limit, 8 model calls, and then it stops with max_steps. Is that enough?

To find out, five gemma3:4b runs were recorded for this lesson with every other rule switched off, up to 12 model calls (reference/c10/m02/record_unbounded.py). These are real runs of the project’s own agent, not constructed:

• T-80001 (the neighbour’s order): after one refused lookup, the model called get_customer_orders 11 times in a row. Each time it got the same empty list.
• T-80009 (an order ID with 5 digits): it alternated, six times each, between get_order with LK-55190 (refused by the contract) and search_policy with “invalid order id”.
• T-90608 (the order list is down, simulated): it called get_customer_orders 11 times, and the service timed out 11 times.
• T-64872 (charged twice): it proposed the same invalid resolution six times, then gave four answers that were cut at the token limit. That run waited 111.9 seconds for the model.
• T-80005 (a late hammock, in French): its first three resolutions were refused, and the fourth was accepted, at call 5.

Four of the five never finished. A step limit of 8 would stop them, but only after 8 calls. Each call resends the whole conversation and can take seconds. And the limit says nothing about why the run went wrong, so nobody learns from it.

The idea: a loop that always ends, for a named reason

Termination means that the loop ends on every path, whatever the model answers. The project’s loop ends for exactly one of these reasons, written into state.stop_reason: finished, max_steps, time, cost, repeated_call, too_many_errors, no_recording or provider_error.

The limits come in two kinds. Budget limits are checked before each model call, and they stop any run that is too long:

• The step limit: at most 8 model calls.
• The time budget: at most 120 seconds spent waiting for the model. In a replay, the loop adds the recorded latency, so a replay stops where the live run would.
• The spending cap: at most US$0.05 per task, from the token counts and the dated prices.

Behaviour rules are checked after each model call, and they stop a run that is stuck, early:

• Repeated-call detection: the same read, with the same arguments, after it already succeeded. The answer cannot change, so the model is stuck.
• Errors in a row: three failures in a row stop the loop. A failure is a refused or failed read, a refused resolution, or an answer with no usable step. A success sets the count back to 0.

The diagram puts the checks in order:

The checks of run_loop in resolver/loop.py, in order. Every path ends in a named stop or goes back to the first check, which counts the steps.

Text description of the diagram

Before each model call, the loop checks the budget: 8 steps, 120 seconds of model time, US$0.05. If one is used up, it stops with max_steps, time or cost.

Then it calls the model. A provider error stops it with provider_error; a missing recording stops it with no_recording.

If the answer has no usable step, the error count goes up by 1. If the answer is finish, the loop checks the resolution: valid, and every order seen. If yes, it stops with finished; if no, the error count goes up by 1. If the answer is a read that already succeeded with the same arguments, it stops with repeated_call. Otherwise run_read runs it: a success sets the error count to 0, an error adds 1.

After an error, if there are 3 in a row, the loop stops with too_many_errors. Otherwise it goes back to the budget check.

The values come from the recordings. The most model calls that any real Azure run needed was 4 for chat-small and 5 for chat-strong, so 8 leaves room. The slowest real task took 72.857 seconds of model time (gemma3:4b), so 120 seconds stops only a run that is far outside the normal. The most expensive chat-strong task cost about US$0.033, so US$0.05 per task stops only a runaway. The rule of three errors matches the prompt, which tells the model: “If a tool fails, you may try it once more.”

Why only a successful repeat stops the loop

The repeated-call rule could stop any repeat. It does not, because a repeat after a failure is often right. On T-90601, the order service times out once (simulated). chat-small asked for get_order LK-645001, got service_timeout, asked again with the same arguments, got the order and finished with the right answer. A rule that stopped every repeat would hand that ticket to a person for nothing.

A failed call that keeps failing is the job of the error rule. On T-90608, the order list times out every time. The third timeout in a row stops the loop, at call 3.

A tiny worked example: T-80009, call by call

Here are the first four calls of the unbounded T-80009 run, with the project’s rules applied:

Call	The model asked for	The loop’s answer	Errors in a row	Repeat of a success?
1	get_order LK-55190	invalid_arguments	1	no
2	search_policy “invalid order id”	ok	0	no
3	get_order LK-55190	invalid_arguments	1	no: the first one failed
4	search_policy “invalid order id”	not run	1	yes: stop, repeated_call

The error rule never fires here, because each successful search sets the count back to 0. The step limit would fire only at call 8. The repeated-call rule catches the pattern at call 4, after 7.99 seconds of model time instead of about 15.

Explore where the loop stops

Each run in the explorer is one of the five unbounded gemma3:4b runs. The explorer applies the rules to the recorded calls, in the same order as run_loop. A check script compares it with the project’s real loop for all 1,080 combinations of the settings.

Explore

Where does the loop stop?

Stop reason

repeated_call

Model calls

3

Model time

16.2 s

Call 3 asked for get_customer_orders with the same arguments as an earlier successful call. The loop stops before it runs. No proposal: the ticket goes to a person. With no rule at all, the recorded run went on to 12 calls, the most that we allowed, and never finished.

• Ran: a successful read or a valid resolution
• Ran: an error (refused, failed or unusable)
• Never asked: the loop had stopped

T-80001: the neighbour's order: stopped by repeated_call after 3 model calls

Call	The model asked for	The loop's answer	Time (s)
1	get_order("LK-615204")	other_customer: No order LK-615204 is available for this customer. (1 in a row)	3.3
2	get_customer_orders()	ok	10.0
3	get_customer_orders()	not run: the loop stops (repeated call)	16.2
4	get_customer_orders()	never asked	-
5	get_customer_orders()	never asked	-
6	get_customer_orders()	never asked	-
7	get_customer_orders()	never asked	-
8	get_customer_orders()	never asked	-
9	get_customer_orders()	never asked	-
10	get_customer_orders()	never asked	-
11	get_customer_orders()	never asked	-
12	get_customer_orders()	never asked	-

Five real runs of gemma3:4b (Ollama, local) on Larkfield tickets, recorded with the stop rules switched off, up to 12 model calls. The explorer applies the rules of resolver/loop.py to the recorded calls.

Text description of the explorer

The explorer shows one recorded run at a time. There are five runs. T-80001 is the neighbour’s order. T-80009 has an order ID with 5 digits. In T-90608, the order list is down (simulated). T-64872 is a double charge, and T-80005 is a late hammock, in French.

The main control is the step limit, from 1 to 12 model calls; the project uses 8. The other controls are the repeated-call rule (off, same successful call, any same call) and the errors-in-a-row rule (off, 3). The last one is the time budget (off, 120 s, 30 s).

A table lists every model call. It shows what the model asked for, the loop’s answer, the errors in a row and the model time so far. A heavy line marks where the loop stops.

With the project’s limits: T-80001 stops at call 3 (repeated_call, 16.2 s). T-80009 stops at call 4 (repeated_call, 8.0 s). T-90608 stops at call 3 (too_many_errors, 8.4 s). T-64872 stops at call 5 (too_many_errors, 39.1 s). T-80005 stops at call 4 (too_many_errors, 27.9 s), one call before the model would have finished.

With the repeated-call rule off, T-80001 runs to the step limit: 8 calls and 35.9 s. With the error rule off, T-90608 also runs to 8 calls. With a time budget of 30 s, T-64872 stops before call 5 (time, 30.5 s).

Look at T-80005 with the project’s limits. The error rule stopped it at call 4, and the model would have finished at call 5 with reply_only. That is the expected outcome, which the course author wrote from Grace’s policy. So the rule has a price: it gives up on some runs that would have recovered. Here the trade is clear: a person reads one ticket, and no model runs for minutes on a stuck task.

Make the loop end in the project

1. Copy resolver/loop.py from the snapshot end-of-m02 over your file. The changes are small: read them in the next section.

2. Open resolver/config.py. In Settings, replace the line max_steps: int = 8 … with three lines:

   resolver/config.py (Settings)

       max_steps: int = 8                # the usage cap, per task
       max_seconds: float = 120.0
       max_cost_usd: float = 0.05

3. In from_env, replace the line max_steps=int(env.get("RESOLVER_MAX_STEPS", "8"))) with:

   resolver/config.py (from_env)

                      max_steps=int(env.get("RESOLVER_MAX_STEPS", "8")),
                      max_seconds=float(env.get("RESOLVER_MAX_SECONDS", "120")),
                      max_cost_usd=float(env.get("RESOLVER_MAX_COST_USD", "0.05")))

4. Replace limits at the end of the class:

   resolver/config.py (limits)

       def limits(self) -> Limits:
           return Limits(max_steps=self.max_steps, max_seconds=self.max_seconds, max_cost_usd=self.max_cost_usd)

5. Copy tests/test_loop.py from the snapshot into your tests folder.

6. Run the tests:

   python -m pytest

   The last line is 35 passed.

What changed in the loop

Limits gains three fields: max_seconds, max_cost_usd and max_errors (3). check_budget runs before every model call and now checks all three budgets:

resolver/loop.py

def check_budget(state: TaskState, limits: Limits) -> None:
    if state.step >= limits.max_steps:
        raise Stop("max_steps")
    if state.usage.model_seconds > limits.max_seconds:
        raise Stop("time")
    if state.usage.cost_usd is not None and state.usage.cost_usd > limits.max_cost_usd:
        raise Stop("cost")

run_loop keeps two new local variables: calls_seen, a dictionary from a call’s signature (the tool name and its compact arguments) to whether it succeeded, and errors_in_a_row. Before a read runs, the loop checks the signature:

resolver/loop.py (in run_loop, before a read runs)

                signature = name + compact(json.loads(args) if isinstance(args, str) and args.strip().startswith("{")
                                           else args)
                if calls_seen.get(signature) is True:
                    state.log("stop", "repeated_call", tool=name, arguments=args)
                    raise Stop("repeated_call")

After the read, calls_seen[signature] = result.ok, and errors_in_a_row goes back to 0 on a success or up by 1 on an error. A refused resolution and an answer with no usable step also add 1. At the end of each step, three errors in a row raise Stop("too_many_errors"). Every Stop ends in the same place: the reason goes into state.stop_reason, and the runner sets the status to stopped.

Watch each rule stop a real run

1. Run T-80006 with gemma3:4b:

   python -m resolver run T-80006 --variant agent_structured --model gemma3:4b

2. Run T-64872 with gemma3:4b, with its trace:

   python -m resolver run T-64872 --variant agent_structured --model gemma3:4b --trace

3. Run T-80009 with a step limit of 2. On macOS and Linux:

   RESOLVER_MAX_STEPS=2 python -m resolver run T-80009

   On Windows, in PowerShell:

   $env:RESOLVER_MAX_STEPS = "2"; python -m resolver run T-80009; Remove-Item Env:RESOLVER_MAX_STEPS

4. Run T-80009 with a time budget of 3 seconds, then with a spending cap of US$0.0005, in the same way:

   RESOLVER_MAX_SECONDS=3 python -m resolver run T-80009
   RESOLVER_MAX_COST_USD=0.0005 python -m resolver run T-80009

Step 1 ends with Stopped: too_many_errors. No proposal: the ticket goes to a person. gemma3:4b sent get_order with a null ID, then proposed two invalid resolutions. Before this lesson, the same command ended with no_recording at step 4. The loop asked the model a fourth time, but the recording run had stopped after three calls, so there is no fourth answer.

Step 2 prints the project’s recording of T-64872, a different run from the one in the explorer:

Run T-64872-agent_structured-gemma3:4b: agent_structured, gemma3:4b, ticket T-64872, status stopped
4 model call(s), 9209 tokens in, 334 out, US$0.00000, 13.508 s waiting for the model

  1  plan      ["Tool: search_policy"]
  1  model     gemma3:4b: search_policy({"query": "double charge terracotta pot…)  (1884 in, 50 out, 2.795 s)
  1  tool      search_policy({"query": "double charge terracotta pots"}) -> ok
  2  plan      ["Tool: get_payments"]
  2  model     gemma3:4b: get_payments({"order_id": "LK-739156"})  (2241 in, 44 out, 2.106 s)
  2  tool      get_payments({"order_id": "LK-739156"}) -> ok
  3  plan      ["Tool: finish"]
  3  model     gemma3:4b: finish({"outcome": "reply_only", "rule": "W7",…)  (2409 in, 193 out, 6.589 s)
  3  proposal  REFUSED: The resolution is not valid: actions.0.create_return_label.sku: Input should be a valid string; act…
  4  plan      ["Tool: search_policy"]
  4  model     gemma3:4b: search_policy({"query": "double charge terracotta pot…)  (2675 in, 47 out, 2.018 s)
  4  stop      repeated_call
  4  stop      repeated_call

The search at call 4 is the same as the successful search at call 1, so the loop stops before it runs. The stop line appears twice in this version of the trace: the loop logs the repeated call, then logs the stop itself.

Steps 3 and 4 stop chat-small’s recorded run of T-80009, which normally finishes after 4 calls:

Stopped: max_steps. No proposal: the ticket goes to a person.
Stopped: time. No proposal: the ticket goes to a person.
Stopped: cost. No proposal: the ticket goes to a person.

The time stop uses the recorded latency. After call 3, the model time is 1.06 + 0.877 + 1.373 = 3.31 seconds, which is more than 3. The cost stop also comes after call 3, at about 0.0007 dollars. The loop checks each budget before a call. So a run can go over a budget by one call, never more.

How gemma3:4b’s 70 runs ended

The loop’s rules matter most for a weak model. This is how the project’s recorded run of gemma3:4b on all 70 tasks ended, with the rules of this lesson:

gemma3:4b in Ollama (local), one JSON step per call, 70 tasks, one run. Real recordings in the project, scored with the project's loop.

Text description of the chart

A bar chart of how the 70 runs ended: finished 23, too_many_errors 37, repeated_call 10, max_steps 0.

There were 37 too_many_errors stops. 34 came after three refused resolutions in a row, 2 after answers cut at the token limit, and 1 after a service timeout. No run reached the step limit: the behaviour rules stopped every stuck run first, after at most 8 calls. chat-small and chat-strong finished 69 of 70 tasks; the 70th was T-80002, stopped by the provider’s content filter (provider_error). So the free local model runs the loop correctly, and the loop protects you from its poor decisions. It does not make them good: gemma3:4b proposed the right resolution on 28 of the 70 tasks.

When to use each rule, and what it costs

Every agent loop needs a step limit and a time budget, because they hold for every model and every failure. Add a spending cap whenever a real provider bills you. Add the behaviour rules when you see stuck runs in your traces; with a small model, you will.

Each rule costs something. A low step limit stops slow but correct runs. The error rule gave up on T-80005 one call too early. The repeated-call rule needs a good signature: {"order_id":"LK-640436"} and {"order_id": "LK-640436"} must count as the same call, so the loop compares compact JSON. And every stop sends a ticket to a person, which costs a person’s time. Choose the values from your recordings, as here, and check them again when you change the model.

Common mistake

'Stop every repeated call, to be safe'

Symptom: Amira tries the stricter rule “any same call” in the explorer. On T-80009 it stops at call 3, one call earlier. So she changes the check in loop.py to if signature in calls_seen:. Now python -m resolver run T-90601 prints Stopped: repeated_call. No proposal: the ticket goes to a person., and test_a_failed_call_may_be_tried_again fails. The order service had timed out once (simulated), and the second call, which would have worked, never ran.

Cause: a repeat after a failure is often the right step. Timeouts and short outages pass, and the prompt allows one more try.

Fix: stop a repeat only after a success, because then the answer cannot change. Let the error rule handle calls that keep failing: three in a row, and the loop stops.

What it means for a real AI service: a stop rule protects you only if you understand why a model repeats itself. Test each rule on real traces, both the stuck ones and the ones that recover.

Module practice: a mock tool loop with predictable results

Build your own test file with a scripted model: a fake model that returns the steps you write, so every result is predictable. tests/scripted.py has one (Scripted, call, finish). The tools are the real read tools on a fresh copy of the practice database, with simulated failures where you ask for them.

1. Make tests/test_my_loop.py. Use ticket T-90103 (the cracked planter, order LK-640436).
2. Write a success case: one get_order, then a finish with a reship. The loop stops with finished after 2 calls.
3. Write two failure cases: a model that calls an unknown tool every time, and a get_order service that always times out (Faults). Both stop with too_many_errors after 3 calls.
4. Write a case that shows the reset: two failed reads, one success, one failure, then finish. It must finish after 5 calls.
5. Run python -m pytest tests/test_my_loop.py.

Show an example answer

tests/test_my_loop.py

"""Module practice: a mock tool loop with predictable success and failure cases.

The model is SCRIPTED (tests/scripted.py): no model wrote these steps. The tools are the real
read tools on a fresh copy of the practice database, with simulated failures where a test asks for them.
"""

from resolver.agent import run_agent
from resolver.data import load_task
from resolver.loop import Limits
from resolver.runner import new_state
from resolver.systems import Faults

from .scripted import Scripted, call, finish

PLANTER = [{"tool": "reship_item", "order_id": "LK-640436", "sku": "PLANTER", "quantity": 1}]


def run(world, script, faults=None, limits=Limits()):
    task = load_task("T-90103")                       # the cracked cedar planter
    state = new_state(task, "agent", "chat-small", "practice")
    return run_agent(state, Scripted(script).complete, world("T-90103", faults), "tools", limits)


def test_success_read_then_finish(world):
    s = run(world, [[call("get_order", order_id="LK-640436")], [finish("resolve", "W5", PLANTER)]])
    assert (s.stop_reason, s.step) == ("finished", 2)
    assert [e.code for e in s.evidence] == ["ok"]


def test_an_unknown_tool_again_and_again_ends_the_loop(world):
    s = run(world, [[call("refund_now", order_id="LK-640436", amount=89)]])
    assert (s.stop_reason, s.step) == ("too_many_errors", 3)
    assert [e.code for e in s.evidence] == ["unknown_tool"] * 3


def test_a_service_that_never_answers_ends_the_loop(world):
    down = Faults([{"tool": "get_order", "mode": "timeout", "times": 99}])     # simulated
    s = run(world, [[call("get_order", order_id="LK-640436")]], faults=down)
    assert (s.stop_reason, s.step) == ("too_many_errors", 3)
    assert [e.code for e in s.evidence] == ["service_timeout"] * 3


def test_a_success_resets_the_error_count(world):
    script = [[call("get_order", order_id="LK-000001")], [call("get_order", order_id="LK-000002")],
              [call("get_order", order_id="LK-640436")], [call("get_order", order_id="LK-000003")],
              [finish("resolve", "W5", PLANTER)]]
    s = run(world, script)
    assert (s.stop_reason, s.step) == ("finished", 5)
    assert [e.ok for e in s.evidence] == [False, False, True, False]

python -m pytest tests/test_my_loop.py prints 4 passed, and the whole suite 39 passed. The scripted model repeats its last step when the script ends, so a one-step script is a model that never changes its mind. The unknown tool is never run, and the failed call is allowed to repeat. So neither case stops with repeated_call: the error rule ends both.

Your turn. Change one condition: run the unknown-tool case with Limits(max_errors=2). Predict the stop reason and the number of calls before you run it. (Answer: too_many_errors after 2 calls.)

Module check

Prove that the loop ends on repeated errors and on unknown tools. Run the two tests of the snapshot and your own:

python -m pytest tests/test_loop.py -k "unknown or repeated_errors" -v
python -m pytest tests/test_my_loop.py

The first command prints 2 passed, 9 deselected. You have finished the module when:

• python -m pytest passes, with your new tests.
• You can explain which rule stops each of the five unbounded runs in the explorer, with the project’s limits. You can also explain why the step limit alone is not enough.
• You can explain why a repeat after a failure is allowed, with T-90601 as the example.

Reset. If your project is broken, rename it and copy the snapshot end-of-m02, as the snapshots README shows. Make the virtual environment again. python -m pytest then prints 35 passed. The snapshot has the guided steps of this module, not the module practice.

Knowledge check

Check your understanding

A model asks for get_order with LK-000001, then LK-000002, then LK-000003. Each order does not exist. Which rule stops the loop, after how many model calls, and why not the repeated-call rule?

Show the answer

The error rule: too_many_errors after 3 model calls. Each call is refused (not_found), so the error count goes 1, 2, 3, and the loop stops at the end of the third step. The repeated-call rule never matches, because the three calls have different arguments, and none of them succeeded. The test test_repeated_errors_end_the_loop checks this case.

Check your understanding

Why does the time budget use the recorded latency in a replay, not the real time on your computer?

Show the answer

A replay answers in milliseconds, so the real time would never reach 120 seconds. Without a live model, you could never test the time budget. With the recorded latency, a replay waits on paper exactly as long as the live run waited. So the budget stops a replay at the same call as the live run. test_the_time_budget checks it with a scripted latency.

Check your understanding

Predict: in the explorer, choose T-90608 and switch the error rule off, with the other limits as in the project. Where does the loop stop, and why does the repeated-call rule not help?

Show the answer

At the step limit: max_steps after 8 model calls. The model calls get_customer_orders again and again, and the service times out every time. The project’s repeated-call rule stops only a repeat of a successful call, and none of these calls succeeded. Only the error rule, or the stricter “any same call” rule, would stop it earlier, at call 3.

Recap

• The model decides whether to stop, so the code must make sure that the loop ends. Every run ends for one named reason in stop_reason.
• Budget limits, before each call: 8 model calls, 120 seconds of model time, US$0.05 per task. Behaviour rules, after each call: a repeat of a successful read, and three errors in a row.
• The values come from the recordings. No Azure run needed more than 5 calls, and no real task took more than 72.9 seconds. No chat-strong task cost more than about US$0.033.
• With the rules off, four of five real gemma3:4b runs never finished in 12 calls. With the rules, they stop after 3 to 5 calls. The cost: T-80005 stopped one call before it would have finished.
• tests/test_loop.py proves termination on a model that never finishes, repeats, calls unknown tools, keeps failing or proposes invalid resolutions. Your own scripted tests prove it again.

You have finished Module 2. Next, the agent makes a plan and changes it when the evidence changes.

Next: Plan and revise