Skip to content
Français

Contain a runaway agent

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, a step limit and a quota stopped Larkfield's runaway agent at a known cost. But that was one loop that we built on purpose, and limits that we chose for it. In this case study, we replay 456 real recorded runs of another team's agent through our gateways. We measure what each control would have stopped, and which good runs it would have stopped by mistake.

What you will be able to do

Choose the limits that contain a runaway agent, from the agent’s own recorded runs. You replay 456 real runs of an open AI support agent through the course’s model gateway and tool gateway. You measure what each control saves, and which good runs it stops by mistake. Then you test your choice on two other models and write a recommendation.

Case studyPortfolio piece

Licence
MIT
Time
About 2.5 hours
Runs in
VS Code, Python 3.12+
Skills used
  • Replay of recorded agent runs
  • Step limits
  • Daily quotas
  • A repeat limit at the tool gateway
  • Strict tool arguments
  • Wrong blocks
  • One measure for a fair comparison
  • A margin for new input
  • Limits recommendation

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

This case study is a project folder, not a notebook. The work runs the course’s own gateways: the replay sends every recorded call through gateway/app.py and toolgate/app.py of your course project. A notebook in Colab would not have them. The folder is small (about 0.5 MB) and adds no package.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

The page teaches one control that the course did not: a repeat limit at the tool gateway.

What you need on your computer

  • The course project at the end of the course, in ~/projects/ai-estate, with its virtual environment. If yours is broken, download the course’s snapshot end-of-m07 and follow its README.
  • The estate’s services do not need to run. The replay starts everything it needs in one process, in a new temporary folder, and deletes it at the end. Your state/ folder does not change.

Tested on macOS (Apple silicon) with Python 3.12.13 and the course project’s packages. The Windows (PowerShell) and Linux commands are written, not run.

Ingrid (she) is the platform lead of a regional phone company (made up for this case study). Her team put the course’s design in place: one model gateway and one tool gateway, and a registration for every agent. Now Tariq (he), an AI engineer, wants to register the company’s first customer-facing agent. It answers customers in the support chat. It looks up their line and bill, fixes data and roaming problems and sends payment requests. When it cannot help, it hands the customer to a person.

Ingrid asks one question before the review: “Which limits do we write in its registration?” Each wrong answer has a cost:

  • A limit that is too loose lets a loop run. The loop spends money, and the customer waits in a chat that goes nowhere.
  • A limit that is too tight stops a conversation that was going well. A satisfied customer goes to a person for nothing.

Tariq has no runs of his own yet. So Ingrid suggests a method for any agent: take recorded runs of a similar agent, and replay them through the gateways with each candidate limit. Then count both errors. The company’s agent will differ, so its own runs must check the limits again after launch.

You will build a limits recommendation from recorded agent runs. You check the data, look at how long runs are, and set a baseline. Then you finish two controls and compare four controls under one measure. You choose a setting with a margin and confirm it through the real gateways. Finally, you test the choice on two other models’ runs, as if the team changed the model.

  1. Download the starter project from the box at the top of this page. Unzip it into ~/projects, so that the folder runaway-replay is next to ai-estate.

  2. In VS Code, choose File > Open Folder, and open runaway-replay. Then open a terminal: Terminal > New Terminal.

  3. Activate the course project’s virtual environment:

    Terminal window
    source ../ai-estate/.venv/bin/activate

    If your course project is in another folder, set ESTATE_PROJECT to that folder (PowerShell: $env:ESTATE_PROJECT = "C:\path\to\ai-estate").

  4. Run the tests: python -m pytest -q. The starter has two parts for you to finish. So five tests fail:

    FAILED tests/test_controls.py::test_a_step_limit_of_4_stops_only_the_loop - a...
    FAILED tests/test_controls.py::test_a_step_limit_of_2_also_stops_the_success
    FAILED tests/test_controls.py::test_a_repeat_limit_of_2_refuses_the_third_identical_call
    FAILED tests/test_gateway.py::test_a_step_limit_through_the_gateway_matches_the_arithmetic
    FAILED tests/test_gateway.py::test_a_repeat_limit_through_the_tool_gateway_matches_the_arithmetic
    5 failed, 9 passed in 6.56s

    If the gateway tests are skipped instead, the project did not find your course project: check ESTATE_PROJECT.

The finished project is in the same box, to compare with your work. Every output on this page comes from it.

The data is the published results of τ²-bench, an open benchmark of AI agents that talk with customers and use tools. Its telecom domain is a phone company’s support desk. A customer has a phone problem, and the agent has 13 tools and a policy to follow. A second model plays the customer.

Ingrid needed four things: the tokens of each model call, the tool calls with their arguments, how each run ended, and some real loops. Open coding-agent runs (SWE-agent, SWE-smith, OpenHands) have no token counts, and each set is 1 to 3 GB. SWE-bench’s experiment files have token counts, but no licence. τ²-bench’s retail runs did not loop. Its telecom runs of gpt-4.1-mini have 32 real loops.

The design day is 456 runs of gpt-4.1-mini: we choose the limits on it. The runs of gpt-4.1 and o4-mini are new input for the end. One run has these fields:

Field What it holds
id, task_id, trial the run; one of 114 tasks; each task was run 4 times
termination_reason, reward how it ended: the benchmark’s stop at 200 messages (a runaway), or a normal end with reward 1 (success) or 0 (failure)
steps[].prompt_tokens, completion_tokens one model call of the agent, with its recorded token counts
steps[].tools[] the tool calls that the model asked for: name, arguments, and error (the tool answered with an error)

replay/data.py checks each file’s SHA-256 before it reads it. Run the comparison’s first part:

Terminal window
python -m replay.compare

The first lines:

Design day: 456 runs of gpt-4.1-mini-2025-04-14 (114 tasks x 4 trials), 13,269 model calls, 8,322 tool calls, 134,545,262 tokens, US$13.73 at chat-small's price
success 200 runs | calls per run: median 19.0, max 44 | US$3.24 (24% of the day), US$0.0162 a run
failure 224 runs | calls per run: median 27.0, max 87 | US$5.97 (43% of the day), US$0.0267 a run
runaway 32 runs | calls per run: median 97.0, max 100 | US$4.52 (33% of the day), US$0.1412 a run

What this tells us. Runaways are 7% of the runs, but a third of the day’s spending. A runaway costs about nine times a successful run. And no successful run made more than 44 model calls, while the runaways made close to 100. The benchmark stopped each runaway at 200 messages, about 100 calls of the agent. So these costs are a floor: without that stop, a loop goes on.

The unit is one run, and the run’s ID goes in X-Run-Id, as in Quotas, budgets and rate limits. The day is the 456 runs one after another, in a fixed shuffled order. It is a constructed day: the benchmark ran its tasks in batches.

The price is one price for every recorded token: the course’s chat-small, US$0.10 in and US$0.50 out per million tokens. The recorded costs in the files include discounts that the token counts do not show. One price compares the controls on how the agent behaves, not on price lists.

The measure. Ingrid will not accept a runaway control that stops a run that would have succeeded. We call that a wrong block. So the measure is: the US$ that a control saves on the day, with no wrong block. Each control gets its tightest setting with no wrong block on the design day, and we compare what that setting saves. Failed runs that a control stops are neutral: they ended badly anyway, and a person had to help.

The replay rule. A recording shows what happened after each answer that the agent got. After a different answer, the recording cannot say what the agent would do next:

A flow. One recorded run: model call i, its tokens, the tool calls it asked for. It goes to the model gateway with X-Run-Id: the step limit counts the calls of this run, the daily quota counts the agent's tokens today. Refused with 429: the recording ends here, the run is stopped, its task goes to a person. Answered: the stand-in provider answers with the recorded token counts, priced as chat-small, and the gateway writes one usage row. Then each tool call goes to the repeat limit: the same tool and arguments, how often in this run? Refused with 429: the run is stopped. Otherwise the tool gateway checks the registered tool, strict arguments and policy, and writes one usage row. Refused, and the recording had an answer: the run is stopped. Refused, and the recording had an error too: the run goes on as recorded. Answered: the run goes on with the next model call.
The replay rule. A run is followed only while the platform gives the agent the same answer as the recording. A different answer ends the recording, and the run counts as stopped.
Text description of the diagram

A recorded run goes to the model gateway, call by call, with its run ID. A refusal by the step limit or the quota stops the run, and its task goes to a person. A stand-in provider answers each other call with the recorded token counts. Each tool call then meets the repeat limit and the tool gateway’s checks. A refusal where the recording had an answer stops the run. A refusal where the recording had an error too lets the run go on.

The course’s provider simulator answers only requests that the course recorded. So a small stand-in provider answers each replayed call with its recorded token counts. The gateway prices, counts and records it like any other call.

  • Successful runs
  • Failed runs
  • Runaways
The design day: 456 runs of gpt-4.1-mini in τ²-bench's telecom domain, counted in bins of 5 calls.
Text description of the chart

Three lines over the number of model calls in a run, in bins of 5. Successful runs, 200 of them: 167 have 6 to 25 calls, and none has more than 44. Failed runs, 224: a little longer, 152 have 21 to 40 calls, and a few up to 87. Runaways, 32: none below 77 calls, and 24 of them between 96 and 100, where the benchmark stopped them.

What this tells us. There is a gap: the longest successful run made 44 calls, and the shortest runaway made 77. A step limit anywhere in that gap stops every runaway and no success. But the gap is a fact of this agent on these tasks. Another agent, or the same agent on harder tasks, may need more calls.

The first runaway that the replay stops is a customer with a mobile-data problem while abroad. The agent asked for the same customer by phone number 32 times, the same line 31 times and the same device 28 times. It wrote to the customer only 6 times in 99 calls. Calls 61 to 72 asked for these tools, in this order:

get_details_by_id, get_customer_by_phone, get_details_by_id, get_details_by_id, get_customer_by_phone,
get_details_by_id, get_details_by_id, get_customer_by_phone, get_details_by_id, get_details_by_id,
get_customer_by_phone, get_details_by_id
  • A runaway (99 calls)
  • A successful run of the same task (20 calls)
Recorded token counts (τ²-bench, gpt-4.1-mini). Each call sends the whole conversation so far, so a long run's calls get bigger.
Text description of the chart

Two lines of prompt tokens per call. The runaway starts at 6,303 tokens and climbs to 23,113 at its last call, 99. The successful run of the same task ends after 20 calls, at 9,554 tokens.

What this tells us. A loop gets more expensive with every call, because each call sends the whole conversation again. This runaway cost US$0.148. Its first 50 calls cost only US$0.054, about a third. So a limit that stops a loop halfway saves more than half of its cost.

A baseline: no limit, then another agent’s limits

Section titled “A baseline: no limit, then another agent’s limits”

The baseline is the design day through the real gateways, with no limit. The run takes about 20 seconds:

Terminal window
python -m replay.gateway --data design
design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | no limit
model calls answered 13,269 | cost US$13.73 (recorded runs to their end: US$13.73, saved US$0.00)
runaway 32 runs, 0 stopped | US$4.52 of US$4.52
success 200 runs, 0 stopped | US$3.24 of US$3.24
failure 224 runs, 0 stopped | US$5.97 of US$5.97
successful runs stopped (wrong blocks): 0
same result as the arithmetic (replay/controls.py): yes | 25.0 s

What this tells us. Every one of the 13,269 recorded calls passed the gateway and was priced. The gateway’s total equals the arithmetic of replay/controls.py, run by run. So the replay works, and nothing in the gateway stops this agent yet.

Tariq’s first idea is to copy the limits of Larkfield’s support workflow from the course: 8 calls a run and 2,000,000 tokens a day.

Terminal window
python -m replay.gateway --data design --max-calls-per-run 8 --tokens-per-day 2000000
design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | step limit 8, quota 2,000,000 tokens a day
model calls answered 289 | cost US$0.21 (recorded runs to their end: US$13.73, saved US$13.53)
runaway 32 runs, 32 stopped | US$0.01 of US$4.52
success 200 runs, 199 stopped | US$0.08 of US$3.24
failure 224 runs, 224 stopped | US$0.12 of US$5.97
successful runs stopped (wrong blocks): 199
first run_step_limit: 429 Run fe7e8ca0-e0dd-446f-9e32-374357c201df has made 8 model calls: the limit is 8. Stop the run and hand the task to a person.
first quota_exhausted: 429 phone-support has used its 2,000,000 tokens for today.
same result as the arithmetic (replay/controls.py): yes | 1.0 s

What this tells us. It “saves” US$13.53 by stopping almost everything: 199 of 200 successful runs. Larkfield’s workflow makes 1 to 5 calls a run; this agent’s median success makes 19. Its day uses 134.5 million tokens, so the quota of 2 million ran out after the first 36 runs. A limit belongs to one agent. It comes from that agent’s own runs, never from another agent’s registration.

Think of a child in a car who asks “Are we there yet?” for the tenth time. Each question is short. The repetition is the sign that something is wrong.

The runaway above did the same: it asked the same tool with the same arguments 32 times. A step limit counts every call of the run, useful or not. A repeat limit counts only the same call: the same tool with the same arguments, in one run. At the limit, the tool gateway refuses the next identical call with 429 repeat_limit, and the run stops. The course’s tool gateway already records a digest of each call’s arguments, so the rule needs no new record. It counts the rows of this run with the same tool and digest. The project adds it in front of the tool gateway, in replay/repeat_rule.py.

A tiny example is the loop in tests/test_controls.py. It makes six calls of get_details_by_id(L7). Each call has 1,000 tokens in and 100 out (US$0.00015 at chat-small’s price).

A run of the same call, get_details_by_id(L7), five times in a column. Each box shows two counters: calls in the run and same call, both 1, 2, 3, 4. With a step limit of 4, call 5 is refused with 429 run_step_limit. With a repeat limit of 2, the tool call of call 3 is refused with 429 repeat_limit, and the run stops after 3 calls.
Two counters on one loop, two settings. The step limit counts every call of the run. The repeat limit counts only the same call, and stops this loop earlier.
Text description of the diagram

The same call five times. Both counters go 1, 2, 3, 4. A step limit of 4 refuses call 5. A repeat limit of 2 refuses the tool call of call 3. The run stops after 3 model calls, because the third call was already answered and paid.

Count it by hand. With a step limit of 4, the gateway refuses the loop’s 5th call. The gateway answered 4 calls, and the limit saved 2 of the 6: US$0.0003. With a repeat limit of 2, the tool gateway refuses the third identical tool call, after the model gateway answered 3 calls. When is a repeat limit wrong? When a good run really needs the same lookup again. For example, the agent checks a line again after the customer restarts the phone.

replay/controls.py replays every run under a Settings object, call by call, as the gateways would. Two checks are missing in the starter.

  1. Open replay/controls.py and find the first TODO (Part 2). It is in the loop over the run’s model calls, before the call is counted. Add the step limit:

    if s.max_calls_per_run and o.calls >= s.max_calls_per_run:
    o.stopped_at, o.stopped_by = i, "run_step_limit"
    break

    The gateway checks the count before the call, so the call that reaches the limit is refused.

  2. Find the second TODO (Part 2), in the loop over the tool calls. Add the repeat limit:

    if s.repeat_limit and seen.get(key, 0) >= s.repeat_limit:
    o.stopped_at, o.stopped_by = i, "repeat_limit"
    break
  3. Run the tests again: python -m pytest -q. The last line says 14 passed.

The two gateway tests now pass too. They replay the first runs of a day through the real gateways and compare each run with your arithmetic. If you wrote > instead of >=, the arithmetic lets one more call through than the gateway, and these tests fail.

Part 3: a fair comparison under one measure

Section titled “Part 3: a fair comparison under one measure”

replay/compare.py gives each control its tightest setting with no wrong block on the design day. It tries every value from the tightest up, and keeps the first one that stops no successful run.

Terminal window
python -m replay.compare
Each control at its tightest setting that stops no successful run (the measure: US$ saved):
control setting saved runaways stopped failures stopped
step limit (model gateway) 44 calls US$ 3.37 32 of 32 15 of 224
repeat limit (tool gateway) 13 times US$ 3.16 32 of 32 4 of 224
daily quota (model gateway) 135 M tokens US$ 0.00 0 of 32 0 of 224
Strict arguments (always on): 0 tool calls refused (all of them errors in the recording too); 0 runs stopped
Writes need a person: 11 of 32 runaways ever write (first write at call 7, median); 156 of 200 successful runs would wait for a person

What this tells us.

  • The step limit saves the most, US$3.37 of US$13.73 (25%). It stops all 32 runaways and 15 long failed runs, and no success. Its setting is the longest successful run: 44 calls.
  • The repeat limit is close: US$3.16. Successful runs repeated one call up to 13 times, so it cannot be tighter. The runaways repeated one call 23 to 63 times.
  • The daily quota saves nothing. A quota that stops no successful run must be as large as the whole day. A quota counts the agent’s work, not one run. When it bites, it stops the good runs that come after the loop too.
  • Strict arguments stopped nothing here. gpt-4.1-mini sent no wrong arguments. The tool gateway keeps the check on for every agent: it is about safety, not loops.
  • Writes that need a person do not contain these loops. Only 11 of the 32 runaways ever wrote, and the loops themselves are reads. But 156 of the 200 successful runs would wait for a person. That is an autonomy decision, as in Autonomy and approvals as policy, not a runaway control.

The quota’s failure has a price. Set it lower, at 120 million tokens, and replay:

design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | quota 120,000,000 tokens a day
model calls answered 11,845 | cost US$12.25 (recorded runs to their end: US$13.73, saved US$1.48)
runaway 32 runs, 4 stopped | US$4.03 of US$4.52
success 200 runs, 23 stopped | US$2.89 of US$3.24
failure 224 runs, 24 stopped | US$5.34 of US$5.97
successful runs stopped (wrong blocks): 23
first quota_exhausted: 429 phone-support has used its 120,000,000 tokens for today.
same result as the arithmetic (replay/controls.py): yes | 21.4 s

It saved US$1.48 and stopped only 4 runaways. But it cut off 23 customers who were being helped: every run after the quota ran out, good or bad. Larkfield’s replayed month showed the same cost of a limit per agent. The loop used the quota, and the normal work was refused.

456 recorded runs of gpt-4.1-mini, replayed through the course's gateways; every token at chat-small's price. The day cost US$13.73 with no limit.
Text description of the chart

Four bars. Step limit 44: US$3.37 saved. Repeat limit 13: US$3.16. A daily quota of 135 million tokens: US$0.00. The chosen step limit of 50, with a margin: US$2.97. Each setting stops no successful run of the day.

The tightest step limit, 44, is the length of one run: the longest success of one day. A setting with no room above it fails on the first longer good run. How fast do wrong blocks grow below it?

The design day (200 successful runs). Below 45 calls, every step of 5 calls stops more successful runs.
Text description of the chart

One line of successful runs stopped, for step limits from 5 to 100 in steps of 5. At 5: all 200. At 10: 173. At 20: 82. At 30: 19. At 35: 8. At 40: 3. From 45: none.

Tariq chooses 50 calls a run: about 10% above the longest successful run. Ingrid asks what the margin costs. Open replay/compare.py, find TODO (Part 4), and write your choice:

chosen = {"max_calls_per_run": 50}

Then replay the design day through the gateways with it:

Terminal window
python -m replay.gateway --data design --max-calls-per-run 50
design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | step limit 50
model calls answered 11,689 | cost US$10.76 (recorded runs to their end: US$13.73, saved US$2.97)
runaway 32 runs, 32 stopped | US$1.72 of US$4.52
success 200 runs, 0 stopped | US$3.24 of US$3.24
failure 224 runs, 11 stopped | US$5.80 of US$5.97
successful runs stopped (wrong blocks): 0
first run_step_limit: 429 Run 767e83fc-50bd-4688-b1b4-ce5f1fc145f6 has made 50 model calls: the limit is 50. Stop the run and hand the task to a person.
same result as the arithmetic (replay/controls.py): yes | 20.5 s

What this tells us. The margin costs US$0.40 of savings: US$2.97 instead of US$3.37. The runaways’ spending falls from US$4.52 to US$1.72, about US$0.05 each before the stop. Every runaway gets the gateway’s message, “Stop the run and hand the task to a person”, after 50 calls instead of 100. And the gateway’s result is the arithmetic’s, run by run.

A limit that fits one model may not fit the next. Changing the model behind a capability is a release, as in Change a mapping safely, and the limits are part of what must still pass. Tariq replays the runs of two other models with the recommended settings. These are the step limit of 50, and a daily quota of 270 million tokens as a ceiling (about twice the design day).

  1. Replay gpt-4.1’s runs:

    Terminal window
    python -m replay.gateway --data gpt-4.1 --max-calls-per-run 50 --tokens-per-day 270000000
    gpt-4.1 (gpt-4.1-2025-04-14): 456 runs through the gateways | step limit 50, quota 270,000,000 tokens a day
    model calls answered 7,639 | cost US$6.20 (recorded runs to their end: US$6.43, saved US$0.23)
    runaway 3 runs, 3 stopped | US$0.15 of US$0.38
    success 156 runs, 0 stopped | US$1.87 of US$1.87
    failure 297 runs, 0 stopped | US$4.18 of US$4.18
    successful runs stopped (wrong blocks): 0
    first run_step_limit: 429 Run 2927f14c-f552-43b4-b3b1-96221e1d8eb9 has made 50 model calls: the limit is 50. Stop the run and hand the task to a person.
    same result as the arithmetic (replay/controls.py): yes | 11.9 s
  2. Replay o4-mini’s runs:

    Terminal window
    python -m replay.gateway --data o4-mini --max-calls-per-run 50 --tokens-per-day 270000000
    o4-mini (o4-mini-2025-04-16): 456 runs through the gateways | step limit 50, quota 270,000,000 tokens a day
    model calls answered 8,070 | cost US$8.01 (recorded runs to their end: US$8.01, saved US$0.00)
    runaway 0 runs, 0 stopped | US$0.00 of US$0.00
    success 192 runs, 0 stopped | US$2.81 of US$2.81
    failure 264 runs, 0 stopped | US$5.20 of US$5.20
    successful runs stopped (wrong blocks): 0
    first invalid_arguments (the recording had an error too): 400 Unknown arguments for phone.get_details_by_id: app_name, permission.
    same result as the arithmetic (replay/controls.py): yes | 13.4 s
  3. Add the repeat limit of 13 to o4-mini’s replay:

    Terminal window
    python -m replay.gateway --data o4-mini --max-calls-per-run 50 --repeat-limit 13
    o4-mini (o4-mini-2025-04-16): 456 runs through the gateways | step limit 50, repeat limit 13
    model calls answered 8,040 | cost US$7.97 (recorded runs to their end: US$8.01, saved US$0.04)
    runaway 0 runs, 0 stopped | US$0.00 of US$0.00
    success 192 runs, 2 stopped | US$2.77 of US$2.81
    failure 264 runs, 1 stopped | US$5.19 of US$5.20
    successful runs stopped (wrong blocks): 2
    first invalid_arguments (the recording had an error too): 400 Unknown arguments for phone.get_details_by_id: app_name, permission.
    first repeat_limit: 429 phone.get_details_by_id with these arguments was called 13 times in run bd003cad-f451-4c8f-bbb9-1d50f355d66b: the limit is 13. Stop the run and hand the task to a person.
    same result as the arithmetic (replay/controls.py): yes | 14.1 s

What this tells us.

  • The step limit of 50 held on both models: no wrong block. It stopped gpt-4.1’s 3 runaways at half their cost. o4-mini had no runaway, so the limit cost nothing.
  • The margin mattered. The longest successful run of both new models made 45 calls. The tightest setting of the design day, 44, would have stopped one good run of each.
  • The repeat limit did not travel. o4-mini’s successful runs repeated one call up to 26 times. So a repeat limit of 13 stopped 2 of them. A rule that follows one model’s habits breaks when the model changes.
  • Strict arguments refused 25 of o4-mini’s tool calls, for example get_details_by_id with arguments that the tool does not have. The benchmark’s tools refused each of them too, so no run changed. The gateway gives the agent the error before the tool server sees the call.

Tariq writes it for Ingrid’s review, in her words:

Five rows: a risk, then the control for it. A loop in the same run: a step limit of 50 calls a run at the model gateway, which stops the run and hands it to a person. A loop that starts a new run for every call: a daily quota of 270 million tokens at the model gateway, a ceiling for the whole agent. The same lookup again and again: a repeat count from the usage records, an alert to the owner, not a stop. A wrong or unknown tool call: registered tools and strict arguments at the tool gateway, always on. A write that changes the customer's account: the approval policy at the tool gateway, decided by risk, not for runaways.
Each risk gets its own control. Only the step limit and the quota are hard stops for loops; the repeat count is an alert.
Text description of the diagram

A loop in one run: a step limit of 50 calls at the model gateway, which hands the customer to a person. A loop with a new run for every call: a daily quota of 270 million tokens. The same lookup again and again: a repeat count, as an alert. A wrong or unknown tool call: registered tools and strict arguments, always on. A write to the customer’s account: the approval policy, by risk.

  1. Register the agent with a step limit of 50 model calls a run. On the replayed day, it stopped all 32 loops after half their cost. It stopped no successful conversation, and it saved US$2.97 of US$13.73 at our price.
  2. Set a daily quota as a ceiling, not as the runaway control. About twice a normal day: 270 million tokens on the replayed day’s numbers. It is for the loop that the step limit cannot see, an agent that starts a new run after each error. A quota tight enough to stop loops would have cut off 23 good conversations.
  3. Watch repeated calls, do not stop on them. Alert the owner when a run makes the same call more than 13 times. As a hard stop, the rule failed on the next model.
  4. Keep strict arguments and the registered tool list on. They cost nothing on this data and stop wrong calls before they reach a system.
  5. Measure again after launch, and at every change of model. These limits come from a benchmark agent with simulated customers. After the first weeks, replay the agent’s own runs from the usage records. Change the registration if the longest good runs get near 50.

What the limits are not for: they do not make a failed conversation succeed. The 224 failed runs cost US$5.97, more than the runaways, and the step limit stopped only 11 of them. Failures are a quality problem: evaluate the agent, as in the evaluation course, and fix its prompt or its tools.

  • The data decides the limit. Larkfield’s limits stopped 199 of 200 good runs of this agent. A limit is a number in one agent’s registration, measured on that agent.
  • A gap makes a step limit cheap. Here the longest success made 44 calls and the shortest runaway 77. Without a gap, every setting trades saved money for wrong blocks, and the comparison must put a price on both.
  • A per-agent quota is a ceiling. It cannot tell a loop from a busy day. It protects the budget, not the customers.
  • One day, one domain, a benchmark. 114 tasks, each run 4 times, with a model as the customer. Real customers are more varied. The day’s order is constructed: the quota’s numbers depend on it, the step and repeat limits do not.
  • The replay stops at the first different answer. After a refusal, a real agent might recover, or might try a third way. The replay cannot know, so it counts the run as stopped.
  • Name the data honestly. These are benchmark runs of other models, priced at our price. Present the limits as a starting point from public data, not as measurements of your own agent.
  • A stop must hand over well. The gateway’s message says “hand the task to a person”. In a customer chat, that means a person receives the conversation and the customer is told, not left waiting. Test that path before launch.
  • Keep the customer’s details out of the records. The usage store keeps a digest of each tool call’s arguments, not the values. That is enough for the repeat count. Do not add the values to see the loops better.
  • Credit the source. The data is MIT-licensed. Keep its licence text and the attribution in your copy, as the dataset card does.

This project shows what platform teams look for. The limits come from evidence, both errors are counted, new input checks the choice, and real controls run the replay.

What to put in the repository:

runaway-replay/
README.md the question, how to run it, the results table and the recommendation
data/dataset.md the source, the licence, the checksums, the changes, the limits
data/trajectories/ the fixed sample and the τ²-bench licence text
LICENSE MIT, for your code
replay/ data, controls, compare, the stand-in provider, the repeat rule, the gateway replay
tests/ 14 tests
results/ your compare.json and the replay outputs
  1. Make a new Git repository from your project folder, and push it to GitHub. .gitignore keeps out results/: add the files you want to show with git add -f.
  2. Write the README’s results table from your own python -m replay.compare output, with the baseline and the new-input check.
  3. Say in the README that the replay needs the course project (ESTATE_PROJECT), and that python -m pytest passes when it is there.

In two sentences, for an interview:

“I replayed 456 open recorded runs of an AI support agent through a model gateway and a tool gateway, to choose its limits. A step limit of 50 calls stopped every loop at half its cost and no good conversation, also on two other models.”

Go further: the project also has the 456 runs of Claude 3.7 Sonnet. Replay them with --data claude-3.7-sonnet, and read the tool_not_allowed refusals: that model called the customer’s phone tools, which the agent does not have. Or reduce another results file of the benchmark with python -I tools/make_sample.py, in a new empty folder. Or put a price on a wrong block (a person’s time per conversation) and choose the step limit that gives the lowest total cost.

Check your understanding

Larkfield's workflow uses a step limit of 8 calls a run, and it works well there. Why did the same limit stop 199 of 200 successful runs of the phone agent?

Show the answer

A limit belongs to one agent and its work. Larkfield’s workflow makes 1 to 5 calls a run, so 8 leaves room. The phone agent’s median successful run makes 19 calls, and the longest 44. A step limit must come from the agent’s own runs: here, a little above 44. In the replay, the quota of 2,000,000 tokens also ran out after 36 runs. The agent uses about 134.5 million tokens a day.

Check your understanding

The tightest safe step limit on the design day was 44 calls, and it saved more than 50. Why did Tariq recommend 50, and what did the new input show?

Show the answer

44 is the length of one run, the longest success of one day. A setting with no room above it fails on the first longer good run. The margin cost US$0.40 of savings on the design day. On the new input, the longest successful runs of gpt-4.1 and o4-mini both made 45 calls. A limit of 44 would have stopped one good run of each, and 50 stopped none.

Check your understanding

Ingrid asks: 'Why not use only the daily quota? It is one number, and it protects the budget.' What does the replay answer?

Show the answer

A quota counts the whole agent’s day, not one run. A quota that stops no good run must be as large as the day, about 135 million tokens, and it then saved nothing. At 120 million, it saved US$1.48 and stopped only 4 runaways. But it cut off 23 successful conversations: every run after the quota ran out. Keep the quota as a ceiling at about twice a normal day, and contain loops with the step limit.

Recap

  • Recorded runs of a similar agent can set its limits before launch. Replay them through the gateways, and count what each control saves and which good runs it stops.
  • On 456 runs of an open support agent, runaways were 7% of the runs and a third of the cost. A step limit of 50 stopped all 32 at half their cost and no successful run.
  • Another agent’s limits stopped 199 of 200 good runs. A per-agent quota saved nothing without cutting off customers. A repeat limit worked on one model and failed on the next.
  • Choose with a margin, and check again on new input: the next model’s longest good run was one call above the tightest setting.

You can now choose an agent’s limits from evidence and defend them in a review. But a registration also lists the tools and the servers an agent may use, and many teams want public MCP servers. Next, you review public MCP servers for registration, from their published information only.

Next: Review public MCP servers for registration