Contain a runaway agent
- Case study 2
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, a step limit and a quota stopped Larkfield's runaway agent at a known cost. But that was one loop that we built on purpose, and limits that we chose for it. In this case study, we replay 456 real recorded runs of another team's agent through our gateways. We measure what each control would have stopped, and which good runs it would have stopped by mistake.
Case study 2 of 3 · Governing AI Agents: Gateways, Registries and ControlsContain a runaway agent
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Choose the limits that contain a runaway agent, from the agent’s own recorded runs. You replay 456 real runs of an open AI support agent through the course’s model gateway and tool gateway. You measure what each control saves, and which good runs it stops by mistake. Then you test your choice on two other models and write a recommendation.
This case study is a project folder, not a notebook. The work runs the course’s own gateways: the replay sends every recorded call through gateway/app.py and toolgate/app.py of your course project. A notebook in Colab would not have them. The folder is small (about 0.5 MB) and adds no package.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The step limit, the quota and what an agent receives when it is stopped, from Quotas, budgets and rate limits.
- The gateway’s path and its usage record, from One door for model calls.
- Registered tools, strict arguments and the usage record of each tool call, from A gateway for tools.
- Writes that need a person, from Autonomy and approvals as policy.
- Why a change of model is a release with checks, from Change a mapping safely.
The page teaches one control that the course did not: a repeat limit at the tool gateway.
What you need on your computer
- The course project at the end of the course, in
~/projects/ai-estate, with its virtual environment. If yours is broken, download the course’s snapshotend-of-m07and follow its README. - The estate’s services do not need to run. The replay starts everything it needs in one process, in a new temporary folder, and deletes it at the end. Your
state/folder does not change.
Tested on macOS (Apple silicon) with Python 3.12.13 and the course project’s packages. The Windows (PowerShell) and Linux commands are written, not run.
The situation
Section titled “The situation”Ingrid (she) is the platform lead of a regional phone company (made up for this case study). Her team put the course’s design in place: one model gateway and one tool gateway, and a registration for every agent. Now Tariq (he), an AI engineer, wants to register the company’s first customer-facing agent. It answers customers in the support chat. It looks up their line and bill, fixes data and roaming problems and sends payment requests. When it cannot help, it hands the customer to a person.
Ingrid asks one question before the review: “Which limits do we write in its registration?” Each wrong answer has a cost:
- A limit that is too loose lets a loop run. The loop spends money, and the customer waits in a chat that goes nowhere.
- A limit that is too tight stops a conversation that was going well. A satisfied customer goes to a person for nothing.
Tariq has no runs of his own yet. So Ingrid suggests a method for any agent: take recorded runs of a similar agent, and replay them through the gateways with each candidate limit. Then count both errors. The company’s agent will differ, so its own runs must check the limits again after launch.
What you will build
Section titled “What you will build”You will build a limits recommendation from recorded agent runs. You check the data, look at how long runs are, and set a baseline. Then you finish two controls and compare four controls under one measure. You choose a setting with a margin and confirm it through the real gateways. Finally, you test the choice on two other models’ runs, as if the team changed the model.
Set up the project
Section titled “Set up the project”-
Download the starter project from the box at the top of this page. Unzip it into
~/projects, so that the folderrunaway-replayis next toai-estate. -
In VS Code, choose File > Open Folder, and open
runaway-replay. Then open a terminal: Terminal > New Terminal. -
Activate the course project’s virtual environment:
Terminal window source ../ai-estate/.venv/bin/activateTerminal window ..\ai-estate\.venv\Scripts\Activate.ps1Terminal window source ../ai-estate/.venv/bin/activateIf your course project is in another folder, set
ESTATE_PROJECTto that folder (PowerShell:$env:ESTATE_PROJECT = "C:\path\to\ai-estate"). -
Run the tests:
python -m pytest -q. The starter has two parts for you to finish. So five tests fail:FAILED tests/test_controls.py::test_a_step_limit_of_4_stops_only_the_loop - a...FAILED tests/test_controls.py::test_a_step_limit_of_2_also_stops_the_successFAILED tests/test_controls.py::test_a_repeat_limit_of_2_refuses_the_third_identical_callFAILED tests/test_gateway.py::test_a_step_limit_through_the_gateway_matches_the_arithmeticFAILED tests/test_gateway.py::test_a_repeat_limit_through_the_tool_gateway_matches_the_arithmetic5 failed, 9 passed in 6.56sIf the gateway tests are skipped instead, the project did not find your course project: check
ESTATE_PROJECT.
The finished project is in the same box, to compare with your work. Every output on this page comes from it.
The data
Section titled “The data”The data is the published results of τ²-bench, an open benchmark of AI agents that talk with customers and use tools. Its telecom domain is a phone company’s support desk. A customer has a phone problem, and the agent has 13 tools and a policy to follow. A second model plays the customer.
Ingrid needed four things: the tokens of each model call, the tool calls with their arguments, how each run ended, and some real loops. Open coding-agent runs (SWE-agent, SWE-smith, OpenHands) have no token counts, and each set is 1 to 3 GB. SWE-bench’s experiment files have token counts, but no licence. τ²-bench’s retail runs did not loop. Its telecom runs of gpt-4.1-mini have 32 real loops.
The design day is 456 runs of gpt-4.1-mini: we choose the limits on it. The runs of gpt-4.1 and o4-mini are new input for the end. One run has these fields:
| Field | What it holds |
|---|---|
id, task_id, trial |
the run; one of 114 tasks; each task was run 4 times |
termination_reason, reward |
how it ended: the benchmark’s stop at 200 messages (a runaway), or a normal end with reward 1 (success) or 0 (failure) |
steps[].prompt_tokens, completion_tokens |
one model call of the agent, with its recorded token counts |
steps[].tools[] |
the tool calls that the model asked for: name, arguments, and error (the tool answered with an error) |
First checks
Section titled “First checks”replay/data.py checks each file’s SHA-256 before it reads it. Run the comparison’s first part:
python -m replay.compareThe first lines:
Design day: 456 runs of gpt-4.1-mini-2025-04-14 (114 tasks x 4 trials), 13,269 model calls, 8,322 tool calls, 134,545,262 tokens, US$13.73 at chat-small's price success 200 runs | calls per run: median 19.0, max 44 | US$3.24 (24% of the day), US$0.0162 a run failure 224 runs | calls per run: median 27.0, max 87 | US$5.97 (43% of the day), US$0.0267 a run runaway 32 runs | calls per run: median 97.0, max 100 | US$4.52 (33% of the day), US$0.1412 a runWhat this tells us. Runaways are 7% of the runs, but a third of the day’s spending. A runaway costs about nine times a successful run. And no successful run made more than 44 model calls, while the runaways made close to 100. The benchmark stopped each runaway at 200 messages, about 100 calls of the agent. So these costs are a floor: without that stop, a loop goes on.
Frame the problem
Section titled “Frame the problem”The unit is one run, and the run’s ID goes in X-Run-Id, as in Quotas, budgets and rate limits. The day is the 456 runs one after another, in a fixed shuffled order. It is a constructed day: the benchmark ran its tasks in batches.
The price is one price for every recorded token: the course’s chat-small, US$0.10 in and US$0.50 out per million tokens. The recorded costs in the files include discounts that the token counts do not show. One price compares the controls on how the agent behaves, not on price lists.
The measure. Ingrid will not accept a runaway control that stops a run that would have succeeded. We call that a wrong block. So the measure is: the US$ that a control saves on the day, with no wrong block. Each control gets its tightest setting with no wrong block on the design day, and we compare what that setting saves. Failed runs that a control stops are neutral: they ended badly anyway, and a person had to help.
The replay rule. A recording shows what happened after each answer that the agent got. After a different answer, the recording cannot say what the agent would do next:
Text description of the diagram
A recorded run goes to the model gateway, call by call, with its run ID. A refusal by the step limit or the quota stops the run, and its task goes to a person. A stand-in provider answers each other call with the recorded token counts. Each tool call then meets the repeat limit and the tool gateway’s checks. A refusal where the recording had an answer stops the run. A refusal where the recording had an error too lets the run go on.
The course’s provider simulator answers only requests that the course recorded. So a small stand-in provider answers each replayed call with its recorded token counts. The gateway prices, counts and records it like any other call.
Explore what the decision needs
Section titled “Explore what the decision needs”How long runs are
Section titled “How long runs are”- Successful runs
- Failed runs
- Runaways
Text description of the chart
Three lines over the number of model calls in a run, in bins of 5. Successful runs, 200 of them: 167 have 6 to 25 calls, and none has more than 44. Failed runs, 224: a little longer, 152 have 21 to 40 calls, and a few up to 87. Runaways, 32: none below 77 calls, and 24 of them between 96 and 100, where the benchmark stopped them.
What this tells us. There is a gap: the longest successful run made 44 calls, and the shortest runaway made 77. A step limit anywhere in that gap stops every runaway and no success. But the gap is a fact of this agent on these tasks. Another agent, or the same agent on harder tasks, may need more calls.
A loop, call by call
Section titled “A loop, call by call”The first runaway that the replay stops is a customer with a mobile-data problem while abroad. The agent asked for the same customer by phone number 32 times, the same line 31 times and the same device 28 times. It wrote to the customer only 6 times in 99 calls. Calls 61 to 72 asked for these tools, in this order:
get_details_by_id, get_customer_by_phone, get_details_by_id, get_details_by_id, get_customer_by_phone,get_details_by_id, get_details_by_id, get_customer_by_phone, get_details_by_id, get_details_by_id,get_customer_by_phone, get_details_by_id- A runaway (99 calls)
- A successful run of the same task (20 calls)
Text description of the chart
Two lines of prompt tokens per call. The runaway starts at 6,303 tokens and climbs to 23,113 at its last call, 99. The successful run of the same task ends after 20 calls, at 9,554 tokens.
What this tells us. A loop gets more expensive with every call, because each call sends the whole conversation again. This runaway cost US$0.148. Its first 50 calls cost only US$0.054, about a third. So a limit that stops a loop halfway saves more than half of its cost.
A baseline: no limit, then another agent’s limits
Section titled “A baseline: no limit, then another agent’s limits”The baseline is the design day through the real gateways, with no limit. The run takes about 20 seconds:
python -m replay.gateway --data designdesign (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | no limitmodel calls answered 13,269 | cost US$13.73 (recorded runs to their end: US$13.73, saved US$0.00) runaway 32 runs, 0 stopped | US$4.52 of US$4.52 success 200 runs, 0 stopped | US$3.24 of US$3.24 failure 224 runs, 0 stopped | US$5.97 of US$5.97successful runs stopped (wrong blocks): 0same result as the arithmetic (replay/controls.py): yes | 25.0 sWhat this tells us. Every one of the 13,269 recorded calls passed the gateway and was priced. The gateway’s total equals the arithmetic of replay/controls.py, run by run. So the replay works, and nothing in the gateway stops this agent yet.
Tariq’s first idea is to copy the limits of Larkfield’s support workflow from the course: 8 calls a run and 2,000,000 tokens a day.
python -m replay.gateway --data design --max-calls-per-run 8 --tokens-per-day 2000000design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | step limit 8, quota 2,000,000 tokens a daymodel calls answered 289 | cost US$0.21 (recorded runs to their end: US$13.73, saved US$13.53) runaway 32 runs, 32 stopped | US$0.01 of US$4.52 success 200 runs, 199 stopped | US$0.08 of US$3.24 failure 224 runs, 224 stopped | US$0.12 of US$5.97successful runs stopped (wrong blocks): 199first run_step_limit: 429 Run fe7e8ca0-e0dd-446f-9e32-374357c201df has made 8 model calls: the limit is 8. Stop the run and hand the task to a person.first quota_exhausted: 429 phone-support has used its 2,000,000 tokens for today.same result as the arithmetic (replay/controls.py): yes | 1.0 sWhat this tells us. It “saves” US$13.53 by stopping almost everything: 199 of 200 successful runs. Larkfield’s workflow makes 1 to 5 calls a run; this agent’s median success makes 19. Its day uses 134.5 million tokens, so the quota of 2 million ran out after the first 36 runs. A limit belongs to one agent. It comes from that agent’s own runs, never from another agent’s registration.
Part 1: a new control, the repeat limit
Section titled “Part 1: a new control, the repeat limit”Think of a child in a car who asks “Are we there yet?” for the tenth time. Each question is short. The repetition is the sign that something is wrong.
The runaway above did the same: it asked the same tool with the same arguments 32 times. A step limit counts every call of the run, useful or not. A repeat limit counts only the same call: the same tool with the same arguments, in one run. At the limit, the tool gateway refuses the next identical call with 429 repeat_limit, and the run stops. The course’s tool gateway already records a digest of each call’s arguments, so the rule needs no new record. It counts the rows of this run with the same tool and digest. The project adds it in front of the tool gateway, in replay/repeat_rule.py.
A tiny example is the loop in tests/test_controls.py. It makes six calls of get_details_by_id(L7). Each call has 1,000 tokens in and 100 out (US$0.00015 at chat-small’s price).
Text description of the diagram
The same call five times. Both counters go 1, 2, 3, 4. A step limit of 4 refuses call 5. A repeat limit of 2 refuses the tool call of call 3. The run stops after 3 model calls, because the third call was already answered and paid.
Count it by hand. With a step limit of 4, the gateway refuses the loop’s 5th call. The gateway answered 4 calls, and the limit saved 2 of the 6: US$0.0003. With a repeat limit of 2, the tool gateway refuses the third identical tool call, after the model gateway answered 3 calls. When is a repeat limit wrong? When a good run really needs the same lookup again. For example, the agent checks a line again after the customer restarts the phone.
Part 2: finish the two controls
Section titled “Part 2: finish the two controls”replay/controls.py replays every run under a Settings object, call by call, as the gateways would. Two checks are missing in the starter.
-
Open
replay/controls.pyand find the firstTODO (Part 2). It is in the loop over the run’s model calls, before the call is counted. Add the step limit:if s.max_calls_per_run and o.calls >= s.max_calls_per_run:o.stopped_at, o.stopped_by = i, "run_step_limit"breakThe gateway checks the count before the call, so the call that reaches the limit is refused.
-
Find the second
TODO (Part 2), in the loop over the tool calls. Add the repeat limit:if s.repeat_limit and seen.get(key, 0) >= s.repeat_limit:o.stopped_at, o.stopped_by = i, "repeat_limit"break -
Run the tests again:
python -m pytest -q. The last line says14 passed.
The two gateway tests now pass too. They replay the first runs of a day through the real gateways and compare each run with your arithmetic. If you wrote > instead of >=, the arithmetic lets one more call through than the gateway, and these tests fail.
Part 3: a fair comparison under one measure
Section titled “Part 3: a fair comparison under one measure”replay/compare.py gives each control its tightest setting with no wrong block on the design day. It tries every value from the tightest up, and keeps the first one that stops no successful run.
python -m replay.compareEach control at its tightest setting that stops no successful run (the measure: US$ saved): control setting saved runaways stopped failures stopped step limit (model gateway) 44 calls US$ 3.37 32 of 32 15 of 224 repeat limit (tool gateway) 13 times US$ 3.16 32 of 32 4 of 224 daily quota (model gateway) 135 M tokens US$ 0.00 0 of 32 0 of 224Strict arguments (always on): 0 tool calls refused (all of them errors in the recording too); 0 runs stoppedWrites need a person: 11 of 32 runaways ever write (first write at call 7, median); 156 of 200 successful runs would wait for a personWhat this tells us.
- The step limit saves the most, US$3.37 of US$13.73 (25%). It stops all 32 runaways and 15 long failed runs, and no success. Its setting is the longest successful run: 44 calls.
- The repeat limit is close: US$3.16. Successful runs repeated one call up to 13 times, so it cannot be tighter. The runaways repeated one call 23 to 63 times.
- The daily quota saves nothing. A quota that stops no successful run must be as large as the whole day. A quota counts the agent’s work, not one run. When it bites, it stops the good runs that come after the loop too.
- Strict arguments stopped nothing here. gpt-4.1-mini sent no wrong arguments. The tool gateway keeps the check on for every agent: it is about safety, not loops.
- Writes that need a person do not contain these loops. Only 11 of the 32 runaways ever wrote, and the loops themselves are reads. But 156 of the 200 successful runs would wait for a person. That is an autonomy decision, as in Autonomy and approvals as policy, not a runaway control.
The quota’s failure has a price. Set it lower, at 120 million tokens, and replay:
design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | quota 120,000,000 tokens a daymodel calls answered 11,845 | cost US$12.25 (recorded runs to their end: US$13.73, saved US$1.48) runaway 32 runs, 4 stopped | US$4.03 of US$4.52 success 200 runs, 23 stopped | US$2.89 of US$3.24 failure 224 runs, 24 stopped | US$5.34 of US$5.97successful runs stopped (wrong blocks): 23first quota_exhausted: 429 phone-support has used its 120,000,000 tokens for today.same result as the arithmetic (replay/controls.py): yes | 21.4 sIt saved US$1.48 and stopped only 4 runaways. But it cut off 23 customers who were being helped: every run after the quota ran out, good or bad. Larkfield’s replayed month showed the same cost of a limit per agent. The loop used the quota, and the normal work was refused.
Text description of the chart
Four bars. Step limit 44: US$3.37 saved. Repeat limit 13: US$3.16. A daily quota of 135 million tokens: US$0.00. The chosen step limit of 50, with a margin: US$2.97. Each setting stops no successful run of the day.
Part 4: the operating point
Section titled “Part 4: the operating point”The tightest step limit, 44, is the length of one run: the longest success of one day. A setting with no room above it fails on the first longer good run. How fast do wrong blocks grow below it?
Text description of the chart
One line of successful runs stopped, for step limits from 5 to 100 in steps of 5. At 5: all 200. At 10: 173. At 20: 82. At 30: 19. At 35: 8. At 40: 3. From 45: none.
Tariq chooses 50 calls a run: about 10% above the longest successful run. Ingrid asks what the margin costs. Open replay/compare.py, find TODO (Part 4), and write your choice:
chosen = {"max_calls_per_run": 50}Then replay the design day through the gateways with it:
python -m replay.gateway --data design --max-calls-per-run 50design (gpt-4.1-mini-2025-04-14): 456 runs through the gateways | step limit 50model calls answered 11,689 | cost US$10.76 (recorded runs to their end: US$13.73, saved US$2.97) runaway 32 runs, 32 stopped | US$1.72 of US$4.52 success 200 runs, 0 stopped | US$3.24 of US$3.24 failure 224 runs, 11 stopped | US$5.80 of US$5.97successful runs stopped (wrong blocks): 0first run_step_limit: 429 Run 767e83fc-50bd-4688-b1b4-ce5f1fc145f6 has made 50 model calls: the limit is 50. Stop the run and hand the task to a person.same result as the arithmetic (replay/controls.py): yes | 20.5 sWhat this tells us. The margin costs US$0.40 of savings: US$2.97 instead of US$3.37. The runaways’ spending falls from US$4.52 to US$1.72, about US$0.05 each before the stop. Every runaway gets the gateway’s message, “Stop the run and hand the task to a person”, after 50 calls instead of 100. And the gateway’s result is the arithmetic’s, run by run.
Part 5: new input, end to end
Section titled “Part 5: new input, end to end”A limit that fits one model may not fit the next. Changing the model behind a capability is a release, as in Change a mapping safely, and the limits are part of what must still pass. Tariq replays the runs of two other models with the recommended settings. These are the step limit of 50, and a daily quota of 270 million tokens as a ceiling (about twice the design day).
-
Replay gpt-4.1’s runs:
Terminal window python -m replay.gateway --data gpt-4.1 --max-calls-per-run 50 --tokens-per-day 270000000gpt-4.1 (gpt-4.1-2025-04-14): 456 runs through the gateways | step limit 50, quota 270,000,000 tokens a daymodel calls answered 7,639 | cost US$6.20 (recorded runs to their end: US$6.43, saved US$0.23)runaway 3 runs, 3 stopped | US$0.15 of US$0.38success 156 runs, 0 stopped | US$1.87 of US$1.87failure 297 runs, 0 stopped | US$4.18 of US$4.18successful runs stopped (wrong blocks): 0first run_step_limit: 429 Run 2927f14c-f552-43b4-b3b1-96221e1d8eb9 has made 50 model calls: the limit is 50. Stop the run and hand the task to a person.same result as the arithmetic (replay/controls.py): yes | 11.9 s -
Replay o4-mini’s runs:
Terminal window python -m replay.gateway --data o4-mini --max-calls-per-run 50 --tokens-per-day 270000000o4-mini (o4-mini-2025-04-16): 456 runs through the gateways | step limit 50, quota 270,000,000 tokens a daymodel calls answered 8,070 | cost US$8.01 (recorded runs to their end: US$8.01, saved US$0.00)runaway 0 runs, 0 stopped | US$0.00 of US$0.00success 192 runs, 0 stopped | US$2.81 of US$2.81failure 264 runs, 0 stopped | US$5.20 of US$5.20successful runs stopped (wrong blocks): 0first invalid_arguments (the recording had an error too): 400 Unknown arguments for phone.get_details_by_id: app_name, permission.same result as the arithmetic (replay/controls.py): yes | 13.4 s -
Add the repeat limit of 13 to o4-mini’s replay:
Terminal window python -m replay.gateway --data o4-mini --max-calls-per-run 50 --repeat-limit 13o4-mini (o4-mini-2025-04-16): 456 runs through the gateways | step limit 50, repeat limit 13model calls answered 8,040 | cost US$7.97 (recorded runs to their end: US$8.01, saved US$0.04)runaway 0 runs, 0 stopped | US$0.00 of US$0.00success 192 runs, 2 stopped | US$2.77 of US$2.81failure 264 runs, 1 stopped | US$5.19 of US$5.20successful runs stopped (wrong blocks): 2first invalid_arguments (the recording had an error too): 400 Unknown arguments for phone.get_details_by_id: app_name, permission.first repeat_limit: 429 phone.get_details_by_id with these arguments was called 13 times in run bd003cad-f451-4c8f-bbb9-1d50f355d66b: the limit is 13. Stop the run and hand the task to a person.same result as the arithmetic (replay/controls.py): yes | 14.1 s
What this tells us.
- The step limit of 50 held on both models: no wrong block. It stopped gpt-4.1’s 3 runaways at half their cost. o4-mini had no runaway, so the limit cost nothing.
- The margin mattered. The longest successful run of both new models made 45 calls. The tightest setting of the design day, 44, would have stopped one good run of each.
- The repeat limit did not travel. o4-mini’s successful runs repeated one call up to 26 times. So a repeat limit of 13 stopped 2 of them. A rule that follows one model’s habits breaks when the model changes.
- Strict arguments refused 25 of o4-mini’s tool calls, for example
get_details_by_idwith arguments that the tool does not have. The benchmark’s tools refused each of them too, so no run changed. The gateway gives the agent the error before the tool server sees the call.
The recommendation
Section titled “The recommendation”Tariq writes it for Ingrid’s review, in her words:
Text description of the diagram
A loop in one run: a step limit of 50 calls at the model gateway, which hands the customer to a person. A loop with a new run for every call: a daily quota of 270 million tokens. The same lookup again and again: a repeat count, as an alert. A wrong or unknown tool call: registered tools and strict arguments, always on. A write to the customer’s account: the approval policy, by risk.
- Register the agent with a step limit of 50 model calls a run. On the replayed day, it stopped all 32 loops after half their cost. It stopped no successful conversation, and it saved US$2.97 of US$13.73 at our price.
- Set a daily quota as a ceiling, not as the runaway control. About twice a normal day: 270 million tokens on the replayed day’s numbers. It is for the loop that the step limit cannot see, an agent that starts a new run after each error. A quota tight enough to stop loops would have cut off 23 good conversations.
- Watch repeated calls, do not stop on them. Alert the owner when a run makes the same call more than 13 times. As a hard stop, the rule failed on the next model.
- Keep strict arguments and the registered tool list on. They cost nothing on this data and stop wrong calls before they reach a system.
- Measure again after launch, and at every change of model. These limits come from a benchmark agent with simulated customers. After the first weeks, replay the agent’s own runs from the usage records. Change the registration if the longest good runs get near 50.
What the limits are not for: they do not make a failed conversation succeed. The 224 failed runs cost US$5.97, more than the runaways, and the step limit stopped only 11 of them. Failures are a quality problem: evaluate the agent, as in the evaluation course, and fix its prompt or its tools.
Insights and limits
Section titled “Insights and limits”- The data decides the limit. Larkfield’s limits stopped 199 of 200 good runs of this agent. A limit is a number in one agent’s registration, measured on that agent.
- A gap makes a step limit cheap. Here the longest success made 44 calls and the shortest runaway 77. Without a gap, every setting trades saved money for wrong blocks, and the comparison must put a price on both.
- A per-agent quota is a ceiling. It cannot tell a loop from a busy day. It protects the budget, not the customers.
- One day, one domain, a benchmark. 114 tasks, each run 4 times, with a model as the customer. Real customers are more varied. The day’s order is constructed: the quota’s numbers depend on it, the step and repeat limits do not.
- The replay stops at the first different answer. After a refusal, a real agent might recover, or might try a third way. The replay cannot know, so it counts the run as stopped.
Responsible use
Section titled “Responsible use”- Name the data honestly. These are benchmark runs of other models, priced at our price. Present the limits as a starting point from public data, not as measurements of your own agent.
- A stop must hand over well. The gateway’s message says “hand the task to a person”. In a customer chat, that means a person receives the conversation and the customer is told, not left waiting. Test that path before launch.
- Keep the customer’s details out of the records. The usage store keeps a digest of each tool call’s arguments, not the values. That is enough for the repeat count. Do not add the values to see the loops better.
- Credit the source. The data is MIT-licensed. Keep its licence text and the attribution in your copy, as the dataset card does.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This project shows what platform teams look for. The limits come from evidence, both errors are counted, new input checks the choice, and real controls run the replay.
What to put in the repository:
runaway-replay/ README.md the question, how to run it, the results table and the recommendation data/dataset.md the source, the licence, the checksums, the changes, the limits data/trajectories/ the fixed sample and the τ²-bench licence text LICENSE MIT, for your code replay/ data, controls, compare, the stand-in provider, the repeat rule, the gateway replay tests/ 14 tests results/ your compare.json and the replay outputs- Make a new Git repository from your project folder, and push it to GitHub.
.gitignorekeeps outresults/: add the files you want to show withgit add -f. - Write the README’s results table from your own
python -m replay.compareoutput, with the baseline and the new-input check. - Say in the README that the replay needs the course project (
ESTATE_PROJECT), and thatpython -m pytestpasses when it is there.
In two sentences, for an interview:
“I replayed 456 open recorded runs of an AI support agent through a model gateway and a tool gateway, to choose its limits. A step limit of 50 calls stopped every loop at half its cost and no good conversation, also on two other models.”
Go further: the project also has the 456 runs of Claude 3.7 Sonnet. Replay them with --data claude-3.7-sonnet, and read the tool_not_allowed refusals: that model called the customer’s phone tools, which the agent does not have. Or reduce another results file of the benchmark with python -I tools/make_sample.py, in a new empty folder. Or put a price on a wrong block (a person’s time per conversation) and choose the step limit that gives the lowest total cost.
Knowledge check
Section titled “Knowledge check”Check your understanding
Larkfield's workflow uses a step limit of 8 calls a run, and it works well there. Why did the same limit stop 199 of 200 successful runs of the phone agent?
Show the answer
A limit belongs to one agent and its work. Larkfield’s workflow makes 1 to 5 calls a run, so 8 leaves room. The phone agent’s median successful run makes 19 calls, and the longest 44. A step limit must come from the agent’s own runs: here, a little above 44. In the replay, the quota of 2,000,000 tokens also ran out after 36 runs. The agent uses about 134.5 million tokens a day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The tightest safe step limit on the design day was 44 calls, and it saved more than 50. Why did Tariq recommend 50, and what did the new input show?
Show the answer
44 is the length of one run, the longest success of one day. A setting with no room above it fails on the first longer good run. The margin cost US$0.40 of savings on the design day. On the new input, the longest successful runs of gpt-4.1 and o4-mini both made 45 calls. A limit of 44 would have stopped one good run of each, and 50 stopped none.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Ingrid asks: 'Why not use only the daily quota? It is one number, and it protects the budget.' What does the replay answer?
Show the answer
A quota counts the whole agent’s day, not one run. A quota that stops no good run must be as large as the day, about 135 million tokens, and it then saved nothing. At 120 million, it saved US$1.48 and stopped only 4 runaways. But it cut off 23 successful conversations: every run after the quota ran out. Keep the quota as a ceiling at about twice a normal day, and contain loops with the step limit.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Recorded runs of a similar agent can set its limits before launch. Replay them through the gateways, and count what each control saves and which good runs it stops.
- On 456 runs of an open support agent, runaways were 7% of the runs and a third of the cost. A step limit of 50 stopped all 32 at half their cost and no successful run.
- Another agent’s limits stopped 199 of 200 good runs. A per-agent quota saved nothing without cutting off customers. A repeat limit worked on one model and failed on the next.
- Choose with a margin, and check again on new input: the next model’s longest good run was one call above the tightest setting.
You can now choose an agent’s limits from evidence and defend them in a review. But a registration also lists the tools and the servers an agent may use, and many teams want public MCP servers. Next, you review public MCP servers for registration, from their published information only.