Quotas, budgets and rate limits
- Lesson 11
- 40 min
- Module 4: The model gateway
Where we are
In the last lesson, we sent every model call through one door that checks the caller and records the cost. But a record only shows the cost after it is spent: an agent that calls a model in a loop goes on until someone reads the records. In this lesson, we add the limits that stop it, and we replay a runaway agent to see where each limit stops it and what it cost first. We start with a family's bank card.
Lesson 11 of 21 · Governing AI Agents: Gateways, Registries and ControlsQuotas, budgets and rate limits
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Stop an agent that calls a model again and again, before it costs much. You set limits per agent, per run and per team. A warning comes before the hard stop, and the answer tells the agent what to do.
Before you startWhat to know and to have ready
What you need to know
- The gateway’s path and its usage record, from One door for model calls.
- What the agents course’s support workflow does in one run, and how its own loop guard works, from the agents course.
What you need on your computer
- The project as the last lesson left it (137 tests pass), with the estate running:
python -m estate up. - One terminal is enough.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A family’s bank card
Section titled “A family’s bank card”A family shares one bank account, and each person has a card. The bank does not wait for the monthly statement to protect the account. A card has a daily limit at the cash machine, so a stolen card cannot empty the account in one day. The account has a monthly budget, and the bank sends a text message when 80% of it is spent. After a few wrong PIN numbers in a short time, the card stops working for a while.
Each limit stops a different accident, at a different speed. The text message is a warning: the card still works. The limits are hard stops: the card does not work until the next day, the next month, or the next minute.
The problem: a loop that nobody watches
Section titled “The problem: a loop that nobody watches”Agents loop. The agents course recorded one: the small local model gemma3:4b, on ticket T-64872, called the same tool again and again. Its run made 4 model calls, 9,209 tokens in and 334 out, in 13.5 s. Then the workflow’s own guard, repeated_call, stopped it. On a local model, the loop cost nothing. The same tokens on strong-reasoning would cost US$0.022.
That loop was short because the app had a guard. Now take a loop that no guard sees. A bad deployment of the workflow sends the first step of ticket T-90103 again and again, 600 times a minute. The request is real: the agents course recorded it on chat-strong, with 2,182 tokens in and 55 out. The loop is constructed: the course caused it on purpose, through the real gateway code, on a replayed clock. With no limit, one hour gives 36,000 calls and US$176.90. Nobody reads the usage store at 2 o’clock in the morning.
The idea: limits that the gateway checks before every call
Section titled “The idea: limits that the gateway checks before every call”A limit is a number in the agent’s registration. The gateway checks it at check 8 of its path, before the request reaches a provider. This course uses four kinds:
- A rate limit caps requests per minute. It protects the providers and the other agents from a burst. Larkfield’s workflow may send 120 a minute.
- A step limit caps model calls per run (the header
X-Run-Id). A run that never ends is the classic loop. The workflow may make 8 calls per run; its real runs make 1 to 5. - A quota caps tokens per day. The workflow may use 2,000,000.
- A budget caps money per month, for the agent (US$25 for the workflow) and for its team (US$40 for the service team). From 80% of the agent’s budget, every answer carries a warning header,
x-gateway-budget-warning. At 100%, the gateway refuses.
The counters live in the shared usage store, per agent and team, per minute, day and month. So every gateway replica reads the same count. The last lesson of this module adds two more replicas, and they still give an agent one budget, not three.
Worked example: which limit stops the loop, and when?
Section titled “Worked example: which limit stops the loop, and when?”One call of the loop costs 2,182 tokens × US$2 per million, plus 55 tokens × US$10 per million. That is US$0.004364 + US$0.00055 = US$0.004914. It uses 2,182 + 55 = 2,237 tokens. The loop sends 600 attempts a minute. Now take one limit at a time.
- Step limit only, the same run ID: the 9th call of the run is refused. 8 calls, 8 × US$0.004914 = US$0.04.
- Rate limit only, a new run ID each time: 120 calls a minute pass. In 60 minutes: 7,200 calls, US$35.38. It slows the loop. It never stops it.
- Quota only: 2,000,000 ÷ 2,237 = 894.06. After 894 calls, the day’s count is 1,999,878 tokens, still under the quota. So the 895th call passes, and then the loop stops: 895 calls, US$4.40, after 895 ÷ 600 = 1.5 minutes.
- Budget only (US$25): 25 ÷ 0.004914 = 5,087.5. So 5,088 calls pass: US$25.00, after 8.5 minutes.
The gateway checks a count before the call and adds the call after it. So the call that crosses the line still passes, and the agent can go a little over, by at most one call.
The course team ran each variant through the real gateway code on a replayed hour. Every number above came out of the run:
Text description of the chart
Six bars, the cost of one hour of the loop. No limit: US$176.90 for 36,000 calls. Step limit only, the same run ID: US$0.04 for 8 calls. Rate limit only: US$35.38 for 7,200 calls. Daily quota only: US$4.40 for 895 calls. Monthly budget only: US$25.00 for 5,088 calls. All limits, the course’s settings: US$4.40 for 895 calls: the rate limit slowed the loop from the 13th second, and the quota stopped it at minute 7.1.
Each limit answers a different question. The step limit is the cheapest stop, but only if the loop keeps its run ID. A bug that starts a new run each time escapes it. The rate limit protects others but never stops the spending. The quota and the budget stop the spending, at different heights. So the registration sets all four, and the first that is reached stops the loop.
What the agent receives, and what it should do
Section titled “What the agent receives, and what it should do”Every limit answers HTTP 429 with a code and a message that says what happened. The code decides what the agent should do next:
Text description of the diagram
When the gateway answers 429, the agent reads the code. For rate_limited, it waits for the time in the retry-after header and sends the request once again. If that is refused too, it stops the run and hands the task to a person. For run_step_limit, it stops the run and hands the task to a person: more calls will not help. For quota_exhausted or budget_exhausted, it stops and does not retry today or this month, and the owner is told. A 200 answer with the header x-gateway-budget-warning means: go on, but alert the owner that 80% of the budget is used.
A retry is right only for a short wait. An agent that retries a quota at once, in a loop, is a second runaway: each refusal still counts as a request in the minute.
See the limits stop a loop
Section titled “See the limits stop a loop”-
Copy the lesson’s files: the runaway tool and the gateway’s tests.
Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m04cp $S/apps/runaway.py apps/cp $S/tests/test_gateway.py tests/Terminal window $S = "..\nextia-learning-labs\C32\snapshots\end-of-m04"Copy-Item $S\apps\runaway.py apps\Copy-Item $S\tests\test_gateway.py tests\Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m04cp $S/apps/runaway.py apps/cp $S/tests/test_gateway.py tests/ -
Run the tests:
python -m pytest -q. The last line says153 passed, 1 deselected. One of the new tests,test_the_budget_is_a_hard_stop_with_a_warning_before, sets the judge’s budget to US$0.01 and checks the warning header before the stop. -
Read the workflow’s limits in its registration:
Terminal window grep -n -e budget_usd_month -e limits registrations/support-workflow.yaml24:budget_usd_month: 2525:limits: {requests_per_minute: 120, tokens_per_day: 2000000, max_calls_per_run: 8}The team budgets are in
config/teams.yaml: the service team has US$40 a month. -
Start a loop that never ends its run. It sends 300 attempts with one run ID:
Terminal window python -m apps.runaway support-workflow --attempts 300 --same-runThe tool first waits for the next minute to start, because the rate limit counts per calendar minute. Then it takes less than a second. The output, a constructed loop of a replayed request:
support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, one run ID for every callWaiting 23 s for the next minute to start (--now starts at once)answered 8 | cost US$0.0393 | 0.2 srefused run_step_limit 112, rate_limited 180first run_step_limit: after 8 answers and US$0.0393 (0.1 s): 429 Run runaway-1792718460 has made 8 model calls: the limit is 8. Stop the run and hand the task to a person.first rate_limited: after 8 answers and US$0.0393 (0.1 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)Your wait and your run ID differ. The step limit stopped the run after 8 answers. The loop went on asking: 112 more refusals, and then the minute was full. Refusals count as requests, so the last 180 attempts got
rate_limitedinstead. -
Now the harder bug: a new run ID for every call.
Terminal window python -m apps.runaway support-workflow --attempts 300support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, a new run ID for every callWaiting 60 s for the next minute to start (--now starts at once)answered 120 | cost US$0.5897 | 0.4 srefused rate_limited 180first rate_limited: after 120 answers and US$0.5897 (0.3 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)The step limit cannot see this loop. The rate limit lets 120 calls through, 120 × US$0.004914 = US$0.5897, and would let 120 more through every minute. In the replayed hour, the quota stopped it at minute 7.1.
What the workflow does when it is stopped
Section titled “What the workflow does when it is stopped”Run the real workflow on a new ticket at once, in the same minute as the last loop:
RESOLVER_PROVIDER=gateway python -m resolver run T-90104 --variant agent_structured --model strong-reasoning$env:RESOLVER_PROVIDER = "gateway"python -m resolver run T-90104 --variant agent_structured --model strong-reasoningRESOLVER_PROVIDER=gateway python -m resolver run T-90104 --variant agent_structured --model strong-reasoningRun T-90104-agent_structured-strong-reasoning-1 | single agent with one JSON step per call (for models without tool calling) | strong-reasoningStopped: provider_error. No proposal: the ticket goes to a person.Last error: HTTP 429: rate limited.The agents course built the workflow to stop on any provider error and hand the ticket to a person. That is safe, but blunt: for a rate limit, one wait of retry-after seconds would have been enough. If the minute has already changed when you run it, the run works: then wait for the next runaway and try again. python -m estate usage --last 1 shows the refusal with check_name rate_limit and the run’s own run_id, T-90104-agent_structured-strong-reasoning-1.
This is the cost of a limit per agent. The loop and the real work share one agent, so the loop’s spending also blocks the real tickets. In the replayed month, the course replayed the same loop on 22 September, with the course’s limits, on top of a normal day. It made 837 calls and cost US$4.09 before the quota stopped it. The day’s real work had used part of the quota already. Then 19 normal workflow calls were refused for the rest of the day, and those tasks went to people.
Your turn: budgets for two teams
Section titled “Your turn: budgets for two teams”A team budget covers every agent of the team. Set small budgets for two teams, predict where each loop stops, and see who else is stopped.
-
Read what each team has spent this month:
Terminal window python -m estate usage --by teamIf you ran every command of this module once, the service team has spent about US$0.68 and the evaluation team US$0.0122. Your numbers differ if you ran more.
-
Open
config/teams.yamland set two budgets:serviceto1.00andevaluationto0.05. Save the file. -
Publish the configuration:
python -m estate config publish --note "practice: small team budgets". -
Predict: how many calls of the workflow’s loop pass before the service team’s budget stops it? One call costs US$0.004914. Write down your number.
-
Run the loop of the workflow, then a loop of the release judge, and one triage ticket:
Terminal window python -m apps.runaway support-workflow --attempts 300python -m apps.runaway release-judge --attempts 30 --nowpython -m apps.replay ticket-triage --units 1 --via gateway -
Put the budgets back to 40 and 10, and publish again with
--note "the teams' budgets again".
Check your prediction
The course’s run:
support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, a new run ID for every callWaiting 59 s for the next minute to start (--now starts at once)answered 66 | cost US$0.3243 | 0.4 srefused budget_exhausted 54, rate_limited 180first budget_exhausted: after 66 answers and US$0.3243 (0.2 s): 429 The service team has spent its US$1.00 for this month.first rate_limited: after 66 answers and US$0.3243 (0.2 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)release-judge: 30 attempts of T-64707 step 1 (strong-reasoning) through the gateway, a new run ID for every callanswered 10 | cost US$0.0409 | 0.1 srefused budget_exhausted 20first budget_exhausted: after 10 answers and US$0.0409 (0.1 s): 429 The evaluation team has spent its US$0.05 for this month.ticket-triage: 2 requests through the gateway | status {429: 2}refused or failed: budget_exhausted x2tokens: 0 in, 0 out | cost: US$0.0000The service team had US$1.00 − US$0.6802 = US$0.3198 left. 0.3198 ÷ 0.004914 = 65.1, so the 66th call passed and crossed the line: 66 calls, US$0.3243. The evaluation team had US$0.0378 left. One judge call costs about US$0.0041, so 10 calls passed. Look at the last line: the triage service did nothing wrong, and it is stopped too, because it is in the service team. A team budget protects the team’s money, and it also lets one agent’s loop stop its neighbours. That is why the workflow has its own budget too, and why a step limit or a quota should stop a loop long before the team budget does. After you publish the budgets again, the triage ticket passes.
Common mistake
A step limit that never stops a loop
Symptom: the workflow’s registration has max_calls_per_run: 8, but python -m apps.runaway support-workflow --attempts 300 answers 120 calls, not 8:
answered 120 | cost US$0.5897 | 0.4 srefused rate_limited 180Cause: the step limit counts calls per run ID, and this loop sends a new run ID with every call. A real bug looks the same: an agent that starts a new run after each error. The step limit cannot see it.
Diagnose: python -m estate usage --by outcome shows more refused:rate_limit and no new refused:step_limit. The run IDs in python -m estate usage --last 3 are all different.
Recover: keep the step limit for runs that never end. Rely on the quota and the budgets for loops that restart. Set the quota near the agent’s real daily use, with room to spare. The workflow’s 45 runs a day (an assumption) use far less than 2,000,000 tokens. To change a limit, edit the registration, register it again and review it. A new version of a high-risk agent waits for a review, as in The agent lifecycle.
Knowledge check
Section titled “Knowledge check”Check your understanding
A team asks for a rate limit only, because a budget 'would stop our agent in production'. What does the replayed runaway say about that choice?
Show the answer
A rate limit slows a loop, but it never stops it. With 120 requests a minute, the replayed hour still made 7,200 calls and cost US$35.38. A budget or a daily quota stops the spending: US$25.00 and US$4.40 in the same hour. The team’s worry is fair. So set the quota and the budget above the agent’s real use, and let the 80% warning give the owner time. Keep the rate limit as well.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The workflow receives 429 run_step_limit, and later 429 rate_limited. What should it do in each case, and why are they different?
Show the answer
run_step_limit: stop this run and hand the task to a person. The run has made more calls than any real run needs, so more calls will not help. rate_limited: wait for the retry-after time and try once again, because the limit resets each minute. If the second try is refused too, stop and hand the task to a person. An agent that retries at once, in a loop, makes the problem worse, because each refusal counts as a request.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: set the evaluation team's budget to US$0.05 above what it has spent, publish, and run python -m apps.runaway release-judge --attempts 30 --now. How many calls pass, what did they cost, and which message stops the rest?
Show the answer
One judge call costs about US$0.0041, so about 13 calls pass: US$0.05 ÷ 0.0041 = 12.2, and the call that crosses the line passes too. They cost a little over US$0.05. The rest get 429 budget_exhausted with “The evaluation team has spent its US$… for this month.” Read the numbers in your own output: the first refusal line gives the number of answers and the cost before it. Then put the budget back and publish again.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The gateway checks four limits before every call: requests per minute, calls per run, tokens per day, and US$ per month. Every replica reads the same shared counters.
- Each limit stops a different loop. The replayed hour cost US$176.90 with no limit. It cost US$0.04 with the step limit, US$35.38 with the rate limit, US$4.40 with the quota and US$25.00 with the budget.
- A warning header comes from 80% of the agent’s budget; the stop at 100% is a 429 with a code. Retry only
rate_limited, once, afterretry-after. - Limits per agent and per team have a cost. A loop uses the same quota and budget as the agent’s real work and its team’s other agents.
Your project now: apps/runaway.py, the gateway’s tests, and 153 tests that pass. The gateway now stops an agent that spends too much. But it still forwards whatever the request contains, even a card number or a key that must not reach a provider. Next, you add guardrails at the gateway, and measure what they cost.