Skip to content
Français

Quotas, budgets and rate limits

  • Lesson 11
  • 40 min
  • Module 4: The model gateway

Where we are

In the last lesson, we sent every model call through one door that checks the caller and records the cost. But a record only shows the cost after it is spent: an agent that calls a model in a loop goes on until someone reads the records. In this lesson, we add the limits that stop it, and we replay a runaway agent to see where each limit stops it and what it cost first. We start with a family's bank card.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Stop an agent that calls a model again and again, before it costs much. You set limits per agent, per run and per team. A warning comes before the hard stop, and the answer tells the agent what to do.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • The gateway’s path and its usage record, from One door for model calls.
  • What the agents course’s support workflow does in one run, and how its own loop guard works, from the agents course.

What you need on your computer

  • The project as the last lesson left it (137 tests pass), with the estate running: python -m estate up.
  • One terminal is enough.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

A family shares one bank account, and each person has a card. The bank does not wait for the monthly statement to protect the account. A card has a daily limit at the cash machine, so a stolen card cannot empty the account in one day. The account has a monthly budget, and the bank sends a text message when 80% of it is spent. After a few wrong PIN numbers in a short time, the card stops working for a while.

Each limit stops a different accident, at a different speed. The text message is a warning: the card still works. The limits are hard stops: the card does not work until the next day, the next month, or the next minute.

Agents loop. The agents course recorded one: the small local model gemma3:4b, on ticket T-64872, called the same tool again and again. Its run made 4 model calls, 9,209 tokens in and 334 out, in 13.5 s. Then the workflow’s own guard, repeated_call, stopped it. On a local model, the loop cost nothing. The same tokens on strong-reasoning would cost US$0.022.

That loop was short because the app had a guard. Now take a loop that no guard sees. A bad deployment of the workflow sends the first step of ticket T-90103 again and again, 600 times a minute. The request is real: the agents course recorded it on chat-strong, with 2,182 tokens in and 55 out. The loop is constructed: the course caused it on purpose, through the real gateway code, on a replayed clock. With no limit, one hour gives 36,000 calls and US$176.90. Nobody reads the usage store at 2 o’clock in the morning.

The idea: limits that the gateway checks before every call

Section titled “The idea: limits that the gateway checks before every call”

A limit is a number in the agent’s registration. The gateway checks it at check 8 of its path, before the request reaches a provider. This course uses four kinds:

  • A rate limit caps requests per minute. It protects the providers and the other agents from a burst. Larkfield’s workflow may send 120 a minute.
  • A step limit caps model calls per run (the header X-Run-Id). A run that never ends is the classic loop. The workflow may make 8 calls per run; its real runs make 1 to 5.
  • A quota caps tokens per day. The workflow may use 2,000,000.
  • A budget caps money per month, for the agent (US$25 for the workflow) and for its team (US$40 for the service team). From 80% of the agent’s budget, every answer carries a warning header, x-gateway-budget-warning. At 100%, the gateway refuses.

The counters live in the shared usage store, per agent and team, per minute, day and month. So every gateway replica reads the same count. The last lesson of this module adds two more replicas, and they still give an agent one budget, not three.

Worked example: which limit stops the loop, and when?

Section titled “Worked example: which limit stops the loop, and when?”

One call of the loop costs 2,182 tokens × US$2 per million, plus 55 tokens × US$10 per million. That is US$0.004364 + US$0.00055 = US$0.004914. It uses 2,182 + 55 = 2,237 tokens. The loop sends 600 attempts a minute. Now take one limit at a time.

  • Step limit only, the same run ID: the 9th call of the run is refused. 8 calls, 8 × US$0.004914 = US$0.04.
  • Rate limit only, a new run ID each time: 120 calls a minute pass. In 60 minutes: 7,200 calls, US$35.38. It slows the loop. It never stops it.
  • Quota only: 2,000,000 ÷ 2,237 = 894.06. After 894 calls, the day’s count is 1,999,878 tokens, still under the quota. So the 895th call passes, and then the loop stops: 895 calls, US$4.40, after 895 ÷ 600 = 1.5 minutes.
  • Budget only (US$25): 25 ÷ 0.004914 = 5,087.5. So 5,088 calls pass: US$25.00, after 8.5 minutes.

The gateway checks a count before the call and adds the call after it. So the call that crosses the line still passes, and the agent can go a little over, by at most one call.

The course team ran each variant through the real gateway code on a replayed hour. Every number above came out of the run:

Simulated loop (600 attempts a minute for one replayed hour) of a real recorded request, through the real gateway code; the course's prices (chat-strong US$2 and US$10 per million tokens).
Text description of the chart

Six bars, the cost of one hour of the loop. No limit: US$176.90 for 36,000 calls. Step limit only, the same run ID: US$0.04 for 8 calls. Rate limit only: US$35.38 for 7,200 calls. Daily quota only: US$4.40 for 895 calls. Monthly budget only: US$25.00 for 5,088 calls. All limits, the course’s settings: US$4.40 for 895 calls: the rate limit slowed the loop from the 13th second, and the quota stopped it at minute 7.1.

Each limit answers a different question. The step limit is the cheapest stop, but only if the loop keeps its run ID. A bug that starts a new run each time escapes it. The rate limit protects others but never stops the spending. The quota and the budget stop the spending, at different heights. So the registration sets all four, and the first that is reached stops the loop.

What the agent receives, and what it should do

Section titled “What the agent receives, and what it should do”

Every limit answers HTTP 429 with a code and a message that says what happened. The code decides what the agent should do next:

A flow. The gateway answers 429. Which code? rate_limited: wait for the retry-after time, then send it once again; if it is 429 again, stop this run and hand the task to a person; if not, go on with the run. run_step_limit: stop this run and hand the task to a person. quota_exhausted or budget_exhausted: stop, do not retry today or this month, and tell the owner. Separately, a 200 answer with x-gateway-budget-warning: go on, and alert the owner that 80% of the budget is used.
What an agent should do with each answer of the limits. Only a rate limit is worth a second try.
Text description of the diagram

When the gateway answers 429, the agent reads the code. For rate_limited, it waits for the time in the retry-after header and sends the request once again. If that is refused too, it stops the run and hands the task to a person. For run_step_limit, it stops the run and hands the task to a person: more calls will not help. For quota_exhausted or budget_exhausted, it stops and does not retry today or this month, and the owner is told. A 200 answer with the header x-gateway-budget-warning means: go on, but alert the owner that 80% of the budget is used.

A retry is right only for a short wait. An agent that retries a quota at once, in a loop, is a second runaway: each refusal still counts as a request in the minute.

  1. Copy the lesson’s files: the runaway tool and the gateway’s tests.

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m04
    cp $S/apps/runaway.py apps/
    cp $S/tests/test_gateway.py tests/
  2. Run the tests: python -m pytest -q. The last line says 153 passed, 1 deselected. One of the new tests, test_the_budget_is_a_hard_stop_with_a_warning_before, sets the judge’s budget to US$0.01 and checks the warning header before the stop.

  3. Read the workflow’s limits in its registration:

    Terminal window
    grep -n -e budget_usd_month -e limits registrations/support-workflow.yaml
    24:budget_usd_month: 25
    25:limits: {requests_per_minute: 120, tokens_per_day: 2000000, max_calls_per_run: 8}

    The team budgets are in config/teams.yaml: the service team has US$40 a month.

  4. Start a loop that never ends its run. It sends 300 attempts with one run ID:

    Terminal window
    python -m apps.runaway support-workflow --attempts 300 --same-run

    The tool first waits for the next minute to start, because the rate limit counts per calendar minute. Then it takes less than a second. The output, a constructed loop of a replayed request:

    support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, one run ID for every call
    Waiting 23 s for the next minute to start (--now starts at once)
    answered 8 | cost US$0.0393 | 0.2 s
    refused run_step_limit 112, rate_limited 180
    first run_step_limit: after 8 answers and US$0.0393 (0.1 s): 429 Run runaway-1792718460 has made 8 model calls: the limit is 8. Stop the run and hand the task to a person.
    first rate_limited: after 8 answers and US$0.0393 (0.1 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)

    Your wait and your run ID differ. The step limit stopped the run after 8 answers. The loop went on asking: 112 more refusals, and then the minute was full. Refusals count as requests, so the last 180 attempts got rate_limited instead.

  5. Now the harder bug: a new run ID for every call.

    Terminal window
    python -m apps.runaway support-workflow --attempts 300
    support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, a new run ID for every call
    Waiting 60 s for the next minute to start (--now starts at once)
    answered 120 | cost US$0.5897 | 0.4 s
    refused rate_limited 180
    first rate_limited: after 120 answers and US$0.5897 (0.3 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)

    The step limit cannot see this loop. The rate limit lets 120 calls through, 120 × US$0.004914 = US$0.5897, and would let 120 more through every minute. In the replayed hour, the quota stopped it at minute 7.1.

Run the real workflow on a new ticket at once, in the same minute as the last loop:

Terminal window
RESOLVER_PROVIDER=gateway python -m resolver run T-90104 --variant agent_structured --model strong-reasoning
Run T-90104-agent_structured-strong-reasoning-1 | single agent with one JSON step per call (for models without tool calling) | strong-reasoning
Stopped: provider_error. No proposal: the ticket goes to a person.
Last error: HTTP 429: rate limited.

The agents course built the workflow to stop on any provider error and hand the ticket to a person. That is safe, but blunt: for a rate limit, one wait of retry-after seconds would have been enough. If the minute has already changed when you run it, the run works: then wait for the next runaway and try again. python -m estate usage --last 1 shows the refusal with check_name rate_limit and the run’s own run_id, T-90104-agent_structured-strong-reasoning-1.

This is the cost of a limit per agent. The loop and the real work share one agent, so the loop’s spending also blocks the real tickets. In the replayed month, the course replayed the same loop on 22 September, with the course’s limits, on top of a normal day. It made 837 calls and cost US$4.09 before the quota stopped it. The day’s real work had used part of the quota already. Then 19 normal workflow calls were refused for the rest of the day, and those tasks went to people.

A team budget covers every agent of the team. Set small budgets for two teams, predict where each loop stops, and see who else is stopped.

  1. Read what each team has spent this month:

    Terminal window
    python -m estate usage --by team

    If you ran every command of this module once, the service team has spent about US$0.68 and the evaluation team US$0.0122. Your numbers differ if you ran more.

  2. Open config/teams.yaml and set two budgets: service to 1.00 and evaluation to 0.05. Save the file.

  3. Publish the configuration: python -m estate config publish --note "practice: small team budgets".

  4. Predict: how many calls of the workflow’s loop pass before the service team’s budget stops it? One call costs US$0.004914. Write down your number.

  5. Run the loop of the workflow, then a loop of the release judge, and one triage ticket:

    Terminal window
    python -m apps.runaway support-workflow --attempts 300
    python -m apps.runaway release-judge --attempts 30 --now
    python -m apps.replay ticket-triage --units 1 --via gateway
  6. Put the budgets back to 40 and 10, and publish again with --note "the teams' budgets again".

Check your prediction

The course’s run:

support-workflow: 300 attempts of T-90103 step 1 (strong-reasoning) through the gateway, a new run ID for every call
Waiting 59 s for the next minute to start (--now starts at once)
answered 66 | cost US$0.3243 | 0.4 s
refused budget_exhausted 54, rate_limited 180
first budget_exhausted: after 66 answers and US$0.3243 (0.2 s): 429 The service team has spent its US$1.00 for this month.
first rate_limited: after 66 answers and US$0.3243 (0.2 s): 429 support-workflow may send 120 requests a minute. (retry-after: 60 s)
release-judge: 30 attempts of T-64707 step 1 (strong-reasoning) through the gateway, a new run ID for every call
answered 10 | cost US$0.0409 | 0.1 s
refused budget_exhausted 20
first budget_exhausted: after 10 answers and US$0.0409 (0.1 s): 429 The evaluation team has spent its US$0.05 for this month.
ticket-triage: 2 requests through the gateway | status {429: 2}
refused or failed: budget_exhausted x2
tokens: 0 in, 0 out | cost: US$0.0000

The service team had US$1.00 − US$0.6802 = US$0.3198 left. 0.3198 ÷ 0.004914 = 65.1, so the 66th call passed and crossed the line: 66 calls, US$0.3243. The evaluation team had US$0.0378 left. One judge call costs about US$0.0041, so 10 calls passed. Look at the last line: the triage service did nothing wrong, and it is stopped too, because it is in the service team. A team budget protects the team’s money, and it also lets one agent’s loop stop its neighbours. That is why the workflow has its own budget too, and why a step limit or a quota should stop a loop long before the team budget does. After you publish the budgets again, the triage ticket passes.

Common mistake

A step limit that never stops a loop

Symptom: the workflow’s registration has max_calls_per_run: 8, but python -m apps.runaway support-workflow --attempts 300 answers 120 calls, not 8:

answered 120 | cost US$0.5897 | 0.4 s
refused rate_limited 180

Cause: the step limit counts calls per run ID, and this loop sends a new run ID with every call. A real bug looks the same: an agent that starts a new run after each error. The step limit cannot see it.

Diagnose: python -m estate usage --by outcome shows more refused:rate_limit and no new refused:step_limit. The run IDs in python -m estate usage --last 3 are all different.

Recover: keep the step limit for runs that never end. Rely on the quota and the budgets for loops that restart. Set the quota near the agent’s real daily use, with room to spare. The workflow’s 45 runs a day (an assumption) use far less than 2,000,000 tokens. To change a limit, edit the registration, register it again and review it. A new version of a high-risk agent waits for a review, as in The agent lifecycle.

Check your understanding

A team asks for a rate limit only, because a budget 'would stop our agent in production'. What does the replayed runaway say about that choice?

Show the answer

A rate limit slows a loop, but it never stops it. With 120 requests a minute, the replayed hour still made 7,200 calls and cost US$35.38. A budget or a daily quota stops the spending: US$25.00 and US$4.40 in the same hour. The team’s worry is fair. So set the quota and the budget above the agent’s real use, and let the 80% warning give the owner time. Keep the rate limit as well.

Check your understanding

The workflow receives 429 run_step_limit, and later 429 rate_limited. What should it do in each case, and why are they different?

Show the answer

run_step_limit: stop this run and hand the task to a person. The run has made more calls than any real run needs, so more calls will not help. rate_limited: wait for the retry-after time and try once again, because the limit resets each minute. If the second try is refused too, stop and hand the task to a person. An agent that retries at once, in a loop, makes the problem worse, because each refusal counts as a request.

Check your understanding

Practical check: set the evaluation team's budget to US$0.05 above what it has spent, publish, and run python -m apps.runaway release-judge --attempts 30 --now. How many calls pass, what did they cost, and which message stops the rest?

Show the answer

One judge call costs about US$0.0041, so about 13 calls pass: US$0.05 ÷ 0.0041 = 12.2, and the call that crosses the line passes too. They cost a little over US$0.05. The rest get 429 budget_exhausted with “The evaluation team has spent its US$… for this month.” Read the numbers in your own output: the first refusal line gives the number of answers and the cost before it. Then put the budget back and publish again.

Recap

  • The gateway checks four limits before every call: requests per minute, calls per run, tokens per day, and US$ per month. Every replica reads the same shared counters.
  • Each limit stops a different loop. The replayed hour cost US$176.90 with no limit. It cost US$0.04 with the step limit, US$35.38 with the rate limit, US$4.40 with the quota and US$25.00 with the budget.
  • A warning header comes from 80% of the agent’s budget; the stop at 100% is a 429 with a code. Retry only rate_limited, once, after retry-after.
  • Limits per agent and per team have a cost. A loop uses the same quota and budget as the agent’s real work and its team’s other agents.

Your project now: apps/runaway.py, the gateway’s tests, and 153 tests that pass. The gateway now stops an agent that spends too much. But it still forwards whatever the request contains, even a card number or a key that must not reach a provider. Next, you add guardrails at the gateway, and measure what they cost.

Next: Guardrails at the gateway