Handle provider limits

Lesson 17
50 min
Module 6: Reliability and cost controls

Lesson 17 of 19 · Building Reliable Applications with LLM APIsHandle provider limits

Course overview

1. Start with one model request

2. Prompting as an engineering task

3. Structured outputs and validation

4. Tool calling safely

5. Conversations and streaming

6. Reliability and cost controls

7. Case studies

What you will be able to do

Keep the assistant working when the provider says “too many requests” or does not answer in time. You retry only what is safe to repeat, wait the right time between attempts, and stop before the agent waits too long.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

The adapter’s errors (ProviderError, ProviderTimeout, RateLimited) and the timeout, from The application boundary.
The outcome failed, and what the agent sees then, from Communicate failure.
The idempotency key in the approval design, from Bound the loop.

What you need on your computer

The project at the end of Module 5. python -m pytest shows 65 passed, 1 deselected. The virtual environment is active.
If your project is not in that state, copy the snapshot end-of-m05 with the steps of Reset the project.

Models and APIs change fast. The responses, headers and prices in this module are a snapshot. The method is what lasts.

A busy phone line

You call a shop, and a recording says: “All lines are busy. Please call again later.” You wait a minute and call again. If the line is still busy, you wait longer before the next try. After a few tries you stop and send an email instead.

Now picture a hundred people who heard the same message at the same moment. If each of them waits exactly one minute, they all call back together, and the lines are busy again. The people who get through are the ones who waited a slightly different time.

The problem: the provider says “not now”

A provider shares its servers between many customers, so it limits how much each one may send. A rate limit is that rule: at most so many requests, or so many tokens, per minute. When you go over it, the provider answers with HTTP 429, Too Many Requests, and does no work.

This happened for real while the course was recorded. The recorder sent tickets to Azure with four workers in parallel. It was not trying to hit the limit, but in about eight minutes it got 42 real HTTP 429 answers. Here is one of them, with its most useful headers:

HTTP 429
{"error": {"message": "Your requests to gpt-6-luna for chat-small in eastus2 have exceeded token rate limit.",
           "type": "too_many_requests", "code": "rate_limit_exceeded"}}

retry-after: 30
retry-after-ms: 511
x-ratelimit-limit-tokens: 100000
x-ratelimit-remaining-tokens: -853
x-ratelimit-limit-requests: 100
x-ratelimit-remaining-requests: 65

The deployment allows 100,000 tokens and 100 requests per minute. Requests were not the problem: 65 of 100 were still free. Tokens were: the four workers had gone 853 tokens over. Each ticket sends about 2,000 tokens, so four workers fill the token limit fast.

The picture shows when the 42 answers came. They did not spread out evenly. They came in groups, about 30 seconds apart.

The 42 real HTTP 429 answers, in 10-second groups, from 22:17:10 Toronto time (four parallel workers, not provoked). The recorder waited 30 s after each one.

Text description of the chart

A histogram of 42 HTTP 429 answers over about 480 seconds, in bins of 10 seconds. The answers come in groups. There are 3 at the start, then groups near 30, 40, 70, 100 and 140 seconds. After a pause, a large group of 19 answers comes from 240 to 310 seconds. Smaller groups follow near 400, 435 and 470 seconds. Between the groups, there are no 429 answers. Many groups are about 30 seconds apart, the same as the recorder’s wait.

The recorder had a simple rule: after a 429, wait the 30 seconds that retry-after asked for, then send the same request again. The requests that failed together waited together, so they came back together. Five of those repeats got another 429. One request, the tool loop of T-65070, got three in a row, 30 seconds apart. All 42 answers said retry-after: 30, whatever the real situation was.

A second kind of failure looks different but needs the same care. A timeout is the most time the code waits for an answer; the project waits 30 seconds. When that time passes, you do not know what happened. The provider may never have received the request. Or it may have done the work, and billed it, but the answer did not arrive.

The idea: retry, but with rules

A retry sends the same request again after a failure. A good retry policy answers three questions: which failures to retry, how long to wait, and when to stop. The picture shows them in order.

A decision tree. The provider call failed. What kind of failure? HTTP 400, 401, 403 or 404, where the request or the key is wrong: do not retry, the same request fails again, and stop. HTTP 429: is there a Retry-After in the answer? No: wait with backoff and jitter, 0.5, 1, 2 seconds and so on, at most 8 seconds. Yes, 8 seconds or less: wait at least what it asks. Yes, more than 8 seconds: stop. Timeout or HTTP 5xx: is the request safe to repeat? A read is, a write only with an idempotency key. If not: do not retry. If yes: backoff with jitter. After a wait: are there attempts left, of 4, and are the waits under 20 seconds? Yes: send the same request again. No: stop. Stop means outcome failed, and a person takes the ticket. — Which failures to retry, how long to wait, and when to stop. The numbers are the project's defaults in assistant/retry.py.

Which failures to retry

Retry a failure only when the next attempt can succeed. A 429 says “not now”, and a 5xx error says “something broke on our side”. Both can pass in a second. A 400 says “this request is wrong”. The real 400 from Describe a tool, where Termchat-strongThe name that Larkfield’s code gives to a larger hosted model: gpt-6.1-sol on Azure. It is often more accurate than chat-small, and it costs about 14 to 18 times more per call.Learn more Request anatomyBuilding Reliable Applications with LLM APIs refused tools, comes back the same way every time. A retry there only adds waiting.

Decide by the type of the error and its status code, never by its words. In Communicate failure, the screen guessed “temporary or not” from the error text, because Outcome.error keeps only text. A provider can change its wording, or answer in another language, and a text match then fails quietly. The retry code runs earlier, while the error object still exists. is_retryable asks two questions that do not depend on words: is it a RateLimited or a ProviderTimeout, and is its status 500 or more? You will see the function in the steps below.

Then ask whether the request is safe to repeat. An operation is idempotent when doing it twice has the same effect as doing it once. Reading an order is idempotent. Analysing a ticket is too, because the assistant only drafts and saves nothing until both gates pass. A refund is not: two refunds cost twice. That is why the approval design in Bound the loop sends an idempotency key, so that a repeated request does the work only once.

The timeout case needs one more sentence. The provider may have done the work, so a retried timeout can cost twice. For a call that costs less than a tenth of a cent, that is acceptable. For an expensive call, count the retries in your budget, as the next lesson does.

How long to wait: backoff and jitter

Exponential backoff means that each wait is twice the one before: 0.5 s, then 1 s, then 2 s. The client “backs off” from a busy server, more and more. A cap stops the waits from growing without end: the project’s cap is 8 s.

Here is the project’s backoff with a base of 0.5 s, for five failed attempts. The numbers are the real output of backoff_s with jitter off:

[0.5, 1.0, 2.0, 4.0, 8.0]

Backoff alone does not fix the phone-line problem. Four workers that fail together still wait 0.5 s together and come back together. Jitter adds randomness: each worker waits a random time between 0 and the backoff. The project uses this “full jitter”, so the second wait is somewhere between 0 and 1 s. The workers now come back at different moments, and fewer of them collide.

When to stop

Every retry policy needs an end. The project stops after 4 attempts (the first call and 3 retries), or when the waits would pass 20 seconds in total. Then the call fails, the outcome is failed, and the agent handles the ticket by hand, as Communicate failure shows. An agent who waits 20 seconds can still work. An agent who waits two minutes has already opened the ticket alone.

Explore: four workers and one busy provider

The explorer runs a simulated provider, not a recording. Four workers send a ticket at the same moment, as the recorder did. The provider starts at most one request every 0.25 s and answers the others with HTTP 429, without a Retry-After header. Each worker follows the project’s policy.

Explore

Retries with backoff and jitter against a rate-limited provider (simulated)

HTTP 429 answers: 6
Tickets answered: 4 of 4
Last worker done at: 3.5 s

Base 0.5 s, jitter off, at most 4 attempts. 10 requests, 6 of them answered with HTTP 429 (simulated). 4 of 4 tickets get an answer; the last worker finishes at 3.5 s. Without jitter, the waiting workers wait the same times, so they come back together, and only one of them can get through each time.

Answer (HTTP 200)
HTTP 429 (simulated)
Waiting before the next attempt

Each worker: attempts (✓ answer, ✗ HTTP 429), waits in seconds, result
Worker	Attempts at (s)	Waits (s)	Result
Worker 1	0 ✓	-	answer at 0 s
Worker 2	0 ✗, 0.5 ✓	0.5	answer at 0.5 s
Worker 3	0 ✗, 0.5 ✗, 1.5 ✓	0.5 + 1 = 1.5	answer at 1.5 s
Worker 4	0 ✗, 0.5 ✗, 1.5 ✗, 3.5 ✓	0.5 + 1 + 2 = 3.5	answer at 3.5 s

A simulated provider: it starts one request every 0.25 s. The waits follow assistant/retry.py: the base doubles for each attempt, at most 8 s, at most 20 s of waits.

Text description of the explorer

The explorer shows four workers on a time line. There are three controls. The base delay is 0.25, 0.5, 1 or 2 seconds. Jitter is on or off, with a button to draw new random waits. The maximum number of attempts is 1 to 6. The readout shows the number of HTTP 429 answers, the tickets answered and the time when the last worker is done. A table lists each worker’s attempts, waits and result.

The starting state: base 0.5 s, jitter off, 4 attempts. Worker 1 gets an answer at 0 s. Workers 2, 3 and 4 get HTTP 429 at 0 s and all wait 0.5 s. At 0.5 s, worker 2 gets through; workers 3 and 4 arrive at the same moment and get 429 again. They both wait 1 s. At 1.5 s, worker 3 gets through, and worker 4 gets 429 again. Worker 4 waits 2 s and gets an answer at 3.5 s, on its fourth and last attempt. In total there are 10 requests and 6 HTTP 429 answers, and all 4 tickets get an answer.

With jitter on, each worker draws its own wait. Over 50 draws, the last worker is done at 1.61 s on average instead of 3.5 s. But in a few unlucky draws, one worker uses all 4 attempts and gives up.

Without jitter, the picture repeats the recorder’s mistake in small. The workers that fail together come back together, and only one can pass each time. Jitter breaks the groups. It is random, so one draw can be unlucky; draw again a few times before you judge it. A larger base delay makes collisions rarer, but every ticket then waits longer.

Retry-After: which number to trust

When the provider says how long to wait, its number is better than any guess. The standard header is Retry-After, in whole seconds (or as a date). Look again at the real 429: it has two answers, and they disagree.

retry-after: 30 says 30 seconds. It was 30 on all 42 answers, whatever happened. It looks like a fixed, safe value.
retry-after-ms: 511 says 0.511 seconds, and you can check it. The limit is 100,000 tokens per 60 seconds, which refills about 1,667 tokens each second. The workers were 853 tokens over, and 853 ÷ 1,667 = 0.51 s.

So trust the precise header when it is there. It was computed for this moment; the other was not. But treat it as the shortest wait, not a promise. After 0.511 s the bucket is back at zero, and the next ticket needs 2,000 tokens more. That is why the project waits at least what the provider asks, or longer if the backoff says so.

Your code must handle both headers. retry-after-ms is not a standard: Azure sent it here, and the OpenAI SDK reads it, but many providers never send it. Some providers send only Retry-After, some send it as a date, and some send nothing. The adapter reads them in order of precision. One rule protects the agent: if the provider asks for more than the 8-second cap, the assistant does not come back early. It stops, and a person takes the ticket.

Build it

Open assistant/providers.py. Add two imports, in alphabetical order with the others:
```
import email.utils
from datetime import datetime, timezone
```

In the same file, add this function just before translate_errors:

def retry_after_s(headers) -> float | None:
    """The wait that a 429 response asks for, in seconds, or None.

    Some providers send `retry-after-ms` (milliseconds, precise) as well as the standard `retry-after`
    (whole seconds, or an HTTP date). Use the precise one when it is there.
    """
    for name, scale in (("retry-after-ms", 0.001), ("retry-after", 1.0)):
        value = headers.get(name)
        if value is None:
            continue
        try:
            return max(0.0, float(value) * scale)
        except ValueError:
            pass
        try:  # retry-after can also be a date: "Fri, 09 Oct 2026 02:21:39 GMT"
            when = email.utils.parsedate_to_datetime(value)
            return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
        except (TypeError, ValueError):
            pass
    return None

In translate_errors, replace the two lines of the RateLimitError branch with one line:

        raise RateLimited("HTTP 429: rate limited.", retry_after_s(e.response.headers)) from e

Make assistant/retry.py:

"""Retry what is safe to repeat, with bounded exponential backoff and jitter.

Analysing a ticket only reads: repeating the request changes nothing, so it is safe to retry.
A request with side effects (a refund, a message to a customer) is not retried without an idempotency key.
"""

import random
import time
from dataclasses import dataclass
from typing import Callable, TypeVar

from .providers import ProviderError, ProviderTimeout, RateLimited

T = TypeVar("T")


@dataclass
class RetryPolicy:
    max_attempts: int = 4      # the first call and up to 3 retries
    base_s: float = 0.5        # the first wait before jitter
    cap_s: float = 8.0         # no single wait is longer than this
    jitter: bool = True        # a random wait between 0 and the backoff, so that clients do not retry together
    max_total_s: float = 20.0  # stop when the waits would add up to more than this


def is_retryable(error: Exception) -> bool:
    if isinstance(error, (RateLimited, ProviderTimeout)):
        return True
    return isinstance(error, ProviderError) and error.status is not None and error.status >= 500


def backoff_s(attempt: int, policy: RetryPolicy, rng: random.Random) -> float:
    """The wait after failed attempt number `attempt` (1, 2, 3 ...): base x 2^(attempt-1), capped, then jitter."""
    wait = min(policy.cap_s, policy.base_s * 2 ** (attempt - 1))
    return rng.uniform(0, wait) if policy.jitter else wait


def call_with_retry(call: Callable[[], T], policy: RetryPolicy = RetryPolicy(),
                    sleep: Callable[[float], None] = time.sleep, rng: random.Random | None = None,
                    on_retry: Callable[[int, Exception, float], None] | None = None) -> T:
    rng = rng or random.Random()
    waited = 0.0
    for attempt in range(1, policy.max_attempts + 1):
        try:
            return call()
        except ProviderError as error:
            if not is_retryable(error) or attempt == policy.max_attempts:
                raise
            wait = backoff_s(attempt, policy, rng)
            if isinstance(error, RateLimited) and error.retry_after is not None:
                if error.retry_after > policy.cap_s:
                    raise  # the provider asks for a longer wait than we accept: stop, do not come back early
                wait = max(wait, error.retry_after)  # never earlier than the provider asked
            if waited + wait > policy.max_total_s:
                raise
            if on_retry:
                on_retry(attempt, error, wait)
            sleep(wait)
            waited += wait
    raise AssertionError("unreachable")

Make assistant/simulate.py. It fails on purpose, so that you can test the policy without a real outage:

"""A SIMULATED provider for tests and demonstrations: it fails on purpose, in a fixed order.

Real rate limits and outages are not provoked for the course. This wrapper raises the same errors that
OpenAICompatibleProvider raises for a real HTTP 429, timeout or 5xx, then lets the next call through.
"""

from .providers import Completion, ProviderError, ProviderTimeout, RateLimited


class SimulatedProvider:
    def __init__(self, inner, script: list[str]):
        """script: one item per call, for example ["rate_limit:2", "timeout", "ok"].
        "rate_limit:N" = HTTP 429 with Retry-After N seconds ("rate_limit" alone has no Retry-After),
        "timeout", "server_error" (HTTP 503), "ok" = ask the inner provider. After the script: always "ok"."""
        self.inner = inner
        self.script = list(script)
        self.calls = 0

    def complete(self, request: dict) -> Completion:
        self.calls += 1
        step = self.script.pop(0) if self.script else "ok"
        if step.startswith("rate_limit"):
            wait = step.partition(":")[2]
            raise RateLimited("HTTP 429: rate limited (simulated).", float(wait) if wait else None)
        if step == "timeout":
            raise ProviderTimeout("The provider did not answer within the timeout (simulated).")
        if step == "server_error":
            raise ProviderError("HTTP 503: service unavailable (simulated).", status=503)
        return self.inner.complete(request)

Open assistant/analyse.py. Add the import below the line that imports from .providers:
```
from .retry import RetryPolicy, call_with_retry
```

In analyse_ticket, add the parameter retry before reasoning_effort, and a small function after request = …:

                   tools: bool = True, book: OrderBook | None = None, limits: Limits = Limits(),
                   retry: RetryPolicy = RetryPolicy(), reasoning_effort: str = "") -> Outcome:
    book = book or OrderBook()
    request = build_request(ticket, model, prompt, response_format() if structured else None,
                            TOOLS if tools else None, reasoning_effort)

    def complete(req: dict) -> Completion:
        return call_with_retry(lambda: provider.complete(req), retry)

In the same function, use complete in both places that called provider.complete:

            result = run_tool_loop(complete, request, ticket.customer_id, book, limits)

            completion = complete(request)

Copy tests/test_retry.py from the snapshot end-of-m06. Then run the tests:
Terminal window
```
python -m pytest
```

Step 9 prints 75 passed, 1 deselected. The ten new tests use a fake sleep that only records the waits, so they run in milliseconds. One of them uses the real headers of the 429 above and expects 0.511 s.

Each retry wraps one model call, not the whole tool loop. In a loop of two calls, a 429 on the second call repeats only that call. The order lookup that came before it does not run again.

Watch the policy work

Make a small script in the project folder, try_retry.py. Its provider fails three times on purpose (simulated), then replays the real recording of T-80008.

"""Watch the retry policy work. The provider is SIMULATED: it fails on purpose, then replays the recording."""
import random
import time

from assistant.context import build_request
from assistant.data import load_ticket
from assistant.providers import MockProvider
from assistant.retry import RetryPolicy, call_with_retry
from assistant.schema import response_format
from assistant.simulate import SimulatedProvider

provider = SimulatedProvider(MockProvider(), ["rate_limit", "timeout", "server_error"])
request = build_request(load_ticket("T-80008"), "chat-small", "v2", response_format())


def show(attempt, error, wait):
    print(f"attempt {attempt}: {error} Wait {wait:.2f} s.")


start = time.monotonic()
completion = call_with_retry(lambda: provider.complete(request), RetryPolicy(), rng=random.Random(8),
                             on_retry=show)
print(f"attempt {provider.calls}: finish_reason {completion.finish_reason}, "
      f"{time.monotonic() - start:.1f} s in total")

random.Random(8) fixes the random draws, so you see the same waits as below. Run it with python try_retry.py. The output:

attempt 1: HTTP 429: rate limited (simulated). Wait 0.11 s.
attempt 2: The provider did not answer within the timeout (simulated). Wait 0.96 s.
attempt 3: HTTP 503: service unavailable (simulated). Wait 0.25 s.
attempt 4: finish_reason stop, 1.3 s in total

The total is measured with a clock, so it can differ by a tenth of a second on your computer. Each wait is random, but never more than the backoff: at most 0.5 s, then 1 s, then 2 s. The fourth attempt is the last one allowed, and it gets the recorded answer.

Your turn

Change one thing at a time in try_retry.py, predict the output, then run it.

Turn jitter off: RetryPolicy(jitter=False).
Put back jitter, and use the script ["rate_limit:0.511"]: a 429 with the real retry-after-ms.
Use the script ["rate_limit:30"]: a provider that asks for 30 seconds.

Show the outputs

Without jitter, the waits are exactly the backoff:

attempt 1: HTTP 429: rate limited (simulated). Wait 0.50 s.
attempt 2: The provider did not answer within the timeout (simulated). Wait 1.00 s.
attempt 3: HTTP 503: service unavailable (simulated). Wait 2.00 s.
attempt 4: finish_reason stop, 3.5 s in total

The provider asked for 0.511 s, and the random backoff was shorter, so the policy waits what the provider asked:
```
attempt 1: HTTP 429: rate limited (simulated). Wait 0.51 s.
attempt 2: finish_reason stop, 0.5 s in total
```
30 s is more than the 8-second cap. The policy does not retry early: it raises the error at once. The script ends with a traceback, and its last line is:
```
assistant.providers.RateLimited: HTTP 429: rate limited (simulated).
```
In analyse_ticket, the same error becomes the outcome failed, and a person takes the ticket.

Delete try_retry.py when you are done. It is not part of the project.

Fall back when retries end

A fallback is what the application does when the retries are used up. TermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps’s fallback is the simplest one. The outcome is failed, and the help-desk app says that the draft is not available. The agent works as they did before the assistant. Nothing is lost, because the assistant only drafts.

Two other fallbacks are common, and both have a cost. A queue tries the ticket again a few minutes later, when the limit has reset. The agent then sees the draft late. A second model can take over, but check that it can do the same job first. chat-strong looks like an obvious backup for Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs. Yet it cannot use tools in TermChat CompletionsThe request and response format of OpenAI’s chat API: a model name, a list of messages with roles, and settings. Azure, Ollama and many other providers accept the same format (“OpenAI-compatible”), so one SDK works with all of them.Learn more Request anatomyBuilding Reliable Applications with LLM APIs: see the real HTTP 400 from Describe a tool. A fallback that was never tested fails at the worst moment.

Retries also cost something even when they succeed. A 429 is free, but its wait is not: with 4 attempts the agent can wait up to 20 s. If 429s are frequent, retries hide a capacity problem. Send fewer requests at once, ask the provider for a higher limit, or use fewer tokens per ticket.

Common mistake

Everyone waits 30 seconds

Symptom: while the course was recorded, the recorder got 42 real HTTP 429 answers in about eight minutes. Five of its repeats got another 429. T-65070 got three in a row, exactly 30 s apart.

Cause: two things together. The recorder waited exactly what retry-after said, 30 s, with no jitter. The requests that failed together came back together and went over the limit again. And it read the rough header, not the precise one: retry-after-ms said 511 ms.

Fix: read the most precise header that the provider sends, and treat it as the shortest wait. Add jitter to every backoff, so that clients do not move in step. Limit the attempts and the total wait. The project’s retry_after_s and call_with_retry do these three things.

What it means for a real AI service: a retry policy is shared behaviour. One client that retries badly wastes its own time. A thousand clients that retry the same way can keep a provider overloaded after the first problem has passed.

Knowledge check

Check your understanding

The assistant gets an HTTP 400 'Unsupported value: temperature'. Should call_with_retry retry it? What does the agent see?

Show the answer

No. A 400 says that the request itself is wrong, so the same request fails again. is_retryable returns false for a status below 500 that is not a 429, and the error is raised at once, on the first attempt. In analyse_ticket it becomes the outcome failed, and the agent handles the ticket by hand. The fix is in the request, not in a retry.

Check your understanding

Predict: jitter is off, the base is 1 s and the cap is 8 s. A provider keeps answering 429 with no Retry-After. How long does one call wait in total before it gives up, and how many attempts does it make?

Show the answer

The waits are 1 s, 2 s and 4 s, which is 7 s in total. Then the fourth attempt fails. It is the last attempt allowed, so call_with_retry raises the error. Four attempts, three waits. The 20-second limit is not reached. With a base of 4 s, the waits would be 4, 8 and 8. After 4 + 8 = 12 s, the next 8 s would pass 20. So the policy stops after the third attempt.

Check your understanding

Why is it safe to retry the analysis of a ticket, but not a call that sends a refund, even after a timeout?

Show the answer

The analysis is idempotent: it reads the ticket and the order and returns a draft, and nothing changes outside the assistant. Doing it twice costs a little more, but the effect is the same. After a timeout, a refund call may have reached the payment system and done its work. Sending it again could refund twice. A write can be retried only with an idempotency key, so that the receiver does the work once whatever the number of requests.

Recap

A rate limit answers HTTP 429 and does no work. During recording, four parallel workers got 42 real 429s in about eight minutes.
Retry only failures that can pass (429, timeouts, 5xx) and requests that are safe to repeat. A 400 is never retried. A write needs an idempotency key.
Exponential backoff doubles the wait (0.5, 1, 2 s, cap 8 s). Jitter makes each wait random, so clients that failed together do not come back together.
The real 429 said retry-after: 30 and retry-after-ms: 511. Trust the precise header as the shortest wait; handle both, because many providers send only one, or none.
The project stops after 4 attempts or 20 s of waits, and when the provider asks for more than 8 s. Then the outcome is failed, and a person takes the ticket.

Next, you log every call: tokens, time and cost.

Next: Measure usage and cost