Backpressure

What you will be able to do

Decide how much waiting work the API accepts. Refuse the rest at once, with an honest time to come back, and tell every accepted caller how long the work will take.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The queue, the worker and GET /v1/queue, from Job lifecycle.
• Why a job waits without spending an attempt when the provider answers 429, from Delivery and retries.
• An open stream: tickets that arrive at their own rate, whatever the service does, from Workload model. scripts/send_tickets.py sends one.

What you need on your computer

• The project at end-of-m04: python -m pytest prints 163 passed, 1 deselected.
• PostgreSQL and Valkey running, and the simulated provider (python -m simulator) in its own terminal.

This lesson adds no code. The overload run is a short load test. Run it only against the services on your own computer, and close other busy programs first. The provider is simulated; the quota is the recorded one (100 chat requests per minute). The measured numbers come from an Apple M4 computer with nothing else busy; your numbers will differ. Tested on macOS (Apple silicon); the Windows commands are not tested.

The waiting list at a busy restaurant

On a Saturday night, a popular restaurant has no free table. The host writes the names of new guests on a waiting list. Tables free up about one every five minutes. When the list has 12 names, a new guest would wait an hour.

A good host does two things. When the list is too long, she says: “We are full tonight. Try again at nine o’clock.” The guest can go somewhere else, or come back later. And she does not let one person write 30 names on the list. The list has a limit for everyone, and a smaller limit for each guest.

A bad host writes every name down and smiles. At midnight, forty people are still waiting, and each of them was told “yes”.

The problem: a queue without a limit only moves the waiting

The previous two lessons made the queue safe: every accepted ticket is saved, and every job runs once in effect. Now look at the same burst as in the first lesson: 10 tickets per second for 30 seconds, with the recorded quota and two workers.

With the queue and no limit, the API accepted all 300 tickets, and each answer took about 13 milliseconds. That looks perfect. But the quota lets only about 50 jobs per minute through. When sending stopped, 225–230 jobs were waiting, and the oldest had waited 23.4 seconds. At 50 per minute, the last of them got its draft reply about five minutes later.

Thirty seconds of burst made a five-minute backlog. Omar’s Black Friday is several hours long. With no limit, a customer told “accepted” at noon could get a draft reply in the evening. The API said “yes” to work that it could not do in a useful time. That broken promise was invisible: no error, no log line, only a number that grew.

There is a second problem. One caller with a broken script could send thousands of tickets and fill the queue. Every other customer would then wait behind that one caller.

The idea: push back early

Backpressure means that a full part of a system pushes back on the part that feeds it. Then work does not pile up without an end. The name comes from pipes: when water cannot flow out fast enough, the pressure rises back toward the source. In an API, pushing back means answering “not now” early, cheaply and honestly.

The project pushes back with two limits:

1. A queue bound. When the queue holds QUEUE_MAX jobs, the API answers 503 queue_full and saves nothing. This limit protects the promise to all customers.
2. A rate limit per caller. One caller may send at most RATE_LIMIT_PER_MINUTE new tickets per minute (30). The 31st answers 429 rate_limited. This limit keeps one caller from filling the queue for everyone.

The two codes mean different things. 429 Too Many Requests says: you sent too much. 503 Service Unavailable says: the service is too busy for everyone now. Both answers carry Retry-After: the number of seconds after which a new try makes sense. A good client waits that long, and does not try again at once.

A tiny example with every number visible. The bound is 100. The queue holds 100 jobs, and 50 jobs finished in the last minute, so the pace is 50 ÷ 60 = 0.83 jobs per second. A new ticket arrives:

• The answer: 503 queue_full; the ticket is not saved.
• Retry-After: 100 jobs ÷ 0.83 jobs per second = 120 seconds. The project keeps the value between 1 and 300 seconds.
• The rate limit: a customer sends their 31st ticket 2 seconds into a minute. The answer is 429, with Retry-After 60 − 2 = 58 seconds, when the next minute starts.

Every check happens before the API saves anything, in this order:

The checks of a new ticket, in the order that the API runs them: the caller's rate, a repeated key, the queue bound, then one transaction.

Text description of the diagram

Read it from the top. The first two checks are about the caller: too many tickets this minute, or a request that the API has seen before. The third check is about everyone: the queue. Only a request that passes all three is saved, and then the ticket and its job are saved together. A refusal saves nothing, so a refused customer can send the same ticket again later without making a copy.

Larkfield: the bound and the rate limit in code

The bound is three lines in accept, inside the same transaction that would save the ticket:

ticket_api/intake.py (accept, part)

stats = jobs.stats(conn)
if stats["queued"] >= settings.queue_max:
    wait = wait_estimate(stats, stats["queued"])
    raise QueueFull(stats["queued"], min(300, max(1, math.ceil(wait or 30))))

wait_estimate divides the waiting jobs by the jobs that finished in the last minute. If no job finished in the last minute, there is no pace to divide by, so the API says 30 seconds.

The rate limit lives in ticket_api/ratelimit.py. It keeps one counter per caller and minute in Valkey, the shared cache of Module 3. All API processes then count together. If the cache does not answer, the limit is off: the cache fails open, because a cache problem must not stop customers from sending tickets.

What each design did with the same burst

The course ran the burst three times with each design: 300 tickets, 10 per second, two workers, the recorded quota.

• saved (202 or 201)
• refused at once (503 queue_full, Retry-After)
• lost (503 ai_busy after the provider refused)

The same burst (300 tickets in 30 s, quota on, 2 workers), three designs; median of 3 passes.

Text description of the chart

Three horizontal bars of 300 tickets each. No queue, with the AI work during the request: 55 saved and 245 lost, answered with 503 ai_busy after the provider refused; their tickets were not saved. A queue without a bound: all 300 saved, none refused, none lost; but 225–230 jobs waited when sending stopped. A queue with a bound of 100: 172 saved and 128 refused at once with 503 queue_full and a Retry-After of 88–128 seconds; none lost.

What this tells us. No design can do more AI work than the quota allows. The designs differ in who waits and who is told the truth:

• No queue: about 245 customers lost their tickets. Their answers took a median of about 150 milliseconds, with a p95 of about 2.5 seconds, and said Retry-After: 1. That hint was far too short for a quota that counts over a minute.
• Queue, no bound: nobody was refused, but the AI work for the last tickets came about five minutes late, and nobody was told.
• Queue, bound 100: 172 customers were accepted, and their work started within about two minutes. 128 were refused in about 12 milliseconds, with a Retry-After of 88–128 seconds: an honest “come back in about two minutes”. The queue peaked at 107, not 100; the common mistake below explains why.

Explore the queue

The explorer replays the burst with other numbers. It is computed from measured rates, not measured. A worker runs 4 jobs at a time, and a job takes 2.2 seconds. The recorded quota lets 50 jobs start at once, and then 50 per minute. For the two settings that the course measured, it shows the measured runs next to its own numbers.

Explore

A queue under a burst of tickets

Accepted (202)

300 of 300

Refused (503)

0

Longest queue

226 jobs

Oldest wait at 30 s

22.6 s

Queue empty

after 5 minutes

300 tickets in 30 s. 2 workers can start about 3.6 jobs/s; after the first 50 jobs, the quota lets only 0.83 jobs/s start. When sending stops, 226 jobs wait, the oldest for 22.6 s. The last job is done after 5 minutes.

• jobs waiting
• oldest job's wait
• sending stops (30 s)

Check: the same settings, measured (3 passes)

	Computed	Measured
Accepted (202)	300	300
Refused (503)	0	0
Longest queue	226	225–230
Oldest job's wait when sending stops	22.6 s	23.4 s
Retry-After	–	–

Computed from measured rates (4 jobs per worker, 2.2 s per job, the recorded quota), checked against the measured runs at 10 tickets per second with 2 workers.

Text description of the explorer

The main control is the arrival rate: 2 to 12 tickets per second, for 30 seconds. Three more controls set the workers (1 to 4), a queue bound (none or 100) and the provider quota (on or off). Two line charts show the jobs waiting and the oldest job’s wait, from 0 to 120 seconds, with a dashed line where sending stops. The readout shows the accepted and refused tickets, the longest queue, the oldest wait at 30 seconds and when the queue is empty again.

It starts at 10 tickets per second, 2 workers, no bound, quota on. The first 50 jobs start quickly; then only 0.83 per second can start. When sending stops, 226 jobs wait, and the oldest has waited 22.6 seconds; the queue is empty after about 5 minutes. The measured runs had 225 to 230 jobs waiting and an oldest wait of 23.4 seconds.

With the bound on, 174 tickets are accepted and 126 refused, with Retry-After 83 to 108 seconds. The queue stays at 100 and is empty after about 2.5 minutes. The measured runs accepted 172 to 173, refused 127 to 128, peaked at 107, and sent Retry-After 88 to 128 seconds.

With the quota on, more workers change almost nothing: 4 workers give the same 174 and 126. With the quota off, workers matter: at 10 tickets per second, 2 workers leave 188 jobs waiting at 30 seconds, and 4 workers leave 77. At 2 tickets per second, one worker keeps up, and at most 6 jobs wait.

What this tells us. A queue absorbs a burst, but it cannot make the provider faster. When the quota is the limit, extra workers only wait for it. The bound decides how long the waiting can get. With 100 jobs at 50 per minute, an accepted ticket waits at most about two minutes. Choose the bound from the wait that you can promise, not from the space in the database.

Tell callers how long the work will take

A 202 answer has two more fields: queued_ahead, the jobs before this one, and estimated_wait_seconds, the time until it starts at last minute’s pace. The help desk’s screen can say “about one minute” instead of only “accepted”.

An estimate is only as good as the pace behind it. In the author’s run, one worker had worked through a burst of 80 tickets for 20 seconds. Then one more ticket arrived:

{"ticket_id":"T-500081","job_id":81,"state":"queued","status_url":"/v1/jobs/81","queued_ahead":43,"estimated_wait_seconds":80.0}
job 81 succeeded attempts 1 started after 23.6 s, finished after 25.6 s

The estimate said 80 seconds. In fact, the job started after 23.6 seconds. “The last minute” held only 20 seconds of work, so the pace looked three times slower than it was. The API computed (43 + 1) ÷ 33 jobs per minute = 80 seconds. This comes from the formula, not from the computer, so it happens on every computer at the start of a burst. The estimate is too high early in a burst and closer later. An estimate that is too high is the safer mistake, but show it as “about”, and never as a promise.

Another choice: save the ticket, skip the AI work

Larkfield’s API refuses the whole ticket when the queue is full. Omar does not like that: “On Black Friday, a customer’s message matters more than its draft reply.” He is right that there is another design. The course describes it but does not build it.

When the queue is full, the API saves the ticket without a job and answers 201, with a note that no AI work is planned. The help desk sees every ticket. People classify these tickets by hand, as they did before the AI work. Later, a small job can look for tickets without AI results and queue them when the queue is short again.

What it costs: the help desk loses its AI help exactly when it is busiest. The API needs a third answer, besides “queued” and “refused”, and the help desk’s screen must show it. And the backfill job is more code to test. Both designs are honest; they protect different things. The course keeps the 503, so that the bound stays easy to see and measure.

Guided practice: the burst with a bound

Run the measured burst on your computer, with a bound of 100 and two workers.

1. Stop the API (Ctrl+C) and any running worker. Load the data again: python -m scripts.load --reset.

2. Restart the simulated provider (Ctrl+C, then python -m simulator), so that its quota is full.

3. Start the API with a bound of 100. Switch off the rate limit, as the measured runs did, so that only the bound refuses tickets:

   macOS / Linux

   QUEUE_MAX=100 RATE_LIMIT_PER_MINUTE=0 fastapi run

   Windows (PowerShell)

   $env:QUEUE_MAX = "100"; $env:RATE_LIMIT_PER_MINUTE = "0"; fastapi run

4. Start two workers, each in its own terminal: python -m ticket_api.worker --name worker-1 and python -m ticket_api.worker --name worker-2.

5. In another terminal, start watching the queue: python -m scripts.queue_watch --seconds 40 --every 5.

6. In the last terminal, send the burst: python -m scripts.send_tickets --rate 10 --seconds 30.

7. When the watch stops, stop the workers and the API with Ctrl+C. In PowerShell, also run Remove-Item Env:QUEUE_MAX, Env:RATE_LIMIT_PER_MINUTE.

Show the expected result

The author’s run, on a busy computer (your numbers will differ):

Sent 300 tickets at 10 per second in 29.9 s: status codes {202: 172, 503: 128}
Retry-After values: {'87': 2, '89': 7, '90': 6, '91': 16, '92': 5, '93': 8, '94': 20, '95': 1, '96': 21, '97': 9, '98': 9, '99': 2, '100': 6, '106': 6, '108': 4, '110': 4, '112': 1, '123': 1}
Answer time: median 14 ms, p95 24 ms

  time  queued  ready  running  oldest s  done/min  dead
     0       0      0        0       0.0         0     0
     5      24     24        8       2.4         8     0
    10      56     56        8       5.6        26     0
    15      87     87        8       8.6        46     0
    20     100     74        8      12.6        63     0
    25     105      0        2      17.6        65     0
    30     102      0        2      22.6        68     0
    35      96      0        2      27.6        74     0

The measured runs on a quiet computer gave 172–173 accepted, 127–128 refused and Retry-After 88–128 seconds. Look at the column ready. From 25 seconds on, almost no queued job is ready: they all wait for the quota, as the previous lesson taught. dead stays at 0.

Then try the rate limit with the normal settings. Start fastapi dev again, and send 31 tickets from one customer in the same minute. The loop prints only the status codes; the last command shows the full answer:

for i in $(seq 1 30); do curl -s -o /dev/null -w "%{http_code} " -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" -d "{\"customer_id\": \"C-0022\", \"subject\": \"Test $i\", \"body\": \"Rate limit test.\"}"; done
curl -s -i -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" -d '{"customer_id": "C-0022", "subject": "One more", "body": "Rate limit test."}'

In PowerShell, write curl.exe and send the requests one by one (written, not tested). The loop prints 202 30 times, or fewer if C-0022 sent tickets earlier in this minute. The 31st answers:

HTTP/1.1 429 Too Many Requests
retry-after: 58
{"error":{"code":"rate_limited","message":"More than 30 new tickets in one minute. Try again in 58 seconds.",…}}

Your retry-after depends on the second of the minute when you send it.

Your turn: twice the workers

Predict first. Run the same burst with four workers instead of two. Do more tickets get accepted? Try it in the explorer first, then on your computer: start two more workers in step 4.

Show the answer

Almost nothing changes. The author’s run with four workers: {202: 168, 503: 132}, Retry-After 96 to 117 seconds, and the queue again at about 100 from 20 seconds on. The explorer computes 174 and 126 for both two and four workers. The quota, not the workers, sets the pace. Four workers start the first 50 jobs a little sooner, and then all of them wait for the quota. This is Tomás’s “add more workers” idea at the wrong moment. More workers help only when the workers are the bottleneck, as the explorer shows with the quota off.

A bound that is not exact

Common mistake

QUEUE_MAX is 100, but the queue holds 107 jobs

Symptom (measured): with QUEUE_MAX=100, the queue’s peak was 107 jobs in all three measured runs. In the author’s run it was 105.

Cause: the check and the insert are two steps. Request A counts 99 queued jobs, so it may insert. At the same moment, request B also counts 99. Both insert, and the queue holds 101. FastAPI runs many requests at once, and with several API processes even more can pass the check together. It is a race: the result depends on which request runs first.

Diagnose: compare the peak in scripts.queue_watch with QUEUE_MAX. A few jobs over the bound under load means a race. A queue far over the bound means that the bound is not set: check QUEUE_MAX in the API’s environment.

Fix, or accept it:

1. For a bound that protects a promise (“at most about two minutes”), 7 extra jobs do not matter. Treat QUEUE_MAX as a soft limit, and write it down.
2. If the bound must be exact, make the check and the insert one step for everyone. For example, take a PostgreSQL advisory lock (pg_advisory_xact_lock) in accept before the count. The cost is that new tickets are then saved one at a time.

When to use backpressure, and what it costs

Bound every queue that a customer’s request fills, and pick the bound from the longest wait that you can promise. Add a rate limit per caller when callers share a queue, so that one of them cannot fill it. Give every refusal a Retry-After, and make it honest: the project computes it from the real pace.

Do not set the bound so low that normal days refuse tickets. Larkfield’s busiest normal hour has 101 tickets: fewer than two per minute, far below the quota. And do not use a rate limit instead of a bound: 40 customers within their limits can still fill the queue together.

The costs are real. Some customers are refused, and the client must handle 429 and 503 and come back later. The limits are settings that someone must choose, measure and change when the quota changes. And a refused ticket is a message that the help desk has not seen yet, which is why Omar’s other design exists.

Module check

This module moved the AI work from the request to a queue. You can now:

• Save a ticket and its job in one transaction, answer 202, and follow the job through its states (Job lifecycle).
• Make a repeated request and a repeated delivery produce one intended effect. Retry with backoff, jitter and a dead letter (Delivery and retries, with the module’s completion check).
• Bound the queue, refuse early with 503 or 429 and Retry-After, and tell callers how long they will wait (this lesson).

Before you go on, check that python -m pytest still prints 163 passed, 1 deselected, and start the API again without QUEUE_MAX and RATE_LIMIT_PER_MINUTE, so that the defaults (500 and 30) apply.

See it end to endQueue and cache for an open-data APICase study

The case study puts a background export job behind an open-data API and measures what the queue changes.

See it end to endCapacity plan from a public traffic traceCase study

The traffic-trace case study adds a 30-minute provider outage: 273 jobs wait, and a queue bound of 100 refuses 177 requests.

Knowledge check

Check your understanding

The queue without a bound accepted all 300 tickets in about 13 ms each. Why is that not a good result?

Show the answer

Accepting was fast, but the work was not. The quota lets about 50 jobs per minute through. After 30 seconds, about 225 to 230 jobs were waiting, and the last one got its AI work about five minutes later. Each customer was told “accepted” without any hint of the wait. In a longer burst, the wait grows without an end. A bound keeps the wait within what the help desk can promise, and tells the others honestly to come back later.

Check your understanding

One customer's script sends 200 tickets in a minute. Which answer does it get, 429 or 503, and why does the difference matter to the other customers?

Show the answer

After 30 tickets in that minute, it gets 429 rate_limited, with Retry-After until the next minute. A 429 says that this caller sent too much; other customers are not affected and still get 202. Without the per-caller limit, the script’s tickets would fill the queue. Then every customer would get 503 queue_full, which says that the service is too busy for everyone. The rate limit keeps one caller’s problem from becoming everyone’s problem.

Check your understanding

Predict with the numbers: the bound is 100, the queue is full, and 25 jobs finished in the last minute. What Retry-After does a new ticket get? With the quota off and four workers, would you expect a bigger or smaller number?

Show the answer

The pace is 25 ÷ 60 = 0.42 jobs per second, so 100 ÷ 0.42 = 240 seconds, within the 1 to 300 limit: Retry-After: 240. With the quota off, four workers finish about 7 jobs per second. The last minute would show far more finished jobs, and a much smaller Retry-After: 100 ÷ 7.3 is about 14 seconds. Retry-After follows the real pace, so it tells callers more when the service is slow.

Recap

• A queue without a bound only moves the waiting: 300 accepted, 225–230 waiting after 30 seconds, the last AI work about five minutes late.
• Backpressure refuses early: 503 queue_full when the queue holds QUEUE_MAX jobs, 429 rate_limited when one caller sends more than 30 per minute. Both save nothing and send Retry-After.
• Retry-After = queued jobs ÷ last minute’s pace (1 to 300 s). With the bound at 100: 172–173 accepted, 127–128 refused with 88–128 seconds, none lost; without a queue, about 245 tickets were lost.
• When the quota is the limit, more workers do not help: the explorer computes 174 accepted with 2 or 4 workers, and the author’s run with 4 workers accepted 168.
• The 202 answer has queued_ahead and estimated_wait_seconds; early in a burst the estimate is too high (80 s said, 23.6 s real). Show it as “about”.
• Another design saves the ticket without AI work when the queue is full. It keeps every message, but loses the AI help when it is most needed.

Next: Failure containment