Failure containment

What you will be able to do

Stop one failing dependency from taking the whole service down with it. Decide how long to wait for the AI provider, when to try again, and when to stop calling it for a while.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The queue, the worker and the job states, from Job lifecycle.
• Retries with backoff, attempts and dead letters, and why a 429 does not spend an attempt, from Delivery and retries.
• The queue bound, 503 queue_full and Retry-After, from Backpressure.

What you need on your computer

• The project ~/projects/ticket-api after Module 4: python -m pytest prints 163 passed, 1 deselected. If your project is broken, copy the snapshot end-of-m04 (see the snapshots).
• PostgreSQL, Azurite and Valkey running: docker compose up -d db azurite cache.
• The clone of the labs repository in ~/projects/nextia-learning-labs, as in Run the AI workload on your computer.

The AI provider in this module is the simulated one on your computer. Its outage and its slow mode are constructed: the real provider did not fail while the course team recorded it. The course team measured the numbers of this module on an Apple M4 computer (10 cores, 16 GB) with nothing else busy. PostgreSQL 18.6 and Valkey 9.1.2 ran in Docker. Your times will differ; the shapes stay the same. Tested on macOS (Apple silicon); the Windows commands are written, not tested.

Calling a shop whose phone is broken

You call a shop to ask about your order. Usually someone answers after three or four rings. Today, the phone rings and rings. How long do you wait? Most people hang up after a minute or so. They do not hold the phone for an hour. That limit is a timeout.

You call again a little later, because perhaps the line was busy. One more try is sensible. Calling again every five seconds for an hour is not. It wastes your afternoon, and if the shop’s phone system is overloaded, your calls make it worse.

After five failed calls, a sensible person stops and does something else. In half an hour, they try once. If someone answers, everything is normal again. If not, they wait another half hour. A fuse box at home works in a similar way. When a circuit has a fault, the breaker switches off, so the fault cannot do more damage. Later, you switch it on again, once, and see if it holds.

These are the three ideas of this lesson: how long to wait, how often to try again, and when to stop trying for a while.

The problem: a provider that does not answer

In Module 4, the AI work moved to a worker. A worker runs 4 jobs at a time. Each job makes three calls to the provider: classify, draft a reply, embed. Now imagine that the provider fails for 20 seconds. Every call returns 503 Service Unavailable after a short wait.

What does the worker do? It has no idea that the provider is down. It sees one failed job, and it schedules a retry, as Module 4 taught it. The next job fails too, and the next. Each failure spends an attempt. A job has 5 attempts (JOB_MAX_ATTEMPTS), then it becomes a dead letter that a person must look at.

Let us count. Thirty tickets are in the queue when the outage starts. If every job tries 5 times before the outage ends, the provider gets 30 × 5 = 150 calls, and all 150 fail. When the provider comes back 20 seconds later, there is nothing left to do: all 30 jobs are dead letters. The customers wrote their tickets, and no draft reply ever comes.

Two things went wrong. The worker kept calling a provider that was clearly down. That wastes its time and adds load to a service that is already in trouble. And it gave up on the jobs because of a problem that was not theirs. A job whose ticket is fine should wait out the outage, not die in it.

There is a third, quieter problem: the time limit. Until now, PROVIDER_TIMEOUT was 30 seconds. A provider that hangs, instead of failing fast, holds each worker slot for 30 seconds per call. With four slots blocked for 30 seconds each, the worker does almost nothing for half a minute.

Idea 1: a timeout that fits the work

A timeout is the longest time that you wait for one call. After it, you stop waiting and treat the call as failed. The question is the number.

Look at what the provider normally does. The course team recorded 100 real calls of each task, one at a time. The slowest draft reply took 2,172 ms, and the slowest classification took 2,143 ms. So a call that has not answered after 10 seconds is very unlikely to be a normal slow call. It is a sign of trouble. From this module, PROVIDER_TIMEOUT is 10 seconds: more than four times the slowest recorded call, and a third of the old 30.

Timeout	A normal slow call (2.2 s)	A hanging provider blocks one worker slot for
30 s (Modules 1 to 4)	passes	30 s per call
10 s (from this module)	passes	10 s per call
1 s (too short)	fails, and is tried again for nothing	1 s per call

A timeout that is too short is a failure that you make yourself. Normal calls fail, are repeated, and cost money twice. A timeout that is too long turns a hanging dependency into a frozen service. So set it from the measured slow end of normal calls, with room to spare. Check it again when the work changes: a longer prompt or a slower model changes the right number.

Idea 2: retry only what is safe, and only a little

A retry is a second try of a call that failed. Before you retry anything, ask one question: what happens if the first try worked after all, and only the answer was lost?

Operation	If it runs twice	Retry?
A call to the AI provider (classify, draft, embed)	You pay for two answers, and nothing else changes	Yes, a few times
POST /v1/tickets without an Idempotency-Key	Two tickets for one customer message	No
POST /v1/tickets with an Idempotency-Key	The same ticket and job (Module 4)	Yes
A job step whose result is saved with the job’s ID	The saved step is not called again (Module 4)	Yes, through the queue

So the provider calls are safe to repeat. But “safe” is not “free”. During an outage, every retry is one more call to a service that is already failing. If every failed call is tried twice more, an outage triples the load on the provider at its worst moment. That is called a retry storm.

The project limits retries in two places. The worker makes no quick retries: a failed job goes back to the queue and waits there, with the growing backoff of Module 4. The API makes one quick retry (PROVIDER_RETRIES=1) for POST /v1/answers, because a person waits for that answer. That retry obeys a retry budget: retries may add at most 10% extra calls (RETRY_BUDGET_RATIO=0.1) over the last 10 seconds, with at least 3 always allowed.

A tiny example shows the budget at work. In 10 seconds, the API makes 100 calls, and the provider is down, so all 100 fail.

• Without a budget, each failure gets its one retry. The provider gets 100 + 100 = 200 calls, and all of them fail.
• With the 10% budget, only 10 retries are allowed. The provider gets 100 + 10 = 110 calls. The other 90 requests fail at once, without a second wait.

When the provider is healthy, only a few calls fail, so the budget almost never refuses a retry. It only acts when many calls fail together, which is exactly when retries do harm.

Idea 3: the circuit breaker

The timeout and the budget still let every request make at least one call. During an outage of 20 seconds, that is still many calls that cannot work, each with its own wait. The fuse box gives the next idea.

A circuit breaker sits in front of the provider client and counts failures in a row. In the project (ticket_api/resilience.py), it has three states:

The circuit breaker of the project: closed, open after 5 failures in a row, half-open after 30 seconds, and one trial call decides.

Text description of the diagram

The diagram has three states. Closed is the normal state: every call goes to the provider, and a success sets the count of failures to 0. After 5 failures in a row, the breaker goes to open. While it is open, a call fails at once: no request is sent, so nobody waits for a timeout. After 30 seconds, the breaker goes to half-open and lets one trial call through. If the trial works, the breaker is closed again. If it fails, the breaker opens for another 30 seconds. A note says what a failure is: a timeout or a 5xx answer. A 429 is not a failure here, because a quota refusal does not mean that the provider is broken. Module 4’s quota waits handle it.

The name comes from the electrical breaker. “Closed” means that the circuit is complete, so calls go through. “Open” means that the circuit is broken, so nothing goes through. This is the opposite of a door, so read the states carefully.

Here is the breaker in the API, in the author’s run. The simulated provider is in outage mode, and six questions arrive one after another:

request 1: 503 in 0.361208 s
request 2: 503 in 0.471282 s
request 3: 503 in 0.187749 s
request 4: 503 in 0.014566 s
request 5: 503 in 0.012967 s
request 6: 503 in 0.012212 s

Count the calls. Requests 1 and 2 each made a call and its one quick retry: 4 failures. Request 3 made one call, the fifth failure in a row, and the breaker opened before its retry. Requests 4 to 6 did not call the provider at all. They failed in about 13 ms instead of 200 to 470 ms. The provider’s own count agrees: "answer":{"503":5}, five calls for six requests. The answer tells the caller when to come back:

HTTP/1.1 503 Service Unavailable
retry-after: 30

{"error":{"code":"ai_unavailable","message":"The AI provider is failing; the API stopped calling it for now. Try again in 30 seconds.", ...}}

These times come from the author’s run on a busy computer; your times will differ. The counts are the same on every computer.

What this tells us. An open breaker turns a slow failure into a fast one. The customer gets a 503 either way, but now in 13 ms with a clear Retry-After, and the provider gets no extra load.

Larkfield: the worker during an outage

The worker uses the same breaker, with one more rule. When the breaker is open, the worker gives the job back to the queue until the breaker may close, and the attempt does not count. The job did nothing wrong, so it waits.

The script scripts/outage_demo.py tests this. It queues 30 new tickets and starts a worker. It switches the simulated provider to outage mode for 20 seconds, then back to normal, and waits until the queue is empty. The quota is on. To keep the run short, the script uses shorter retry waits than the defaults: 1 to 8 seconds instead of 2 to 60. The measured run, with the breaker on, gave the same result in all 3 passes:

Outage over after 20 s (simulated).
Circuit breaker: on (opened 1 time(s))
Calls that reached the provider during the outage: 8 (answered {'200': 0, '429': 0, '503': 8})
Jobs at the end (65 s): succeeded 30

Eight calls reached the provider in 20 seconds. Five failures in a row opened the breaker; the other calls had already started. Then the worker stopped calling and gave each job back. After 30 seconds, the trial call worked, because the outage had ended, and the worker did all 30 jobs.

With --breaker off, the same outage gave this, also in all 3 passes:

Outage over after 20 s (simulated).
Circuit breaker: off
Calls that reached the provider during the outage: 150 (answered {'200': 0, '429': 0, '503': 150})
Jobs at the end (65 s): dead_letter 30

Calls that reached the simulated provider during a 20-second outage, with 30 jobs in the queue. The same in all 3 passes: 8 with the breaker, 150 without it.

Text description of the chart

A column chart with two columns. With the circuit breaker on, 8 calls reached the provider during the 20-second outage. With the breaker off, 150 calls reached it: 30 jobs times 5 attempts. Every one of these calls failed. Each pass of the measurement gave exactly the same counts.

• Breaker on
• Breaker off

The 30 jobs, 45 seconds after the outage ended: done (succeeded) or dead letters. With the breaker, all 30 succeeded; without it, all 30 are dead letters.

Text description of the chart

A column chart with four columns, on an axis from 0 to 30 jobs. With the breaker on, 30 jobs succeeded and 0 are dead letters. With the breaker off, 0 jobs succeeded and 30 are dead letters. The provider was back after 20 seconds in both runs, but without the breaker no job was left to use it.

What this tells us. The breaker changed two numbers, and the second matters more to customers. Fewer calls protect the provider and the account’s quota. But the breaker also gave all 30 tickets their AI work, about a minute late, instead of none. Omar asks what customers see. With the breaker, the help desk sees each draft reply arrive after the outage. Without it, 30 tickets have no draft until someone finds the dead letters and retries each one with POST /v1/jobs/{id}/retry.

When to use each one, and what it costs

Tool	Use it	It costs
Timeout	On every network call	Too short: normal calls fail. Too long: a hang freezes the worker
Quick retry with a budget	Where a person waits, for calls that are safe to repeat	A little extra load; a longer wait for the person in a real outage
Queue retry with backoff	For background work	The work arrives later
Circuit breaker	In front of a dependency that can fail as a whole	30 s without AI work after a short glitch; thresholds to choose

Sam asks: “Why does each process have its own breaker? Four workers must each see five failures.” That is true, and it is a choice. A breaker per process needs no shared state, and it keeps working when Valkey is down. The cost is a few more failed calls at the start of an outage: up to 5 per process. A shared breaker in Valkey would learn faster. But it adds a dependency to the code that protects you from dependencies, so the project keeps it simple.

Two more limits. The breaker cannot tell a provider outage from your own network problem, because both look like timeouts. And a breaker protects the user only when the work can wait. The API’s 503 ai_unavailable is honest: it says what failed and when to come back.

Guided practice: an outage, with and without the breaker

1. In ~/projects/ticket-api, with the virtual environment active, copy this lesson’s files from the snapshot end-of-m05. worker.py also has the graceful stop of Graceful shutdown, which comes later in this module.

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m05
   for f in README.md ticket_api/resilience.py ticket_api/provider.py ticket_api/config.py ticket_api/errors.py ticket_api/main.py ticket_api/worker.py scripts/outage_demo.py tests/test_resilience.py; do cp "$S/$f" "$f"; done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m05"
   foreach ($f in 'README.md', 'ticket_api\resilience.py', 'ticket_api\provider.py', 'ticket_api\config.py', 'ticket_api\errors.py', 'ticket_api\main.py', 'ticket_api\worker.py', 'scripts\outage_demo.py', 'tests\test_resilience.py') { Copy-Item "$S\$f" $f }

2. Run the tests: python -m pytest. Expect 172 passed, 1 deselected.

3. Run the new tests alone, with their names: python -m pytest tests/test_resilience.py -v. Expect 9 passed.

4. Stop any worker that still runs from Module 4, because the demo starts its own. In terminal 1, start the simulated provider: python -m simulator.

5. In terminal 2, load the data again, so that the demo counts only its own jobs: python -m scripts.load --reset.

6. Run the outage with the breaker. It takes about 65 seconds. LOG_LEVEL=WARNING hides the worker’s log lines.

   macOS / Linux

   LOG_LEVEL=WARNING python -m scripts.outage_demo --breaker on

   Windows (PowerShell)

   $env:LOG_LEVEL = "WARNING"; python -m scripts.outage_demo --breaker on

7. Run python -m scripts.load --reset again. Then run step 6 with --breaker off.

8. Now see the breaker in the API. Stop the simulator with Ctrl+C, and start it in outage mode: python -m simulator --mode outage. In terminal 2, start the API: fastapi dev.

9. In terminal 3, send six questions, one after another. On Windows, type curl.exe, and send the request six times by hand (written, not tested):

   for i in 1 2 3 4 5 6; do curl -s -o /dev/null -w "request $i: %{http_code} in %{time_total} s\n" -X POST http://127.0.0.1:8000/v1/answers -H "Content-Type: application/json" -H "X-Customer-ID: C-0003" -d "{\"question\": \"Where is my parcel number $i?\"}"; done

10. Send one more question with -i in place of -s -o /dev/null -w …. Read its retry-after header and its error code.

Show the expected result

Steps 6 and 7 print the two outputs of this lesson: 8 calls and succeeded 30 with the breaker; 150 calls and dead_letter 30 without it. The calls are the same on every computer. The time in brackets, (65 s), can be a little longer on a busy computer. If step 7 counts more than 30 jobs, you forgot --reset: the script counts every ticket whose subject starts with Outage test.

Step 9 prints six 503 lines. The first three take a few hundred milliseconds. The last three take about 10 to 20 ms, because the breaker is open. Step 10 shows 503, a retry-after: header with a number of seconds up to 30, and "code":"ai_unavailable". If the first requests already answer fast, the breaker was open before you started: wait 30 seconds and try again. When you finish, stop the API, and start the simulator again without --mode outage.

Your turn: a slow provider

An outage is easy to see. A slow provider is harder. Change one condition: start the simulator with python -m simulator --mode slow. Every answer now takes 8 times longer, so a classification takes about 7.3 seconds and a draft reply about 8.8 seconds. These are medians, so some calls take longer than 10 seconds.

Predict first. With PROVIDER_TIMEOUT=10, will jobs fail? Will the breaker open? Then start a worker (python -m ticket_api.worker) and the API (fastapi dev). Send 8 tickets with python -m scripts.send_tickets --rate 1 --seconds 8, and watch with python -m scripts.queue_watch --every 5.

Show the answer

In the author’s run, all 8 jobs succeeded. Three draft replies took more than 10 seconds, so those three jobs were retried and succeeded at attempt 2. The worker’s log said job 2: retried (attempt 1) and later job 2: succeeded (attempt 2). The breaker did not open: timeouts and successes were mixed, so there were never 5 failures in a row. The queue’s oldest job reached about 31 seconds.

The simulator’s count shows two costs of a timeout. It answered 8 classifications but 11 draft replies: the provider finished the 3 slow calls that the worker had stopped waiting for. With a real provider, you pay for those answers. And the classifications were not called again, because their results were saved with the job’s ID (Module 4).

With the old PROVIDER_TIMEOUT=30, these jobs would succeed at attempt 1. But each slow call would hold a worker slot for up to 30 seconds. Neither number is right forever. A slow provider is a reason to measure again, as in Find the bottleneck.

Common mistake

Just retry until the provider answers

Symptom: during the outage, Tomás says: “The provider will come back. Turn off the breaker, raise JOB_MAX_ATTEMPTS to 50 and keep retrying.” With the breaker off and 5 attempts, the outage demo prints Calls that reached the provider during the outage: 150 and Jobs at the end (65 s): dead_letter 30. The provider was back after 20 seconds, and no job was left to use it.

Cause: each failed call spends an attempt. The retry waits (1 to 8 seconds in the demo, up to 60 by default) are shorter than many outages. So the attempts run out during the outage. More attempts only move the problem. With 50 attempts, the jobs survive a longer outage, but the provider gets up to 50 calls per job while it is down. With 300 jobs in the queue, that is up to 15,000 failing calls, and the quota counts every one of them. A real retry storm can also keep a recovering provider down.

Diagnose: look at the pattern of the failures, not at one job. python -m scripts.queue_watch shows dead rising for many jobs at once. The worker’s log shows ProviderUnavailable or ProviderTimeout for every job, not only for some. The simulator’s /admin/stats (or a real provider’s status page) shows only 503s. When every call fails, the dependency is down, and more calls cannot help.

Fix:

1. Keep the breaker on (BREAKER_FAILURES=5). While it is open, jobs wait without spending attempts.
2. Keep the attempts bounded (JOB_MAX_ATTEMPTS=5). The bound protects you from the job that can never work.
3. After the outage, retry the dead letters on purpose: POST /v1/jobs/{id}/retry.

For an AI service: an AI provider counts quota per call, and some providers also bill failed calls. Unbounded retries turn an outage into a bill and a quota problem that lasts after the outage.

Knowledge check

Check your understanding

The recorded real provider answered a draft reply in at most 2,172 ms. Amira sets PROVIDER_TIMEOUT=2. What will she see on a normal day, and what is a better value?

Show the answer

On a normal day, some calls will fail. The 100 recorded draft replies had a p95 of 1,514 ms and a maximum of 2,172 ms. So the slowest normal calls pass 2 seconds. Each of them times out, the job is retried, and the provider does the work twice: two answers to pay for. A good timeout sits well above the slowest normal call and well below “the worker is frozen”. The project’s 10 seconds is more than four times the slowest recorded call. She should check it again when the prompt or the model changes.

Check your understanding

Predict: the circuit breaker of a worker is open, and 12 jobs are ready. What happens to them, and do they spend attempts?

Show the answer

The worker takes them, but no call is sent: the breaker raises BreakerOpen at once. The worker gives each job back to the queue until the breaker may close, and the attempt does not count. So the jobs wait without losing attempts. After 30 seconds, one trial call goes through. If the provider is back, the breaker closes and the jobs run normally. That is why all 30 jobs of the outage demo succeeded.

Check your understanding

Which of these may be retried automatically, and why: a classify call to the provider; POST /v1/tickets without an Idempotency-Key; POST /v1/tickets with one?

Show the answer

The classify call: yes, a few times. Running it twice costs a second answer and changes nothing else. POST /v1/tickets without a key: no. If the first request was saved and only its answer was lost, a retry makes a second ticket and a second job. With an Idempotency-Key: yes. The API returns the same ticket and job, with Idempotent-Replayed: true. The rule is the one of Module 4: retry only what is safe to repeat, and make writes safe with an idempotency key.

Recap

• A timeout limits one call. Set it from the slow end of normal calls (at most 2,172 ms in the recordings), with room. The project uses 10 seconds instead of 30.
• Retry only what is safe to repeat. Provider calls are safe; writes are safe only with an idempotency key. The worker retries through the queue. The API makes one quick retry inside a retry budget of 10% extra calls.
• A circuit breaker opens after 5 failures in a row (timeouts or 5xx, not 429). It fails at once for 30 seconds, then lets one trial call decide. In the API, an open breaker answered 503 ai_unavailable with Retry-After in about 13 ms instead of hundreds.
• In a simulated 20-second outage with 30 jobs, the breaker cut the provider calls from 150 to 8. All 30 jobs succeeded afterwards, instead of becoming 30 dead letters.
• Retrying until the provider answers turns an outage into a storm of calls and a pile of dead letters.

Next, you decide how many workers to run, and let the queue tell you when to add or remove them.

Next: Scaling signals