Failure and stampede behaviour

What you will be able to do

Keep the answers correct and the AI provider safe in the bad moments. The cache can be empty, down, slow, or hold an old answer. Stop a crowd of requests from making the same answer at the same moment.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Keys, generations and TTLs, and the life of a cached answer, from Keys and invalidation.
• The provider’s quota of 100 chat requests per minute, from Dependency limits.

What you need on your computer

• The project after Keys and invalidation: python -m pytest prints 148 passed, 1 deselected.
• Valkey, PostgreSQL and Azurite running (docker compose up -d db azurite cache), the simulated provider (python -m simulator) and the API (fastapi dev), each in its own terminal.

After this lesson, your project matches the snapshot end-of-m03. Some failures here are constructed: you stop or freeze the cache on purpose. Tested on macOS (Apple silicon); the Windows commands are not tested. The times in the outputs are from the author’s computer while it was busy: your numbers will differ.

A bakery at opening time

A bakery opens at seven, and the shelves are empty. Twenty customers come in, and all of them want the same new cake. If each baker starts a cake for each customer, twenty cakes go into the oven. The oven is full, the other bread waits, and nobody needed nineteen of the cakes.

A better bakery says: “One cake is in the oven. Please wait a few minutes.” One baker bakes, and the twenty customers share the result.

One morning, the display fridge breaks. The bakery does not close. It sells straight from the oven, more slowly, until someone repairs the fridge. The fridge was a convenience, not the bakery.

The problem: a stampede

A cache is empty more often than you think. It is empty after a restart, because Valkey keeps nothing on disk in this project. A popular key disappears when its TTL ends. A document change gives every answer a new key at once. And on Black Friday, a new promotion brings a new question that everyone asks.

Then many requests miss the same key at the same moment. Each one calls the provider for the same answer. This is a cache stampede: a crowd of misses for one key, all refreshing it together. Twenty requests for one new question, with no protection, sent 20 calls to the provider in the course team’s run. Each of those calls used the quota, and only one result was needed.

The idea: one request computes, the others wait

The stampede guard gives each key a lock. The first request that misses takes the lock and calls the provider. Every other request that misses the same key sees the lock, and waits for the result to appear in the cache. Valkey makes the lock safe with one command, SET key:lock token NX PX 10000: “set only if it does not exist, and expire after 10 seconds”. Only one request can win it.

Twenty requests for one new question. Without the guard, each one calls the provider; with it, one calls and nineteen wait. The bar lengths show the shape; the totals are measured (medians of 5 passes).

Text description of the diagram

The top group shows the run without the guard. It has 20 blue bars, one call to the provider each, and their lengths vary like the provider’s times. The batch ends when the slowest call ends. The bottom group shows the run with the guard. One blue bar is the single call. The 19 grey bars are requests that wait and then read the cached answer. The measured totals are 20 calls in 1.96 seconds without the guard, and 1 call in 1.22 seconds with it.

The guard also made the batch faster. Without it, the twenty answers are ready only when the slowest of twenty provider calls ends. With it, there is one call, plus at most 50 ms while the others check the cache again. The simulated provider does not slow down with load. The real one did: replies took about 7% longer at 16 calls in flight. So a real stampede is worse than this run.

Calls that reached the simulated provider, medians of 5 passes. The guard turned 20 calls into 1.

Text description of the chart

Two bars. Without the stampede guard, 20 requests for the same new question sent 20 calls to the provider, in 1.96 seconds. With the guard, they sent 1 call, in 1.22 seconds.

Larkfield: the guard in code

The guard is in ticket_api/cache.py and answers.py since the last lesson. The lock stores a random token, so that only its owner can remove it:

ticket_api/cache.py (Cache.lock and Cache.unlock, the core)

    async def lock(self, key: str, seconds: float) -> str | None:
        """A token if this caller may compute the value of `key`; None if another does."""
        token = uuid.uuid4().hex
        if await self.client.set(f"{key}:lock", token, nx=True, px=int(seconds * 1000)):
            return token
        return None

    async def unlock(self, key: str, token: str) -> None:
        # Delete the lock only if it is still ours (it may have expired and been taken).
        script = ("if redis.call('get', KEYS[1]) == ARGV[1] then "
                  "return redis.call('del', KEYS[1]) else return 0 end")
        await self.client.eval(script, 1, f"{key}:lock", token)

Why the token? The lock expires after CACHE_LOCK_SECONDS (10 s), so a crashed request cannot hold it forever. But a very slow request may still be working when its lock expires and another request takes it. Without the token, the slow request would then delete the other request’s lock. The small script compares and deletes in one step inside Valkey, so nothing can happen between the two.

The other requests wait in answers.wait_for. Every 50 ms, each one reads the key again, for at most CACHE_LOCK_SECONDS. When the answer appears, it returns it with X-Cache: wait. If the computing request fails, the waiters find nothing. Each one keeps checking until its 10 seconds end, then calls the provider itself. That path is read from the code; the course did not measure it.

When the cache is down: fail open

What should the API do when Valkey does not answer? There are two choices. A service fails closed when it refuses to work without a helper, for example with 503. It fails open when it carries on without the helper. For a cache, the answer follows from the first lesson: a cache is a copy, and the API must give a correct answer without it. So the ticket API fails open.

Fail open: a cache error turns every request into a miss for 5 seconds, then the API tries again. The answers stay correct; the provider does the extra work.

Text description of the diagram

The API is in one of three states. Normally the cache answers. When a cache call fails, or takes longer than CACHE_TIMEOUT (50 ms), the API skips the cache for 5 seconds. During that time every request asks the provider and answers with X-Cache: off. After 5 seconds, the next request tries the cache again. If it answers, the cache is back; if not, the API skips it for another 5 seconds. The note explains the one risk: a change that happens while the cache is unreachable cannot bump the generation.

Fail open is not free. Every request becomes a miss, so the provider gets all of them. In the median pass of the measured run, 214 of 400 questions called the provider with the cache, and all 400 would without it. That is almost twice the use of the quota at the worst moment. Module 5 adds the tools that protect the provider itself: timeouts and a circuit breaker.

The timeout matters as much as the decision. A GET on Valkey took a median of 178 µs in the course team’s measurement. CACHE_TIMEOUT is 50 ms, about 280 times longer. So the API treats a late cache like a cache that is down. The skip for 5 seconds then stops each request from paying the 50 ms again.

A stale answer: when an event is missed

There is one case that fail open does not fix. A new ticket bumps the customer’s generation with one cache command. If the cache does not answer at that moment, the bump is lost. The API logs it and goes on: the ticket itself is saved in PostgreSQL, which is what matters. The author froze the cache on purpose while C-0003 opened a ticket, then let it run again:

x-cache: miss
{"answer":"... Your latest ticket is T-30194 (open): Change my email address.","scope":"customer:C-0003",...}
HTTP/1.1 201 Created
{"ticket_id":"T-500001","customer_id":"C-0003",...
x-cache: hit
{"answer":"... Your latest ticket is T-30194 (open): Change my email address.","scope":"customer:C-0003",...,"cached":true}
WARNING ticket_api: cache invalidation of customer:C-0003 failed: Timeout reading from 127.0.0.1:6379

After the cache came back, C-0003 got her old answer, without T-500001. It is still her own answer: a lost event never sends data to the wrong person. It only keeps an old answer longer. This is the TTL’s second job: it limits a stale answer to 10 minutes. If 10 minutes of staleness is too long for a result, give it a shorter TTL. If no staleness is acceptable, do not cache that result.

Guided practice: a stampede, then a cache that is down

1. Copy the last file of this module from the snapshot end-of-m03:

   macOS / Linux

   cp ../nextia-learning-labs/C21/snapshots/end-of-m03/scripts/stampede.py scripts/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C21\snapshots\end-of-m03\scripts\stampede.py scripts\

2. With the simulator and the API running, send 20 requests for one new question at once: python -m scripts.stampede. Run it twice. Each run asks a question that was never asked before.

3. Stop the API with Ctrl+C. Start it again with the guard off:

   macOS / Linux

   CACHE_STAMPEDE_GUARD=false fastapi dev

   Windows (PowerShell)

   $env:CACHE_STAMPEDE_GUARD = "false"; fastapi dev

4. Run python -m scripts.stampede again, twice.

5. Stop the API. Start it with the default settings: fastapi dev. On Windows, first remove the setting: Remove-Item Env:CACHE_STAMPEDE_GUARD.

6. Stop the cache: docker compose stop cache. Then ask C-0003’s question twice, with the curl command of Keys and invalidation, step 7. Look at the API’s terminal.

7. Run python -m scripts.cache stats.

8. Start the cache: docker compose start cache. Wait 5 seconds, then ask the same question twice again.

Show the expected result

Step 2, the author’s run. One request missed and called the provider; nineteen waited for its result:

20 requests for the same new question: 1.45 s, status codes {200: 20}
Cache: {'miss': 1, 'wait': 19}
Calls that reached the provider: 1

Step 4, with the guard off. Every request missed and called the provider:

20 requests for the same new question: 1.65 s, status codes {200: 20}
Cache: {'miss': 20}
Calls that reached the provider: 20

Your seconds will differ from run to run; the counts stay the same. Step 6: both answers are 200 with x-cache: off. The first one tried the cache and failed. The second one skipped it, so its server-timing has no cache part. The API’s terminal shows one warning:

WARNING ticket_api: cache not available (Error 61 connecting to 127.0.0.1:6379. Connection refused.): working without it

On Windows and Linux, the error number and text differ. Step 7 prints The cache does not answer: … Connection refused.. With Docker: docker compose up -d cache. Step 8: the first answer is x-cache: miss, and the second x-cache: hit. The cache came back empty, because Valkey keeps nothing on disk here. An empty cache serves nothing old, so a restart is always safe for correctness.

Your turn: a lock shorter than the work

The lock and the wait last CACHE_LOCK_SECONDS, 10 seconds by default. One answer from the provider takes about a second. Predict what happens with a lock of 0.3 seconds. How many calls reach the provider? Then try it: start the API with CACHE_LOCK_SECONDS=0.3 fastapi dev, and run python -m scripts.stampede. Afterwards, start the API with the default settings again.

Show the answer

All 20 requests call the provider, as with no guard. In the author’s three runs: Cache: {'miss': 20} and Calls that reached the provider: 20 each time. The waiting requests give up after 0.3 seconds, before the first answer is ready. Then each one calls the provider itself. A lock must last longer than the slowest normal computation. Too long is safer than too short: a long lock only matters when the computing request crashes.

Common mistake

A cache that hangs instead of failing

Symptom (constructed): the API is slow, although the provider is fine. Every answer takes more than 6 seconds, and the API log shows cache not available (Timeout reading from …). The author froze Valkey on purpose with docker compose pause cache. A frozen server accepts the connection but never answers. Then the author started the API with a long timeout, CACHE_TIMEOUT=5, and ran python -m scripts.stampede:

20 requests for the same new question: 6.62 s, status codes {200: 20}
Cache: {'off': 20}
Calls that reached the provider: 20

With the default timeout of 0.05 seconds and the same frozen cache, the same 20 requests took 1.67 s.

Cause: a missing service fails fast: the connection is refused at once. A hung service is worse. Each request waits for the whole timeout before it gives up. Here, every request that arrived together waited 5 seconds for the cache, then about a second for the provider.

Diagnose: read the server-timing header. One request in the author’s run showed db;dur=6.3, cache;dur=5001.8, answer;dur=1110.8: 5 seconds were spent waiting for the cache. A cache time near CACHE_TIMEOUT means the cache does not answer.

Fix:

1. Keep the cache’s timeout short, close to what a slow but healthy cache needs. The project uses 50 ms, about 280 times a normal GET.
2. After a failure, skip the cache for a while, as Cache.available() does for 5 seconds. Then only one request in 5 seconds pays the timeout.
3. Unfreeze the cache with docker compose unpause cache, and start the API with the default settings.

When to use these protections, and what they cost

Use the stampede guard for results that are slow to make and asked by many callers at once: here, answers to popular questions. It costs one more cache command per miss, a few reads while requests wait, and one more setting to choose with care. For a cheap result, a stampede does no harm, and the guard is not worth it.

Fail open is right for any cache that only holds copies. It moves the load to the provider when the cache is down, so the provider needs its own protection. Fail closed is right when the stored data is the only source of a decision. The cache’s timeout should be short, and a failure should pause further tries. A missed invalidation is the last risk. The TTL limits it, so choose each TTL as “the longest time a stale answer is acceptable”.

Knowledge check

Check your understanding

Twenty requests miss the same key at once. With the guard, only one calls the provider. Why was the batch also faster (1.22 s against 1.96 s)?

Show the answer

Without the guard, the batch ends when the slowest of 20 provider calls ends, and the provider’s times vary. With the guard, there is only one call; the other requests return within about 50 ms after it, when they read the cache again. A real provider also gets slower with more calls in flight, which the simulated one does not model. So the gap can be larger in real life.

Check your understanding

Valkey stops for 30 seconds during Black Friday. What do customers see, and what does the provider see?

Show the answer

Customers still get correct answers with status 200, marked X-Cache: off. Answers that were hits now take about a second instead of 14 ms. The provider gets every question instead of about half of them: 400 calls instead of 214 in the measured mix. So the quota runs out sooner, which Module 5’s timeouts and circuit breaker must handle. After the restart, the cache is empty and fills again.

Check your understanding

Predict: a customer opens a ticket while Valkey is frozen for a moment. A minute later, she asks a question. Can she see another customer's data? Can she see old data?

Show the answer

She cannot see another customer’s data: her key still holds her own scope, customer:<id>. She can see old data. The generation bump failed. So her old answer, without the new ticket, stays in use until its TTL ends, at most 10 minutes after the cache stored it. The API logged cache invalidation of customer:… failed. A lost invalidation affects time, never identity.

Recap

• A cache stampede is many misses for one key at once. Without a guard, 20 requests for one new question made 20 provider calls; with the guard, 1 call (medians of 5 passes).
• The guard is a lock per key (SET … NX with a token and a 10-second expiry). One request computes; the others wait and answer with X-Cache: wait. The lock must last longer than the work.
• The cache fails open. When it is down or slower than 50 ms, every request is a miss with X-Cache: off, and the API skips the cache for 5 seconds. Answers stay correct; the provider gets all the load.
• A hung cache is worse than a missing one. A long timeout made 20 answers take 6.62 s instead of 1.67 s.
• A lost invalidation leaves an old answer until its TTL ends, but never another person’s answer.
• Module 3 is complete: your project matches the snapshot end-of-m03, and python -m pytest prints 148 passed, 1 deselected. Next, the slow AI work moves to a queue.

Next: Job lifecycle