Keys and invalidation

What you will be able to do

Build the cache so that a key holds everything that changes the answer, including who asks. Then make a change of the data turn the old answers into misses at once, instead of waiting for them to expire.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The cache plan, hits, misses and TTLs, from Choose cacheable work.
• Async endpoints and why a waiting call must not block the server, from I/O versus CPU work.
• Several API processes, from Horizontal scaling.

What you need on your computer

• The project after Choose cacheable work: python -m pytest prints 137 passed, 1 deselected, and docs/cache-plan.md exists.
• Docker running, with the database and Azurite: docker compose up -d db azurite.

After this lesson, your project matches the snapshot end-of-m03, except for one script of the next lesson. Tested on macOS (Apple silicon); the Windows commands are not tested. The AI provider is the simulated one. The times in the outputs are from the author’s computer while it was busy: your numbers will differ.

A shelf at the pharmacy

A pharmacy keeps a shelf of filled prescriptions, waiting to be picked up. Each bag has a label with the patient’s name, the medicine and the prescription number. A bag labelled only “cough medicine” could go to anyone who asks for cough medicine. That is why the name is on the label.

Sometimes the doctor changes a dose. The pharmacist does not search the shelf for the old bag. She writes a new prescription number, and from now on only bags with the new number are handed out. The old bag stays on the shelf, unused, until the weekly clean-up throws it away.

The label is a cache key, and the new prescription number is a way to make old copies unused. This lesson builds both.

The problem: the wrong person, the wrong time

The last lesson found two ways for a copy to be wrong. Tomás proposed a key with the question text only. With that key, customer C-0022 asks “Where is my parcel?” and receives C-0003’s answer, with C-0003’s ticket in it. That is the wrong person: a leak of private data, which you will see in a real run below.

The plan also showed the wrong time. C-0003 opens a new ticket, but her cached answer still names the old one for up to 10 minutes. A TTL limits how long a wrong answer lives. It does not stop the wrong answer.

The idea: a key holds everything that changes the answer

Ask of each input: “If this changes, may the answer change?” If yes, the input belongs in the key. Two requests that may get different answers must build different keys. Here is one real key of the project, with its parts:

Every part of the key is an input that changes the answer. The scope (red) decides who receives it; the generation (green) decides when it becomes old.

Text description of the diagram

The key has eight parts, separated by colons. The first two say whose key it is and what kind of result it holds. The next three are versions: of all documents together, of the model and of the prompt. Then come the identity scope, customer:C-0003, and her generation, g1. The last part is a hash of the question after the API turns it into lower case with single spaces.

Three parts need a reason:

• The identity scope comes from the request’s caller, never from the question text. A request from the shop’s website for a signed-in customer carries the header X-Customer-ID: C-0003. A request without it comes from the help desk’s own tool, so the caller is staff. The API key proves that the request comes from one of Larkfield’s apps; the header names the caller. In this course, the header stands for a real sign-in, which the authentication course adds. Its lesson File and retrieval access applies the same rule to cache keys and permissions.
• The API normalizes the question, then hashes it. “Where is my parcel?” and “ where IS my parcel?” are the same question, so they get the same key. The hash turns any question into 32 characters, so long questions make short keys.
• The versions make a new prompt, a new model or a changed document produce new keys. No old answer is ever read with the new prompt.

Invalidation by generations, not by deleting

Invalidation means making cached copies unused when the data that they came from changes. The obvious way is to delete them. But to delete C-0003’s answers, the API must first find every key that belongs to her. With many keys, the search is slow, and a request can store a new old answer during the search.

The project uses the pharmacy’s way instead: a generation number per scope. Valkey keeps ta:gen:customer:C-0003, starting at 0. Each answer key contains the current value. A new ticket or message of C-0003 adds 1, with one INCR command. Here is a tiny example, with every value visible:

Step	Event	ta:gen:customer:C-0003	Key ends with	Result
1	C-0003 asks “Where is my parcel?”	0	:g0:cc3ce500…	miss: stored, “latest ticket T-30194”
2	She asks again	0	:g0:cc3ce500…	hit
3	She opens ticket T-500001	0 → 1	–	–
4	She asks again	1	:g1:cc3ce500…	miss: stored, “latest ticket T-500001”

At step 4, the key changed, so the old answer is never read again. Nobody deleted it; its TTL removes it within 10 minutes. The documents work the same way. The key holds a fingerprint of all documents, so any change of any document makes every answer key new. The whole path of one question, with the event that changes it:

The life of a cached answer: build the key, hit or miss, store with a TTL. A change of the data makes the next key a new one. The lock and the 'off' path are the next lesson's topic.

Text description of the diagram

The main path runs left to right. The API first reads the context from PostgreSQL, because the key needs the documents’ fingerprint. Then it builds the key and asks Valkey for it. A hit answers at once. A miss takes a lock, calls the provider, and stores the answer with a TTL of 600 seconds. The amber event box is the invalidation. A new ticket or message adds 1 to the customer’s generation, and a document change changes the fingerprint. Both make the next key a new one, and the old key expires by itself. The red path, the lock and the wait belong to the next lesson.

So a TTL still has a job: it is the cleaner. It removes old keys, and it is the upper limit on staleness when an event is missed. The project uses 10 minutes for answers (CACHE_TTL_SECONDS=600) and 24 hours for embeddings, which are never wrong.

Why a shared cache, and why Valkey

In Horizontal scaling, the API ran as four processes. A Python dictionary in each process would be four separate caches. Each one would see a quarter of the questions, so the hit rate would fall. Worse, a new ticket handled by process 1 would bump the generation in process 1 only. Processes 2 to 4 would keep serving the old answer.

So the cache is a separate server that every process shares. The project uses Valkey 9.1, an open-source fork of Redis under the BSD licence. It speaks the same protocol, so the client is the redis Python package. The course team measured both servers through the same client. A GET took a median of 178.4 µs on Valkey and 177.7 µs on Redis 8.8: the same speed. The choice is the licence. Redis 8 comes under the AGPLv3, RSALv2 or SSPL licences, and Valkey under BSD.

The client is the async one, redis.asyncio. A normal, blocking cache call inside an async endpoint would stall the whole server, the mistake of I/O versus CPU work.

Larkfield: the key and the events in code

Amira’s ticket_api/cache.py builds the key from its parts:

ticket_api/cache.py

def answer_key(scope: str, generation: int, question: str, versions: dict) -> str:
    """The key of a cached answer. Everything that changes the answer is in it."""
    v = ":".join(f"{k}={versions[k]}" for k in sorted(versions))
    return f"{PREFIX}:answer:v1:{v}:{scope}:g{generation}:{digest(normalize_question(question))}"

ticket_api/answers.py takes the scope from the header, never from the body:

ticket_api/answers.py (parts)

def scope_of(customer_id: str | None) -> str:
    return f"customer:{customer_id}" if customer_id else "staff"

    # in answer_question():
    scope = scope_of(x_customer_id)
    with timer("db"):
        ctx = await run_in_threadpool(context, db, q.question, x_customer_id)
    ...
        versions = {"prompt": ai.ANSWER_PROMPT, "model": ai.CHAT_MODEL, "docs": ctx["docs_version"]}
        with timer("cache"):
            key = answer_key(scope, await cache.generation(scope), q.question, versions)
            result = await cache.get(key)

The context query runs before the cache. It checks that the customer exists, finds the documents that the caller may read, and computes the documents’ fingerprint for the key. So a hit still costs this database work. That is why a hit took 13.8 ms and not 1 ms: the cache saves the provider call, not the database work before it.

Two endpoints change the data that answers depend on, so each one bumps the customer’s generation after it saves:

ticket_api/intake.py (create_ticket, at the end)

    if cache is not None:
        # The customer's latest tickets are part of their answers: those answers are old now.
        await cache.invalidate(f"customer:{new.customer_id}")

add_message in ticket_api/tickets.py does the same for the customer of the ticket. The embedding cache in intake.py needs no scope and no generation: its key is the model, the dimensions and a hash of the exact text.

Guided practice: build the cache and watch the keys

You copy this lesson’s files from the snapshot, then run the cache and read its keys. The copy adds or replaces 22 files, including the project’s README.md. If you changed one of them in an earlier “Your turn”, keep a copy of your version first.

1. In ~/projects/ticket-api, with the virtual environment active, copy the files:

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m03
   for f in .env.example compose.yaml pyproject.toml requirements.txt requirements-run.txt \
     requirements-lock.txt contract/openapi.json simulator/model.py ticket_api/ai.py \
     ticket_api/answers.py ticket_api/cache.py ticket_api/config.py ticket_api/intake.py \
     ticket_api/main.py ticket_api/repository.py ticket_api/tickets.py scripts/cache.py \
     scripts/leak_demo.py loadtest/questions.py tests/conftest.py tests/test_cache.py README.md; do
     cp "$S/$f" "$f"
   done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m03"
   $files = '.env.example', 'compose.yaml', 'pyproject.toml', 'requirements.txt', 'requirements-run.txt',
     'requirements-lock.txt', 'contract\openapi.json', 'simulator\model.py', 'ticket_api\ai.py',
     'ticket_api\answers.py', 'ticket_api\cache.py', 'ticket_api\config.py', 'ticket_api\intake.py',
     'ticket_api\main.py', 'ticket_api\repository.py', 'ticket_api\tickets.py', 'scripts\cache.py',
     'scripts\leak_demo.py', 'loadtest\questions.py', 'tests\conftest.py', 'tests\test_cache.py', 'README.md'
   foreach ($f in $files) { Copy-Item "$S\$f" $f }

2. Add the cache’s two lines to your .env. The API uses database 0 of Valkey; the tests use database 15 and empty it:

   .env (add at the end)

   CACHE_URL=redis://127.0.0.1:6379/0
   TEST_CACHE_URL=redis://127.0.0.1:6379/15

3. Install the new package, redis 8.1.0: python -m pip install -r requirements.txt.

4. Start Valkey: docker compose up -d cache. It listens on 127.0.0.1:6379 and keeps nothing on disk.

5. Load the data again, so that new tickets start at T-500001: python -m scripts.load --reset. Then run the tests: python -m pytest. Expect 148 passed, 1 deselected.

6. Start the simulated provider in a second terminal (python -m simulator), and the API in a third (fastapi dev). Both need the virtual environment.

7. Ask the same question twice as customer C-0003:

   macOS / Linux

   curl -i -X POST http://127.0.0.1:8000/v1/answers -H "Content-Type: application/json" \
     -H "X-Customer-ID: C-0003" -d '{"question": "Where is my parcel and how do I track it?"}'

   Windows (PowerShell)

   $body = '{"question": "Where is my parcel and how do I track it?"}'
   $r = Invoke-WebRequest -Method Post -Uri http://127.0.0.1:8000/v1/answers -ContentType "application/json" -Headers @{ "X-Customer-ID" = "C-0003" } -Body $body
   $r.Headers["X-Cache"]; $r.Headers["Server-Timing"]; $r.Content

8. List the keys: python -m scripts.cache keys.

9. Ask the same question as C-0022, and once without the header (as staff). List the keys again.

10. Open a new ticket for C-0003, then ask her question again:

    curl -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" \
      -d '{"customer_id": "C-0003", "subject": "Parcel not delivered", "body": "My parcel for order LK-100200 did not arrive. Where is it?"}'

    On Windows, use Invoke-WebRequest as in step 7, without the header.

11. See her generation: docker compose exec cache valkey-cli get ta:gen:customer:C-0003.

Show the expected result

Step 7, the author’s run (your times will differ). The first answer is a miss, with the provider’s time in server-timing. The second is a hit, with no answer part:

HTTP/1.1 200 OK
...
x-cache: miss
server-timing: db;dur=11.0, cache;dur=3.3, answer;dur=941.1

{"answer":"The document \"Help centre: frequently asked questions\" has the details. Your latest ticket is T-30194 (open): Change my email address.","scope":"customer:C-0003","sources":["help-centre-faq","delivery-policy"],"cached":false}

HTTP/1.1 200 OK
...
x-cache: hit
server-timing: db;dur=9.2, cache;dur=1.5

{"answer":"The document \"Help centre: frequently asked questions\" has the details. Your latest ticket is T-30194 (open): Change my email address.","scope":"customer:C-0003","sources":["help-centre-faq","delivery-policy"],"cached":true}

The answer is short and simple, because the provider is the simulated one: it names the first document and the customer’s latest ticket. Step 8, then step 9 with three keys, one per scope:

ta:answer:v1:docs=75cf862216c3:model=chat-small:prompt=answer-v1:customer:C-0003:g0:cc3ce5009832630860551d4c6b1eb7b3  (598 s left)
ta:answer:v1:docs=75cf862216c3:model=chat-small:prompt=answer-v1:customer:C-0022:g0:cc3ce5009832630860551d4c6b1eb7b3  (599 s left)
ta:answer:v1:docs=75cf862216c3:model=chat-small:prompt=answer-v1:staff:g0:cc3ce5009832630860551d4c6b1eb7b3  (600 s left)

C-0022’s answer names her own ticket, T-30171, a resolved question about a garden hose. The staff answer has no ticket and uses a staff document, “Courier claims for damaged or lost parcels”. In step 10, the new ticket is T-500001. The next answer is a miss that names it:

x-cache: miss
{"answer":"The document \"Help centre: frequently asked questions\" has the details. Your latest ticket is T-500001 (open): Parcel not delivered.","scope":"customer:C-0003",...,"cached":false}

Step 11 prints "1": one new ticket, one bump. The key list now has C-0003’s g0 key and her new g1 key. Nothing deleted the g0 key; it is never read again and expires. The key list also shows ta:embedding:v1:embed-small:384:… with 24 hours left, from the new ticket’s embedding, and ta:gen:customer:C-0003 with no expiry.

Completion check: one customer’s answer never reaches another

The module’s completion check asks you to show that one user’s private result cannot be served to another. scripts.leak_demo runs the API inside the script and asks the same question as C-0003, then as C-0022. It does this twice. The first time, it swaps in a wrong key, on purpose: the question only. The second time, it uses the project’s key. Run it after an empty cache, and hide the API’s log lines:

python -m scripts.cache flush
python -m scripts.leak_demo 2>/dev/null

On Windows, write 2>$null instead of 2>/dev/null. The output:

With the WRONG key (the question only):
  C-0003 asks -> cache miss: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30194 (open): Change my email address.
  C-0022 asks -> cache hit: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30194 (open): Change my email address.
With the project's key (identity scope in the key):
  C-0003 asks -> cache miss: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30194 (open): Change my email address.
  C-0022 asks -> cache miss: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30171 (resolved): Question about the garden hose (25 m).

With the wrong key, C-0022 got a hit, and the hit told her about C-0003’s ticket T-30194. With the project’s key, her request missed and she got her own ticket, T-30171. If you ran step 10 first, C-0003’s ticket is T-500001 in both blocks; the leak looks the same. The test suite checks the same rule on every run:

python -m pytest tests/test_cache.py -k "never_served or without_the_scope" -v

Expect 2 passed, 9 deselected. test_one_customers_answer_is_never_served_to_another checks that C-0022’s answer is a miss and contains none of C-0003’s ticket numbers. test_a_key_without_the_scope_leaks builds the wrong key and checks that the leak happens, so the demo cannot quietly stop showing it.

Your turn: keys for a document-answering service

Design the keys for a different service, on paper. Use the four questions of the last lesson and the parts of the key diagram.

A city council runs a service that answers questions about its documents. Residents ask about public documents, and an answer may name the resident’s own open applications. Council employees may also see internal documents. Answers come in English or Welsh, as the caller chooses. The documents change a few times a week, and the prompt has a version.

Write the key of a cached answer, part by part. Then list the events that make an answer old, and choose a TTL. Then decide: does the embedding of a document’s paragraph need a scope?

Show one answer

One good key: council:answer:v1:docs=<fingerprint>:model=<model>:prompt=<version>:lang=<en|cy>:<scope>:g<generation>:<hash of the normalized question>. The scope is resident:<id> or employee, taken from the sign-in and never from the question. The language is in the key, because it changes the answer.

Events: a resident’s new or changed application bumps gen:resident:<id>. A document change changes the fingerprint, which makes every answer new. A new prompt or model changes its version. A TTL of a few minutes to an hour is a limit for a missed event. Shorter is safer for answers that name applications.

The embedding of a paragraph needs no identity scope: it depends only on the text and the model. But an internal paragraph must never be searched for a resident. That rule belongs in the search’s filter, not in the embedding’s key.

Common mistake

Tomás: 'cache the answers by question text'

Symptom (constructed): the hit rate looks excellent, 97% in the last lesson’s prediction. A customer then reports that the help desk told her about a ticket she never opened. The leak demo above shows it: C-0022 received C-0003’s ticket T-30194.

Cause: the key held only the question. Two callers with different rights built the same key, so the second one read the first one’s copy. The permission rules shaped the answer when it was made, for the first caller. A copy carries no rules: whoever builds the same key reads it.

Diagnose:

1. List the keys with python -m scripts.cache keys. A key with no customer: or staff part is the warning sign.
2. Ask of the result: “May two callers get different answers?” Here, yes: each customer’s answer names their own tickets, and staff answers use staff documents.
3. Look at the response: the scope field of a cached answer must equal the caller’s own scope. In the demo, a hit for C-0022 would carry customer:C-0003.

Fix: put the scope into the key, taken from the caller and never from the request body. Keep a test that fails when two callers share a copy, as tests/test_cache.py does. Accept the lower hit rate, 46.5% instead of 97%: those extra hits were wrong answers.

When to use generations, and what they cost

Use a generation when many keys depend on one piece of data, such as all of a customer’s answers. Use a version in the key when the change is global: a new prompt, a new model or a document change. Delete a key directly only when you know exactly which one is wrong.

Generations cost one extra cache read per request, to learn the current number. Old keys use memory until their TTL ends. The project’s Valkey has a limit of 256 MB and drops the least recently used keys when it is full. An event that the API misses still leaves a stale answer until the TTL, which is the next lesson’s topic.

See it end to endQueue and cache for an open-data APICase study

The case study adds a cache to an open-data API under load and chooses its keys and TTLs with the same questions.

Knowledge check

Check your understanding

Why does the API take the scope from the X-Customer-ID header, and not from a field in the JSON body or from the question?

Show the answer

The scope must name the caller, and only the caller’s identity can decide what they may see. In this course, the header stands for the shop website’s sign-in, which the authentication course makes real. A question text or a body field is just data that any caller can write. If a caller could set their own scope, they could build another customer’s key and read that customer’s cached answer.

Check your understanding

A document's text changes. How do the cached answers become old, without one delete?

Show the answer

Every answer key holds docs=<fingerprint>, a short hash of all documents’ IDs, versions and checksums. When any document changes, the fingerprint changes, so every request builds a new key and misses. The old keys are never read again and expire when their 10-minute TTL ends.

Check your understanding

Predict: the API runs as 4 processes, each with its own in-memory cache. C-0003 opens a ticket through process 1. What can she see when her next question goes to process 3?

Show the answer

An old answer. Process 1 bumped the generation in its own memory only. Process 3 still has generation 0 and its old copy, so it can serve the answer with her previous ticket until its TTL ends. A shared cache such as Valkey keeps one generation for all processes, so every process builds the new key.

Recap

• A key holds everything that changes the answer. Here, that means the versions of the documents, the model and the prompt, the caller’s identity scope, a generation, and the question’s hash.
• The scope comes from the caller (X-Customer-ID, or staff without it), never from the question.
• Invalidation by generations: a new ticket or message runs one INCR, the next key is new, and the old key expires by its TTL. A document change changes the fingerprint.
• The cache is shared (Valkey, BSD licence, the same speed as Redis) and async, so all processes see the same copies and generations.
• Completion check: with the wrong key, C-0022 received C-0003’s ticket T-30194. With the project’s key, she got her own, T-30171. tests/test_cache.py checks it on every run.

Next: Failure and stampede behaviour