Choose cacheable work

What you will be able to do

Decide which results of the ticket API to keep a copy of, who may receive each copy, and how long it stays correct. Make this plan before you write any cache code. A copy that goes to the wrong person, or stays after it is wrong, is worse than no copy.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The AI work of a new ticket and the simulated AI provider, from Run the AI workload on your computer.
• Percentiles: p50 is the median, and p95 is the time that 95% of requests stay under, from Latency and capacity.
• The provider’s quota of 100 chat requests per minute, from Dependency limits.

What you need on your computer

• The project ~/projects/ticket-api after Module 2. python -m pytest prints 137 passed, 1 deselected. If your project is broken, copy the snapshot end-of-m02, as the snapshots README explains.
• The course files in ~/projects/nextia-learning-labs. Get the newest ones with git -C ~/projects/nextia-learning-labs pull.

This lesson adds one file, docs/cache-plan.md, and changes no code. The next lesson builds the cache. The course team measured the times in this module on an Apple M4 computer with nothing else busy. They used the simulated provider with its quota switched off. Your times will differ, but the shape stays the same.

A card at the front desk

Think of the front desk of a large shop. Every day, about fifty people ask “What are your opening hours?” At first, the clerk looks it up each time. Then she writes the answer on a card and keeps it on the desk. Now she reads the card aloud in two seconds.

Another question is “Where is my parcel?” The clerk looks it up for Mrs Diaz and writes “Parcel 4471, out for delivery” on a card. Then Mr Lee asks the same question. If she reads him the card, he gets Mrs Diaz’s parcel number. The question was the same, but the right answer was not.

On a public holiday, the shop opens late. The opening-hours card is now wrong, and nobody changed it. So every card raises three questions. Is the answer worth writing down? Who may hear it? When does it become wrong?

The problem: the same slow answer, made again and again

This module adds a new feature to the ticket API: customers and staff can ask the help desk a question. POST /v1/answers takes a question and returns a short answer. The API reads the documents that the caller may see and, for a customer, their latest tickets. Then it sends all of it to the AI provider, which writes the answer.

Each answer is one chat call to the provider. In the measured run, an answer from the provider took a median of 1,125 ms. The provider is the simulated one, whose times come from real recorded replies. On Black Friday, Omar expects many people to ask the same dozen questions. Without a copy, 400 questions mean 400 calls. Each call costs time, money and a part of the quota of 100 chat requests per minute.

So the help desk has the clerk’s problem: it makes the same slow answer again and again.

The idea: a cache

A cache is a fast store that keeps a copy of a result. The next request with the same inputs reads the copy instead of making the result again. The name comes from the French word for a hiding place: a store kept close at hand.

• A cache hit is a request that finds its copy. A cache miss finds none: it makes the result and usually stores a copy for the next request.
• The hit rate is the share of requests that are hits.
• A key is the label of a copy. Two requests share a copy only when they build the same key.
• A TTL (time to live) is how long a copy may stay. After that, the cache deletes it.

The card on the desk is a cache entry. The question that the clerk hears is the key, and a note “until the end of the month” is a TTL. A cache is always a copy. The database stays the source of truth, and the API must give a correct answer when the copy is missing.

A tiny example: ten questions, two keys

Ten questions arrive in this order. Each answer from the provider takes about 1,125 ms, and a hit takes about 14 ms. The first key is the question and who asks, because a customer’s answer names their own latest ticket.

#	Who asks	Question	Key: question + who	Key: question only
1	C-0003	Where is my parcel?	miss	miss
2	C-0022	Where is my parcel?	miss	hit: C-0003’s answer
3	C-0003	Where is my parcel?	hit	hit
4	staff	How do I return a damaged item?	miss	miss
5	C-0022	How do I return a damaged item?	miss	hit: the staff answer
6	C-0003	Where is my parcel?	hit	hit
7	C-0022	Where is my parcel?	hit	hit: C-0003’s answer
8	staff	How do I return a damaged item?	hit	hit
9	C-0003	How do I return a damaged item?	miss	hit: the staff answer
10	C-0022	Where is my parcel?	hit	hit: C-0003’s answer

With the key “question + who”, 5 of 10 are hits: a hit rate of 50%. The ten requests take about 5 × 1,125 + 5 × 14 = 5,695 ms in total, against 11,250 ms without a cache. With “question only”, 8 of 10 are hits. That looks better, but five of those eight hits give a person someone else’s answer. A higher hit rate is worth nothing when the answers are wrong.

What a cache can buy, and what it cannot

The course team measured the finished cache of this module with 400 questions per pass, 8 at a time, in 5 passes. The questions came from the help desk’s 12 common questions. The common ones came more often: question number k had the weight 1/k. Nine of ten askers were customers, from 40 practice customers. The provider was the simulated one with its quota off, so that the quota did not hide the cache’s effect.

The hit rate was 46.5%. A hit took a median of 13.8 ms and a miss 1,125 ms: about 80 times longer. Now look at the whole run:

• Without the cache
• With the cache

All requests of the question workload, medians of 5 passes (simulated provider, quota off). The median fell by about 200 ms; the p95 fell by 50 ms.

Text description of the chart

Four bars. The median (p50) answer time was 1,115 ms without the cache and 920 ms with it. The p95 was 1,537 ms without the cache and 1,487 ms with it. Both pairs of bars are almost the same height.

Almost half of the answers became 80 times faster. Yet the median fell only from 1,115 to 920 ms, and the p95 barely moved: from 1,537 to 1,487 ms. Sort the 400 answers of one pass by time, and the reason is clear:

With 46.5% hits, the middle answer and the slow answers are all misses. A cache makes hits fast; it does not make a miss faster.

Text description of the diagram

Each strip stands for 400 answers, the fastest on the left. With the cache, the hits fill the left 46.5% of the strip. The p50 is the answer in the middle, at 50%. That position is past the hits, so the p50 is one of the fastest misses: 920 ms. The p95 is the answer at 95%, deep in the misses: 1,487 ms. Without the cache, the whole strip is misses, with a p50 of 1,115 ms and a p95 of 1,537 ms.

A percentile picks one position in the sorted list. With fewer than 50% hits, the middle position is still a miss. The p95 stays a miss until more than 95% of requests hit. So Omar’s question, “What will customers see?”, has an honest answer. About half of them get an answer in 14 ms instead of a second, and the slowest answers stay as slow as before.

What the cache really saves is work. In the median pass, only 214 of 400 questions called the provider, instead of all 400. On Black Friday, that means quota and money. The slow tail needs other tools: a queue in Module 4, and timeouts in Module 5.

Sort the results: what, for whom, how long

Amira does not cache everything. Every copy needs a rule for its key, a rule for when it becomes old, and memory. So she asks four questions about each result of the API:

Four questions for each result. The first two decide if a copy is worth it. The last two decide its key and its lifetime.

Text description of the diagram

The first two questions can end the flow with “do not cache”. A result that is cheap to make gains nothing from a copy. A result that never comes back with the same inputs leaves a copy that nobody reads. The last two questions do not stop the flow; they add to the key. “Who may see it?” adds the caller’s identity scope. “What makes it old?” adds a version or a generation, and sets a TTL as a limit.

The identity scope is the group of callers who may receive the same copy. In this project, it is customer:C-0022 for one customer, or staff for the help desk’s staff. Amira’s answers for Larkfield’s results are in docs/cache-plan.md:

Result	Cache it?	Scope in the key	TTL	Made old by
Ticket categories	no: fixed in the code, costs nothing	–	–	a new release
A document’s text	no: reading it is fast	–	–	a new version
An embedding of a ticket’s text	yes	none needed	24 hours	never wrong; a new model gets a new key
A ticket’s classification and draft reply	no: made once and stored in PostgreSQL	–	–	–
An answer to a question	yes	customer:<id> or staff	10 minutes	a new ticket or message of that customer; any document change
Similar tickets of a ticket	not now: the search takes milliseconds	–	–	–

Two rows need a reason. The embedding of a text depends only on the text and the model. Only a caller who already has the text can build its key, so the copy reveals nothing new, and no scope is needed. The classification is not a cache at all. It is made once per ticket and saved in the ai_runs table, as the record of what the AI said. A record is the source of truth; a cache is a copy that you can lose.

“Why not cache everything?” Sam asks. Because each copy can be wrong in two ways: for the wrong person, and at the wrong time. Each one also costs memory. And the cache becomes one more service that can fail, which is the topic of Failure and stampede behaviour.

Guided practice: read the plan and predict the hit rate

Before you build a cache, predict how often it can hit. The first ask of each key is always a miss, and every repeat could be a hit. So the best possible hit rate is 1 − (different keys ÷ requests). You do not need the API for this.

1. In ~/projects/ticket-api, with the virtual environment active, copy the plan from the snapshot end-of-m03:

   macOS / Linux

   cp ../nextia-learning-labs/C21/snapshots/end-of-m03/docs/cache-plan.md docs/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C21\snapshots\end-of-m03\docs\cache-plan.md docs\

2. Open docs/cache-plan.md in VS Code. Read the row “An answer to a question”, and the three rules at the end of the file.

3. Make a file hitrate.py in ~/projects, next to the project folder and not in it. It is a tool for this lesson only. Paste this code. It builds the same question mix as the measured run, with the same seeds:

   ~/projects/hitrate.py

   """How often could a cache answer, for the question mix of this course? (No API needed.)"""
   import random
   
   QUESTIONS = [
       "Where is my parcel and how do I track it?", "How do I return a damaged item?",
       "When will I get my refund?", "How do I reset my password?", "Can I change or cancel my order?",
       "How does the warranty work?", "Do you deliver large items like sheds?", "How do gift cards work?",
       "What are your opening hours?", "How do I change my email address?",
       "Is my garden furniture covered if it breaks?", "How do I use a discount code?",
   ]
   WEIGHTS = [1 / (k + 1) for k in range(len(QUESTIONS))]  # question k is asked with weight 1/k
   
   for seed in range(1, 6):
       rng = random.Random(seed)
       asks = []
       for _ in range(400):
           question = rng.choices(QUESTIONS, weights=WEIGHTS)[0]
           who = f"customer:C-{rng.randint(1, 40):04d}" if rng.random() < 0.9 else "staff"
           asks.append((question, who))
       # The first ask of each key is a miss; every repeat could be a hit.
       by_question = 1 - len({q for q, _ in asks}) / len(asks)
       by_question_and_who = 1 - len(set(asks)) / len(asks)
       print(f"pass {seed}: key = question only {by_question:.1%} | key = question + who {by_question_and_who:.1%}")

4. Predict the two hit rates. Then run the file from the project folder: python ../hitrate.py.

Show the expected result

pass 1: key = question only 97.0% | key = question + who 45.2%
pass 2: key = question only 97.0% | key = question + who 48.0%
pass 3: key = question only 97.0% | key = question + who 47.0%
pass 4: key = question only 97.0% | key = question + who 46.5%
pass 5: key = question only 97.0% | key = question + who 44.5%

The numbers are the same on every computer, because the seeds fix the mix. The median pass gives 46.5%, the hit rate that the real cache reached. So the measured cache hit every time that it could. The key “question only” would reach 97%: with only 12 questions, only 12 of 400 asks are first asks. That key is wrong, as the tiny example showed. The gap between 97% and 46.5% is the price of correct keys.

Your turn: change the crowd

Change one line of hitrate.py, and predict before you run it.

1. Black Friday brings more customers. Change rng.randint(1, 40) to rng.randint(1, 400): 400 different customers ask the 400 questions. Does the hit rate with “question + who” go up or down?
2. Put 40 back. Now the same 40 customers ask ten times as often: change range(400) to range(4000).

Show the answer

1. It falls to between 12.7% and 17.5% in the five passes. More people means more different keys, and the first ask of each key is a miss. The key “question only” stays at 97%, because it ignores who asks.
2. It rises to about 88%, because the same keys come back again and again. In the real cache, the TTL and the next lesson’s generations limit this. A copy lives at most 10 minutes, and a customer’s new ticket makes their copies old. So write the mix down with every hit rate that you report. A hit rate belongs to the workload, not to the cache alone.

Common mistake

The plan forgets an input of the answer

Symptom (constructed): a customer opens a new ticket, then asks “Where is my parcel?” The answer still names her previous ticket. The author made this case on purpose, with the cache’s invalidation switched off, so only the 10-minute TTL could end the copy.

Constructed: no invalidation (the TTL only)
  before: cache miss: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30194 (open): Change my email address.
  new ticket T-500001 for C-0003
  after:  cache hit: The document "Help centre: frequently asked questions" has the details. Your latest ticket is T-30194 (open): Change my email address.

Cause: the answer depends on the customer’s latest tickets, but the plan did not say so. The copy was made before T-500001 existed, and it stays for up to 10 minutes.

Diagnose: write down every input of the result. Here they are the question, the documents that the caller may read, the caller’s tickets, the prompt and the model. For each input, ask: “Which event changes it?” An input with no event in the column “Made old by” is a stale answer that waits to happen.

Fix: add the event to the plan, as the row in docs/cache-plan.md does: “a new ticket or message of that customer”. The next lesson turns that event into code. A shorter TTL alone is not a fix. It only makes the wrong answer live a shorter time.

When to cache, and what it costs

Cache a result when it is slow or costly to make and the same inputs come back often. You must also be able to name every input that changes it. Do not cache a result that is cheap, unique to each request, or the record of what happened. A record belongs in the database.

A cache costs memory, one more service to run, and two new ways to be wrong. Most of all, it costs care: every key needs a scope and a way to become old. In return, it saves calls to the provider, quota and money, and it makes hits very fast. It does not make the slowest requests faster.

Knowledge check

Check your understanding

The cache made 46.5% of answers about 80 times faster. Why did the p95 fall only from 1,537 to 1,487 ms?

Show the answer

The p95 is the answer at the 95% position when all answers are sorted by time. Hits fill only the fastest 46.5% of the list, so the 95% position is still a miss. A cache does not make a miss faster: the miss still calls the provider. The p95 can only fall a lot when more than 95% of requests hit, or when the misses themselves become faster.

Check your understanding

Tomás says: 'Cache the answers by question text. The hit rate would be 97% instead of 46.5%.' What do you answer?

Show the answer

The extra hits come from serving one person’s answer to other people. In this API, a customer’s answer names their own latest ticket, and a staff answer can use staff-only documents. With the question as the only key, a customer would receive another customer’s ticket, or a staff answer. The scope must be in the key, and the lower hit rate is the price of correct answers.

Check your understanding

Predict: the help desk adds a result 'the number of open tickets of a customer'. Its query takes 2 ms. Should it go into the cache?

Show the answer

No. The first of the four questions stops it: a 2 ms query is cheap, so a copy saves almost nothing. The copy would still need the scope customer:<id>, and an event that makes it old: every new or closed ticket. The cost of correct keys is larger than the gain.

Recap

• A cache keeps a copy of a result. A hit finds the copy, a miss makes the result, and a TTL limits how long a copy lives.
• Ask four questions of each result: is it costly, does it repeat, who may see it, and what makes it old.
• In this API, an answer depends on who asks, so its key holds the identity scope: customer:<id> or staff. An embedding needs no scope.
• Measured: a hit took 13.8 ms and a miss 1,125 ms, with a hit rate of 46.5%. The median fell from 1,115 to 920 ms; the p95 only from 1,537 to 1,487 ms, because the slow answers are misses.
• The cache saves provider calls: in the median pass, 214 of 400 questions called the provider. The key “question only” would hit 97%, by serving wrong answers.
• Every input of a result needs an event that makes its copy old. docs/cache-plan.md is the plan, and the next lesson builds it.

Next: Keys and invalidation