Caching and batching

What you will be able to do

Stop paying again for work that the system has already done. You find the reusable work in the assistant and decide how long a saved answer stays right. Then you measure what a cache hit and a batched call really save. The answer cache has been part of the design since Module 2; this lesson sets its hit share and its value.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The answer cache and its key (tenant, audience, documents version, question), from Data and identity flows: A tiny example.
• Hits, misses and why a cache does not make a miss faster, from Choose cacheable work in Scaling APIs and AI Workloads.
• The option matrix, from Model routing.
• The provider quota and the busiest minute, from Workload arithmetic: Check the provider’s limit.

What you need on your computer

• Your folder design-pack from the last lesson: options.py and the new design.toml copied in, and python -m pytest -q printing 35 passed.

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A card box and a postman

A school librarian hears the same question every week: “When is the library open in the holidays?” The first time, she opens the calendar folder and reads for a few minutes. Then she writes the answer on a card and keeps it at the desk. The next time, the answer takes two seconds. When the calendar changes, she throws the card away, or it would give a wrong answer quickly.

The postman on her street works in a different way. He does not drive to the street once for each letter. He takes all the street’s letters in one trip. Each letter still needs its stamp, but the trips are fewer.

The card box is a cache. The postman’s bag is a batch. Both save work that the system would otherwise repeat; they save different things.

The problem: the same work, again and again

Every question to the assistant pays for the full path. It embeds the question (0.34 s), searches and reranks (0.32 s), and asks chat-small to write the answer (1.52 s). That is 2.18 seconds, and about US$0.000175 of tokens and retries. Suppose that agents at Larkfield ask “How many days do I have to return an item?” twenty times a day. The assistant does the same work twenty times.

At the base volume, that is a small waste. In the high case, it becomes a limit. Without a cache, the high case’s busiest minute needs 148.8 chat calls, and the provider’s quota is 100 a minute. The provider would refuse part of those calls.

Ingestion has a second kind of waste. A new document is cut into chunks, and each chunk needs an embedding. Sent one chunk per call, the course team’s run needed 2.81 seconds per document. Most of that time was the trips, not the work.

The idea: caching

A cache keeps the result of work so that the system can reuse it. A request first looks in the cache. A hit finds a saved result and returns it. A miss does the full work and then saves the result. The hit share, or hit rate, is the share of requests that are hits. The word comes from the French cacher, “to hide”: a store that is out of sight.

Each saved result is found by a key. The key must hold every input that changes the result. Module 2 chose the answer cache’s key: tenant, audience, documents version and question. Because the documents version is in the key, a new version of a policy makes the old answers stop matching. Nobody needs to delete them.

That answers the question of freshness: how long can a saved answer stay? As long as every input in its key is the same. Two inputs are not in the documents: the prompt and the model. So the key also needs the prompt and model versions, or the release of a new prompt must empty the cache. A time limit, for example a week, adds a safety net for anything that the key misses.

Find the reusable work

Kwame asks two questions about each kind of work in the design. Does the same input come again? Until when is the old result right?

Work	When does the same input come again?	The result stays right while…	Where it is kept
An answer	the same question, tenant and audience	the documents, prompt and model versions are the same	the answer cache in PostgreSQL
Chunk embeddings and a document’s summary	never: they are made once per document version	the document version is the same	the chunk and document rows; they already work like a cache
A question’s embedding	the same question text	the embedding model is the same	not worth a cache: all of them cost US$0.01 a month
The start of a long prompt	the same instructions and passages	the provider decides	the provider’s own prompt cache

Only the first row is a real decision. The second is already in the design, and the third saves less than a cent.

A tiny example: ten questions in a morning

Ten questions reach the assistant in a morning. Three of them repeat an earlier question with the same key. So 7 are misses and 3 are hits.

• Time, added up: 7 × 2.18 + 3 × 0.014 = 15.26 + 0.04 = 15.30 s, against 21.80 s without a cache.
• Model cost: 7 × US$0.000171 = US$0.001197, against US$0.00171. Each hit saves one chat call.
• The median wait: sort the ten times: 0.014, 0.014, 0.014, 2.18, 2.18 … The middle of the list is still a miss: 2.18 s.

The hits are fast, but the typical question does not feel faster. The median changes only when more than half of the questions are hits. The scaling course saw the same thing in Choose cacheable work.

The answer cache on the request path. A hit skips three steps and the model call; a miss pays for all of them and then saves the answer.

Text description of the diagram

A question arrives with its tenant, audience and documents version. The API looks in the answer cache in PostgreSQL for the same key. On a hit, it returns the saved answer in about 0.014 s, without a model call. (That hit time was measured in the scaling course, with Valkey as the cache.) On a miss, it embeds the question in 0.34 s and searches in 0.32 s. Then chat-small answers in 1.52 s for US$0.000171. It saves the answer under the key and returns it: 2.18 s in all.

Larkfield: what the cache is worth

Nobody knows how often Larkfield’s questions repeat: the assistant does not run yet. So the hit share is an assumption, with a range: 0% in the low case, 10% in the base case, 30% in the high case. The scaling course’s constructed ticket mix reached 46.5% with a correct key; the real shops will be different. The course measured the time and the money that one hit saves. Kwame keeps the two kinds of number apart.

Guided practice: price the cache

1. In your folder design-pack, with the environment active, run the options again and read the second table:

   python -m costmodel options

   Caching answers: the hit share is an ASSUMPTION; the time saved per hit is measured
     a miss takes 2.18 s (embed + search + answer, medians); a hit takes 0.014 s
      hit share  US$/month    saved  median wait s
             0%      52.67     0.00           2.18
            10%      51.90     0.77           2.18
            30%      50.37     2.30           2.18
            50%      48.83     3.83           2.18
     The median wait does not change until more than half of the questions are hits.

   This is the second of the three tables that the command prints. Each row is the whole design priced again with one hit share.

2. Print the monthly cost with the design’s own assumption, 10% in the base case:

   python -m costmodel costs

   The last lines:

     Total                                         51.90
   Biggest driver: Compute: API (53% of the total)

   In the last module, the same command printed US$52.67. The difference is the cache: from now on, the course quotes US$51.90.

3. Look at the provider’s quota in the high case:

   python -m costmodel workload --scenario high

   The line to read:

     chat calls, busiest min.           104.2 (provider quota: 100 per minute)

   Without a cache, the same line said 148.8. The high case assumes 30% hits, so 30% fewer chat calls.

4. Open design.toml, and change the base value of the hit share from 0.1 to 0.3:

   cache_hit_share = [0.0, 0.3, 0.3]

   Run python -m costmodel costs again. The total is now 50.37. Then change the line back to [0.0, 0.1, 0.3].

Read the result

Three numbers tell the story:

The monthly cost of the base case at four hit shares. The design's assumption, 10%, is the second column. Even half of the questions from the cache saves only 7%.

Text description of the chart

A column chart with four columns. With 0% hits, the base case costs US$52.67 a month. With 10% hits, the design’s assumption, US$51.90. With 30% hits, US$50.37. With 50% hits, US$48.83. The columns are almost the same height.

The money is small. With 10% hits, the cache saves US$0.77 a month; with 50%, US$3.83. A hit saves only the variable cost of a question, about US$0.000175. The variable part is 13% of the bill. The always-on API replica and the database cost the same with or without hits.

The typical wait does not change. Below 50% hits, the median question is a miss: 2.18 seconds. Only the people whose question was a hit get a fast answer.

The quota is where the cache matters. In the high case, each hit is a chat call that the provider does not see:

The high case's busiest minute at four hit shares, and the provider's quota of 100 calls a minute. 30% hits bring the calls close to the quota, but not under it.

Text description of the chart

A column chart with five columns. In the high case’s busiest minute, with 0% hits, the assistant needs 148.8 chat calls. With 10% hits, it needs 133.9. With 30% hits, the design’s high assumption, it needs 104.2, and with 46.5% hits, 79.6. The last column is the provider’s quota: 100 calls a minute. The first three columns are above the quota.

The cache brings the high case from 148.8 to 104.2 calls, but 104.2 is still above 100. The cache helps, but it is not the whole answer to the quota. Growth plan in Module 5 plans the rest.

So Kwame writes in the proposal: “The answer cache stays, because it costs almost nothing: some rows in the database we already pay for. It saves little money. Its value is headroom under the provider’s quota. We measure the real hit share in the first month and update the range.”

The provider’s own prompt cache

Model providers have a cache too. The prompt cache keeps the start of a request that the provider has seen recently. Its price list has a lower price for input tokens that it reports as cached: US$0.01 per million for chat-small, against US$0.10. It also lists a price for writing to the cache, US$0.125 per million. The course did not find out when the provider uses each price; its page says when. We measured only what the API reported.

The course team sent two requests four times each. The first was long: the question with the whole document collection, 17,852 tokens. The second was the assistant’s normal answer request: 863 tokens.

Four calls of a long request, L1 to L4 (17,852 tokens), and four of the normal answer request, S1 to S4 (863 tokens). From the second call, the provider reported 17,849 of the long request's tokens as cached; the calls were not faster. The short request was never cached.

Text description of the chart

A column chart with eight columns. The long request with the whole collection took 1.47 s on the first call, with 0 cached tokens. Then 1.89 s, 3.59 s and 1.47 s, each with 17,849 of 17,852 tokens reported as cached. The normal answer request took 1.86, 1.77, 1.39 and 1.71 s, with 0 cached tokens each time.

This was a surprise. From the second call on, the provider reported 17,849 of the 17,852 tokens as cached, but the calls were not faster. The short answer request was never cached in four calls. So the provider’s prompt cache saves money on long prompts that repeat, but not time, and not on the assistant’s short answer prompts.

Batching

The postman’s idea is batching: put many small requests into one call. Each call has a fixed part, the trip: the network round trip and the provider’s start of work. A batch pays that fixed part once.

The course team embedded the 8 longest documents, 61 chunks, in two ways, three passes each. Then it compared the median time per document:

Embedding one document: one call per chunk, or all its chunks in one call. The same 3,439 tokens and the same US$0.000069 for the 8 documents; 6.4 times faster in one call.

Text description of the chart

A column chart with two columns. With one call per chunk, embedding a document took 2.81 s (median). With one call per document, it took 0.44 s. Both ways sent the same 3,439 tokens and cost the same US$0.000069 for the 8 documents.

One call per document is 6.4 times faster. It sends the same 3,439 tokens and costs the same US$0.000069, because the price is per token, not per call. Batching saved time, not money. The design’s ingestion sequence from Module 2 already embeds all chunks of a document in one call.

Some work can also wait. A document’s summary is made in the background; nobody waits for it. For work that accepts a delay, the price list has a Flex price for chat-small. It is half the normal price: US$0.05 and US$0.25 per million tokens. That would bring a summary from about US$0.00012 to about US$0.00006. With 11 changed documents a month, the saving is less than a tenth of a cent. The course did not test Flex. Treat it as an option for a large ingestion, such as a new tenant’s whole collection.

When to use caching and batching

Cache work that repeats with the same inputs, when a fast wrong answer is impossible because the key holds every input. Write down the freshness rule next to the key. Do not cache answers whose inputs you cannot put in a key, such as a person’s live order status. The costs are storage, a few lines of code, and one more risk: a key that misses an input gives a fast, wrong answer. Data and identity flows showed what happens when the tenant is missing.

Batch work that comes in many small pieces and does not need an answer one piece at a time: embeddings at ingestion, summaries, reports. Do not make a person wait for a batch to fill. The costs: a failed batch must be retried as a whole or split, and a provider limits the size of one request.

Common mistakes

Common mistake

30% hits do not cut the bill by 30%

Symptom: Tomás expects the monthly cost to fall from US$52.67 to about US$37 with 30% hits. The calculator says US$50.37.

Cause: a hit saves only the variable cost of a question: tokens and retries, about US$0.000175. The fixed part of the bill, about 85% without a cache, does not depend on hits.

Fix: price a cache as you price usage: only the variable lines change. Then look at the lines that do change in a busy minute, such as the calls against the quota.

Common mistake

Old answers after a prompt change

Symptom: Amira releases a new prompt that cites the section of each policy. A day later, Grace still sees answers without sections.

Cause: the answers come from the cache. The key holds the tenant, the audience, the documents version and the question, but not the prompt version. Every cached answer still matches.

Fix: add the prompt and model versions to the key, or empty the cache in the release steps. Add a test: change the prompt version, ask a cached question, expect a miss.

Your turn: the scaling course’s hit share

The scaling course’s constructed ticket mix reached a hit share of 46.5% with a correct key. Suppose Larkfield’s questions repeat as often. Predict three numbers for the base case: the monthly cost, the all-in cost per question and the chat calls in the busiest minute.

Then change the line in design.toml to cache_hit_share = [0.0, 0.465, 0.465]. The high value must be at least the base value, or the pack refuses the range. Run python -m costmodel units and python -m costmodel workload, and change the line back to [0.0, 0.1, 0.3].

Show the answer

Only the variable part shrinks. The 43,950 questions now bring 46.5% hits instead of 10%. That is 0.365 × 43,950 = 16,042 more hits. Each one saves about US$0.000175, so the total is about US$2.80 below US$51.90.

Unit economics, scenario base, month 0
  total                        US$      49.10 per month (fixed 92%, variable 8%)
  per question (variable only) US$   0.000094
  per question (everything)    US$   0.001117
  per document change          US$   0.000169 (embed + summary + worker)
  Larkfield                    US$      26.43 per month
  Bramble Books                US$      22.68 per month

The workload command prints chat calls, busiest min. 8.6. A very good cache cuts the model calls almost in half, and the bill by about 5%. The fixed share rises to 92%: the more the cache saves, the more the bill is the always-on parts. Change the line back to [0.0, 0.1, 0.3].

See it end to endDesign review of an open document-question serviceCase study

The design review keys a shared answer cache by the question and the collection version, because its answers are public. It shows that the cache saves about a dollar a month, but its real value is the quota.

Knowledge check

Check your understanding

Predict: with 30% of the questions from the cache, does the median question get its answer faster than 2.18 seconds? Explain with the sorted list of times.

Show the answer

No. Sort the times: 30% of them are hits at about 0.014 s, and 70% are misses at about 2.18 s. The middle of the list, at 50%, falls in the misses. The median stays 2.18 s. It changes only when more than half of the questions are hits. The people with a hit get a fast answer; the typical person does not.

Check your understanding

Mei asks why the answer cache stays in the design if it saves only US$0.77 a month in the base case. Give her two reasons, with numbers.

Show the answer

First, it costs almost nothing. The saved answers are rows in the PostgreSQL server that the design already pays for: no new store, backup or security rule. Second, its value is in the busy minute, not in the bill. In the high case, it brings the chat calls from 148.8 to 104.2 a minute, against a quota of 100. And a hit returns in about 0.014 s instead of 2.18 s for the people who ask a repeated question.

Check your understanding

The course team embedded documents chunk by chunk and in one call per document. One way was 6.4 times faster. Which way, and why did it not save money?

Show the answer

One call per document: 0.44 s against 2.81 s per document. Each call has a fixed part: the network round trip and the provider’s start of work. One call per document pays it once, not once per chunk. It did not save money, because the price is per token. Both ways sent the same 3,439 tokens: US$0.000069 for the 8 documents.

Recap

• A cache keeps the result of work for reuse; a hit finds it, a miss does the work. The key holds every input that changes the result, and the freshness rule says how long a result stays right.
• The answer cache’s key: tenant, audience, documents version, question, plus the prompt and model versions. Chunk embeddings and summaries are already made once per document version.
• The hit share is an assumption: 0 / 10 / 30%. A hit (0.014 s, scaling course) skips 2.18 s and about US$0.000175. With 10% hits, the base case costs US$51.90. Even 50% hits would save only US$3.83.
• Below 50% hits, the median wait does not change. The cache’s real value is quota headroom: in the high case, 148.8 → 104.2 calls a minute.
• The provider’s prompt cache reported 17,849 of 17,852 tokens as cached for a long repeated request. The calls were not faster, and the short answer request was never cached.
• Batching: one embedding call per document is 6.4 times faster than one per chunk, for the same tokens and cost. Flex prices for work that can wait are half price; not tested here.

Next: Build versus buy