Context and memory

What you will be able to do

Know what a model can use when it answers one request, and what your application must keep for it. See why a very large context window does not mean that the model uses everything in it.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Tokens, and why token counts differ between models, from From text to tokens.
• How a model writes an answer one token at a time, from Predicting continuations.
• What training puts into a model, from Training stages.
• Percentages and averages, from Describe a dataset in Essential Mathematics and Statistics for AI.

What you need

• A browser. Nothing to install.

Hosted models change often: your own results will differ.

The expert on the phone

Imagine that Larkfield can call an expert on the phone. The expert has read thousands of books and knows a lot about gardens, shops and customers. But the expert has two strange limits. First, they forget every call when they hang up. Second, during a call they can only look at the papers on one small desk.

So before each call, someone at Larkfield must put the right papers on the desk. These are the policy, the customer’s message, and a note of the last call. If a paper is not on the desk, the expert cannot use it, however clever they are. And if the desk holds a thousand pages, the expert may not find the one line that matters.

A language model works in the same way. This lesson separates the four places where “memory” can live, and shows which ones the model owns and which ones your application owns.

The problem: the second message

Camille tests the assistant with a short conversation, as a real customer would write it. Amira sends the three messages to Claude Haiku 5.5. Her first version of the application sends only the newest message each time.

1. “My order L-20431 is 7 business days late. What can you do?”
2. “OK. And if it still has not come next week, can I get my money back instead?”
3. “Thanks. Please send that refund to my new card, the old one has expired.”

The first answer is good: free shipping, because the order is more than 5 business days late. The second answer shows the problem. The model replies:

"If you tell us your order number, we can check how late it is."

The customer gave the order number one message ago. The model did not “forget” it. It never received it, because the application did not send it again. “But the chat on my screen shows the whole conversation,” says Omar. It does, because the application shows it. The model sees only what is in the current request.

The idea: four kinds of memory

People use the words “the model’s memory” for four different things. Only one of them is inside the model.

1. The weights. In Training stages, you saw that training sets millions or billions of numbers inside the model, its weights (also called parameters). They hold what the model learned from its training text, as patterns. After training, they do not change. When you send a ticket, the weights do not learn from it. This is the expert’s knowledge from years of reading.
2. The context window. This is the desk. The context window is the largest number of tokens that the model can handle in one request. It counts the tokens you send in, plus the tokens it writes out. Apart from its weights, the model can use only what is in the window.
3. Application history. This is the note of the last call. To continue a conversation, your application keeps the earlier messages and replies, and resends them with each new request. The model reads them again every time.
4. Retrieval. This is a colleague who finds the right page in the files. Your application searches its own documents, such as the policy or a product handbook, and pastes the relevant part into the request. A later course, RAG: Building AI That Uses Your Data, builds it; here you only need the idea.

The memory boundary. Your application owns the history, the documents and what goes into each request. The model owns only its fixed weights, and it sees only the window of this one request.

Text description of the diagram

The diagram has three areas. On the left, your application holds two stores: the chat history (earlier turns) and the policy and handbook. An arrow labelled “resends” goes from the history to a prompt builder (code you write). An arrow labelled “retrieval: finds and pastes” goes from the documents to the prompt builder.

In the middle, the prompt builder sends one request into the context window, which is for this request only. From top to bottom, the window holds the instructions, a policy excerpt, the earlier turns and the new ticket. A dashed box holds the answer (output) that the model will write.

On the right, the model has its weights, learned in training and fixed after it. The weights read the window and write the answer, token by token, into the same window. A note says that the model keeps nothing between requests.

The line between the two sides is the important part. Everything on the left is your code and your data, so you decide what goes in. Everything on the right is fixed for each request. If an answer needs a fact, that fact must be in the weights from training, or your application must put it in the window.

A tiny worked example: the window grows with history

Amira runs the same three messages again. This time, her application resends the earlier messages and replies. Here are the input tokens that Claude Haiku 5.5 reported for each request. The first request, the instructions and the policy with message 1, is 776 tokens. (A test ticket in other lessons is about 796 tokens on Claude: the same instructions and policy, with a ticket instead of this message.)

Request	What the application sends	Input tokens
1	Instructions and policy, message 1	776
2	The same, plus reply 1 and message 2	938
3	The same, plus reply 2 and message 3	1,082

With the history, the second reply is correct. It names order L-20431. About the 15-day rule for lost orders, it says: “Your order is currently 7 business days late, so it does not yet meet that threshold.”

The table also shows the cost of memory. The application sends every earlier message again in each new request, and you pay for it again. In a conversation of 30 messages, the application sends the first message 30 times. The window also has a limit, so the application must cut or summarise a very long conversation. The model cannot do that for you between requests.

How large is the window?

Current hosted models have large windows. Anthropic’s model overview lists a window of 1 million tokens for Claude Haiku, Sonnet and Opus 5.5. At most 128,000 of them can be output. Google’s page for Gemini 3.5 Flash lists 1,048,576 input tokens. It also lists 65,536 output tokens. Ollama’s library lists 128,000 tokens for Gemma 3 4B.

Models change fast. Every number in this module is a snapshot. The method of each test is what lasts.

Remember from From text to tokens that the same text is a different number of tokens on each model. So “1 million tokens” holds a different amount of text on each one.

The running example: put the whole handbook in the window?

Larkfield also has a product handbook: one short sheet per product, with its warranty, its warehouse and its care notes. Tomás has an idea. “The window is 1 million tokens. Why build retrieval? Paste the whole handbook into every request. Then the model knows everything.”

Grace asks Amira to test the idea before anyone builds it. Amira writes a synthetic handbook for the test: up to 3,000 product sheets that look alike, such as “Product LF-27611: terracotta compost bin. Warranty: 12 months. Ships from Winnipeg.” In each request, she asks for the warranty of 10 products. The 10 sheets are spread through the handbook, from near the start to near the end. An answer counts only if the number of months is exact.

The same 800 sheets are 28,600 tokens with OpenAI’s o200k tokenizer, 32,100 on Gemini and 47,400 on Claude. So the chart counts sheets, not tokens.

Predict first: will a model that accepts 128,000 tokens find 10 facts in a handbook of 16,000 tokens?

• Claude Haiku 5.5
• Gemini 3.5 Flash
• Gemma 3 4B (local)
• Mistral 7B (local)

Share of the 10 warranty lookups that each model answered exactly, by the size of the handbook in the request. Each point is 1 to 3 requests (10 to 30 lookups). Claude Haiku 5.5 and Gemini 3.5 Flash stay at 100% (the two lines overlap). Mistral 7B stays near 100% up to 400 sheets; Gemma 3 4B falls to 0% at 800 sheets.

Text description of the chart

The x axis is the number of product sheets in the handbook, on a log scale from 10 to about 4,000. The y axis is the share of the lookups answered exactly, from 0% to 100%.

Claude Haiku 5.5 found every lookup at every size: 25, 100, 200, 400, 800 and 3,000 sheets. On Claude, 3,000 sheets are 177,400 tokens. Gemini 3.5 Flash also found every lookup at 100, 400, 800 and 3,000 sheets (120,100 tokens on Gemini).

Gemma 3 4B ran on Larkfield’s office computer, a Mac mini M4 with 16 GB of memory. It found 100% at 25 sheets (1,200 tokens) and 87% at 100 sheets (4,200 tokens). It found 50% at 200 sheets (8,200 tokens), 25% at 400 sheets (16,200 tokens) and 0% at 800 sheets (32,100 tokens).

Mistral 7B, run the same way, found 100% at 25 and 100 sheets and 90% at 200 sheets. At 400 sheets (19,100 tokens on Mistral), it found 100%.

The answer to the prediction is no. Gemma 3 4B accepts 128,000 tokens, but at 16,200 tokens it found only one fact in four. And it never said “I am not sure”. At 32,100 tokens it still returned 10 numbers in clean JSON, and all 10 were wrong.

Now compare Mistral 7B. Ollama’s library lists a window of only 32,000 tokens for it, a quarter of Gemma’s. Yet it found 39 of the 40 lookups up to 19,100 tokens. The advertised window size did not predict which model reads long input well. Only the test showed it.

The model’s limit is not always the limit you get. For the chart, Amira set the window size in Ollama large enough for each request. In one more run, she sent the 200-sheet handbook (about 8,200 tokens) to Gemma 3 4B without that setting. Ollama used its smaller default and kept only the last 2,051 tokens of the request, with no error. The model still answered all 10 lookups, and only the 2 products near the end were right. So check the window that your tool really uses, not only the number on the model’s page.

The two hosted models did much better: every lookup was correct, up to 177,400 tokens. So should Tomás paste the handbook into Claude? Amira tries a harder question on the same handbooks: “How many products ship from Halifax and have a warranty of 48 months?” To answer it, the model must check every sheet, not find one line.

Model	Handbook	Correct counts	Model’s answers
Claude Haiku 5.5	100 sheets	2, 3, 2	2, 3, 2
Claude Haiku 5.5	800 sheets	14, 21, 11	14, 21, 11, after 5,100 to 14,100 output tokens of thinking
Claude Haiku 5.5	3,000 sheets	50, 63	no answer: it used all 16,000 output tokens and stopped
Gemini 3.5 Flash	800 sheets	14, 21	14; then no answer (16,000 tokens used)
Gemini 3.5 Flash	3,000 sheets	50	no answer (16,000 tokens used)

The window held the whole handbook every time. But a question that needs all of it can still fail, or cost so many output tokens that no answer comes.

Omar asks about the money. One normal ticket is about 800 tokens. With the 3,000-sheet handbook, every request to Claude Haiku 5.5 is 177,400 input tokens. Above 100,000 input tokens, Haiku 5.5 costs $0.50 per million input tokens instead of $0.10 (Anthropic’s pricing page). So the input of each ticket costs about $0.089. That is more than 1,000 times the $0.00008 of a normal ticket. On Gemini 3.5 Flash, at $1.50 per million input tokens, the 120,100 tokens cost $0.18 per ticket. Retrieval sends only the 2 or 3 sheets that the ticket needs.

Common mistake

Equating context-window size with reliable recall

Symptom: Tomás reads “128K context window” for Gemma 3 4B and plans to send the whole 400-sheet handbook (16,200 tokens) with each ticket. The model accepts the request, and the answers look complete. In the test, 15 of the 20 warranty lookups are wrong, and nothing in the output shows which ones.

Cause: the window size is the most that the model accepts, not the amount that it uses well. Research shows the same pattern. Liu and others (Lost in the Middle, 2023) wrote that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts.” The RULER study (Hsieh and others, NVIDIA, 2024) tested models that claim 32,000 tokens or more. It found that “only half of them can maintain satisfactory performance at the length of 32K.”

Fix: send less, and test. Use retrieval to paste only the sheets that the ticket needs. Before you rely on a long window, run a small recall test like Amira’s. Use your documents, your length and your kind of question. A lookup and a count are different tasks. Put the instructions and the most important facts near the start or the end of the request.

What it means for a real AI service: the context window is a limit, like the size of a desk. It does not tell you how well the model reads what is on the desk.

Which kind of memory for which job

The four kinds of memory do not compete. Each one has a job, an owner and a cost.

Kind	Who controls it	How long it lasts	What it costs	Use it for
Weights	The model provider, in training	Until the next model version	Nothing per request	General language and knowledge. Never Larkfield’s rules: they are not in the training text, and they change.
Context window	You, in each request	One request	Every token, every time	The instructions, the ticket, and the facts this answer needs
Application history	Your application	As long as you store it	Resent and paid for in each request	Conversations. Cut or summarise long ones.
Retrieval	Your application	As long as your documents	A search, plus the pasted tokens	Policies, handbooks, anything large or often changed (the RAG course)

“Then why not train the model on our policy, so it is in the weights?” asks Omar. Training is slow and costly, and the policy changes. A model trained in June still knows the June policy in October. The window always holds the version that you send today.

Check your understanding

A customer writes to the assistant on Monday. On Tuesday, the same customer writes again: as I told you yesterday. Explain what the model knows about Monday, and what the application must do.

Show the answer

The model knows nothing about Monday. Its weights did not change after training, and Monday’s request is gone. The model sees only Tuesday’s request. If Monday’s messages matter, the application must store them and resend them, or a summary of them, in Tuesday’s request. That is application history, and it costs input tokens in every request.

Check your understanding

Predict: Amira sends the 200-sheet handbook (about 8,200 tokens) to Gemma 3 4B, but asks for only one product, which is in the first lines. Do you expect a more or a less reliable answer than in the test? Why test it anyway?

Show the answer

Probably more reliable. In the test, Gemma 3 4B found the product nearest the start in every run up to 400 sheets. And one question is easier than ten. Research such as Lost in the Middle also finds the start and the end easiest. But this is a guess from a small test, not a rule. Real tickets ask about products anywhere in the handbook. Test the real question at the real length before you rely on it.

Check your understanding

Larkfield changes its return window from 30 to 45 days on 1 November. Which kind of memory should carry the new rule to the assistant, and which kind cannot?

Show the answer

The context window should carry it: put the new policy in each request, through retrieval or in the instructions. The weights cannot. They hold only what the model saw in training, and they do not change when you send a request. The new rule did not exist when the provider trained the model.

Recap

• The weights hold what the model learned in training. Nothing changes them after training, and the model keeps nothing between requests.
• The context window is the most tokens that one request can hold, in and out. Claude 5.5 models and Gemini 3.5 Flash accept about 1 million tokens; Gemma 3 4B accepts 128,000.
• Application history is your code resending earlier turns. In Amira’s test, the input grew from 776 to 938 to 1,082 tokens over three messages.
• Retrieval is your code finding the relevant documents and pasting them in.
• Window size is not reliable recall. Gemma 3 4B (128,000-token window) found 50% of the facts at 8,200 tokens and none at 32,100. Mistral 7B (32,000-token window) found 39 of 40 up to 19,100 tokens. Two hosted models found every fact at 120,000 to 177,000 tokens, but gave no answer to a count over the same handbook. Test your own task at your own length.

Next, you will see how some models spend extra tokens to think before they answer, and what changes when the input is a photo.

Next: Reasoning and multimodality