Manage conversation state

Lesson 14
50 min
Module 5: Conversations and streaming

Lesson 14 of 19 · Building Reliable Applications with LLM APIsManage conversation state

Course overview

1. Start with one model request

2. Prompting as an engineering task

3. Structured outputs and validation

4. Tool calling safely

5. Conversations and streaming

6. Reliability and cost controls

7. Case studies

What you will be able to do

Let an agent ask the assistant follow-up questions about a ticket, such as “make the reply shorter”. You keep each agent’s conversation apart, and you decide how much history each request may carry.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

Why a model keeps nothing between requests, and why the context window grows with history, from Context and memory in the language models course.
Messages and roles (system, user, assistant), from Request anatomy.
The two gates (schema, then business rules), from Validate meaning.

What you need on your computer

The project at the end of Module 4. python -m pytest shows 59 passed, 1 deselected. The virtual environment is active.
The snapshot end-of-m05 from the labs repository. You copy two files from it.
If your project is broken, copy the snapshot end-of-m04 instead, with the steps of Reset the project.

Models and APIs change fast. The responses, token counts and timings in this module are a snapshot. The method is what lasts.

Two clerks, one notepad

TermCamilleAn agent at Larkfield’s help desk. Camille uses the assistant’s draft replies to answer customers. and TermGraceThe lead of Larkfield’s help desk. She writes the support policy and decides what a good answer to a customer is. share a desk at TermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps’s help desk. To save paper, they also share one notepad. Camille takes a call about a hammock and writes notes. Then Grace takes a call about a kettle and writes her notes below. Camille’s customer calls back: “Can you make that shorter?” Camille looks at the notepad. The last note is about a kettle.

Nobody would run a desk like this. Each call gets its own page. But the same mistake is easy to make in code, and there it is much harder to see.

The application keeps the history

In the language models course, Context and memory showed that a model keeps nothing between two requests. To continue a conversation, the application keeps the earlier messages and sends them again with each new request. This lesson is about the code that keeps them.

Camille tried it with ticket T-80005. Sophie writes in French that her hammock (order LK-266415) has not arrived. The assistant drafted a reply. Then Camille asked two follow-up questions. Each request carried the whole conversation so far:

Ticket T-80005 and two follow-up questions, chat-small (gpt-6-luna on Azure), real recordings. Every request resends the system message, the ticket and all earlier turns.

Text description of the chart

Three bars. Request 1 has 2 messages, the system message and the ticket: 1652 input tokens. Request 2 adds the first answer and “Make the reply shorter: two sentences.”: 1749 tokens. Request 3 adds the shorter answer and “Now write the same reply in English for my colleague”: 1851 tokens.

Two things stand out. First, the system message (the instructions and Grace’s policy) is most of every request. Second, each turn adds about 100 tokens, and every later request pays for them again. The provider reported most of these tokens as TermCached tokensInput tokens that the provider kept from an earlier request that started with the same text, such as the same instructions and policy. Many providers charge less for them.Learn more Request anatomyBuilding Reliable Applications with LLM APIs (1,746 of 1,851 in request 3). The start of each request was the same as before. Many providers charge less for cached input tokens: check your provider’s price page to see what that means for the bill.

So the application owns the conversation. That raises two questions: whose conversation is it, and how much of it do you send?

The problem: one list for the whole help desk

TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses.’s first version of the follow-up feature kept the history in one Python list. It worked in her tests, because she was the only user. Here is the idea, cut down to a script that you can run with the mock:

import json

from assistant.context import MAX_TOKENS, build_request
from assistant.data import load_tickets
from assistant.providers import MockProvider

tickets = load_tickets()
provider = MockProvider()
SYSTEM = build_request(tickets["T-80005"], "chat-small", "v2")["messages"][0]["content"]

history = []  # one list for the whole help desk: the bug


def ask(agent, text):
    history.append({"role": "user", "content": text})
    request = {"model": "chat-small", "messages": [{"role": "system", "content": SYSTEM}] + history,
               "max_completion_tokens": MAX_TOKENS}
    completion = provider.complete(request)
    history.append({"role": "assistant", "content": completion.text})
    answer = json.loads(completion.text)
    print(f"{agent}: {len(request['messages'])} messages sent, {completion.input_tokens} tokens in")
    print(f"   {answer['language']} | {answer['team']} | {answer['reply']}")


def opening(ticket_id):
    """The first user message for a ticket: the ticket text between <ticket> tags."""
    return build_request(tickets[ticket_id], "chat-small", "v2")["messages"][1]["content"]


ask("camille", opening("T-80005"))   # the French ticket: a hammock that has not arrived
ask("grace", opening("T-64704"))     # an English ticket: a kettle with a burning smell
ask("camille", "Make the reply shorter: two sentences.")

The output. Each answer is a real response of Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs (gpt-6-luna on Azure), replayed by the mock:

camille: 2 messages sent, 1652 tokens in
   fr | delivery | Je suis désolé que votre commande ne soit pas encore arrivée. Notre équipe de livraison pourra vérifier où elle se trouve.
grace: 4 messages sent, 1808 tokens in
   en | warranty | I’m sorry this happened, and thank you for stopping use of the kettle. A member of our team will contact you about the next steps.
camille: 6 messages sent, 1909 tokens in
   en | warranty | Please stop using the kettle. A member of our team will contact you about next steps.

Camille asked to shorten Sophie’s hammock reply. She got a short reply about a kettle, in English, for the warranty team. The model did what the messages asked. The newest draft in the conversation was the kettle reply, so it shortened that one.

The wrong reply is the visible part. The hidden part is worse. Grace’s request had 4 messages: it carried Sophie’s ticket, her name and her order number into a conversation about another customer. With 45 tickets a day and several agents, every request would carry pieces of other customers’ tickets. No error appears, and every answer is valid JSON. The business rules check an answer against the ticket that the code gives them, and nothing here tells the code that the conversation is wrong. Only a careful agent would notice.

The idea: one conversation per agent and ticket

The fix is the notepad rule: one page per call. In code, the conversation state is the list of messages that belongs to one conversation. You store it under a key that names its owner. For Larkfield, the owner is a pair: the agent and the ticket. Camille’s follow-ups about T-80005 never meet Grace’s work on T-64704. Two agents on the same ticket do not mix their requests either.

Two panels. Left, one list, the bug: Camille's ticket T-80005 and its French answer, then Grace's ticket T-64704 and its English answer, then Camille's 'Make the reply shorter: two sentences.' Camille gets an English warranty reply: 'Please stop using the kettle.' Right, separate lists, the fix: the list for camille and T-80005 holds the ticket, the answer and the follow-up, and Camille gets the French delivery reply 'Désolé pour le retard. ...'. A separate list for grace and T-64704 holds the kettle ticket and its answer. — The same three steps with one shared list (the bug) and with one list per agent and ticket (the fix). All answers are real responses of chat-small.

The project’s ConversationStore does this with a dictionary. Its key is (agent_id, ticket_id), and each value is a Conversation. In a real help-desk app, the store would be a database table with the same key, so that the history survives a restart. The idea does not change.

The second idea: a token budget

The history grows with every turn. Without a limit, a long conversation becomes slow and expensive, and in the end it does not fit in the TermContext windowThe maximum number of tokens that a model can read and write in one call. The instructions, the conversation and the answer must all fit in it.Learn more Context and memoryLLMs: How They Work and How to Choose One. So the application sets a token budget: the largest number of input tokens that it allows for one request. You choose the budget for cost and speed, far below the window.

When the conversation is larger than the budget, the application must choose what to send. There are three common ways:

Way	What it sends	Good for	Risk
Cut	the newest turns that fit; the oldest are left out	short follow-ups about the latest draft	the oldest turn may be the one that matters
Select	chosen turns: the ticket always, then the newest	a conversation about one document	you must know which turns matter
Summarise	a short summary of the old turns, made by a model call, then the newest turns	long conversations	one more call; the summary is model output and can drop or invent facts

The project uses the simplest rule: cut. Conversation.messages() always sends the system message and the newest turn. Then it walks back through the older turns and adds each one while the total stays within the budget. It counts tokens with a rough estimate, about 4 characters per token, because the exact count is known only after the provider answers.

Here it is on Camille’s real conversation, before the third request, with a budget of 1,900 estimated tokens:

Six boxes from top to bottom. The system message, instructions and policy, 1,724 estimated tokens, always sent. Turn 1, user, the ticket, 45, cut. Turn 2, assistant, the first answer, 86, cut. Turn 3, user, Make the reply shorter, 9, sent. Turn 4, assistant, the shorter answer, 77, sent. Turn 5, user, Now in English, 19, the newest, always sent. A note: budget 1,900. Walk back from the newest turn: 19, 77 and 9 fit with the system's 1,724. Adding 86 would make 1,915: stop. Sent: 1,829. Cut: turns 1 and 2. — The cut rule of assistant/history.py on Camille's conversation with a budget of 1,900 estimated tokens. Dashed boxes are cut. Token numbers are the project's estimate (characters ÷ 4).

Look at what the cut removed: the ticket. The model would get “Make the reply shorter”, a short French answer and “Now in English”, without Sophie’s message. For a help desk, the oldest turn is often the most important one. The project keeps the simple rule, so that you can see this failure. In the module practice at the end of this module, you detect it and stop with a clear message instead.

The estimate is rough, too. It said 1,864 tokens for Camille’s second request, and the provider counted 1,749. Keep a margin, and use the provider’s usage numbers when you need exact counts.

Build it in the project

Make the file assistant/history.py:

"""Conversation state: the model remembers nothing, so the application keeps and resends the history.

An agent can ask follow-up questions about a ticket ("make the reply shorter"). Each (agent, ticket)
pair has its own conversation, and only the newest turns that fit in the token budget are sent.
"""

from dataclasses import dataclass, field


def estimate_tokens(text: str) -> int:
    """A rough count: about 4 characters per token for English. Use the provider's usage for exact numbers."""
    return max(1, len(text) // 4)


@dataclass
class Conversation:
    system: str
    budget_tokens: int = 3000
    turns: list[dict] = field(default_factory=list)

    def add(self, role: str, content: str) -> None:
        self.turns.append({"role": role, "content": content})

    def messages(self) -> list[dict]:
        """The system message, then the newest turns that fit in the budget (the oldest are cut first)."""
        used = estimate_tokens(self.system)
        kept = []
        for turn in reversed(self.turns):
            cost = estimate_tokens(turn["content"])
            if kept and used + cost > self.budget_tokens:  # the newest turn is always kept
                break
            kept.insert(0, turn)
            used += cost
        return [{"role": "system", "content": self.system}] + kept

    def dropped(self) -> int:
        return len(self.turns) - (len(self.messages()) - 1)


class ConversationStore:
    """One conversation per agent and ticket. Never one shared list for everybody."""

    def __init__(self, system: str, budget_tokens: int = 3000):
        self.system = system
        self.budget_tokens = budget_tokens
        self._conversations: dict[tuple[str, str], Conversation] = {}

    def get(self, agent_id: str, ticket_id: str) -> Conversation:
        key = (agent_id, ticket_id)
        if key not in self._conversations:
            self._conversations[key] = Conversation(self.system, self.budget_tokens)
        return self._conversations[key]

Copy the recorded conversations from the snapshot end-of-m05 into your project. The mock needs them. In the project folder, on macOS or Linux:
Terminal window
```
cp ../nextia-learning-labs/C08/snapshots/end-of-m05/recordings/conversation.jsonl recordings/
```
On Windows, in PowerShell:
Terminal window
```
Copy-Item ..\nextia-learning-labs\C08\snapshots\end-of-m05\recordings\conversation.jsonl recordings\
```
Change the path if your copy of the labs repository is in another folder.
Make shared_history.py from the problem section. Run it with python shared_history.py. Camille gets the kettle reply.

Make separate_history.py. It is the same script with the store instead of the list:

import json

from assistant.context import MAX_TOKENS, build_request
from assistant.data import load_tickets
from assistant.history import ConversationStore
from assistant.providers import MockProvider

tickets = load_tickets()
provider = MockProvider()
store = ConversationStore(build_request(tickets["T-80005"], "chat-small", "v2")["messages"][0]["content"])


def ask(agent, ticket_id, text):
    conversation = store.get(agent, ticket_id)  # one conversation per agent and ticket
    conversation.add("user", text)
    request = {"model": "chat-small", "messages": conversation.messages(), "max_completion_tokens": MAX_TOKENS}
    completion = provider.complete(request)
    conversation.add("assistant", completion.text)
    answer = json.loads(completion.text)
    print(f"{agent} on {ticket_id}: {len(request['messages'])} messages sent, {completion.input_tokens} tokens in")
    print(f"   {answer['language']} | {answer['team']} | {answer['reply']}")


def opening(ticket_id):
    """The first user message for a ticket: the ticket text between <ticket> tags."""
    return build_request(tickets[ticket_id], "chat-small", "v2")["messages"][1]["content"]


ask("camille", "T-80005", opening("T-80005"))
ask("grace", "T-64704", opening("T-64704"))
ask("camille", "T-80005", "Make the reply shorter: two sentences.")
ask("camille", "T-80005", "Now write the same reply in English for my colleague, so that I can check it.")

Run it with python separate_history.py.
Copy tests/test_history.py from the snapshot end-of-m05 into your tests folder, as in step 2. Then run the whole suite with python -m pytest.

The output of step 5 (real responses of chat-small):

camille on T-80005: 2 messages sent, 1652 tokens in
   fr | delivery | Je suis désolé que votre commande ne soit pas encore arrivée. Notre équipe de livraison pourra vérifier où elle se trouve.
grace on T-64704: 2 messages sent, 1672 tokens in
   en | warranty | I'm sorry this happened, and I'm glad you've stopped using the kettle. Please continue not to use it. A member of our team will contact you about the next steps and your safety concern.
camille on T-80005: 4 messages sent, 1749 tokens in
   fr | delivery | Désolé pour le retard. Notre équipe de livraison vérifiera où se trouve votre commande.
camille on T-80005: 6 messages sent, 1851 tokens in
   fr | delivery | Désolé pour le retard. Notre équipe de livraison vérifiera où se trouve votre commande.

Grace’s request now has 2 messages, not 4, and nothing from Sophie’s ticket. Camille gets a two-sentence French reply about her hammock. Step 6 prints 62 passed, 1 deselected. The three new tests check the cut rule, the separate conversations and the recorded conversation.

Look at the last line, though. Camille asked for English and got the same French reply again. The FailureCase below explains why.

When to keep history, and what it costs

Keep a conversation only where people really ask follow-up questions. The assistant’s main job, the analysis of one ticket, needs no history: each ticket is one request. The follow-up feature is extra, and each follow-up costs a full request. In Camille’s conversation, each one sent about 1,750 to 1,850 input tokens to change two sentences.

Choose the key with care. (agent, ticket) keeps both people and tickets apart. A key with only the ticket would let two agents’ requests mix. A key with only the agent would bring back the notepad bug as soon as one agent opens a second ticket.

Decide also when a conversation ends. A closed ticket does not need its history any more. Delete it, to save storage and because it holds a customer’s words.

Your turn: predict the cut

Use Camille’s conversation before the third request, as in the diagram. Predict how many turns messages() sends, and how many it cuts, with a budget of 3,000, then 1,800. Then check your answer with a short script. Build the conversation as tests/test_history.py does, and replay the two recorded requests with the mock. For each budget, set conversation.budget_tokens and print len(conversation.messages()) - 1 and conversation.dropped().

Show the answer

budget 3000: 5 turns sent, 0 cut, 1960 tokens (estimated)
budget 1800: 1 turns sent, 4 cut, 1743 tokens (estimated)

With 3,000, everything fits: 1,724 for the system message plus 236 for the five turns. With 1,800, the system message and the newest turn already make 1,743. The next turn back, the 77-token answer, would make 1,820. So only the newest turn is sent: “Now write the same reply in English”. The model would not even know which reply Camille means. A budget that is too small makes the conversation useless, not only cheaper.

Common mistake

'The model will follow the agent's latest message'

Symptom: Camille asks, “Now write the same reply in English for my colleague, so that I can check it.” The recorded answer (chat-small) is the same French JSON as before: "language":"fr", and the reply “Désolé pour le retard. Notre équipe de livraison vérifiera où se trouve votre commande.” TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. says that the model “ignored” Camille.

Cause: the follow-up went into a conversation whose system message is the ticket-analysis prompt. That message sets the contract: one JSON object, and “a draft reply to the customer in the ticket’s language”. The system message outranks a user’s request, by design. For the same reason, a customer cannot change the rules by writing “ignore your instructions” in a ticket. Here, the model kept the contract. That was right for a customer reply and wrong for Camille’s task.

Fix: give each follow-up task its own request design. A translation for a colleague is a different task. It needs its own system message, such as “translate this draft into English for an agent; it is not for the customer”. It also needs its own output contract and checks. The ticket-analysis conversation keeps its rules.

What it means for a real AI service: the system message is part of the conversation state. When the task changes, change the instructions in code. Do not expect a chat message to override them.

Knowledge check

Check your understanding

In the shared-list bug, every answer was valid JSON and no error appeared. Name two harms of the bug that no gate would catch.

Show the answer

First, Camille got a reply for the wrong ticket: a kettle reply, in English, for Sophie’s French hammock ticket. The gates check an answer against the ticket that the code gives them, and nothing tells them that the conversation is mixed. Second, Grace’s request carried Sophie’s ticket, her name and her order number to the model inside another customer’s conversation. That is a privacy problem even when the answer looks right. Both harms come from the conversation’s key, not from the model.

Check your understanding

Why does the project key a conversation by (agent, ticket), and not by ticket alone?

Show the answer

With the ticket alone as the key, two agents who work on the same ticket would share one list. One agent’s follow-up (“make it shorter”) would then change the draft that the other agent reads. Each would get answers to questions they never asked. With the agent alone, one agent’s conversations about different tickets would mix, which is the notepad bug again. The pair keeps both apart.

Check your understanding

Predict: Conversation cuts the oldest turns first. In a long follow-up conversation about T-80005, which turn is cut first, and why is that a problem here?

Show the answer

Turn 1, the ticket itself, is cut first, because it is the oldest turn. The model then gets the follow-up questions and the drafts, but not the customer’s words. It may still answer, with confidence, without the facts that it needs. That is why the module practice detects a missing ticket and stops with a clear message. It is also why “select” (always keep the ticket) or “summarise” can be better than “cut”.

Recap

The model keeps nothing between requests. The application keeps the conversation and resends it: Camille’s three requests had 1,652, 1,749, 1,851 input tokens.
Keep each conversation separate. One shared list gave Camille a kettle reply for her hammock ticket (a real recording). It also sent Sophie’s ticket inside Grace’s request. The project keys each conversation by (agent, ticket).
A token budget limits the input of one request. You can cut (oldest first), select (always keep what matters) or summarise (one more call, and model output to check). The project cuts, so the ticket goes first: the module practice handles that.
Token counts in code are an estimate (about 4 characters per token). Keep a margin, and use the provider’s usage for exact numbers.
The system message belongs to the conversation state. A new task, such as a translation for a colleague, needs its own instructions.

Next, show the answer while it arrives, and learn why the text on the screen is not data until the stream is complete.

Next: Stream responses