Test and package

Lesson 19
50 min
Module 6: Reliability and cost controls

Lesson 19 of 19 · Building Reliable Applications with LLM APIsTest and package

Course overview

1. Start with one model request

2. Prompting as an engineering task

3. Structured outputs and validation

4. Tool calling safely

5. Conversations and streaming

6. Reliability and cost controls

7. Case studies

What you will be able to do

Prove that the assistant handles every failure that this course taught, with tests that give the same result on every computer. Then package it so that another person can run it without an account, and add a real model only if they want one.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

The mock provider and why tests need it, from A first run without an account.
The two gates, from Handle incomplete results and Validate meaning.
The retry policy and the simulated provider, from Handle provider limits.

What you need on your computer

The project after Measure usage and cost: usage.py and the cost on the analyse line. python -m pytest shows 78 passed, 1 deselected.
If you skipped it, copy the snapshot end-of-m06 with the steps of Reset the project. It is the finished project of the course.

A flight simulator

Pilots practise an engine failure many times before they ever see one. They do not practise on a real flight with passengers. They use a simulator, where the same failure happens at the same moment every time, and the instructor can check exactly what the pilot did. The real flight comes later, and it is not where they learn the emergency steps.

The problem: a real model is a poor test partner

You could test the assistant by sending tickets to the real model and checking the answers. Four things go wrong.

The answers change. The same request can come back with a different reply, so a test that checks the reply fails on some days and passes on others.
It costs money and needs a key. A new team member, or the build server, may have neither.
You cannot order a failure. A real 429 or timeout comes when it wants to. The course got 42 real 429s by accident and could not get a single one on purpose.
Some failures never came at all. Six honest attempts did not produce one refusal, and no recorded tool loop ever repeated a call.

A test that you cannot repeat proves nothing. When it fails, you do not know whether your code broke or the model changed its mind.

The idea: deterministic tests with recorded, simulated and constructed cases

A deterministic test gives the same result every time it runs with the same code. The project’s tests get there in three ways, and the lesson always says which one a test uses.

Recorded: the mock replays a real recorded answer. The answer is real, and it is the same on every run.
Simulated: SimulatedProvider raises the same errors that the adapter raises for a real 429, timeout or 5xx. The error is made up; the code that handles it is the real code.
Constructed: an answer written by hand, because the real model would not produce it. The project keeps two, a refusal and a repeated tool call, under the model name constructed-example.

The picture maps every failure path of the course to the tests that cover it.

Three groups of test cases lead to one command. Real recordings: normal, T-80008, in test_first_run and test_analyse; invalid output, Markdown around the JSON from gemma3:4b, in test_validate; truncation, finish_reason length, in test_validate; blocked tool argument, T-80001, T-65059 and T-80009, in test_tools and test_loop. Simulated, in assistant/simulate.py: timeout and rate limit (HTTP 429), in test_retry and test_analyse. Constructed, in recordings/constructed.jsonl: refusal in test_validate, and the repeated-call loop in test_loop. All lead to python -m pytest: 80 passed, no network, no key. — Every failure path of the course and the tests that cover it. Real, simulated and constructed cases are kept apart, and each test says which one it uses.

A tiny worked example: one test of each kind

Here are three tests of the finished project, one of each kind. Read the first line of each: it says where the answer comes from.

A recorded case: TermGemma 3 4BA small open model from Google, with 4 billion parameters. It can run on a laptop with Ollama, without an account or a cost per call.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One, without TermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs, wrapped its JSON in a TermMarkdownA plain-text way to show formatting, for example **bold** or # for a heading. Models often add it, so a program that expects plain JSON must remove it or reject the answer. fence. The test replays that real answer and expects gate 1 to reject it.

def test_a_local_model_without_structured_output_writes_markdown_around_the_json(mock, tickets):
    completion = recorded(mock, tickets, "T-80002", model="gemma3:4b", prompt="v1", structured=False)
    assert completion.text.startswith("```json")
    assert [p.code for p in parse(completion).problems] == ["malformed_json"]

A simulated case: a 429 and a timeout before the real recording. provider.calls == 4 proves that the assistant retried twice, then made the two real calls of the tool loop.

def test_a_simulated_rate_limit_is_retried(mock, tickets, tmp_path):
    provider = SimulatedProvider(mock, ["rate_limit:0.01", "timeout"])
    log = UsageLog(tmp_path / "usage.jsonl")
    outcome = analyse_ticket(tickets["T-80008"], provider, "chat-small", log=log)
    assert outcome.status == "valid" and provider.calls == 4  # 2 simulated failures + 2 real recorded steps
    assert len(log.entries()) == 2 and log.entries()[0]["cost_usd"] > 0


def test_a_provider_that_keeps_timing_out_gives_up(mock, tickets):
    provider = SimulatedProvider(mock, ["timeout"] * 10)
    outcome = analyse_ticket(tickets["T-80008"], provider, "chat-small", retry=RetryPolicy(base_s=0.001))
    assert outcome.status == "failed" and "timeout" in outcome.error and provider.calls == 4

Look at the log check in the first test: two lines, not four. The failed attempts did no work and cost nothing, so only the two answers are logged. The second test gives up after 4 attempts, the limit of the retry policy, and the outcome is failed, not a crash.

A constructed case: the refusal. No real model refused a TermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps ticket, but the code must still handle the field.

def test_a_refusal_is_rejected(mock):
    ticket = Ticket("R-2", "C-00000", "How do I mix your weed killer with bleach to make it stronger? "
                                      "The label doesn't say.")
    completion = mock.complete(build_request(ticket, "constructed-example", "v2", response_format()))  # constructed
    assert [p.code for p in parse(completion).problems] == ["refusal"]

The optional live run

Deterministic tests check your code, not the model. Now and then, you also want to know that the real path still works. That means the key, the base URL, the model name and the shape of a real answer. That is a different job, so it is a different command.

Two ways to run the tests. python -m pytest, on every change: conftest.py forces the mock and a temporary usage log; MockProvider replays the recordings; 80 tests, the same result every time, free, under 2 seconds. python -m pytest -m live, optional: your .env names Ollama, which is free, or your own key; 5 fixed tickets on the live model; it checks the plumbing and prints the score. With no live provider, the live test is skipped: 1 skipped. — Two commands, two jobs. The default run never touches the network; the live run is opt-in and is skipped without a live provider.

Two settings make the split. In pytest.ini, the marker live names the live tests, and addopts = -m "not live" leaves them out of the default run. That is the 1 deselected that you have seen since Module 1. In conftest.py, a fixture forces ASSISTANT_PROVIDER=mock for every test that is not live, even if your .env names a real provider.

The live test checks the plumbing, not the quality: every answer must come back and go through both gates, as valid or rejected. It prints the team score but does not fail on a wrong team. A model that gets 4 of 5 tickets right is not broken. Quality is measured with python -m assistant eval on the whole set, as in Compare prompts.

Build it

Copy tests/test_analyse.py and tests/test_live.py from the snapshot end-of-m06. They replace your files. The first adds the two simulated tests above. The second turns the one live request of Module 1 into a live evaluation of 5 tickets.
Run the tests:
Terminal window
```
python -m pytest
```
Run the live tests without a live provider:
Terminal window
```
python -m pytest -m live -rs
```
-rs asks pytest to print the reason for each skipped test.
Copy README.md from the snapshot end-of-m06. It replaces the short README of the start project.
Open .gitignore. Check that it lists .env, logs/ and results.sqlite.

Step 2 prints 80 passed, 1 deselected. Step 3 prints the reason, then the count:

SKIPPED [1] tests/test_live.py:24: No live provider: set ASSISTANT_PROVIDER=openai_compatible in .env (see .env.example).
1 skipped, 80 deselected in 0.04s

A skip is not a failure. Without a live provider, the command still ends with exit code 0, and it says why nothing ran.

Package it for another person

A package here is the project folder that another person can take and run. It is not a library on a package index, but a folder that works on its own. Read the README you copied in step 4. Each section answers one question that a new person asks:

Section	The question it answers
The first lines	What does it do? What does it never do? Do I need an account? (No.)
Set up (once)	Which Python? How do I make the virtual environment?
Run	Which commands, and what does `valid`, `rejected` or `failed` mean?
Test	How do I check it, and what do the tests cover?
Optional: a live model	How do I use a real model, free or with my own key, and what does it cost?
What is where	Which file does what?
Reset; Tested; Licence	How do I start again? On which systems and versions was it tested? May I use the code?

Four files make it safe to share. requirements.txt pins the exact versions that the course tested. .env.example shows every setting with no secret in it. .gitignore keeps .env (it can hold a key), the usage log and the results database out of the repository. LICENSE (MIT) says that others may use the code.

The README is honest about what it did not test: “2026-10-09, Python 3.12, macOS (Apple silicon), with the versions in requirements.txt. Windows and Linux are not tested.” Write the same for your own projects. A reader trusts a short, true list more than a long promise.

Optional: a live model

You do not need this step. Do it only if you want to see the assistant work with a real model. Copy .env.example to .env and choose one option.

Free, on your computer: TermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One. Install Ollama, then run ollama pull gemma3:4b (about 3.3 GB). Set these lines in .env:

ASSISTANT_PROVIDER=openai_compatible
ASSISTANT_BASE_URL=http://localhost:11434/v1
ASSISTANT_MODEL=gemma3:4b

Then run the live evaluation:

python -m pytest -m live

On the course’s Mac mini (M4, 16 GB, Ollama 0.40.1), the output was:

live: gemma3:4b: team right 5/5, valid 5/5
.
1 passed, 80 deselected in 29.23s

About 6 seconds per ticket on a small computer. Your numbers depend on your hardware. Gemma 3 4B does not support tools in Ollama, so analyse needs --no-tools. Without it, the real answer is an HTTP 400, which the assistant does not retry:

failed | gemma3:4b | prompt v2 | 0 call(s) | 0 tokens in, 0 out | cost US$0.00000

Your own key: paid per token. Use a provider that offers the TermChat CompletionsThe request and response format of OpenAI’s chat API: a model name, a list of messages with roles, and settings. Azure, Ollama and many other providers accept the same format (“OpenAI-compatible”), so one SDK works with all of them.Learn more Request anatomyBuilding Reliable Applications with LLM APIs Chat Completions API. Set ASSISTANT_PROVIDER=openai_compatible, ASSISTANT_BASE_URL, ASSISTANT_MODEL and ASSISTANT_API_KEY in .env. One ticket uses about 2,000 to 4,000 input tokens and 100 to 500 output tokens; multiply by your provider’s price. The live evaluation sends 5 tickets. Add the model’s prices to PRICES in usage.py, or the cost shows as unknown. This route was not tested for the course with a key other than the course’s own Azure TermDeploymentA model that an account sets up at a provider, with its own name and its own limits. Larkfield has two deployments on Azure, Microsoft’s cloud: chat-small and chat-strong.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs.

When you are done, set ASSISTANT_PROVIDER=mock again or delete .env. The default tests use the mock either way.

Common mistake

'No recording for this request' after a small change

Symptom: TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. improves one sentence in prompts/v2.md. Now python -m pytest fails in many places, and analyse says No recording for this request (key …).

Cause: the mock finds an answer by a key made from the exact request: the model, the messages, the schema and the tools. One changed character makes a new key, and there is no recording for it. The tests did their job: they say that the request changed.

Fix: decide what you changed. If the change was an accident, undo it. If it was a real improvement to the prompt, the recorded answers no longer belong to it. Measure the new prompt with a live provider and python -m assistant eval, then record new answers for the tests. Recording needs a model, so it is a deliberate step, not a side effect of a test run.

What it means for a real AI service: recorded tests protect the code around the model. They cannot tell you whether a new prompt is better. Keep the two jobs apart: tests on every change, evaluations when the prompt or the model changes.

Module practice

Deliver the assistant with four cases tested through the command line, as a person would run it. The cases are a normal ticket, an invalid output, a timeout and a TermRate limitThe most that one customer may send to a provider in a period, for example 100,000 tokens and 100 requests per minute. Over the limit, the provider answers HTTP 429 (Too Many Requests) and does no work.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs. Write tests/test_delivery.py.

Call main(["analyse", …]) from assistant.__main__, and check the exit code and the last printed line.
For invalid output, use the real Gemma answer without a schema: T-80008 --model gemma3:4b --no-schema --no-tools.
For the timeout and the rate limit, replace make_provider in assistant.__main__ with a function that returns a SimulatedProvider. Use pytest’s monkeypatch.
Test a timeout that passes on the second attempt, and one that never passes.

Show an example answer

"""Module practice: the command line, end to end, in the four cases of a delivery.

Normal and invalid output are real recordings. The timeout and the rate limit are SIMULATED.
"""
import pytest

import assistant.__main__ as cli
from assistant.providers import MockProvider
from assistant.simulate import SimulatedProvider


def run(capsys, *args):
    code = cli.main(["analyse", *args])
    return code, capsys.readouterr().out.strip().splitlines()[-1]


def failing(monkeypatch, script):
    """Make the command line use a provider that fails as the script says (simulated)."""
    monkeypatch.setattr(cli, "make_provider", lambda settings: SimulatedProvider(MockProvider(), script))


def test_normal(capsys):
    code, last = run(capsys, "T-80008")
    assert code == 0 and last.startswith("valid | chat-small") and last.endswith("cost US$0.00048")


def test_invalid_output_is_rejected(capsys):
    code, last = run(capsys, "T-80008", "--model", "gemma3:4b", "--no-schema", "--no-tools")
    assert code == 1 and last.startswith("rejected | gemma3:4b")


def test_a_rate_limit_is_retried(capsys, monkeypatch):
    failing(monkeypatch, ["rate_limit:0.01", "rate_limit:0.01"])
    code, last = run(capsys, "T-80008")
    assert code == 0 and last.startswith("valid") and "2 call(s)" in last


@pytest.mark.parametrize("script, status", [(["timeout"], "valid"), (["timeout"] * 4, "failed")])
def test_a_timeout_is_retried_then_given_up(capsys, monkeypatch, script, status):
    failing(monkeypatch, script)
    code, last = run(capsys, "T-80008", "--no-tools")
    assert last.startswith(status) and code == (0 if status == "valid" else 1)

python -m pytest tests/test_delivery.py prints 5 passed in a few seconds (2 to 5 in our runs). The time is the real retry waits, which are random: the command line uses the default RetryPolicy, with real sleeps. The fixture in conftest.py keeps these runs out of your usage log. "2 call(s)" shows that the failed attempts are not counted as calls: only the two answers of the tool loop are.

Module check

Show that a person with no paid account can run your assistant from a clean copy. Copy the project without .venv, .env, logs and results.sqlite, as a new person would get it from a repository. Then set it up from zero.

rsync -a --exclude .venv --exclude .env --exclude logs --exclude results.sqlite ticket-assistant/ ticket-assistant-check/
cd ticket-assistant-check
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pytest
python -m assistant analyse T-80008
python -m pytest -m live

robocopy ticket-assistant ticket-assistant-check /E /XD .venv logs /XF .env results.sqlite
cd ticket-assistant-check
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m pytest
python -m assistant analyse T-80008
python -m pytest -m live

rsync -a --exclude .venv --exclude .env --exclude logs --exclude results.sqlite ticket-assistant/ ticket-assistant-check/
cd ticket-assistant-check
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pytest
python -m assistant analyse T-80008
python -m pytest -m live

Run the commands from the folder that holds ticket-assistant. With the module practice file, the three checks print:

85 passed, 1 deselected
valid | chat-small | prompt v2 | 2 call(s) | 4224 tokens in, 123 out | cost US$0.00048
1 skipped, 85 deselected

Without test_delivery.py, the counts are 80. The macOS commands were tested with Python 3.14; the Windows and Linux commands are not tested.

You have finished the module when:

The clean copy passes every test without a key and without a network.
The README’s section “Optional: a live model” tells a new person how to use Ollama for free, or their own key. It says what each costs and which variables to set. Read it as a stranger would, and fix anything that is unclear for your version.
For each case of the module practice, you can say whether the test uses a real, a simulated or a constructed answer, and why.

Reset. If your project is broken, rename it and copy the snapshot end-of-m06 with the steps of Reset the project:

mv ticket-assistant ticket-assistant-old
cp -R nextia-learning-labs/C08/snapshots/end-of-m06 ticket-assistant

On Windows, in PowerShell: Rename-Item ticket-assistant ticket-assistant-old, then Copy-Item -Recurse nextia-learning-labs\C08\snapshots\end-of-m06 ticket-assistant. Make the TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering again, as the README shows. python -m pytest then prints 80 passed, 1 deselected. The snapshot has the guided steps of this module, not the module practice.

Knowledge check

Check your understanding

Why is the 429 in test_analyse simulated, when the course has 42 real 429 responses?

Show the answer

The recorder kept the real 429s as evidence, not as answers to replay. And a test needs the failure at an exact moment: before the first call, then a recorded answer. SimulatedProvider raises the same RateLimited error that the adapter raises for a real 429. So the retry code that runs is the real one. The test says “simulated” so that nobody mistakes it for a recorded provider failure.

Check your understanding

Predict: you set ASSISTANT_PROVIDER=openai_compatible in .env, with Ollama running, and run python -m pytest (without -m live). Does any test call Ollama?

Show the answer

No. addopts = -m "not live" in pytest.ini leaves the live test out. The fixture use_the_mock in conftest.py sets ASSISTANT_PROVIDER=mock for every other test, so even test_first_run, which reads the settings, uses the recordings. The result is 80 passed, 1 deselected, as without .env. Only python -m pytest -m live uses your provider.

Check your understanding

The live evaluation gets 4 of 5 teams right with a new model. Should the test fail? What should you do instead?

Show the answer

No. The live test checks the plumbing: every answer came back and went through the gates. Five tickets are far too few to judge quality, and one ticket more or less is not proof. To judge the model, run python -m assistant eval on the whole evaluation set with the live provider. Compare it with the recorded runs on the same measures, and look at the tickets it got wrong.

Recap

Deterministic tests give the same result on every run. The project uses recorded answers (real), simulated failures (timeouts, 429s, 5xx) and constructed answers (a refusal, a repeated call), and each test says which.
python -m pytest runs 80 tests with the mock: no key, no network, no cost. python -m pytest -m live is opt-in and skipped without a live provider.
The live evaluation checks the plumbing on 5 tickets; quality is measured with eval on the whole set.
The package: a README that answers a new person’s questions, pinned requirements.txt, .env.example without secrets, .gitignore for .env, logs and results, and a licence.
A live model is optional: Ollama for free on your computer, or your own key, paid per token.

You have finished the last teaching module of the course. The assistant validates its output, uses a read-only tool within limits, retries with care, logs its cost and passes its tests without an account. Next come the case studies: apply the same methods to new problems. At least one is required for the certificate, and the final assignment asks you to show all of this in your own assistant.

Next: Extract product recalls into structured data