Keep the documentation true

What you will be able to do

Make it impossible to change the API’s contract by accident. A test compares the API with a reviewed copy of its contract, and a second test sends the documented example. If either one no longer matches, the test fails before any client sees the change.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• How to write and run pytest tests, from Focused tests: From assert to a test.
• The documented examples, from Documentation people can use: Worked example: the models.

What you need on your computer

• The documented API, from Documentation people can use: Guided practice: document the API.
• requirements.txt and requirements-lock.txt in the project. You made them in step 4 of Environment and first endpoint: Step 1: make the project. In this lesson, you add the test tools to them.
• The development server runs, as in Environment and first endpoint: Step 3: start the server.
• If your project is missing or broken, copy the snapshot end-of-m03, the project at the end of Module 3, as in Use a snapshot. Then do Documentation people can use: Guided practice: document the API again.

Tested with Python 3.14.6, FastAPI 0.142.2, pytest 9.1.1 and httpx2 2.13.1 on macOS.

Two ways documentation goes wrong

The documentation is generated from the code, so it changes when the code changes. That is good, until a change is a mistake:

1. The contract changes, and nobody decided it. A “small clean-up” renames model_version to version. The documentation shows the new name, and is correct. But every client still reads the old name, and breaks, as in Resource and endpoint design. The documentation was true, and the change was still wrong.
2. The documentation and the behaviour disagree. One case is an example that the API rejects. Another is a response that the code changes in a handler without telling FastAPI, as the 422 shape did in this module.

A test can catch both. For the first, keep a reviewed copy of the contract in the project, and compare the API with it. For the second, send the documented examples and compare the results.

Install the test tools

1. Add the tools for working on the API to requirements.txt:

   requirements.txt

   # Needed to run the API
   fastapi[standard]==0.142.2
   
   # Needed to work on the API
   pytest==9.1.1
   httpx2==2.13.1

   FastAPI’s test client sends requests to the application without a server. It uses the package httpx2 for that.

2. Install them, and update the lock file:

   python -m pip install -r requirements.txt
   python -m pip freeze > requirements-lock.txt

3. Make the folders tests and contract.

Worked example: the reviewed contract

A small module writes the current OpenAPI document to contract/openapi.json, formatted, so that it is easy to read and compare:

ticket_api/export_contract.py

"""Write the current OpenAPI document to contract/openapi.json."""

import json
from pathlib import Path

from ticket_api.main import app

path = Path("contract/openapi.json")
path.parent.mkdir(exist_ok=True)
path.write_text(json.dumps(app.openapi(), indent=2) + "\n", encoding="utf-8")
print(f"Wrote {path}")

app.openapi() returns the same document that /openapi.json serves, as a dictionary. Amira runs it once, reads the file to check that it says what she wants, and keeps it in the project:

Terminal

python -m ticket_api.export_contract

Output

Wrote contract/openapi.json

From now on, contract/openapi.json is the contract that she has reviewed. If she uses Git, she commits it, and every change to it appears in a pull request for a reviewer to see.

Then two tests:

tests/test_contract.py

import json
from pathlib import Path

from fastapi.testclient import TestClient

from ticket_api.main import app

CONTRACT = Path(__file__).parent.parent / "contract" / "openapi.json"
client = TestClient(app)


def test_openapi_matches_the_reviewed_contract():
    current = json.dumps(client.get("/openapi.json").json(), indent=2) + "\n"
    reviewed = CONTRACT.read_text(encoding="utf-8")
    assert current == reviewed, (
        "The API contract changed. If the change is intended, run "
        "python -m ticket_api.export_contract and review the difference."
    )


def test_documented_example_gives_the_documented_result():
    schemas = client.get("/openapi.json").json()["components"]["schemas"]
    request_example = schemas["TicketIn"]["examples"][0]
    response_example = schemas["Classification"]["examples"][0]
    response = client.post("/v1/classify", json=request_example)
    assert response.status_code == 200
    assert response.json() == response_example

Part	Why
TestClient(app)	Sends requests to the application in memory, with no server and no port. Module 6 uses it for all endpoint tests.
json.dumps(..., indent=2)	Compares the documents as formatted text, so a failure shows the changed lines, not two long dictionaries.
The message in assert	Says what to do when the change is intended. A test failure must explain itself.
The second test	Takes the examples from the document itself, so it checks exactly what a developer sees in /docs.

Terminal

python -m pytest

Output (last lines)

tests/test_contract.py ..                                                [100%]

============================== 2 passed in 0.14s ===============================

Catch an accidental change

To check the test, Amira makes the “small clean-up”. In models.py, she renames model_version to version in Classification and in its example. In routes.py, she changes model_version=classifier.version to version=classifier.version. The API still works, and /docs is correct. The test fails:

Output (shortened)

E       AssertionError: The API contract changed. If the change is intended, run python -m ticket_api.export_contract and review the difference.
E         Skipping 4118 identical leading characters in diff, use -v to show
E         -          "model_version": {
E         ?           ------
E         +          "version": {
E                       "type": "string",
...
FAILED tests/test_contract.py::test_openapi_matches_the_reviewed_contract - A...
========================= 1 failed, 1 passed in 0.13s ==========================

The - line is the reviewed contract, and the + line is the API now. The rename is visible, and it is a breaking change, so she undoes it.

When a change is intended, for example a new optional field, the steps are:

1. Run python -m ticket_api.export_contract to write the new contract.
2. Read the difference. With Git: git diff contract/openapi.json. Without Git, compare the file with your last copy in VS Code (File › Compare Active File With…).
3. Decide if it is breaking (Module 3). If yes, it needs a new version, not only a new file.
4. Run the tests again. They pass, and the reviewed contract matches the API.

Decide what to expose in production

The documentation pages are useful for anyone who builds a client. They also show an attacker every endpoint and every field. What to do depends on who the API is for:

API for	Documentation in production
The public, or many outside developers	Keep /docs and /openapi.json public. They are the product’s manual.
Your own frontend, or a few known teams	Turn the pages off, and give the teams contract/openapi.json instead.
Only other services inside your company	Turn them off.

Turning them off is not a security control. Every endpoint must still check its input and its access, because an attacker can find endpoints without documentation. It only removes a map. Amira makes it a setting, so each place where the API runs can decide:

ticket_api/main.py (changes)

show_docs = os.environ.get("SHOW_DOCS", "true").lower() == "true"

app = FastAPI(
    # ... title, version, description and openapi_tags as before
    docs_url="/docs" if show_docs else None,
    redoc_url="/redoc" if show_docs else None,
    openapi_url="/openapi.json" if show_docs else None,
)

None turns a page off. With SHOW_DOCS=false, the three paths return 404 in the usual error shape, and the API works as before.

Guided practice: protect the contract

1. Install the test tools, and make ticket_api/export_contract.py and tests/test_contract.py.
2. Run python -m ticket_api.export_contract. Open contract/openapi.json, and find the Classification schema and its example.
3. Run python -m pytest. Both tests pass.
4. Rename model_version to version, as in the worked example. Run the tests. The contract test fails, and shows the rename.
5. Undo the rename. Run the tests again. Both pass.
6. Change the TicketIn example’s subject to "Hello", without a change to the Classification example. Run the tests. Which test fails now, and why? Then undo it.
7. Add the SHOW_DOCS setting. Start the server with SHOW_DOCS=false ($env:SHOW_DOCS = "false" in PowerShell), and check that /docs returns 404 and /health still works. Then start it again without the variable.

Show the answer for step 6

Both tests fail. The contract test fails because the example is part of the document, so the document changed. The example test fails because "Hello" is classified as other, which is not the documented response. The documentation would show a request and a response that do not belong together.

Your turn: an intended change

The help desk asks for a new optional response field, explanation: a short text that says why the classifier chose the category, for example "matched: charged, invoice". Add it, so that the contract changes on purpose:

1. Add the field to Classification, with a description and a value in the example. Set it in the route. (The classifier can return the matched words in Prediction.)
2. Run the tests, and read the failure.
3. Decide: is this change breaking? What new version does the API get?
4. Update the reviewed contract, review the difference, and run the tests again.

Show the answer to step 3

It is not breaking: old clients do not read explanation, and every field that they read is still there. So the path stays /v1, and the version goes from 1.0.0 to 1.1.0. If you made it required for requests instead, old clients would fail, and it would be a breaking change. Remove the field again before the next module, and export the contract again, so that your project matches the course.

Common mistake

Updating the contract to make the test pass

Symptom: the contract test fails, and the quickest fix is to run export_contract and run the tests again. They pass. A week later, a client breaks.

Cause: the test only shows that the contract changed. Running the export accepts any change without reading it. The reviewed copy is only useful if someone reviews it.

Fix:

1. Before you export, read the test’s diff, and say in one sentence what changed and why.
2. Decide if it is breaking. If it is, it needs a new path version or an agreement with every client, not only a new file.
3. If you use Git, keep the change to contract/openapi.json in the same commit as the code that caused it, so the reviewer sees both.

Check your understanding

The tests pass on your computer, but fail on a colleague's computer with a difference in the contract. Neither of you changed the code. What could be different?

Show the answer

The packages. A different version of FastAPI or Pydantic can write the same models slightly differently in the OpenAPI document. For example, it can add a new field in a schema. That is why the project has a lock file: install from requirements-lock.txt, so everyone has the same versions. Another cause: SHOW_DOCS=false in one terminal, which removes /openapi.json.

Check your understanding

Module check: name the two tests in this lesson, and for each one, a change that makes it fail.

Show the answer

test_openapi_matches_the_reviewed_contract fails for any change to the contract: a renamed field, a new endpoint, a different limit, a changed description or example. test_documented_example_gives_the_documented_result fails when the documented request no longer gives the documented response. Examples are a changed classifier rule, a new required field, or an example that is not valid. Together, they make sure that a change to the contract is a decision, and that the examples in /docs work when sent unchanged.

See it end to endServe a house-price modelCase study

The house-price case study keeps a reviewed OpenAPI contract and a test that fails when the code drifts from it.

See it end to endBuild a data API for city service requestsCase study

The data-API case study does the same for an API with filters and pagination.

Recap

• Keep a reviewed copy of the OpenAPI document in the project, and compare the API with it in a test.
• Compare as formatted text, so a failure shows exactly which lines changed.
• Test the documented examples by sending them, so /docs never shows a request that fails.
• When the contract test fails, read the difference and decide. Do not export only to make the test pass.
• Expose the documentation in production only when the API’s users need it. Hiding it is not security.

Your API now documents itself, and the documentation stays true. In the next module, you use the same OpenAPI document in Postman.

Next: First requests in Postman