Contract and compatibility tests

What you will be able to do

Pin what the server promises, and prove who can use it. You record the server’s contract in a file, and a test compares the server with it after every change. Then you build a compatibility matrix: each client, each transport and each protocol era, with the same checks. Every row names its era.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Tool descriptions and schemas, from Tools.
• Protocol errors and tool errors, from Inspect behaviour.
• The two protocol eras, from Version awareness.
• Why a description is part of the attack surface, from Untrusted inputs and outputs.

Your project

• ~/projects/support-mcp, with the virtual environment active: python -m pytest -q prints 86 passed.
• Node.js 22.19 or later, for MCP Inspector.

A printed timetable

A bus company prints its timetable. Shops, schools and hotels copy it and pin it on their walls. One day, the company moves the 8:10 bus to 8:05. The company’s staff know. Nobody tells the hotel. The next morning, a guest misses the bus.

The bus still runs, so the company sees no problem. The real problem is that other people copied a promise, and the promise changed quietly. A careful company keeps a copy of what it published. Before any change, it compares the new plan with that copy, and it tells everyone who copied it.

The problem: a quiet change in what clients read

An MCP server publishes a promise too. Every client that connects reads the tool names, the descriptions, the input and output schemas, the annotations, the resource template and the prompt. A host passes the descriptions to its model. A partner’s app may build its screens from the schemas.

Here is a change that looks harmless. Amira tidies the description of get_ticket and removes its last sentence, “The text is the customer’s words.” The tool still works. The 86 tests still pass:

# end-of-m06, with the sentence removed from get_ticket's docstring
$ python -m pytest -q
..............                                                           [100%]
86 passed in 11.46s

But that sentence was part of the defence from Untrusted inputs and outputs. It tells every model that the ticket text comes from a customer, not from the help desk. Now it is gone, and nothing noticed.

The second problem is about clients. “It works with MCP” says nothing about which client. You ran MCP Inspector in Inspect behaviour, and the course team first wrote down “Inspector: pass”. Later, they looked at the messages. Inspector 2.10.1 had connected with the older initialize handshake, revision 2025-11-25. The “pass” said nothing about the current revision, 2026-07-28.

The idea: record the promise, and test each client

Two tools answer these two problems.

A contract is everything that a client can see of the server: the answers of discovery, written to a file. A contract test compares the running server with that file after every change. Any difference fails the test, until a person reads the difference and accepts it. The file is the company’s copy of its published timetable.

The contract file holds the shapes. It does not hold the errors. So a second test pins the error shapes: what a validation failure, an unknown tool and a missing resource look like to a client.

A compatibility matrix is a table. Each row is one client, one transport and one protocol era. Each row runs the same checks: discovery, a valid call, a validation failure, a resource read and the prompt. A row passes only if all its checks pass. And each row names its era, because a pass in one era proves nothing about the other.

Record the contract

1. Copy the contract script and its tests from the snapshot end-of-m07:

   macOS / Linux

   S=../nextia-learning-labs/C28/snapshots/end-of-m07
   cp "$S/scripts/contract.py" scripts/
   cp "$S/tests/test_contract.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C28\snapshots\end-of-m07"
   Copy-Item "$S\scripts\contract.py" scripts\
   Copy-Item "$S\tests\test_contract.py" tests\

2. Compare the server with its contract. There is no contract file yet, so every line is new:

   python -m scripts.contract

   --- contract
   +++ server
   @@ -0,0 +1,317 @@
   +{
   +  "prompts": [
   …

   The script exits with code 1: the server and the file differ.

3. Read the 317 lines once. They are exactly what a client receives. Then accept them:

   python -m scripts.contract --write

   Wrote capabilities.json: 3 tools, 1 resource template, 1 prompt

4. Compare again:

   python -m scripts.contract

   The server matches contract/capabilities.json.

5. Run the three new tests:

   python -m pytest -q tests/test_contract.py

   ...                                                                      [100%]
   3 passed in 0.21s

The script asks the server for its discovery answers, in process, with the SDK’s own client. It writes them as sorted JSON, so the same server always gives the same file:

scripts/contract.py (the contract)

async def current() -> dict:
    async with Client(build_server()) as c:
        tools = (await c.list_tools()).tools
        templates = (await c.list_resource_templates()).resource_templates
        prompts = (await c.list_prompts()).prompts
        return {
            "protocol": c.protocol_version,
            "tools": [t.model_dump(mode="json", by_alias=True, exclude_none=True) for t in tools],
            "resourceTemplates": [t.model_dump(mode="json", by_alias=True, exclude_none=True) for t in templates],
            "prompts": [p.model_dump(mode="json", by_alias=True, exclude_none=True) for p in prompts],
        }

Your file equals the snapshot’s contract/capabilities.json, byte for byte. Commit it with the code. From now on, a change to the contract shows up as a diff of this file in every review.

Catch the quiet change

Now remove the sentence again. Open support_mcp/server.py, find get_ticket, and delete “The text is the customer’s words.” from its docstring. Then compare:

python -m scripts.contract

--- contract
+++ server
@@ -123,7 +123,7 @@
         "openWorldHint": false,
         "readOnlyHint": true
       },
-      "description": "Get one support ticket of this organization by its ID (for example T-30002): the customer's first name, subject, text, status and the last 3 messages. The text is the customer's words.",
+      "description": "Get one support ticket of this organization by its ID (for example T-30002): the customer's first name, subject, text, status and the last 3 messages.",
       "inputSchema": {

The 86 older tests missed this change. The contract script shows the old line and the new line, and exits with 1. The test fails too:

$ python -m pytest -q tests/test_contract.py
F..                                                                      [100%]
FAILED tests/test_contract.py::test_the_server_matches_the_recorded_contract
1 failed, 2 passed in 0.23s

Put the sentence back, and the script prints The server matches contract/capabilities.json. again.

The failure does not mean “you broke something”. It means “you changed a promise: decide”. Sometimes the change is right. Then run --write, and the diff of contract/capabilities.json goes into the review. At Larkfield, Kwame reads every such diff. He asks one question: which clients see this change, and do they need to know?

Pin the error shapes

The contract file holds no errors. The third test pins them:

tests/test_contract.py (the error shapes)

def test_error_shapes_are_stable():
    async def go():
        async with Client(build_server()) as c:
            out = {
                "validation": (await c.call_tool("get_ticket", {"ticket_id": "x"})).is_error,
                "unknown_tool": (await c.call_tool("nope", {})).is_error,
            }
            with pytest.raises(MCPError) as e:
                await c.read_resource("policy://larkfield/no-such-policy")
            out["resource_not_found"] = e.value.error.code
            return out

    assert anyio.run(go) == {"validation": True, "unknown_tool": True, "resource_not_found": -32602}

Why does this matter? The course team tried a one-word change in server.py: ResourceError instead of ResourceNotFoundError for a missing policy. The contract script said The server matches contract/capabilities.json. The error-shape test failed:

E       AssertionError: assert {'validation'...ound': -32603} == {'validation'...ound': -32602}
E         Differing items:
E         {'resource_not_found': -32603} != {'resource_not_found': -32602}

On the wire, the message and data.uri stayed the same. Only the code changed: −32603, “internal error”, instead of −32602, “invalid parameters”. A client that shows “this policy does not exist” for −32602 would now show “the server is broken”.

The test also pins a known difference. For an unknown tool, the specification asks for a protocol error, −32602. SDK 2.3.0 answers with a tool result, isError: true, “Unknown tool: nope”. The test records what this server does today. If an SDK upgrade changes it, the test fails, and you find out before your clients do.

Two real changes and two checks. Each check sees one kind of change, so run both after every change.

Text description of the diagram

The change	Contract file (python -m scripts.contract)	Error-shape test
get_ticket’s description loses “The text is the customer’s words.” (the 86 older tests pass)	caught: exit 1, the old and the new description	not seen: passed
ResourceError instead of ResourceNotFoundError for a missing policy	not seen: “The server matches contract/capabilities.json.”	caught: −32603 instead of −32602, same message and data.uri

One era is not every era

Back to Inspector. The course team ran the same Inspector command twice against the stdio server: a get_ticket call. The second run added --protocol-era modern. The project’s scripts/tee.py, which you meet in the next lesson, recorded every message:

The same Inspector command, recorded in both eras. Its default is the legacy handshake.

Text description of the diagram

• Default (legacy handshake, 2025-11-25): initialize, its result, notifications/initialized, tools/list and tools/call. No request carries _meta. The tool list is 4,315 bytes; the call’s result is 1,213 bytes.
• --protocol-era modern (2026-07-28): server/discover, then subscriptions/listen (a stream for change notifications, see Optional and evolving features), then tools/list and tools/call with _meta. The tool list is 4,460 bytes, and the call’s result is 1,325 bytes, because each result carries resultType and the server’s name. When Inspector disconnects, the listen stream ends with error −32000.

Both runs pass. But they test different protocols. So the matrix needs a row for each era, and each row must say which one it is.

Build the matrix

1. Copy the compatibility script, its test and the checklist:

   macOS / Linux

   cp "$S/scripts/compat.py" scripts/
   cp "$S/tests/test_compat.py" tests/
   cp "$S/docs/compatibility.md" docs/

   Windows (PowerShell)

   Copy-Item "$S\scripts\compat.py" scripts\
   Copy-Item "$S\tests\test_compat.py" tests\
   Copy-Item "$S\docs\compatibility.md" docs\

2. Run the matrix. The script starts the server in process, as a stdio child process and over Streamable HTTP, in both eras. The server’s log lines go to stderr; hide them with 2>/dev/null (PowerShell: 2>$null):

   python -m scripts.compat 2>/dev/null

   MCP Python SDK 2.3.0 Client
   transport | mode | protocol | discovery | valid_call | validation_failure | resource_read | prompt | host_answer | no_token_refused
   in-process | auto | 2026-07-28 | pass | pass | pass | pass | pass | pass | -
   stdio | auto | 2026-07-28 | pass | pass | pass | pass | pass | pass | -
   Streamable HTTP | auto | 2026-07-28 | pass | pass | pass | pass | pass | pass | pass
   in-process | legacy | 2025-11-25 | pass | pass | pass | pass | pass | pass | -
   stdio | legacy | 2025-11-25 | pass | pass | pass | pass | pass | pass | -
   Streamable HTTP | legacy | 2025-11-25 | pass | pass | pass | pass | pass | pass | pass

   Six rows, and the protocol column names the revision of each connection. host_answer runs the course’s host with the mock model on the same connection. On HTTP, no_token_refused checks the 401 without a token.

3. Run Inspector in its default era, the legacy handshake:

   npx @modelcontextprotocol/inspector@2.10.1 --cli python -m support_mcp -- --method tools/call --tool-name get_ticket --tool-arg ticket_id=T-30002

   {
     "content": [
       {
         "type": "text",
         "text": "{\n  \"ticket_id\": \"T-30002\",\n  \"customer_name\": \"Hugo\", …"
       }
     ],
     "structuredContent": {
       "ticket_id": "T-30002",
   …

4. Run it again in the current era:

   npx @modelcontextprotocol/inspector@2.10.1 --cli python -m support_mcp -- --protocol-era modern --method tools/call --tool-name get_ticket --tool-arg ticket_id=T-30002

   {
     "_meta": {
       "io.modelcontextprotocol/serverInfo": {
         "name": "support-mcp",
         "version": "1.0.0"
       }
     },
     "content": [
   …

   The _meta block at the top is the visible sign of revision 2026-07-28: every result names the server. Repeat the other checks from Inspect behaviour in both eras: tools/list, the validation failure with --tool-arg 'ticket_id="30002"' (exit code 5, tool_is_error), resources/read and prompts/get.

5. Run all the tests:

   python -m pytest -q

   90 passed in 12.30s

   Four tests are new: three in test_contract.py, and test_compat.py, which runs the six rows.

The PowerShell quoting of --tool-arg 'ticket_id="30002"' is different, and the course did not test it on Windows.

An older client

Version awareness promised a test with an older client. The SDK’s legacy mode speaks the old handshake, but it is still the code of SDK 2.3.0. A real older host ships an older library. So the course team installed the older SDK line, version 1.30.0, in a separate virtual environment. They connected its ClientSession to support-mcp and ran the same five checks. This is a real older library, not a constructed one:

client: MCP Python SDK 1.30.0 ClientSession (stdio)
protocol: 2025-11-25
server: support-mcp 1.0.0
discovery: pass
valid_call: pass
validation_failure: pass
resource_read: pass
prompt: pass

Over Streamable HTTP, with a valid token, the result was the same: five passes. The server’s log showed Created new transport with session ID …. The legacy era over HTTP uses a session; revision 2026-07-28 has none. You do not need to repeat this run. To try it, create a second virtual environment and install mcp==1.30.0 there. Never mix the two SDK lines in one environment.

The matrix, with every era named

docs/compatibility.md collects all rows: the six automatic ones, the older client, Inspector in both eras and the course’s host. Desktop assistants and code editors are not tested by the course: the course shows their configuration only.

The compatibility matrix. Blue: the current era. Grey dashed: the legacy handshake.

Text description of the diagram

Client	Transport	Era	Checks	How tested
SDK 2.3.0 Client, auto	in-process, stdio, Streamable HTTP	current, 2026-07-28	5 of 5 pass; no token: 401	scripts.compat
SDK 2.3.0 Client, legacy	in-process, stdio, Streamable HTTP	legacy handshake, 2025-11-25	5 of 5 pass; no token: 401	scripts.compat
SDK 1.30.0 ClientSession (older client)	stdio, Streamable HTTP	legacy handshake, 2025-11-25	5 of 5 pass	by hand, once
Inspector 2.10.1 CLI, default	stdio	legacy handshake, 2025-11-25	5 of 5 pass	by hand, once
Inspector 2.10.1 CLI, --protocol-era modern	stdio	current, 2026-07-28	5 of 5 pass	by hand, once
The course’s host	stdio, Streamable HTTP	2026-07-28; 2025-11-25 with --legacy	call, refusals, both eras	pytest
Desktop assistants, code editors	—	—	not tested by the course	configuration only

The revisions 2024-11-05, 2025-03-26 and 2025-06-18 use the same handshake. SDK 2.3.0 accepts them, but no client with those revisions was tested. The matrix says so, and does not claim them.

When to use these tests, and what they cost

Use a contract test for every server that someone else connects to. It costs one file and one test, and it runs in a fraction of a second. The real cost is discipline: never run --write without reading the diff. Otherwise the test only records your mistakes.

The contract file also changes when you upgrade the SDK, even if your code does not change. A new SDK may write a schema in a different way. That failure is useful: it shows exactly what your clients will see after the upgrade.

The matrix costs more. scripts.compat takes a few seconds, and it runs with your tests. The by-hand rows, the older client and Inspector, can go stale: they are true for the versions in the row, and nothing more. So each row names the client’s version and the era. When a version changes, run the row again.

Do not add rows that you cannot test. “Works with every MCP client” is a promise that nobody can check. A short, honest matrix helps a partner more than a long, hopeful one.

Common mistakes

Common mistake

The contract test fails after an SDK upgrade, and nothing in your code changed

Symptom: test_the_server_matches_the_recorded_contract fails after you change the SDK version.

Cause: the new SDK sends the same tools in a slightly different form: an extra field, a different title, another order.

Fix: run python -m scripts.contract, and read the diff. If only the SDK’s form changed, accept it with --write, and name the SDK version in the commit message. If a description or a bound changed, find out why before you accept it.

Common mistake

Inspector passes, but a current client fails

Symptom: your Inspector checks pass, but a client with revision 2026-07-28 gets an error.

Cause: Inspector 2.10.1 uses the legacy handshake by default. Your checks tested revision 2025-11-25 only.

Fix: run every Inspector check again with --protocol-era modern. Look for the _meta block in the output: it shows that the current era was used.

Reset. If your project is broken, copy the snapshot end-of-m06 as in Where MCP fits, with end-of-m06 instead of start, and repeat this lesson’s steps. Then python -m pytest -q prints 90 passed.

See it end to endAn MCP server over a real open datasetCase study

The case study pins its server’s contract in contract/capabilities.json. When a real model showed that a bound was too large, the contract test showed the fix as a diff to review.

See it end to endSecurity review of an MCP integrationCase study

The security review pins the reviewed tool descriptions in contract/descriptions.json, with a test: when a later fix changes a description, the test fails until a person reviews the new text. It also measures attacks and normal tasks in one table: 13 of 13 attacks reached their goal on the prototype, and 0 of 13 after five fixes, with 5 of 5 normal tasks.

Knowledge check

Check your understanding

Amira removes one sentence from a tool's description, and all 86 older tests pass. Which check fails, what does its output show, and what should Amira do next?

Show the answer

The contract test fails. python -m scripts.contract exits with 1 and prints a diff with the old and the new description of the tool, and test_the_server_matches_the_recorded_contract fails. The older tests passed because none of them pins the text of a description. Amira reads the diff and decides. If the sentence was part of a defence, such as “The text is the customer’s words.”, she puts it back. If the change is wanted, she runs --write, and the diff of contract/capabilities.json goes into the review. Then Kwame can ask who needs to know.

Check your understanding

A colleague writes 'Inspector 2.10.1: pass' in the compatibility checklist. What is missing, and how do you complete the row?

Show the answer

The era is missing. Inspector 2.10.1 connects with the legacy initialize handshake, revision 2025-11-25, unless you pass --protocol-era modern. So the row tested 2025-11-25 only. Write two rows: the default, legacy handshake 2025-11-25, and --protocol-era modern, current 2026-07-28. Run the same five checks in each: discovery, a valid call, a validation failure, a resource read and the prompt. You can see the era in the output: a current-era result starts with a _meta block that names the server.

Recap

• A contract is what clients see after discovery. contract/capabilities.json records it, and a test fails on any difference, until a person reads the diff and accepts it.
• The error shapes need their own test: a one-word change turned −32602 into −32603, and the contract file did not change.
• A compatibility matrix runs the same checks for each client, transport and era. Each row names its era.
• Inspector 2.10.1 uses the legacy handshake by default: test it in both eras. An older client, SDK 1.30.0, passed with revision 2025-11-25 over stdio and HTTP.
• Your project now has 90 passed.

Next: Resource limits and observability