Check parity

Lesson 9
40 min
Module 3: Build the prediction service

Lesson 9 of 15 · Deploying AI Models for Real UsersCheck parity

Course overview

1. Define the inference contract

2. Choose a serving pattern

3. Build the prediction service

4. Choose infrastructure and measure

5. Release, monitor and recover

6. Case studies

What you will be able to do

Prove that the running service gives the same scores as training for fixed tickets: normal, missing, extreme and unknown inputs. Then find out at once when it does not.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

The 14 parity cases in parity_cases.csv and the self-check of load_bundle(), from Bundle model dependencies: The parity cases and Artifact trust and integrity: Six gates, in order.
How the pipeline fills missing numbers (median imputation with indicators), from Preprocess with a pipeline: Problem 2: gaps in Machine Learning: From Problem to Reliable Model.
The service and its routes, from Load and initialize and Connect the API.

What you need on your computer

The project ~/projects/escalation-service with escalation/routes.py and tests/test_api.py, from Connect the API: routes.py. python -m pytest gives 28 passed.
MODEL_SHA256 set in your terminal, as in Load and initialize: Start the service.
If your project is missing or broken, copy the snapshot end-of-m02 and do the two earlier lessons of this module again. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

The same bread in a second shop

A bakery opens a second shop. The new bakers get the same recipe and the same flour. Each morning, the owner buys a few fixed items in both shops, the same loaf and the same cake, and compares them side by side. She does not taste one loaf and decide that everything is fine. A fault in the new oven shows only in the items that need the oven’s top heat.

Your service is the second shop. Training scored tickets in one way, with a table of rows and predict_proba(). The service scores them in another way: JSON, the contract, to_frame(), then the pipeline. The two paths must give the same score for the same ticket.

The problem: one good score proves little

Tomás changed one line in a copy of the service, to “clean” the input: to_frame() now fills a missing number with 0. Then he checked the service: the start-up log, /ready and the example ticket.

INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...) in 660 ms
{"status":"ready","model_version":"1.0.0"}
{"ticket_id":"T-120001","score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[]}

The bundle passed all six checks, including the self-check. /ready says ready. The example ticket gets exactly the right score. “It works. Ship it.”

It does not work. The example ticket has no missing value, so the change does not touch it. Send a guest customer, with customer_tenure_days null, and the service answers 0.250 instead of 0.145. The ticket is now flagged and goes to the senior team. The service treats a guest as a customer of 0 days, which the model learned to see as risky. The self-check did not see this, because it scores the cases with the pipeline directly, not through the service’s own code.

A service that does something that training did not do has training-serving skew. The model is right, but it gets different numbers than the ones it learned from. Skew gives no error. Only a comparison of scores shows it.

Parity: the same tickets, the same scores

The check for skew is parity. Send fixed tickets through the running service, and compare each served score with the training score of the same ticket. Module 1 saved these tickets in the bundle: parity_cases.csv, with an expected_score for each, computed by make_bundle.py.

Two paths for the same tickets. Training side, make_bundle.py, once: the 14 parity tickets as a table go to pipeline.predict_proba, all 14 rows together, which gives expected_score in parity_cases.csv. Serving side, the running service: the same ticket as JSON goes through the contract check, to_frame, preprocessing then the model on one row, and gives the score in the response. Both scores go to one question: is the difference at most the tolerance, 1e-9? Yes: parity, the service does what training did. No: training-serving skew, find the step that differs. — Parity compares the two paths on the same tickets. Every step of the serving side is a place where skew can enter.

The 14 cases are not random. Each group tests one kind of input where a service can go wrong:

6 normal tickets with no missing value: the everyday case.
3 missing: order_value null; a guest with customer_tenure_days null; both.
3 extreme: the edges of the contract. T-900004 has every risk at its maximum and scores exactly 1.0.
2 unknown: channel social; region overseas with team installation.

Why not exactly the same?

You might expect each difference to be 0. Here is one real case from the container. The training score and the served score of ticket T-116942:

training  0.12958362401471518
served    0.12958362401471521

They agree in the first 15 digits and differ only at the end, by 3e-17. A computer stores a score with about 16 significant digits, and the last digit can change for harmless reasons. Training scored 14 rows together, and the service scores one row, so the additions happen in a different order. A different processor (the Linux container instead of macOS) can use different instructions. And pandas’ fast CSV reader can lose the last digit of a 17-digit number when it reads parity_cases.csv. It reads the training score above as 0.1295836240147151, so check_parity.py prints 1.1e-16, not 3e-17. Each effect is about 1e-17 to 1e-16: a 1 in the 16th or 17th decimal place.

So parity compares with a tolerance: the largest difference that still counts as “the same”. The service uses 1e-9. Choose it between two numbers. Rounding noise is about 1e-16, so the tolerance must be far above it: 1e-9 is ten million times larger. The decision compares a score with 0.19, which has two decimals. So you must catch a real fault long before it reaches 0.01, and 1e-9 is ten million times smaller. A tolerance of 0 would fail on noise. A tolerance of 0.01 would pass a real fault that moves every score by 0.009.

check_parity.py

Make the folder parity and the file parity/check_parity.py. It sends each parity case to the running service:

"""Compare the scores of a running service with the scores the model gave in training.

    python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0

Sends every parity case of the bundle to POST /v1/score and compares the
served score with expected_score, the score that the training code gave the
same ticket. A difference larger than the tolerance means the serving path
does something that training did not do (training-serving skew).
"""

import argparse
import json
import os
import sys
import urllib.request
from pathlib import Path

import pandas as pd

TOLERANCE = 1e-9


def as_json_value(value):
    if pd.isna(value):
        return None
    return value.item() if hasattr(value, "item") else value


def served_score(url: str, ticket: dict, key: str | None) -> float:
    request = urllib.request.Request(
        f"{url}/v1/score",
        data=json.dumps(ticket).encode(),
        headers={"Content-Type": "application/json", **({"X-API-Key": key} if key else {})},
    )
    with urllib.request.urlopen(request, timeout=10) as response:
        return json.load(response)["score"]

as_json_value() turns an empty CSV cell into null, so a missing value reaches the service as the contract expects. The script uses only the standard library and pandas, so it runs anywhere the service’s packages are. If API_KEY is set in the terminal, it sends the key.

The second part reads the cases, compares and prints one line per case:

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("url")
    parser.add_argument("bundle", type=Path)
    parser.add_argument("--tolerance", type=float, default=TOLERANCE)
    args = parser.parse_args()

    cases = pd.read_csv(args.bundle / "parity_cases.csv", keep_default_na=False, na_values=[""],
                        dtype={"priority": "Int64", "word_count": "Int64", "customer_tenure_days": "Int64",
                               "prior_tickets_90d": "Int64", "created_hour": "Int64",
                               "prior_escalations_90d": "Int64"})
    key = os.environ.get("API_KEY")
    failed = 0
    print(f"{'case':8} {'ticket':9} {'expected':>10} {'served':>10} {'difference':>10}")
    for row in cases.to_dict("records"):
        expected = row.pop("expected_score")
        case = row.pop("case")
        ticket = {k: as_json_value(v) for k, v in row.items()}
        served = served_score(args.url, ticket, key)
        difference = abs(served - expected)
        ok = difference <= args.tolerance
        failed += not ok
        print(f"{case:8} {row['ticket_id']:9} {expected:10.6f} {served:10.6f} {difference:10.1e}"
              f"{'' if ok else '  FAIL'}")
    print(f"{len(cases) - failed} of {len(cases)} cases match within {args.tolerance:g}.")
    return 1 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

Two details matter. keep_default_na=False, na_values=[""] treats only an empty cell as missing, so a real category such as unknown stays text. The Int64 types keep whole numbers as whole numbers, so priority is sent as 2, not 2.0. The exit code is 1 if any case fails, so a release script can stop on it.

Worked example: check the running service

Start the service in one terminal with fastapi run, as in Load and initialize. In a second terminal, run the check:

python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0

The output:

case     ticket      expected     served difference
normal   T-116940    0.187598   0.187598    2.8e-17
normal   T-116941    0.064735   0.064735    0.0e+00
normal   T-116942    0.129584   0.129584    8.3e-17
normal   T-116943    0.179840   0.179840    5.6e-17
normal   T-116944    0.244735   0.244735    0.0e+00
normal   T-116945    0.113802   0.113802    8.3e-17
missing  T-900001    0.037929   0.037929    7.6e-17
missing  T-900002    0.144791   0.144791    5.6e-17
missing  T-900003    0.075978   0.075978    1.4e-17
extreme  T-900004    1.000000   1.000000    0.0e+00
extreme  T-900005    0.039165   0.039165    0.0e+00
extreme  T-900006    0.094671   0.094671    5.6e-17
unknown  T-900007    0.177527   0.177527    0.0e+00
unknown  T-900008    0.111725   0.111725    5.6e-17
14 of 14 cases match within 1e-09.

All 14 cases match, and the exit code is 0. Even on the computer that trained the model, 9 of the 14 differences are not zero: this is the rounding noise. Your differences can be different, but each one is far below 1e-9.

The service will run for real in a container, which you build in Module 4. Amira already built it, started it with 1 CPU and ran the same command against it. You do not have the image yet, so only read her output (shortened):

case     ticket      expected     served difference
normal   T-116940    0.187598   0.187598    2.8e-17
normal   T-116941    0.064735   0.064735    0.0e+00
normal   T-116942    0.129584   0.129584    1.1e-16
...
missing  T-900001    0.037929   0.037929    9.7e-17
...
14 of 14 cases match within 1e-09.

Linux on the container’s processor gives slightly different last digits: up to 1.1e-16. Still ten million times below the tolerance. You build this image and run this check yourself in Container resource settings: Check parity in the container. Run parity wherever the service runs, because each new place can bring its own small differences, or a large one.

Parity in the test suite

check_parity.py checks a running service from the outside. The same check also belongs in the tests, so every change to the code runs it. Make tests/test_parity.py:

"""The served scores equal the training scores for every parity case."""

import pandas as pd
import pytest

from tests.conftest import BUNDLE

INTS = ["priority", "word_count", "customer_tenure_days", "prior_tickets_90d", "created_hour", "prior_escalations_90d"]
CASES = pd.read_csv(BUNDLE / "parity_cases.csv", keep_default_na=False, na_values=[""], dtype={c: "Int64" for c in INTS})


def as_json(row: dict) -> dict:
    return {k: (None if pd.isna(v) else (v.item() if hasattr(v, "item") else v)) for k, v in row.items()}


@pytest.mark.parametrize("row", CASES.to_dict("records"), ids=lambda r: f"{r['case']}-{r['ticket_id']}")
def test_served_score_matches_training(client, row):
    expected = row.pop("expected_score")
    row.pop("case")
    served = client.post("/v1/score", json=as_json(row)).json()["score"]
    assert served == pytest.approx(expected, abs=1e-9)

parametrize makes one test per case, named by group and ticket, so a failure names the case at once. pytest.approx(expected, abs=1e-9) is the tolerance. The client fixture sends the tickets through the whole app: the contract, to_frame(), the route and the response. Run the suite:

python -m pytest

The output:

..........................................                               [100%]
42 passed in 3.44s

42 tests: 28 from before and 14 parity tests. This is the state of the snapshot end-of-m03.

Explore: which mistakes does parity catch?

The explorer shows real runs. Amira made four copies of the service: the correct one, and three copies with one serving mistake each. She ran each copy in Docker with 1 CPU and sent the 14 cases, with the same comparison as check_parity.py.

Explore

Parity: which cases does each serving mistake break?

Cases within the tolerance: 14 of 14
Largest difference: 1.1e-16
Flags that change: 0 of 14

The service as you built it: missing numbers stay NaN, and the pipeline's imputer fills them. 14 of 14 cases match within 1e-09. The differences are not all zero, but the largest, 1.1e-16, is far below the tolerance.

pass: within the tolerance
FAIL: above the tolerance

Show the exact values

No mistake: 14 of 14 cases pass
Case	Ticket	Expected	Served	Difference	Result
normal	`T-116940`	0.187598	0.187598	2.8e-17	pass
normal	`T-116941`	0.064735	0.064735	0	pass
normal	`T-116942`	0.129584	0.129584	1.1e-16	pass
normal	`T-116943`	0.179840	0.179840	2.8e-17	pass
normal	`T-116944`	0.244735	0.244735	0	pass
normal	`T-116945`	0.113802	0.113802	4.2e-17	pass
missing	`T-900001`	0.037929	0.037929	9.7e-17	pass
missing	`T-900002`	0.144791	0.144791	5.6e-17	pass
missing	`T-900003`	0.075978	0.075978	2.8e-17	pass
extreme	`T-900004`	1.000000	1.000000	0	pass
extreme	`T-900005`	0.039165	0.039165	1.4e-17	pass
extreme	`T-900006`	0.094671	0.094671	5.6e-17	pass
unknown	`T-900007`	0.177527	0.177527	0	pass
unknown	`T-900008`	0.111725	0.111725	5.6e-17	pass

The 14 parity cases served by the container, with one serving mistake switched on. Each dot is one case: its difference from the training score on a log scale. Dots right of the dashed line fail.

Text description of the explorer

The explorer starts with no mistake. 14 of 14 cases are within the tolerance of 1e-09. The largest difference is 1.1e-16 (T-116942). No flag changes.

Fill missing numbers with 0 (to_frame() fills a missing number with 0): 11 of 14 pass. The three missing-value cases fail. T-900001 differs by 5.4e-03 (0.032546 instead of 0.037929). T-900002 differs by 1.1e-01 (0.250070 instead of 0.144791). T-900003 differs by 4.5e-02 (0.121413 instead of 0.075978). T-900002, the guest, is now flagged. The normal, extreme and unknown cases still pass.

Round the score to 4 decimals (the response rounds the score): 1 of 14 pass. Only T-900004 passes, because its score is exactly 1.0. The other 13 differ by 1.7e-06 to 4.0e-05. No flag changes.

Replace a new category with “unknown” (a category value that the model never saw becomes the text unknown before scoring): 12 of 14 pass. The two unknown-category cases fail. T-900007 differs by 3.3e-02 (0.144458 instead of 0.177527). T-900008 differs by 2.1e-02 (0.090970 instead of 0.111725). No flag changes.

Each mistake fails a different group, and no single case catches all three. Look at what each one teaches:

Fill with 0 breaks only the missing-value cases. In training, the imputer filled a missing number with the median of the training data and set an indicator column, “this value was missing”. The model learned what a missing tenure means. A 0 tells it something else: a brand-new customer.
Round to 4 decimals looks tidy and changes no flag today. It still fails 13 cases, and that is right: the service no longer returns the model’s score. A score of 0.18996 would show as 0.19 next to a flag of false. If you want rounding, round in the client’s display, not in the API.
Replace with “unknown” looks safe, because “unknown” sounds neutral. But unknown is a real category in the training data: tickets with no recorded channel or region. The model learned its own weight for it. A value that the model never saw gets no weight at all. They are not the same, and only the two unknown cases show it.

One success is not readiness

Tomás’s check was one ticket, one /ready and a passing self-check. Every one of them passed with the fill-with-0 mistake. A model service fails quietly: it answers 200 with a number that looks right. So a check must look at the inputs where serving code makes choices: missing values, edges and unknown categories. The 14 cases are cheap: they run in under a second in the tests.

Parity has limits. It catches only skew that the cases cover. If a new kind of input appears, such as a new field, add a case in the next bundle version. Parity also says nothing about the quality of the model on today’s tickets. That is monitoring, in Monitor model health. Run it on every code change (pytest) and after every deployment (check_parity.py against the real service).

Guided practice: run parity twice

Write parity/check_parity.py and tests/test_parity.py as above.
Run python -m pytest. The last line says 42 passed.
Start the service with fastapi run.
In a second terminal, run python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0.
Check the last line: 14 of 14 cases match within 1e-09.
Run echo $? (PowerShell: $LASTEXITCODE). It gives 0.
Stop the service with Ctrl+C.

Show the expected output of step 4

Fourteen lines with a difference each, then 14 of 14 cases match within 1e-09. Your differences can be different from the worked example, but each one is 1e-15 or smaller. If a line ends with FAIL, read the next section.

Your turn: round the score

In escalation/routes.py, in the return ScoreOut(...) of score(), change score=result.score, to score=round(result.score, 4),. Predict first: how many parity tests fail, and which one passes? Then run python -m pytest tests/test_parity.py -q. Undo the change, and run the tests again.

Show the answer

FAILED tests/test_parity.py::test_served_score_matches_training[normal-T-116940]
...
FAILED tests/test_parity.py::test_served_score_matches_training[unknown-T-900008]
13 failed, 1 passed in 1.21s

Only extreme-T-900004 passes: its score is exactly 1.0, and rounding does not change it. Every other score changes in the fifth decimal or later (1.7e-06 to 4.0e-05), which is far above the tolerance. After you undo the change, 14 passed.

Common mistake

Three missing-value cases fail parity

Symptom: after a change to contract.py, the service starts and /ready says ready, but check_parity.py prints:

case     ticket      expected     served difference
...
missing  T-900001    0.037929   0.032546    5.4e-03  FAIL
missing  T-900002    0.144791   0.250070    1.1e-01  FAIL
missing  T-900003    0.075978   0.121413    4.5e-02  FAIL
...
11 of 14 cases match within 1e-09.

The exit code is 1. python -m pytest also fails the three tests missing-T-900001, missing-T-900002 and missing-T-900003, and the contract test test_the_frame_has_the_training_columns_in_order, which also checks that a missing number stays NaN.

Cause: to_frame() fills missing numbers before the pipeline sees them:

            frame[name] = frame[name].astype("float64")
            frame[name] = frame[name].astype("float64").fillna(0)

Training never saw 0 in place of a missing value. Its imputer filled each gap with the training median and set the indicator. The failing group names the cause: only the cases with a missing value fail, so look at the code that handles missing values.

Fix: remove .fillna(0). The service must pass missing numbers to the pipeline as NaN, so the pipeline’s own imputer does the same as in training. The rule in general: the service checks and shapes the input, but every transformation that training learned stays inside the pipeline.

For a real AI service: this is the most common kind of skew: a “small cleanup” in serving code that training never did. It gives no error and changes no healthy-looking metric. Here, the guest ticket T-900002 moved from 0.145 to 0.250, and the service flagged it. Grace’s team would review guests that the model never meant to flag. Parity cases with missing values catch it before release.

Check your understanding

Why does parity use a tolerance of 1e-9, and not 0 or 0.01?

Show the answer

The served scores differ from the training scores by up to about 1e-16. The cause is rounding in the last digit: another order of additions, another processor, or reading a 17-digit number from CSV. A tolerance of 0 would fail on this noise. The decision compares a score with 0.19, so you must catch a real fault long before it reaches 0.01. A tolerance of 0.01 would pass a fault that moves every score by 0.009. 1e-9 is ten million times above the noise and ten million times below 0.01.

Check your understanding

The self-check in load_bundle() already compares the parity scores at start-up. Why did it not catch the fill-with-0 mistake?

Show the answer

The self-check scores the cases with the pipeline directly, from the CSV file. It proves that the bundle is the right one and works with the installed packages. It does not use the service’s own code: the contract, to_frame() and the route. The mistake was in to_frame(), so only a check that sends tickets through the service, check_parity.py or test_parity.py, can see it.

Module practice

The module practice is to wrap the classifier in the API. You did the guided part in this module: the lifespan, the routes and the parity checks. Now prove it on your own, from a clean copy.

Task: copy the snapshot end-of-m02 to a new folder, ~/projects/escalation-check. Without copying from end-of-m03, add the files of this module: escalation/main.py, escalation/errors.py, escalation/security.py, escalation/routes.py, pyproject.toml, parity/check_parity.py, tests/test_api.py and tests/test_parity.py. Use the lessons where you need them. Then collect four pieces of evidence that the service is ready: the tests, the start-up log, the model information and the parity check.

Show a model answer

In the new folder, make a .venv and install requirements.txt, as in Module 1. Then:

python -m pytest
export MODEL_SHA256=$(shasum -a 256 bundles/escalation-1.0.0/SHA256SUMS | cut -c1-64)
fastapi run

curl http://127.0.0.1:8000/v1/model
python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0

The four pieces of evidence:

42 passed.
The log line model 1.0.0 loaded (bundle …) in … ms, with the first 12 characters of your digest.
/v1/model gives "model_version":"1.0.0", a bundle_sha256 that starts with the same 12 characters, and "threshold":0.19.
14 of 14 cases match within 1e-09. and exit code 0.

If a parity case fails, its group points to the code to check. Missing values: to_frame(). Categories: the contract or the scoring. Every case: the route’s response.

Module check

Check your understanding

Retrieval: which work does escalation-service do once per worker at start-up, and which work does it do in every request?

Show the answer

Once per worker, in the lifespan: import the packages, read the settings, and run load_bundle() with its checks and the self-check, which is also the warm-up. Then the port opens. In every request: check the API key and the contract, then frame, prepare and model. Then answer with the version and threshold, and write one log line.

Check your understanding

Predict: a developer changes the service so that a category value that the model never saw becomes the text unknown before scoring. Which parity cases fail, and why does this change the score?

Show the answer

Only the two unknown-category cases fail: T-900007 (social) and T-900008 (overseas, installation), by 3.3e-02 and 2.1e-02 in the measured run. unknown is a real category in the training data for channel and region, with its own learned weight. A value that the model never saw gets no column from the one-hot encoder, so no weight. Changing it to unknown gives it the weight of tickets with an unrecorded channel or region.

Check your understanding

Completion check: show that your service passes parity for normal, missing and extreme inputs. What do you run, and what output proves it?

Show the answer

Start the service, then run python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0. The proof has three parts. No line ends with FAIL, including the six normal, three missing and three extreme cases. The last line is 14 of 14 cases match within 1e-09. The exit code is 0. python -m pytest tests/test_parity.py gives 14 passed for the same cases through the app in the tests.

Recap

In this module, you built the prediction service:

main.py does the slow work once per worker, in the lifespan: the bundle is checked, loaded and warmed up before the port opens. A wrong bundle stops the service with exit code 3.
routes.py checks each ticket with the contract and answers with the score, the flag, the threshold, the model version and warnings. Every error has one shape, and every score has one log line.
Training-serving skew is a difference between how training and the service prepare the same input. Parity catches it: the 14 fixed cases, served and compared with the training scores within a tolerance of 1e-9. The real differences are about 1e-16.
One good prediction proves little. Each serving mistake broke a different group of cases.
The tests: 42 passed.

Reset. To start Module 4 from a clean state:

Stop every fastapi run with Ctrl+C.
Delete ../edited-bundle if it is still there, and ~/projects/escalation-check if you do not need it.
If your project is broken, copy the snapshot end-of-m03 and run python -m pytest: 42 passed.

You made no Docker containers or images in this module, so there is nothing to remove in Docker.

Next, you choose where the service runs: CPU or GPU, a container platform or a managed endpoint.

Next: CPU, GPU and managed endpoints