Release a new house-price model

What you will be able to do

Release a new version of a model without surprising the people who use it. You take the house-price API from Build and Deploy Your First API and make each model a versioned bundle. Then you run a retrained model in the shadow, on a replay of real requests. You compare the two models with checks that suit a price model, promote the new one, and practise the way back.

Case studyPortfolio piece

Dataset

House sales in King County, USA, 2014–2015 (OpenML 42092)

Licence

CC0 1.0 (data); MIT (code)

Time

About 3 hours

Runs in

VS Code, Python 3.12+

Skills used

• Versioned bundles with a pinned digest
• Shadow deployment on a replay
• Release checks for a regression model
• Promotion and a rollback drill
• Parity and contract checks after a release
• One image with two models

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is the finished project of the API course’s case study, with a script that makes a replay and one that sends it. You add the bundles, the shadow, the container, the comparison and the release records.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

The lessons that this case study uses most:

• The bundle and its digest, from Bundle model dependencies and Artifact trust and integrity.
• Parity cases and their tolerance, from Check parity.
• The container with limits, from Container resource settings.
• Shadow scoring, release records and rollback, from Version and rollout and Rollback and retirement.
• The API that you start from: Serve a house-price model, the case study of the API course. You do not need to have done it: the starter is its finished project, and this page explains what you need from it.

What you need on your computer

• Python 3.12 or newer, VS Code with the Python extension, and curl, as in Environment and first endpoint.
• Docker Desktop, as in Container resource settings.
• About 1.2 GB of free disk space (the virtual environment is about 360 MB, the image about 690 MB) and an internet connection.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The packages: FastAPI 0.142.2, scikit-learn 1.9.1, NumPy 2.5.3, SciPy 1.18.1, joblib 1.6.0, pandas 3.0.6 and pytest 9.1.1. The Windows and Linux steps are not tested.

The situation

The house-price API from the API course runs. Hannah Osei’s valuation team at the lender uses it every day, through the intake tool that Elif Demir maintains. Its model, kc-hgb-2015-02, learned from sales up to February 2015.

The team’s analyst now retrains the model every month. Hannah wants the newer prices; Elif wants her tool to keep working with no change. Daniel Ruiz, from compliance, asks: “How do we know that the new model is not worse? And if it is, how fast can we go back?” Kenji Mori, the developer, owns the release. You are Kenji again.

What you will build

Part	What you add	The question it answers
1	make_bundle.py, bundles/<version>/, BUNDLES in config.py	Which model files may the API load at all?
2	A shadow model and a prediction log	How does a new model answer real requests without anybody seeing its answers?
3	Dockerfile, compose.yaml, scripts/check_release.py	Does the service run in a container, and does it serve the version that we released?
4	The first candidate, in the shadow, and scripts/compare.py	Is this candidate safe to release? (It is not.)
5	The second candidate, the decision, the promotion, RELEASES.md	Is this one safe, and who decides?
6	A rollback drill	Can we go back, and how long does it take?

The idea: a new model is a release

A restaurant that changes a recipe cooks the new dish next to the old one for a week, and the team tastes both. The guests still get the old dish. The menu changes only when the team agrees, and the old recipe stays in the drawer.

The problem: one house, or one test, cannot decide

The retrained model learned from one more month, March 2015. The API course’s typical house sold on 26 April 2015 for 425,000 dollars:

Model	Estimate
Version 1, kc-hgb-2015-02 (live)	422,000
Version 2, kc-hgb-2015-03 (new)	379,000

The new model moved it by −10%, away from the sale price. Is it worse? Three more houses from the same weeks:

House	Sale price	Version 1	Version 2	Error of version 1	Error of version 2	Move
A	235,000	235,000	239,000	0	4,000	+1.7%
B	945,000	1,086,000	1,115,000	141,000	170,000	+2.7%
C	1,400,000	1,880,000	1,428,000	480,000	28,000	−24.0%
D (the typical house)	425,000	422,000	379,000	3,000	46,000	−10.2%

The error is the distance from the sale price, with no sign. Its average over the houses is the mean absolute error (MAE): version 1, (0 + 141,000 + 480,000 + 3,000) ÷ 4 = 156,000 dollars; version 2, (4,000 + 170,000 + 28,000 + 46,000) ÷ 4 = 62,000 dollars. The move is how much the estimate changes: 379,000 ÷ 422,000 − 1 = −10.2% for house D.

On these four houses, version 2 looks 2.5 times better, only because of house C, and two estimates moved by more than 10%. On 2,817 houses, you will see that the models are almost equal. A large offline test does not decide alone either: in Part 4, a broken candidate gets the same test error as version 1, and passes every unit test.

The idea: shadow, compare, decide, promote, keep the way back

The week of tasting is a shadow deployment: the candidate estimates every request, the client gets only the live answer, and the log keeps both. You do not need to wait for new traffic: you replay the intake tool’s recorded requests, so every candidate sees the same houses.

A price model has no threshold and no flag, so the checks of module 5 (flag rate against capacity) do not apply. The checks ask two questions instead:

1. Is the new error lower? The MAE on the newer sales.
2. How much do the answers change? The share of estimates that move by more than 10%, and the largest median move of one area. A loan officer who saw 422,000 last month and sees 379,000 today needs a reason.

Hannah and Kenji agree on the limits before any shadow runs. A limit chosen after the results tends to be the one that the candidate passes.

Check	Limit
Error	The candidate’s MAE is at most 1% above the live MAE.
Moves	At most 5% of estimates move by more than 10%.
Areas	No zip code (with 10 sales or more) has a median move of more than 5%.
Contract and parity	The tests pass, and the running service passes check_release.py after the release.
A person	Kenji reads the largest moves and explains them to Hannah, who decides.

Why 10%? It is inside the model’s own range (in the API course, 80% of houses sold between 0.884 and 1.279 times the estimate), but it is the change that a loan officer notices. Hannah’s team can explain a few such moves, not hundreds.

The release path of this case study. The environment chooses which reviewed bundle is live or in the shadow; only a reviewed change in the code can add a bundle.

Text description of the diagram

A candidate goes through six steps. First, the analyst trains it with train.py. Second, make_bundle.py makes a bundle and prints its digest. Third, the digest goes into BUNDLES in config.py, in a reviewed change. Fourth, the candidate runs in the shadow: both models estimate each request, and only the live one answers. A replay of 2,819 requests feeds the comparison, the fifth step, which checks the error, the moves and the areas. If a check fails, the candidate is rejected and retired. If all checks pass and a person has read the largest moves, the sixth step makes it live, and check_release.py confirms the release. The previous version stays in the image and in BUNDLES, so a rollback is one setting.

The data and the time split

The data is the API course’s: 21,613 real house sales in King County, Washington, from 2 May 2014 to 27 May 2015. The source is House Sales in King County, USA (harlfoxem, Kaggle 2016; OpenML 42092), under CC0 1.0. Hannah, Elif, Kenji, Daniel and the lender are invented. The sales are real.

A release story needs a “today”. The data stops on 27 May 2015, so the case study pretends that today is the end of May. Each house goes where its first sale goes, as in the API course:

Part	Houses first sold	Sales	Use
Version 1 learned from	before 1 March 2015	16,959	The API course’s model, unchanged
Version 2 learns from	before 1 April 2015	18,794	one more month: March
The replay	1 April to 27 May 2015	2,819	the intake tool’s requests, and later their sale prices

Two months, because May alone has only 646 sales. Neither model learned from these houses, but they are also version 2’s offline test set. The shadow still adds something: it sends each house through the real API, and Part 4 shows what that catches.

Set up the starter project

1. Download the starter project with the button in the box at the top of this page. Unzip it into your projects folder. You get a folder house-price-api. If you still have the project from the API course, keep it somewhere else: this is a new copy.

2. In VS Code, open the folder: File › Open Folder…, then choose house-price-api.

3. Open a terminal in VS Code: Terminal › New Terminal.

4. Make and activate a virtual environment:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1

   Linux

   python3 -m venv .venv
   source .venv/bin/activate

5. Install the exact versions of the packages. This takes about 20 seconds to a few minutes:

   python -m pip install -r requirements-lock.txt

6. Choose the environment in VS Code. Open the Command Palette (Ctrl+Shift+P, or Cmd+Shift+P on macOS). Choose Python: Select Interpreter, then the one in .venv.

7. Run the tests:

   python -m pytest

8. Make the replay. The script downloads the data (0.7 MB) and checks its SHA-256:

   python replay/make_replay.py

The output of the last two steps:

Output

...
tests/test_model_file.py ......                                          [100%]

============================== 49 passed in 1.16s ==============================

Downloading https://github.com/Nextia-Intelligence/nextia-learning-labs/raw/main/C05/case-studies/data/king_county_house_sales.csv.gz
wrote replay/requests-2015-04-05.jsonl and replay/sales-2015-04-05.csv: 2819 sales from 2015-04-01 to 2015-05-27

Open the two replay files. Each line of requests-2015-04-05.jsonl is one request as the intake tool sent it, with a sale ID such as 2800031-20150401 (house ID and sale date). sales-2015-04-05.csv has the prices. They are separate on purpose: the service never sees a price.

New in the starter	What it does
training/train.py	The analyst’s script, now with --until (the first day that the model must not see), --version, --data and --out. It writes a candidate to candidates/<version>/.
training/simulate_export.py	Makes a changed copy of the data. Part 4 explains it.
replay/make_replay.py	Writes the two replay files.
scripts/replay.py	Sends the requests to a running API, one at a time, and keeps each answer’s request ID.

Part 1: versioned bundles with a pinned digest

The problem: one model path, one hash

The API of the API course knows one file, model/house_price.joblib, and one hash. A shadow needs two models at once, and a rollback needs the previous model, unchanged.

The idea: a folder per version, one digest per folder

Use the bundle of module 1: one folder per version, never changed, with a digest that comes from outside the bundle.

One difference: the escalation service reads its digest from .env. This project keeps the API course’s rule instead: anyone who can change the server’s settings could set the digest of their own file. So the code holds the reviewed versions with their digests, and the environment only chooses among them.

Step 1: make the bundle

Make make_bundle.py in the project folder:

make_bundle.py

"""Make a versioned model bundle from a folder that training/train.py wrote.

    python make_bundle.py candidates/kc-hgb-2015-03

Reads house_price.joblib and model_card.json from the folder, and writes
bundles/<model_version>/ with four files:

    model.joblib        the model, copied byte for byte
    model_card.json     its card, copied byte for byte (GET /v1/model serves it)
    parity_cases.json   fixed houses and the estimates this model gives them
    SHA256SUMS          the SHA-256 of each file above

It prints the bundle digest: the SHA-256 of SHA256SUMS. Add it to BUNDLES in
house_price_api/config.py, in a reviewed change, before the service may load
the bundle. A version is never changed: if the folder exists, the script stops.

It loads the joblib file to compute the parity estimates, and loading can run
code: run it only on a model that you trained, or that comes from a source
that you trust.
"""

import argparse
import hashlib
import json
import shutil
import sys
from pathlib import Path

import joblib

from house_price_api.estimator import FEATURES
from house_price_api.models import TYPICAL_HOUSE, HouseIn

FILES = ["model.joblib", "model_card.json", "parity_cases.json"]
GOLDEN = Path("tests/golden_houses.json")

# Fixed houses that every copy of a model must estimate the same way: the
# three golden houses of C16, and houses at the edges of the contract.
EDGE_CASES = {
    "never_renovated_new": {"yr_built": 2014, "yr_renovated": None},
    "renovated_old": {"yr_built": 1905, "yr_renovated": 2010},
    "smallest": {
        "bedrooms": 0, "bathrooms": 0.0, "sqft_living": 250, "sqft_lot": 500,
        "floors": 1.0, "view": 0, "condition": 1, "grade": 1, "sqft_basement": 0,
        "yr_built": 1900, "sqft_living15": 400, "sqft_lot15": 600,
    },
    "largest": {
        "bedrooms": 11, "bathrooms": 8.0, "sqft_living": 14000, "sqft_lot": 1100000,
        "floors": 3.5, "waterfront": True, "view": 4, "condition": 5, "grade": 13,
        "sqft_basement": 5000, "yr_built": 2015, "yr_renovated": 2015,
        "sqft_living15": 6500, "sqft_lot15": 900000,
    },
    "south_west_corner": {"lat": 47.15, "long": -122.52},
    "north_east_corner": {"lat": 47.78, "long": -121.31},
}


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def parity_cases(model) -> list[dict]:
    golden = json.loads(GOLDEN.read_text("utf-8"))
    houses = {g["name"]: g["house"] for g in golden}
    houses |= {name: {**TYPICAL_HOUSE, **change} for name, change in EDGE_CASES.items()}
    cases = []
    for name, house in houses.items():
        inputs = HouseIn(**house).model_inputs()  # every case must pass the contract
        [estimate] = model.predict([[inputs[f] for f in FEATURES]]).tolist()
        cases.append({"name": name, "house": house, "expected_estimate": estimate})
    return cases


def main() -> None:
    parser = argparse.ArgumentParser(description="Make a model bundle.")
    parser.add_argument("source", type=Path, help="a folder with house_price.joblib and model_card.json")
    args = parser.parse_args()

    card = json.loads((args.source / "model_card.json").read_text("utf-8"))
    model_file = args.source / "house_price.joblib"
    if sha256(model_file) != card["sha256"]:
        sys.exit("Stopped: model_card.json describes another model file.")
    if card["features"] != FEATURES:
        sys.exit("Stopped: the model was trained on other features than the API sends.")

    out = Path("bundles") / card["model_version"]
    if out.exists():
        sys.exit(f"Stopped: {out} exists. A version is never changed: choose a new version.")
    out.mkdir(parents=True)
    shutil.copyfile(model_file, out / "model.joblib")
    shutil.copyfile(args.source / "model_card.json", out / "model_card.json")

    cases = parity_cases(joblib.load(model_file))
    (out / "parity_cases.json").write_text(json.dumps(cases, indent=2) + "\n", "utf-8")

    lines = [f"{sha256(out / name)}  {name}" for name in FILES]
    (out / "SHA256SUMS").write_text("\n".join(lines) + "\n", "utf-8")

    print(f"Made {out}: {len(cases)} parity cases.")
    print(f"Bundle digest: {sha256(out / 'SHA256SUMS')}")


if __name__ == "__main__":
    main()

Part	Why
The parity cases are houses in the API’s terms	yr_renovated is null, waterfront is true or false, and sqft_above is missing, as in a request. So the check goes through HouseIn.model_inputs(), the API’s own translation. A mistake in that translation is training-serving skew, from Check parity.
The edge cases	A new house never renovated, an old house renovated, the smallest and the largest house that the contract allows, and two corners of the model’s area.
if out.exists()	A version is never changed. A new model needs a new version.

Make the bundle of the live model from the API course’s model/ folder:

python make_bundle.py model

Output

Made bundles/kc-hgb-2015-02: 9 parity cases.
Bundle digest: 29752760649bf79193867d4a5887ee47c361aa913315d831b0e9fafe6f50d1cf

Open bundles/kc-hgb-2015-02/SHA256SUMS:

bundles/kc-hgb-2015-02/SHA256SUMS

8d0483d05cfca4752118c5c6ef4c47a9531c7230981cc880c27d85def9dbfdec  model.joblib
3389bc75812971ba87b3908026409d0eb78eca0eea40affdc3771fec31d35e1f  model_card.json
96bb75911713fdfe7aa1d2c12524216e292cfc6a16e93c2a06de2960cd1787d4  parity_cases.json

The model file is the API course’s reviewed file (8d0483d05cfc…). Unlike the escalation service, your digest is the same as ours: training is reproducible byte for byte, and the bundle holds no date. Run the command again: it stops with A version is never changed: choose a new version.

The bundle replaces model/. Delete the folder model/. Also delete scripts/check_deployment.py and tests/test_model_file.py: both check the single model file, and Part 3 and Step 4 replace them.

Step 2: the reviewed versions

Replace house_price_api/config.py:

house_price_api/config.py

import os
from dataclasses import dataclass, field
from pathlib import Path

PROJECT = Path(__file__).resolve().parent.parent

# The reviewed bundles: model version -> bundle digest (the SHA-256 of its
# SHA256SUMS file). The service loads only these. A new bundle needs a new
# line here, in a reviewed change; the environment only chooses among them.
BUNDLES = {
    "kc-hgb-2015-02": "29752760649bf79193867d4a5887ee47c361aa913315d831b0e9fafe6f50d1cf",
}
DEFAULT_VERSION = "kc-hgb-2015-02"


@dataclass(frozen=True)
class Settings:
    api_key: str | None = field(repr=False)
    allowed_origins: list[str]
    bundles_dir: Path
    model_version: str
    shadow_version: str | None
    reviewed: dict[str, str]
    estimate_timeout: float
    log_level: str
    show_docs: bool


def load_settings() -> Settings:
    """Read the settings from environment variables, with safe defaults."""
    origins = os.environ.get("ALLOWED_ORIGINS", "")
    return Settings(
        api_key=os.environ.get("API_KEY") or None,
        allowed_origins=[o.strip() for o in origins.split(",") if o.strip()],
        bundles_dir=Path(os.environ.get("BUNDLES_DIR", PROJECT / "bundles")),
        model_version=os.environ.get("MODEL_VERSION", DEFAULT_VERSION),
        shadow_version=os.environ.get("SHADOW_VERSION") or None,
        reviewed=BUNDLES,
        estimate_timeout=float(os.environ.get("ESTIMATE_TIMEOUT", "2.0")),
        log_level=os.environ.get("LOG_LEVEL", "INFO").upper(),
        show_docs=os.environ.get("SHOW_DOCS", "true").lower() == "true",
    )

MODEL_VERSION chooses the live model, and SHADOW_VERSION the shadow (Part 2). Both must be keys of BUNDLES. reviewed is in the settings only so that the tests can give another list; on a server it is always BUNDLES.

Step 3: load a bundle, gate by gate

In house_price_api/estimator.py, the bundle loader replaces the API course’s load_estimator. Make three changes:

• Add from importlib.metadata import version to the imports.
• Add the two constants below, above class ModelUnavailable.
• Give HouseEstimator a third argument, digest. Its __init__ starts like this; the rest stays as it is:

house_price_api/estimator.py (changes)

# The packages that must have the versions that saved the model.
PACKAGES = ["scikit-learn", "numpy", "joblib"]
# The self-check allows this relative difference: 1e-9 of 400,000 USD is 0.0004 USD.
TOLERANCE = 1e-9


class HouseEstimator:
    """The loaded model, its card, and the digest of its bundle."""

    def __init__(self, model, card: dict, digest: str = ""):
        self.model = model
        self.card = card
        self.digest = digest
        # ... the other lines of __init__ stay as they are

Then replace the whole function load_estimator with these four functions:

house_price_api/estimator.py (replaces load_estimator)

def check_files(path: Path, expected_digest: str) -> None:
    """Gates 1 and 2: this is the reviewed bundle, and no file in it changed."""
    sums = path / "SHA256SUMS"
    if not sums.is_file():
        raise ModelUnavailable(f"no bundle at {path}")
    digest = file_sha256(sums)
    if digest != expected_digest:
        raise ModelUnavailable(
            f"{path.name}: the bundle digest is {digest[:12]}..., "
            f"but config.BUNDLES expects {expected_digest[:12]}..."
        )
    for line in sums.read_text("utf-8").splitlines():
        expected, name = line.split()
        if file_sha256(path / name) != expected:
            raise ModelUnavailable(f"{path.name}/{name} changed after the bundle was made")


def check_card(path: Path, card: dict) -> None:
    """Gate 3: the card names this model file, this version and these packages."""
    if card["model_version"] != path.name:
        raise ModelUnavailable(f"{path.name}: the card says version {card['model_version']}")
    if card["sha256"] != file_sha256(path / "model.joblib"):
        raise ModelUnavailable(f"{path.name}: the card describes another model file")
    for package in PACKAGES:
        tested, installed = card["trained_with"][package], version(package)
        if installed != tested:
            raise ModelUnavailable(
                f"{path.name}: {package} {installed} is installed, "
                f"but the model was saved with {tested}"
            )
    if card["features"] != FEATURES:
        raise ModelUnavailable(f"{path.name}: the model expects other features")


def self_check(estimator: HouseEstimator, path: Path) -> None:
    """Gate 5: the parity cases get the estimates that they got at bundling."""
    # Imported here: models.py does not need the estimator, and the self-check
    # must turn each house into model inputs exactly as a request does.
    from house_price_api.models import HouseIn

    cases = json.loads((path / "parity_cases.json").read_text("utf-8"))
    rows = [HouseIn(**case["house"]).model_inputs() for case in cases]
    for case, result in zip(cases, estimator.estimate(rows)):
        expected = case["expected_estimate"]
        if abs(result.value - expected) > TOLERANCE * expected:
            raise ModelUnavailable(
                f"{path.name}: self-check failed for {case['name']}: "
                f"{result.value:.2f} instead of {expected:.2f}"
            )


def load_bundle(path: Path, expected_digest: str) -> HouseEstimator:
    """Check a bundle, then load it. Raise ModelUnavailable if anything is wrong."""
    check_files(path, expected_digest)
    try:
        card = json.loads((path / "model_card.json").read_text("utf-8"))
    except (OSError, ValueError) as error:
        raise ModelUnavailable(f"cannot read model_card.json: {error}") from None
    check_card(path, card)
    # Every byte is the reviewed one. Only now is it safe to load the model:
    # joblib uses pickle, which can run code from the file (gate 4).
    with warnings.catch_warnings():
        warnings.simplefilter("error", InconsistentVersionWarning)
        try:
            model = joblib.load(path / "model.joblib")
        except InconsistentVersionWarning as warning:
            raise ModelUnavailable(
                f"the model was saved with scikit-learn "
                f"{warning.original_sklearn_version}, this server has "
                f"{warning.current_sklearn_version}"
            ) from None
    estimator = HouseEstimator(model, card, file_sha256(path / "SHA256SUMS"))
    self_check(estimator, path)
    return estimator

The gates run in the order of module 1: digest, files, card and packages, only then joblib.load, then the self-check. Any failure raises ModelUnavailable, so the API keeps the API course’s behaviour: /health stays 200, and /ready and the estimates give 503 model_unavailable.

Step 4: choose a version at start-up

In house_price_api/main.py, the import of the estimator changes, a small function chooses the bundle, and the start-up block uses it:

house_price_api/main.py (changes)

from house_price_api.estimator import (
    READY_CHECK_HOUSE,
    HouseEstimator,
    ModelUnavailable,
    load_bundle,
)


def load_version(settings: Settings, version: str) -> HouseEstimator:
    """Load the bundle of a reviewed version: only versions in config.BUNDLES."""
    if version not in settings.reviewed:
        raise ModelUnavailable(f"{version} is not a reviewed bundle (config.BUNDLES)")
    return load_bundle(settings.bundles_dir / version, settings.reviewed[version])


    # in create_app: the start-up block becomes
    started = time.perf_counter()
    try:
        app.state.estimator = load_version(settings, settings.model_version)
        app.state.model_problem = None
        logger.info(
            "model %s loaded (bundle %s...) in %.0f ms, memory %s MB",
            app.state.estimator.version,
            app.state.estimator.digest[:12],
            (time.perf_counter() - started) * 1000,
            memory_mb(),
        )
    except ModelUnavailable as problem:
        app.state.estimator = None
        app.state.model_problem = str(problem)
        logger.error("model not loaded: %s", problem)

Put load_version at the top level of the file, before def create_app. The log line now names the live digest.

Step 5: test the gates

In tests/conftest.py, change the import to from house_price_api.config import BUNDLES, PROJECT, Settings, and replace the two model lines of TEST_SETTINGS:

tests/conftest.py (change)

    bundles_dir=PROJECT / "bundles",
    model_version="kc-hgb-2015-02",  # the golden answers in golden_houses.json are this version's
    shadow_version=None,
    reviewed=BUNDLES,

In tests/test_estimates.py, change the first line of test_no_model_gives_503_everywhere_but_health to client = make_client(bundles_dir=tmp_path): an empty folder has no bundle.

Then make tests/test_bundle.py. Each test breaks one gate. The fixture no_unpickling fails the test if joblib.load runs before the checks:

tests/test_bundle.py

import json
import logging
import shutil
from pathlib import Path

import joblib
import pytest

from house_price_api.config import BUNDLES, PROJECT
from house_price_api.estimator import file_sha256, load_bundle

CONTRACT = PROJECT / "contract" / "openapi.json"
LIVE = "kc-hgb-2015-02"


def assert_not_ready(client):
    """Liveness stays 200; readiness gives 503, without internals."""
    assert client.get("/health").status_code == 200
    response = client.get("/ready")
    assert response.status_code == 503
    assert response.json()["error"]["code"] == "model_unavailable"


def changed_copy(tmp_path: Path, name: str, change) -> str:
    """Copy the live bundle, change one file, write new sums. Return the new digest."""
    folder = tmp_path / LIVE
    shutil.copytree(PROJECT / "bundles" / LIVE, folder)
    path = folder / name
    data = json.loads(path.read_text("utf-8"))
    change(data)
    path.write_text(json.dumps(data, indent=2) + "\n", "utf-8")
    sums = [f"{file_sha256(folder / n)}  {n}" for n in ("model.joblib", "model_card.json", "parity_cases.json")]
    (folder / "SHA256SUMS").write_text("\n".join(sums) + "\n", "utf-8")
    return file_sha256(folder / "SHA256SUMS")


@pytest.fixture
def no_unpickling(monkeypatch):
    """Fail the test if anything calls joblib.load."""

    def refuse(*args, **kwargs):
        raise AssertionError("joblib.load ran before the checks passed")

    monkeypatch.setattr(joblib, "load", refuse)


@pytest.mark.parametrize("version", BUNDLES)
def test_every_reviewed_bundle_loads_and_passes_its_self_check(version):
    path = PROJECT / "bundles" / version
    assert file_sha256(path / "SHA256SUMS") == BUNDLES[version]
    assert load_bundle(path, BUNDLES[version]).version == version


@pytest.mark.parametrize("version", BUNDLES)
def test_a_new_model_is_not_a_new_contract(make_client, version):
    current = json.dumps(make_client(model_version=version).get("/openapi.json").json(), indent=2)
    assert current + "\n" == CONTRACT.read_text("utf-8")


def test_wrong_digest_stops_before_loading(make_client, no_unpickling):
    assert_not_ready(make_client(reviewed={LIVE: "0" * 64}))


def test_unreviewed_version_is_not_loaded(make_client, no_unpickling):
    assert_not_ready(make_client(model_version="kc-hgb-2015-04"))


def test_changed_file_is_not_loaded(make_client, tmp_path, no_unpickling):
    shutil.copytree(PROJECT / "bundles" / LIVE, tmp_path / LIVE)
    with (tmp_path / LIVE / "parity_cases.json").open("a") as file:
        file.write(" ")
    assert_not_ready(make_client(bundles_dir=tmp_path))


def test_other_package_version_is_not_loaded(make_client, tmp_path, no_unpickling):
    def older(card):
        card["trained_with"]["scikit-learn"] = "1.0.0"

    digest = changed_copy(tmp_path, "model_card.json", older)
    assert_not_ready(make_client(bundles_dir=tmp_path, reviewed={LIVE: digest}))


def test_self_check_catches_another_estimate(make_client, tmp_path):
    def one_dollar_more(cases):
        cases[0]["expected_estimate"] += 1

    digest = changed_copy(tmp_path, "parity_cases.json", one_dollar_more)
    assert_not_ready(make_client(bundles_dir=tmp_path, reviewed={LIVE: digest}))

changed_copy plays a colleague who changes a file and writes new sums. The test gives the new digest as reviewed, so the next gate must catch the change. The parametrized tests run once for each version in BUNDLES, so a new bundle needs no new test. Run the tests:

python -m pytest

Output (last lines)

tests/test_bundle.py .......                                             [ 14%]
tests/test_contract.py ..                                                [ 18%]
tests/test_estimates.py .......................................          [ 96%]
tests/test_health.py ..                                                  [100%]

============================== 50 passed in 1.11s ==============================

Seven bundle tests replace the six tests of the old model file.

Step 6: see it on the server

Start the API with fastapi run. In a second terminal, run curl -s http://127.0.0.1:8000/ready: the answer is {"status":"ready"}. Stop the API, start it again with a version that is not reviewed, and call /ready again:

macOS

MODEL_VERSION=kc-hgb-2015-03 fastapi run

Windows

$env:MODEL_VERSION = "kc-hgb-2015-03"
fastapi run

Linux

MODEL_VERSION=kc-hgb-2015-03 fastapi run

The log of the first start, then of the second (shortened)

INFO house_price_api: model kc-hgb-2015-02 loaded (bundle 29752760649b...) in 296 ms, memory 234 MB
...
ERROR house_price_api: model not loaded: kc-hgb-2015-03 is not a reviewed bundle (config.BUNDLES)
WARNING house_price_api: model unavailable: kc-hgb-2015-03 is not a reviewed bundle (config.BUNDLES) id=e234cb73ae13
INFO house_price_api: GET /ready 503 5ms id=e234cb73ae13

A setting can choose only what the code trusts. Stop the API. On Windows, remove the setting with Remove-Item Env:MODEL_VERSION.

Part 2: a shadow that never answers

The problem: two answers, one client

The service must estimate each house twice, but Elif’s tool must get exactly one answer, the live one, in the same shape as before. And a broken candidate must never break a live answer.

Step 1: load the shadow

In main.py, add this block in create_app, after the start-up block of Part 1:

house_price_api/main.py (add)

    # The shadow: a candidate that estimates every house too, for the log
    # only. If it cannot load, the live model still serves.
    app.state.shadow = None
    if settings.shadow_version:
        try:
            app.state.shadow = load_version(settings, settings.shadow_version)
            logger.info(
                "shadow %s loaded (bundle %s...), memory %s MB",
                app.state.shadow.version,
                app.state.shadow.digest[:12],
                memory_mb(),
            )
        except ModelUnavailable as problem:
            logger.error("shadow not loaded: %s", problem)

If the shadow fails a gate, the log says so and the live model serves as before.

Step 2: estimate twice, answer once, log both

In routes.py, add import json and import logging at the top, two loggers below router = ..., and a function above estimate_houses:

house_price_api/routes.py (add)

logger = logging.getLogger("house_price_api")
predictions = logging.getLogger("house_price_api.predictions")


def shadow_values(shadow: HouseEstimator, rows: list[dict]) -> list[float] | None:
    """The candidate's estimates. They go to the log, never to the client."""
    try:
        return [estimate.value for estimate in shadow.estimate(rows)]
    except Exception:  # a broken candidate must never break a live answer
        logger.exception("shadow %s failed", shadow.version)
        return None

Then, in estimate_houses, add these lines after the except TimeoutError block and before return [:

house_price_api/routes.py (add to estimate_houses)

    shadow = request.app.state.shadow
    shadowed = await run_in_threadpool(shadow_values, shadow, rows) if shadow else None
    for item, estimate in enumerate(estimates):
        record = {
            "request_id": request.state.request_id,
            "item": item,
            "model_version": estimator.version,
            "estimate": round(estimate.value, 2),
        }
        if shadowed is not None:
            record |= {
                "shadow_version": shadow.version,
                "shadow_estimate": round(shadowed[item], 2),
            }
        predictions.info(json.dumps(record))

Part	Why
After the live estimate, outside its time limit	The live answer is ready first. A slow shadow cannot turn a live answer into a 503.
One log line per house, with the request ID	item is the house’s place in a batch. compare.py joins the line with the replay’s answers through the request ID, which the client also gets in X-Request-ID.
No house description in the line	The API course promised that the log identifies no house or person. The line has an ID, versions and two numbers.

The return value, and so the contract, is unchanged. Add two tests at the end of tests/test_bundle.py. Any reviewed version can be the shadow, even the live one, so they work before there is a second model:

tests/test_bundle.py (add)

def test_shadow_is_logged_but_never_answers(make_client, house, caplog):
    caplog.set_level(logging.INFO, logger="house_price_api.predictions")
    # Any reviewed version can be the shadow, even the live one.
    client = make_client(shadow_version=LIVE)
    response = client.post("/v1/estimates", json=house)
    assert response.json()["model_version"] == LIVE
    [line] = [r.getMessage() for r in caplog.records if r.name == "house_price_api.predictions"]
    record = json.loads(line)
    assert record["request_id"] == response.headers["X-Request-ID"]
    assert record["model_version"] == LIVE
    assert record["shadow_version"] == LIVE
    assert record["shadow_estimate"] == record["estimate"]


def test_a_broken_shadow_does_not_stop_the_live_model(make_client, house):
    client = make_client(shadow_version="kc-hgb-2015-04")
    assert client.get("/ready").status_code == 200
    assert client.post("/v1/estimates", json=house).json()["model_version"] == LIVE

Run python -m pytest: 52 passed. Then see one prediction line. Start the API with the live model as its own shadow, and send the typical house from a second terminal:

macOS

SHADOW_VERSION=kc-hgb-2015-02 fastapi run

Windows

$env:SHADOW_VERSION = "kc-hgb-2015-02"
fastapi run

Linux

SHADOW_VERSION=kc-hgb-2015-02 fastapi run

Second terminal (on Windows: curl.exe, with the @requests/... part in quotes)

curl -s -X POST http://127.0.0.1:8000/v1/estimates -H "Content-Type: application/json" -d @requests/typical-house.json

Output, then the API's log

{"estimate_usd":422000,"range_usd":{"low":373000,"high":540000},"model_version":"kc-hgb-2015-02","prices_as_of":"2015-02-28","warnings":[]}

INFO house_price_api: model kc-hgb-2015-02 loaded (bundle 29752760649b...) in 260 ms, memory 240 MB
INFO house_price_api: shadow kc-hgb-2015-02 loaded (bundle 29752760649b...), memory 240 MB
INFO house_price_api.predictions: {"request_id": "e59dbefcf396", "item": 0, "model_version": "kc-hgb-2015-02", "estimate": 421936.2, "shadow_version": "kc-hgb-2015-02", "shadow_estimate": 421936.2}

The client gets the API course’s answer, field for field; the log has both estimates, unrounded, with the request ID. Stop the API (on Windows, also Remove-Item Env:SHADOW_VERSION).

The shadow is not free. We measured one estimate 2,000 times inside the container of Part 3 (1 CPU). It took 1.0 ms with the live model only, and 2.1 ms with the shadow too. For a valuation team, 1 ms is nothing. For a service near its limit, shadow only for a few days, or only a sample of requests.

Part 3: one image, two models

Now the service runs in a container with the limits of Container resource settings.

Step 1: the files

First, the pinned packages for the image, made inside the base image, as in module 4. The API does not need pandas.

docker run --rm python:3.14.8-slim-trixie sh -c 'pip install -q --root-user-action=ignore --disable-pip-version-check "fastapi[standard-no-fastapi-cloud-cli]==0.142.2" scikit-learn==1.9.1 numpy==2.5.3 scipy==1.18.1 joblib==1.6.0 && pip freeze' > requirements-run.txt

On Windows PowerShell, run the same command, then check that requirements-run.txt has 57 lines and no strange characters at the start. The command took 33 seconds for us. Then make four files:

Dockerfile

FROM python:3.14.8-slim-trixie

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_ROOT_USER_ACTION=ignore

RUN useradd --create-home --uid 10001 app

WORKDIR /app

COPY requirements-run.txt .
RUN pip install -r requirements-run.txt

# Every reviewed bundle goes into the image. MODEL_VERSION chooses the live
# one when the container starts, so a rollback needs no new image.
COPY bundles ./bundles
COPY pyproject.toml .
COPY house_price_api ./house_price_api

USER app

EXPOSE 8000
CMD ["fastapi", "run", "--host", "0.0.0.0", "--port", "8000"]

.dockerignore

# Send only what the image needs. Data, candidates, tests, .env and .venv stay out.
*
!requirements-run.txt
!pyproject.toml
!house_price_api/
!bundles/
**/__pycache__/

compose.yaml

# The house-price API on this computer, with the limits of a small cloud
# container. The settings come from .env (copy .env.example).
services:
  api:
    image: house-price-api:${IMAGE_TAG:?Set IMAGE_TAG in .env}
    build: .
    ports:
      - "127.0.0.1:8000:8000"
    environment:
      MODEL_VERSION: ${MODEL_VERSION:?Set MODEL_VERSION in .env}
      SHADOW_VERSION: ${SHADOW_VERSION:-}
      API_KEY: ${API_KEY:-}
      SHOW_DOCS: "true"
    cpus: 1.0
    mem_limit: 512m
    memswap_limit: 512m  # the same value: no swap, as in most cloud containers
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/ready')"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 10s

.env.example

# Copy this file to .env. Compose reads it; Git and the image do not.
# IMAGE_TAG names the image: the code and the bundles in it.
# MODEL_VERSION is the live model; SHADOW_VERSION the candidate (empty: none).
# Both must be in the image and in BUNDLES in house_price_api/config.py.
IMAGE_TAG=1.1.0
MODEL_VERSION=kc-hgb-2015-02
SHADOW_VERSION=
API_KEY=local-dev-key

The image holds every reviewed bundle: the design decision of this case study. The shadow and the rollback both need a second model ready, so a promotion and a rollback change a setting, not the image. The image tag names one set of code and bundles, and gets a new number each time a bundle is added or removed.

Step 2: start it and check the release

Copy .env.example to .env (in VS Code, or cp .env.example .env; on Windows, Copy-Item .env.example .env). Then build and start:

docker compose up -d --build --wait
docker compose logs api

Output (the API's lines)

INFO house_price_api: model kc-hgb-2015-02 loaded (bundle 29752760649b...) in 120 ms, memory 167 MB
INFO house_price_api: GET /ready 200 18ms id=26c3b47d63bd

The first build took about 26 seconds. The image is 687 MB: the pip install layer is 380 MB, the bundles layer 545 kB.

Now a script for after every release and every rollback. It asks the running service four questions and compares the answers with your project’s files. Make scripts/check_release.py:

scripts/check_release.py

"""Check that a running API serves the version that you released, unchanged.

    python scripts/check_release.py http://127.0.0.1:8000 kc-hgb-2015-02

Run it after every promotion and every rollback. Set HOUSE_API_KEY first if
the API needs a key. It checks four things against the files in this project:

1. /ready is 200.
2. /v1/model names the version, and the SHA-256 of its model file.
3. Parity: each parity case of bundles/<version>/ gets the estimate that the
   bundle recorded, rounded to 1,000 USD as the API rounds it.
4. The contract: /openapi.json equals contract/openapi.json, byte for byte.

The exit code is 0 if every check passed, and 1 if one failed.
"""

import json
import os
import sys
from pathlib import Path

import httpx

PROJECT = Path(__file__).resolve().parent.parent
failures = []


def check(passed: bool, text: str) -> None:
    print(f"  {'PASS' if passed else 'FAIL'}  {text}")
    if not passed:
        failures.append(text)


def main() -> int:
    if len(sys.argv) != 3:
        print(__doc__)
        return 2
    url, version = sys.argv[1], sys.argv[2]
    bundle = PROJECT / "bundles" / version
    card = json.loads((bundle / "model_card.json").read_text("utf-8"))
    cases = json.loads((bundle / "parity_cases.json").read_text("utf-8"))
    key = os.environ.get("HOUSE_API_KEY")
    api = httpx.Client(base_url=url, timeout=10, headers={"X-API-Key": key} if key else {})

    print(f"Release check: expected {version}")
    check(api.get("/ready").status_code == 200, "/ready is 200")

    served = api.get("/v1/model").json()
    check(
        served.get("model_version") == version and served.get("sha256") == card["sha256"],
        f"/v1/model names {served.get('model_version')}, model file {served.get('sha256', '?')[:12]}...",
    )

    matched = 0
    for case in cases:
        answer = api.post("/v1/estimates", json=case["house"]).json()
        expected = int(round(case["expected_estimate"], -3))
        if answer.get("estimate_usd") == expected:
            matched += 1
        else:
            print(f"        {case['name']}: got {answer.get('estimate_usd')}, expected {expected}")
    check(matched == len(cases), f"parity: {matched} of {len(cases)} cases give the bundle's estimates")

    response = api.get("/openapi.json")
    if response.status_code == 404:
        print("  INFO  /openapi.json is hidden (SHOW_DOCS=false): the contract was not compared")
    else:
        reviewed = (PROJECT / "contract" / "openapi.json").read_text("utf-8")
        check(json.dumps(response.json(), indent=2) + "\n" == reviewed,
              "contract: /openapi.json equals contract/openapi.json")

    print("All checks passed." if not failures else f"{len(failures)} check(s) failed.")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())

Parity over HTTP compares rounded estimates, because the API rounds to 1,000 dollars. The start-up self-check already compared the unrounded ones, with a tolerance of 1e-9. Give the script the key, and run it. On Windows, set the key with $env:HOUSE_API_KEY = "local-dev-key":

export HOUSE_API_KEY=local-dev-key
python scripts/check_release.py http://127.0.0.1:8000 kc-hgb-2015-02

Output

Release check: expected kc-hgb-2015-02
  PASS  /ready is 200
  PASS  /v1/model names kc-hgb-2015-02, model file 8d0483d05cfc...
  PASS  parity: 9 of 9 cases give the bundle's estimates
  PASS  contract: /openapi.json equals contract/openapi.json
All checks passed.

This output is the evidence for a release record.

Step 3: a first replay, without a shadow

Send the 2,819 recorded requests to the container:

python scripts/replay.py http://127.0.0.1:8000 replay/requests-2015-04-05.jsonl replay/out/answers.csv

Output

Sent 2819 requests in 13.0 s: 2817 answered 200, refused 422 invalid_request: 2.
Time per request: median 3.4 ms, 95th percentile 9.9 ms.
Wrote replay/out/answers.csv

Two real requests are outside the contract: a lot of 1,164,794 sq ft, and a sqft_living15 of 399. A new model is not a new contract, so every comparison uses the 2,817 answered requests. The times varied by about 1 ms from run to run.

Part 4: the first candidate

The analyst’s candidate

This month, the county changed the export that the analyst downloads: same columns, same sales. We simulate this new export with a script; the real county data never had this change.

python training/simulate_export.py
python training/train.py --data data/county-export-2015-04-01.csv.gz --until 2015-04-01 --version kc-hgb-2015-03-rc1

Output

wrote data/county-export-2015-04-01.csv.gz: 21613 sales
data: county-export-2015-04-01.csv.gz, SHA-256 50d99029d343... (not the reviewed course file)
train 18794 sales | test 2819 sales
trained in 2.7 s
test: {'median_error_pct': 7.1, 'mape_pct': 13.1, 'within_10pct': 0.482, 'over_by_10pct_or_more': 0.12}
range: 0.890 to 1.295 times the estimate (80% of test sales inside)
wrote candidates/kc-hgb-2015-03-rc1/ (house_price.joblib 0.39 MB, model_card.json)
sha256 3cd6b0831257a4b353fb6b3748bd4fbb99ecbb9500041b46a87b846cc0cbd35a

-rc1 means release candidate: a version that may go live, or not. Its offline error (MAPE) is 13.1%, the same as version 1’s on these houses. Make its bundle, and add its line to BUNDLES:

python make_bundle.py candidates/kc-hgb-2015-03-rc1

Output

Made bundles/kc-hgb-2015-03-rc1: 9 parity cases.
Bundle digest: 10652d2d82d7e619c49756ebb9c5c8f9ec261c73c7df645136c2b4f89f001275

house_price_api/config.py (change)

BUNDLES = {
    "kc-hgb-2015-02": "29752760649bf79193867d4a5887ee47c361aa913315d831b0e9fafe6f50d1cf",
    "kc-hgb-2015-03-rc1": "10652d2d82d7e619c49756ebb9c5c8f9ec261c73c7df645136c2b4f89f001275",
}

Run python -m pytest: 54 passed. The candidate passes its self-check, and its contract is the reviewed one.

Predict before you look. The offline error is the same as version 1’s, and every test passes. Will the candidate pass the three release checks?

Run it in the shadow

In .env, set IMAGE_TAG=1.1.1 and SHADOW_VERSION=kc-hgb-2015-03-rc1. The image needs the new bundle, so build it again. Only the layers from COPY bundles on are new; the 380 MB pip install layer comes from the cache, and the build took 9 seconds:

docker compose up -d --build --wait
python scripts/replay.py http://127.0.0.1:8000 replay/requests-2015-04-05.jsonl replay/out/answers.csv

The log has the two start-up lines, and a prediction line for every answered request:

The API's log (shortened)

INFO house_price_api: model kc-hgb-2015-02 loaded (bundle 29752760649b...) in 199 ms, memory 168 MB
INFO house_price_api: shadow kc-hgb-2015-03-rc1 loaded (bundle 10652d2d82d7...), memory 170 MB
...
INFO house_price_api.predictions: {"request_id": "435263bd860e", "item": 0, "model_version": "kc-hgb-2015-02", "estimate": 235254.36, "shadow_version": "kc-hgb-2015-03-rc1", "shadow_estimate": 262660.79}

Compare: the release checks

Now join three files: the replay’s answers (sale ID, request ID), the log (request ID, both estimates) and the sales (sale ID, price). Make scripts/compare.py:

scripts/compare.py

"""Compare the live model and the shadow on a replay, and run the release checks.

    python scripts/compare.py replay/out/answers.csv replay/out/service.log replay/sales-2015-04-05.csv

answers.csv comes from replay.py, service.log is the API's log during the
replay, and the sales file has the prices that arrived later. The script
joins the three, prints the error of each model on the sale prices and how
much the estimates move, runs the release checks, and lists the largest
moves. It writes the joined rows to compare-<shadow version>.csv next to
answers.csv.

A regression model has no threshold to check. So the checks look at two
things: is the new model's error lower, and how much do the answers that
clients see change? A person reads the largest moves before deciding.
"""

import json
import sys
from pathlib import Path

import numpy as np
import pandas as pd

# The release checks, agreed with the valuation team before the shadow ran.
MAX_ERROR_INCREASE = 0.01  # the shadow's MAE may be at most 1% above the live MAE
BIG_MOVE = 0.10  # an estimate "moves" when it changes by more than 10%
MAX_MOVED_SHARE = 0.05  # at most 5% of estimates may move
MAX_AREA_MOVE = 0.05  # no zip code's median estimate may move by more than 5%
MIN_AREA_SALES = 10  # areas with fewer sales are too small to judge
MARKER = "house_price_api.predictions: "


def read_log(path: Path) -> pd.DataFrame:
    """The prediction lines of the log, one row per estimated house."""
    rows = [json.loads(line.split(MARKER, 1)[1]) for line in path.read_text("utf-8").splitlines() if MARKER in line]
    return pd.DataFrame(rows)


def errors(price: pd.Series, estimate: pd.Series) -> dict:
    pct = (price - estimate) / price
    return {
        "Mean absolute error (USD)": f"{np.mean(np.abs(price - estimate)):,.0f}",
        "Within 10% of the price": f"{np.mean(np.abs(pct) <= 0.10):.1%}",
        "Median error (+ = too low)": f"{np.median(pct):+.1%}",
    }


def main() -> int:
    if len(sys.argv) != 4:
        print(__doc__)
        return 2
    answers_file, log_file, sales_file = map(Path, sys.argv[1:])
    answers = pd.read_csv(answers_file, dtype={"sale_id": str})
    answers = answers[answers["status"] == 200].drop(columns="model_version")
    log = read_log(log_file)
    if "shadow_estimate" not in log:
        print("The log has no shadow estimates: start the API with SHADOW_VERSION.")
        return 2
    sales = pd.read_csv(sales_file, dtype={"sale_id": str, "zipcode": str})
    rows = (
        answers.merge(log, on="request_id", validate="one_to_one")
        .merge(sales, on="sale_id", validate="one_to_one")
    )
    rows["move"] = rows["shadow_estimate"] / rows["estimate"] - 1
    live, shadow = rows["model_version"].iloc[0], rows["shadow_version"].iloc[0]
    print(f"Joined {len(rows)} answers with the log and the sale prices.")
    print(f"Live {live}, shadow {shadow}.\n")

    live_errors, shadow_errors = errors(rows["price"], rows["estimate"]), errors(rows["price"], rows["shadow_estimate"])
    print(f"{'':28}{'live':>12}{'shadow':>12}")
    for name in live_errors:
        print(f"{name:28}{live_errors[name]:>12}{shadow_errors[name]:>12}")

    mae_live = np.mean(np.abs(rows["price"] - rows["estimate"]))
    mae_shadow = np.mean(np.abs(rows["price"] - rows["shadow_estimate"]))
    change = mae_shadow / mae_live - 1
    moved = rows["move"].abs() > BIG_MOVE
    areas = rows.groupby("zipcode")["move"].agg(["median", "size"])
    areas = areas[areas["size"] >= MIN_AREA_SALES]
    worst = areas["median"].abs().idxmax()
    checks = [
        (change <= MAX_ERROR_INCREASE,
         f"error: the shadow's MAE is at most {MAX_ERROR_INCREASE:.0%} above the live MAE ({change:+.1%})"),
        (moved.mean() <= MAX_MOVED_SHARE,
         f"moves: at most {MAX_MOVED_SHARE:.0%} of estimates move by more than {BIG_MOVE:.0%} "
         f"({moved.mean():.1%}, {moved.sum()} houses)"),
        (abs(areas.loc[worst, "median"]) <= MAX_AREA_MOVE,
         f"areas: no zip code's median estimate moves by more than {MAX_AREA_MOVE:.0%} "
         f"(largest {worst}: {areas.loc[worst, 'median']:+.1%}, {areas.loc[worst, 'size']} sales)"),
    ]
    print("\nRelease checks")
    for passed, text in checks:
        print(f"  {'PASS' if passed else 'FAIL'}  {text}")

    print("\nThe 5 largest moves (read them before you decide)")
    print(f"  {'sale':22}{'live':>11}{'shadow':>11}{'move':>8}{'price':>11}  closer")
    for row in rows.reindex(rows["move"].abs().sort_values(ascending=False).index).head(5).itertuples():
        closer = "shadow" if abs(row.price - row.shadow_estimate) < abs(row.price - row.estimate) else "live"
        print(f"  {row.sale_id:22}{row.estimate:>11,.0f}{row.shadow_estimate:>11,.0f}"
              f"{row.move:>+8.1%}{row.price:>11,}  {closer}")

    out = answers_file.with_name(f"compare-{shadow}.csv")
    rows[["sale_id", "date", "zipcode", "price", "estimate", "shadow_estimate", "move"]].to_csv(out, index=False)
    print(f"\nWrote {out}")
    return 0 if all(passed for passed, _ in checks) else 1


if __name__ == "__main__":
    sys.exit(main())

Part	Why
The limits at the top, with a comment	They were agreed before the shadow ran. A reviewer sees them in one place.
validate="one_to_one"	Each answer has exactly one log line and one sale. A duplicate or a lost line stops the script, instead of a wrong number.
The areas use zipcode from the sales file	The API never gets the zip code. The comparison may use it, because it runs on your computer, after the fact.
The exit code	0 if all checks pass, 1 if not. A pipeline can use it later, for example in the CI/CD course.

Save the log, then compare:

macOS

docker compose logs --no-log-prefix api > replay/out/service.log
python scripts/compare.py replay/out/answers.csv replay/out/service.log replay/sales-2015-04-05.csv

Windows

docker compose logs --no-log-prefix api | Out-File -Encoding utf8 replay/out/service.log
python scripts/compare.py replay/out/answers.csv replay/out/service.log replay/sales-2015-04-05.csv

Linux

docker compose logs --no-log-prefix api > replay/out/service.log
python scripts/compare.py replay/out/answers.csv replay/out/service.log replay/sales-2015-04-05.csv

Output

Joined 2817 answers with the log and the sale prices.
Live kc-hgb-2015-02, shadow kc-hgb-2015-03-rc1.

                                    live      shadow
Mean absolute error (USD)         77,300      93,864
Within 10% of the price            48.2%       37.1%
Median error (+ = too low)         +7.4%      +10.8%

Release checks
  FAIL  error: the shadow's MAE is at most 1% above the live MAE (+21.4%)
  FAIL  moves: at most 5% of estimates move by more than 10% (23.7%, 667 houses)
  FAIL  areas: no zip code's median estimate moves by more than 5% (largest 98092: -11.9%, 52 sales)

The 5 largest moves (read them before you decide)
  sale                         live     shadow    move      price  closer
  5428000070-20150511       779,716    553,007  -29.1%    770,000  live
  2822059181-20150413       273,872    353,301  +29.0%    306,000  live
  1921069101-20150508       408,910    295,427  -27.8%    399,000  live
  1720069075-20150508       549,665    402,908  -26.7%    530,000  live
  9808100150-20150402     2,411,350  1,785,070  -26.0%  3,345,000  live

Wrote replay/out/compare-kc-hgb-2015-03-rc1.csv

All three checks fail, and for the five largest moves the live model was closer every time. Offline, the same candidate had the same error as version 1. So the problem is not what the model learned: it is between the training data and the API.

Find the cause

Read the three largest moves in the replay file:

grep -E "5428000070-20150511|2822059181-20150413|1921069101-20150508" replay/requests-2015-04-05.jsonl

On Windows, use Select-String -Pattern "5428000070-20150511|2822059181-20150413|1921069101-20150508" replay/requests-2015-04-05.jsonl.

Output (shortened to the fields that matter)

{"sale_id": "2822059181-20150413", "house": {..., "yr_built": 1910, "yr_renovated": null, ...}}
{"sale_id": "1921069101-20150508", "house": {..., "yr_built": 2008, "yr_renovated": null, ...}}
{"sale_id": "5428000070-20150511", "house": {..., "yr_built": 2013, "yr_renovated": null, ...}}

All three were never renovated. The two new houses dropped by 28% and 29%, and the old one rose by 29%. So look at yr_renovated in the two data files. Count the houses with the code 0, “never renovated”:

python -c "import pandas as pd; [print(f, (pd.read_csv('data/' + f).yr_renovated == 0).sum()) for f in ['king_county_house_sales.csv.gz', 'county-export-2015-04-01.csv.gz']]"

Output

king_county_house_sales.csv.gz 20699
county-export-2015-04-01.csv.gz 0

For a house never renovated, the new export writes the build year, not 0. But the API still sends 0 for “never”, as the contract says. So, for the 96% of houses never renovated, rc1 sees a value at serving time that it never saw in training. A tree reads 0 as “the last work was long ago”: very wrong for a new house. rc1’s median move for houses never renovated, by build period:

Built	Houses	Median move	Moved by more than 10%
1900–1939	364	+0.9%	5.8%
1940–1959	504	+1.8%	10.7%
1960–1979	620	-0.3%	6.1%
1980–1999	582	-7.4%	32.3%
2000–2015	647	-10.7%	54.7%

This is training-serving skew in the other direction: here, training changed a value that the service did not. The parity cases could not catch it, because the candidate itself computed their expected estimates. Zip code 98092 moved most because 54% of its replayed houses were built in 1990 or later (all houses: 34%).

Common mistake

The candidate passed every test, and the shadow failed it

Symptom: a candidate has the same offline error as the live model and passes every gate and test. In the shadow, all three release checks fail.

Cause: the training data codes yr_renovated differently from the API. The offline test used the same export, so it could not see the difference. Every check that uses only the candidate agrees with the candidate.

Fix:

1. Reject the candidate. Remove its line from BUNDLES, delete bundles/kc-hgb-2015-03-rc1/, and record the decision in RELEASES.md (Part 5). It is then retired: no image from now on contains it, and no setting can make it live.

2. Stop the cause where it starts. In training/train.py, add a check that the data uses the API’s code, and call it in load_data, after the line that fixes the 33 bedrooms:

   training/train.py (add)

   def check_codes(houses: pd.DataFrame) -> None:
       """Stop if a column uses another code than the API sends to the model."""
       if not (houses["yr_renovated"] == 0).any():
           raise SystemExit(
               "Stopped: yr_renovated is never 0 in this file. The API sends 0 for "
               "'never renovated', so the model must learn from that code."
           )

   Train with the export again. The output is now: Stopped: yr_renovated is never 0 in this file. The API sends 0 for 'never renovated', so the model must learn from that code.

3. Ask the analyst to convert the export to the old code, or to use the reviewed data. Do not change the API to send the build year: that would change what null means in the contract, and every client would need to know.

What it means for a real AI service: offline metrics and unit tests check a model against its own data. A release must also check it against the requests that it will serve, next to the model that it replaces. A shadow on real requests is the cheapest place to find this: no client saw a single wrong estimate.

Part 5: the second candidate, the decision and the promotion

Train, bundle and test

The analyst trains again, from the reviewed course data:

python training/train.py --until 2015-04-01 --version kc-hgb-2015-03
python make_bundle.py candidates/kc-hgb-2015-03

Output

train 18794 sales | test 2819 sales
trained in 3.1 s
test: {'median_error_pct': 6.9, 'mape_pct': 12.9, 'within_10pct': 0.479, 'over_by_10pct_or_more': 0.119}
range: 0.893 to 1.288 times the estimate (80% of test sales inside)
wrote candidates/kc-hgb-2015-03/ (house_price.joblib 0.52 MB, model_card.json)
sha256 054ed960e883b16905447d38db386da87ffbf9240f7c7fc574a9f44861aa4ccb
Made bundles/kc-hgb-2015-03: 9 parity cases.
Bundle digest: 1ed11041ebcd4ccb751b599269c119d4d1d1978b620ac87e04ccf17c83dca30f

Offline, version 2’s typical error is 12.9%, against 13.1% for version 1. You removed the rc1 line from BUNDLES in the fix of Part 4, so add the new one:

house_price_api/config.py (change)

BUNDLES = {
    "kc-hgb-2015-02": "29752760649bf79193867d4a5887ee47c361aa913315d831b0e9fafe6f50d1cf",
    "kc-hgb-2015-03": "1ed11041ebcd4ccb751b599269c119d4d1d1978b620ac87e04ccf17c83dca30f",
}

Run python -m pytest: 54 passed. Now open bundles/kc-hgb-2015-03/parity_cases.json and compare its estimates with version 1’s. The typical house moves from 421,936 to 379,196 (−10.1%), the waterfront house from 2,155,965 to 1,841,795 (−14.6%), and the largest house of the contract by −24%.

Predict before you look. Seven of the nine parity cases moved by more than 5%. What share of the 2,817 real houses will move by more than 10%?

Shadow and compare

In .env, set IMAGE_TAG=1.1.2 and SHADOW_VERSION=kc-hgb-2015-03. The image 1.1.2 holds versions 1 and 2, and not rc1. Then the same four commands as in Part 4:

docker compose up -d --build --wait
python scripts/replay.py http://127.0.0.1:8000 replay/requests-2015-04-05.jsonl replay/out/answers.csv
docker compose logs --no-log-prefix api > replay/out/service.log
python scripts/compare.py replay/out/answers.csv replay/out/service.log replay/sales-2015-04-05.csv

On Windows, save the log with Out-File -Encoding utf8, as in Part 4.

Output of compare.py

Joined 2817 answers with the log and the sale prices.
Live kc-hgb-2015-02, shadow kc-hgb-2015-03.

                                    live      shadow
Mean absolute error (USD)         77,300      76,960
Within 10% of the price            48.2%       47.9%
Median error (+ = too low)         +7.4%       +6.9%

Release checks
  PASS  error: the shadow's MAE is at most 1% above the live MAE (-0.4%)
  PASS  moves: at most 5% of estimates move by more than 10% (2.9%, 82 houses)
  PASS  areas: no zip code's median estimate moves by more than 5% (largest 98014: -2.8%, 14 sales)

The 5 largest moves (read them before you decide)
  sale                         live     shadow    move      price  closer
  9253900408-20150408     1,879,833  1,427,749  -24.0%  1,400,000  shadow
  853200040-20150428      2,168,502  2,665,208  +22.9%  2,408,000  live
  3052700385-20150414       694,199    554,286  -20.2%    765,000  live
  4134300175-20150417     2,095,206  1,674,514  -20.1%  1,851,000  shadow
  4385700250-20150407     1,638,151  1,313,107  -19.8%  1,800,000  live

Wrote replay/out/compare-kc-hgb-2015-03.csv

The two candidates side by side, on the same 2,817 requests. Follow 10% on the bottom axis up to each line:

• Version 2 (kc-hgb-2015-03)
• rc1 (new export)

The share of the 2,817 replayed estimates that move by more than a given size, against the live model. At 10%, the limit of the release check, version 2 moves 2.9% of estimates and rc1 moves 23.7%. The limit is 5%.

Text description of the chart

A line chart with two falling lines. The horizontal axis is the size of a move, from 0% to 30%. The vertical axis is the share of the 2,817 estimates that move by more than that size, from 0% to 100%. Both lines start at 100% at 0%, because nearly every estimate changes a little. The line of version 2 falls fast: about 15% of its estimates move by more than 5%, and 2.9% by more than 10%. The dashed line of rc1 falls slowly: 23.7% of its estimates move by more than 10%, and it reaches 0% only near 30%.

The mean absolute error of each model on the 2,817 answered requests of April and May 2015, against their sale prices. rc1 is 21% worse than the live model; version 2 is 0.4% better.

Text description of the chart

A column chart with three columns: the live model (version 1), 77,300 dollars; rc1, 93,864 dollars; version 2, 76,960 dollars. The columns of the live model and version 2 look the same height; rc1’s column is clearly taller.

All three checks pass. Version 2 is a little better on average (MAE 340 dollars lower), not on every measure (47.9% within 10%, against 48.2%). The parity cases were a poor guide: they moved a lot, but 84.5% of the real estimates moved by 5% or less.

Read the largest moves

The checks pass; now a person reads. Four of the five largest moves are houses estimated at 1 million dollars or more. In that group, 14.7% of the 156 houses moved by more than 10%, against 2.2% of the others. Few expensive houses sell each month, and the API course’s API already adds the warning high_estimate to them.

Did the moves help? For 39 of the 82 houses that moved by more than 10%, version 2 was closer to the sale price: about half. So these moves are uncertainty, not corrections. Kenji writes this down for Hannah.

Decide and promote

Hannah decides to promote. The gain in error is small, and she says so in the record. The reason to release is one more month of prices: prices_as_of becomes 2015-03-31.

To promote, change two lines in .env: MODEL_VERSION=kc-hgb-2015-03 and SHADOW_VERSION= (empty). Keep IMAGE_TAG=1.1.2: the image does not change. Then restart and check the release:

docker compose up -d --wait
python scripts/check_release.py http://127.0.0.1:8000 kc-hgb-2015-03
curl -s -X POST http://127.0.0.1:8000/v1/estimates -H "X-API-Key: local-dev-key" -H "Content-Type: application/json" -d @requests/typical-house.json

Output

Release check: expected kc-hgb-2015-03
  PASS  /ready is 200
  PASS  /v1/model names kc-hgb-2015-03, model file 054ed960e883...
  PASS  parity: 9 of 9 cases give the bundle's estimates
  PASS  contract: /openapi.json equals contract/openapi.json
All checks passed.
{"estimate_usd":379000,"range_usd":{"low":339000,"high":488000},"model_version":"kc-hgb-2015-03","prices_as_of":"2015-03-31","warnings":[]}

Version 2 is live, with the same fields and a new model_version and prices_as_of. Elif’s tool needs no change.

Keep the record

Make RELEASES.md: the model registry of module 5, kept by hand. Below its table come the agreed checks, and one entry per event with its evidence and the person who decided. The finished project has the complete file. The table:

RELEASES.md (start)

| Version | Bundle digest | Status | Trained with sales to | Image |
|---|---|---|---|---|
| kc-hgb-2015-02 | `2975276064...` | previous: the rollback target | 2015-02-28 | 1.1.2 |
| kc-hgb-2015-03-rc1 | `10652d2d82...` | rejected, retired | 2015-03-31 (new export) | 1.1.1 only |
| kc-hgb-2015-03 | `1ed11041eb...` | **live** | 2015-03-31 | 1.1.2 |

(The finished file has the full digests.) rc1’s entry stays after its retirement: it is evidence that the checks work.

Part 6: the rollback drill

The problem: a way back that nobody tried

Daniel asked how fast you can go back. An untested rollback is a hope, not a plan. A drill answers with a measured number, on a quiet day.

Step 1: go back to version 1

In .env, set MODEL_VERSION=kc-hgb-2015-02. Then:

docker compose up -d --wait
python scripts/check_release.py http://127.0.0.1:8000 kc-hgb-2015-02

Output of check_release.py

Release check: expected kc-hgb-2015-02
  PASS  /ready is 200
  PASS  /v1/model names kc-hgb-2015-02, model file 8d0483d05cfc...
  PASS  parity: 9 of 9 cases give the bundle's estimates
  PASS  contract: /openapi.json equals contract/openapi.json
All checks passed.

docker compose up -d --wait returned after 6.4 seconds (the health check runs every 5 seconds). A small script that asked /v1/model every 20 ms saw version 2 until 1.23 s, no answer until 2.81 s, then version 1: a gap of about 1.6 seconds. It takes seconds because the previous bundle is already in the image and already reviewed.

Step 2: return to version 2

Set MODEL_VERSION=kc-hgb-2015-03 again, and run the same two commands with kc-hgb-2015-03. Then add the drill to RELEASES.md: the date, the 6.4 s, the 1.6 s gap and the two release checks.

The gap is the single-container trap: up -d stops the old container before the new one is ready. A platform with revisions keeps the old one serving (Operational configuration in the Docker course; Rollback strategy in the CI/CD course, if you took it).

Common mistake

A typo in MODEL_VERSION, and no estimates at all

Symptom: after a change in .env, docker compose up -d --wait waits about 22 seconds, then stops with container ... is unhealthy and exit code 1. /health answers {"status":"ok"}, but every estimate gets 503 model_unavailable. The log says:

ERROR house_price_api: model not loaded: kc-hgb-2015-2 is not a reviewed bundle (config.BUNDLES)

Cause: MODEL_VERSION=kc-hgb-2015-2 is missing a 0. The API refused it, as designed: it is not a reviewed version. But with one container, the old one is already gone. So a typo in a rollback, the moment of most stress, means no estimates at all.

Fix:

1. Correct the line in .env, and run docker compose up -d --wait again. In our run, the service was healthy 6.3 seconds later.
2. Copy the version from RELEASES.md instead of typing it.
3. On a real platform, use revisions: the old version keeps serving while the new one fails its health check.

What it means for a real AI service: the safety check worked: no unknown model was loaded. But a check that stops the start-up also stops the service. Plan the release so that the old version serves until the new one is ready.

Clean up

Stop the container and remove the three images. The bundles, the code and RELEASES.md stay in your project.

docker compose down
docker image rm house-price-api:1.1.0 house-price-api:1.1.1 house-price-api:1.1.2

On a real registry, keep 1.1.2 (the live and the previous version) and delete 1.1.1, the only image with the retired rc1.

Our run-through

We ran every step of this page from starter.zip in a new folder with a new virtual environment. The commands and full outputs are in the course’s repository (reference/c17/case-studies/house-price-release/RUNS.md).

Check	Result
Tests after each part	49 (start), 50, 52, then 54 passed; the finished project: 54 passed
Bundles	The same digest as on this page for every bundle (training is reproducible)
Replay	2,817 of 2,819 requests answered; 2 refused by the contract
Release checks	rc1: all three fail; version 2: all three pass
Rollback drill	Healthy after 6.4 s; about 1.6 s without an answer
Typo in MODEL_VERSION	Unhealthy after 22 s, exit code 1; healthy 6.3 s after the fix
Not tested	Windows and Linux; a platform with revisions; a cloud deployment

Responsible use

A new model changes numbers that people act on. The release is where you decide whether they should.

• Not for decisions about people. As in the API course, the estimate is a first number for a valuation team. A model release does not change that limit, and the card of every version keeps the same not_for list.
• The area check measures change, not fairness. In the United States, an area’s prices are tied to who lives there, through a long history of discrimination. A release that moves one zip code’s estimates by 12% moves a community’s numbers. The areas check makes that visible and stops it. It does not show whether either model treats areas fairly. That needs a review by qualified people, with better data.
• The shadow log is data about requests. It has request IDs, versions and estimates, and no house description. Keep it only as long as the comparison needs it, and delete it after the decision, under the lender’s own data rules.
• Old prices stay old. Version 2 knows prices up to March 2015. Every answer says so in prices_as_of.

Make it a portfolio piece

This project shows an employer that you can change a model in production without surprising anyone. It has pinned bundles, a shadow, release checks agreed in advance, a recorded decision and a measured rollback. Publish it as a repository under your name.

house-price-api/
  README.md               what it is, how to run it, how a release works
  RELEASES.md             the registry: versions, digests, status, evidence, decisions
  MODEL_CARD.md           both versions, their test results and limits
  LICENSE                 MIT, for the code
  Dockerfile, compose.yaml, .dockerignore, .env.example, requirements-run.txt
  make_bundle.py          a model folder -> bundles/<version>/ with SHA256SUMS
  bundles/                kc-hgb-2015-02 and kc-hgb-2015-03 (model, card, parity cases, sums)
  house_price_api/        the API: config.py (BUNDLES), estimator.py (the five gates), routes.py (the shadow)
  training/               train.py (with the code check) and simulate_export.py
  replay/, scripts/       make_replay.py, replay.py, compare.py, check_release.py
  tests/                  54 tests

1. Make sure that .gitignore lists .venv/, .env, data/, candidates/ and replay/out/. The finished project’s .gitignore does.
2. Make the repository: git init, git add ., git commit -m "House price API with versioned model releases".
3. Check with git ls-files bundles that both bundles are in it, with their model.joblib files, and that no .env file is.
4. Add the two charts of this page as images, or the output of compare.py, to the README. Write the release decision in your own words.

How to describe it in an interview, in two sentences: “I released a retrained house-price model behind a live API, with pinned bundles and a shadow on 2,800 replayed requests. My release checks rejected a candidate with a good offline score whose data coded a column differently, and I measured the rollback at six seconds.”

Go further:

1. Keep the previous version in the shadow for a week after a promotion. If the new one surprises you, the log already has the old estimates to compare.
2. A canary. Send 10% of real requests to the new version before all of them, as in Progressive release in the CI/CD course. The answers of a canary are real, so its checks must pass first in the shadow.
3. Compare in the pipeline. Run compare.py in CI on every new bundle, and use its exit code to block the merge of a new line in BUNDLES.
4. Monitor after the release. When next month’s sale prices arrive, compute the share within 10% for the live version. The API course’s model card says to stop below 45%.

Knowledge check

Check your understanding

Version 2 moves the typical golden house from 422,000 to 379,000, away from its sale price of 425,000, so the API course's golden test would fail for it. Why is this not a reason to reject version 2?

Show the answer

One house cannot measure a model. On the 2,817 replayed houses, version 2’s error is 0.4% lower, and only 2.9% of its estimates move by more than 10%. The typical house is one of those 82. The golden tests make every change visible; they do not decide. A retrain will fail them, and a person then updates their expected answers in a reviewed change. The decision comes from the agreed checks on many real requests, and from reading the largest moves.

Check your understanding

rc1 had the same offline error as version 1, a valid bundle and 54 passing tests. Why did only the shadow catch it?

Show the answer

Every earlier check compared rc1 with its own data. The offline test used the same export, with the build year in yr_renovated. The parity cases were computed by rc1 itself, so they agreed with it. The shadow was the first check that sent real requests through the API’s translation, which sends 0 for “never renovated”, and compared the result with the live model and with the sale prices. That showed the skew: 23.7% of estimates moved by more than 10%, and the error rose by 21%.

Check your understanding

Why does the code hold the digests in BUNDLES, while the environment only holds MODEL_VERSION? What could go wrong if .env held the digest, as in the escalation service?

Show the answer

The code changes only through a reviewed change, so every model that can be live has passed a person’s review. The environment can then choose quickly, for a promotion or a rollback, without a new image. But it can choose only among reviewed bundles. If .env held the digest, anyone who can change the server’s settings could set the digest of their own bundle. The service would then load it. A typo in MODEL_VERSION makes the service refuse to start the model, which is safe but stops the estimates (the second failure case).

Recap

• A new model is a release: a bundle that never changes, with one digest. The code lists the reviewed digests; the environment only chooses among them.
• A shadow estimates every request with the candidate, answers with the live model only, and logs both. It must never break a live answer, and it doubled the model work (1.0 to 2.1 ms).
• A regression model has no threshold. Its release checks are the error on newer sales (MAE), the share of estimates that move by more than 10%, and the largest move of one area. Agree on them before the shadow runs, then let a person read the largest moves.
• A candidate can pass every offline test and still be wrong for the requests it will serve: rc1’s data coded yr_renovated differently.
• Promote and roll back with one setting, then check the running service. Our rollback took 6.4 s, with about 1.6 s without an answer in a single container.
• Record every candidate, decision and drill in RELEASES.md, including the rejected one.

Next, the third case study: a large file of hospital discharges, scored as a queued job.

Next: Score hospital readmissions asynchronously