Artifact trust and integrity

What you will be able to do

Load a model bundle only after you prove that it is the exact bundle you expect. A changed byte gives wrong scores, and a model file can run code.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The bundle and its four files, from Bundle model dependencies.
• A SHA-256 hash as a fingerprint, and a pickle file that runs code: the API course’s case study Serve a house-price model: Part 2.
• An image digest, from Build efficiently: Pin the versions in the Docker course.

What you need on your computer

• ~/projects/escalation-service with escalation/contract.py, escalation/bundle.py, make_bundle.py and bundles/escalation-1.0.0/, from Bundle model dependencies: Make the bundle.
• tests/conftest.py and tests/test_contract.py, from Prediction inputs and outputs: Test the contract.
• If your project is missing or broken, you can copy the finished module, end-of-m01, and read along. Use a snapshot has the steps.

Tested with Python 3.14.6 on macOS, Apple silicon. The Windows and Linux steps are not tested.

A seal on a medicine bottle

A new bottle of medicine has a seal under its cap. If the seal is broken, you do not take the medicine, even if it looks normal. You cannot see a change in a pill, so you check the seal instead. And you trust the seal because it comes from the factory, not from whoever handed you the bottle.

A model file is the same. You cannot see a change in it, and a changed file can still look and work normally.

The problem: one byte, and code that runs

Amira copies model.joblib and changes one byte of the copy: the byte that holds the sign of one coefficient (the one for team = payment). Then she scores the example ticket with both files. The script is one_byte.py, in the project folder:

one_byte.py

"""Change one byte of a copy of the model, then compare hashes and scores."""

import hashlib
from pathlib import Path

import joblib

from escalation.contract import TicketIn, to_frame

MODEL = Path("bundles/escalation-1.0.0/model.joblib")
COPY = Path("changed.joblib")

original = MODEL.read_bytes()
pipeline = joblib.load(MODEL)
position = original.find(pipeline[-1].coef_.tobytes()) + 7 * 8 + 7  # coefficient 8, its last byte
changed = bytearray(original)
changed[position] ^= 0x80  # change one bit of one byte: the sign of the number
COPY.write_bytes(changed)

example = TicketIn.model_config["json_schema_extra"]["examples"][0]
frame = to_frame([TicketIn(**example)])
for name, data, path in [("original", original, MODEL), ("changed ", bytes(changed), COPY)]:
    score = joblib.load(path).predict_proba(frame)[0, 1]
    print(f"{name} {hashlib.sha256(data).hexdigest()}  score {score:.4f}")
print(f"Bytes that differ: {sum(a != b for a, b in zip(original, changed))} of {len(original)}")
COPY.unlink()

Run it with python one_byte.py. The output:

original a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736  score 0.3507
changed  d794091256b80c7d48c16b56fd37b300f639749a312cca46a344fb561c5afacb  score 0.2007
Bytes that differ: 1 of 5354

One byte of 5,354, and the score fell from 0.3507 to 0.2007. The changed file loaded, scored and gave no warning. This ticket is still flagged, just above 0.19. On all 1,400 July tickets, the changed file flags 15.6% instead of 20.1%, and 64 tickets lose their flag. A broken disk, a bad copy or a person could make this change. Nobody would see it in the scores.

The second problem is worse. You saw in the machine learning course and the API course that a joblib file holds instructions, and that loading it runs them. A harmless file can call print while it loads; a harmful one can call anything, with the rights of your service. So the check must come before joblib.load. Checking the scores after loading is too late.

The idea: a fingerprint for every file

A hash is a fingerprint of some bytes: SHA-256 reads every byte and returns 64 characters. The same bytes always give the same hash. Different bytes give a completely different hash, even when one character differs:

Python

import hashlib

for text in ["threshold 0.19", "threshold 0.18"]:
    print(text, hashlib.sha256(text.encode()).hexdigest())

The output:

threshold 0.19 a3ab6ef04db90010e235ae18413f3f93b37b44558f6b1676a22ca071a2cf3f17
threshold 0.18 442296fdd032932d01be065454160a7fb142bd90c52e1d01029b07e7cf238c7f

The model file showed the same: a4acd259… became d7940912…. Nobody can change a file and keep its hash. So if you know the right hash, you can check a file before you open it.

A bundle has four files. Their four hashes go into one more file, SHA256SUMS, one line per file. Then the hash of SHA256SUMS is one fingerprint for the whole bundle: the bundle digest. A change in any byte of any file changes that file’s hash, so it changes SHA256SUMS, so it changes the digest. It works like an image digest in the Docker course: 64 characters name exact content.

Provenance: where the digest comes from

Where does the service get the right digest? Not from the bundle. Think about the person who changed model.joblib. They can also compute the new hash and write it into SHA256SUMS. A check against a value inside the bundle then passes. The broken seal was replaced by a new seal.

So the expected digest must come from a trusted place outside the bundle. In this course, that is the setting MODEL_SHA256. Priya writes it down when she makes the bundle, and a reviewer approves it in the release record (Module 5) and in the deployment settings. To change the model, a person must change both the bundle and that reviewed setting.

This is provenance: who made an artifact, from what, and how you got it. Part of it is inside the bundle: metadata.json records when the model was trained, the hashes of its training data, and its package versions. The proof that the bundle is the one Priya made comes from outside.

The digest is pinned to the service from outside the bundle. Changing the model needs a change in two places, and one of them is reviewed.

Text description of the diagram

The diagram of the last lesson. make_bundle.py turns the machine learning course’s model/ folder into bundles/escalation-1.0.0/: model.joblib, contract.json, metadata.json, parity_cases.csv and SHA256SUMS, the hash of each other file. escalation-service reads the bundle with load_bundle at start-up. The digest, 8f233029aa4c..., is the hash of SHA256SUMS. It is kept in a trusted place outside the bundle, as MODEL_SHA256, and pinned to the service with a dashed arrow.

Six gates, in order

load_bundle checks a bundle in six steps. Each step is a gate: if it fails, the loader raises BundleError with a message that names the gate, and the service does not start. The order is the point.

The loader checks the bytes before it loads anything, and the model's behaviour after. A no at any gate stops the service.

Text description of the diagram

load_bundle gets a folder and the expected digest, MODEL_SHA256. Three gates check the bytes while nothing is loaded yet. Gate 1, the digest: the hash of SHA256SUMS must equal MODEL_SHA256; if not, “wrong bundle”. Gate 2, the file hashes: each file must match its line in SHA256SUMS; if not, “a file changed”. Gate 3, the versions: the installed scikit-learn, numpy, pandas and joblib must equal metadata.json; if not, “other versions”.

Only then, step 4: joblib.load. Two gates check the loaded model. Gate 5, the features: the pipeline’s feature_names_in_ must equal the contract’s list and FEATURES; if not, “other features”. Gate 6, the self-check: the 14 parity cases must score within 1e-9 of their expected scores; if not, “other scores”. If every gate says yes, the bundle is ready to score. A no at any gate raises BundleError, and the service does not start.

Gates 1 and 2 come first because they need only bytes, and bytes are safe to read. Gate 3 comes before the load because an object pickled with other versions can load and still compute differently. Gates 5 and 6 need the loaded pipeline, so they come last. The self-check also proves the whole chain: these packages, this pipeline and this contract give the scores that training gave.

Tomás asks: “Gate 1 already proves the bytes. Why gate 2?” Gate 1 proves that SHA256SUMS is the right list. Gate 2 proves that the files match the list. Without gate 2, a changed model.joblib next to the original SHA256SUMS would pass.

Write the loader

Replace escalation/bundle.py with the full loader. The first part has the names, the error and the Bundle class:

escalation/bundle.py

"""Load a model bundle safely, and refuse a bundle that is not the one we expect.

A bundle is a folder with everything the model needs to run:

    model.joblib       the fitted pipeline (preprocessing + model)
    contract.json      the features, their order and rules, and the threshold
    metadata.json      the version, the training data and the tested versions
    parity_cases.csv   fixed tickets and the scores they got in training
    SHA256SUMS         the SHA-256 of each file above

A joblib file can run code when it is loaded. So the checks run in this order,
and nothing is unpickled until the bytes are proven to be the expected ones.
"""

import hashlib
import json
from dataclasses import dataclass
from importlib.metadata import version
from pathlib import Path

import joblib
import numpy as np
import pandas as pd

from escalation.contract import FEATURES

FILES = ["model.joblib", "contract.json", "metadata.json", "parity_cases.csv"]
PACKAGES = ["scikit-learn", "numpy", "pandas", "joblib"]
TOLERANCE = 1e-9


class BundleError(Exception):
    """The bundle is not safe or not right to serve. The service must not start."""


@dataclass
class Bundle:
    path: Path
    digest: str
    pipeline: object
    contract: dict
    metadata: dict

    @property
    def version(self) -> str:
        return self.metadata["model_version"]

    @property
    def threshold(self) -> float:
        return self.contract["output"]["threshold"]

    @property
    def known(self) -> dict[str, list[str]]:
        return self.contract["known_categories"]

The second part, at the end of the same file, has the gates and load_bundle:

escalation/bundle.py (continued)

def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def bundle_digest(path: Path) -> str:
    """One fingerprint for the whole bundle: the SHA-256 of its SHA256SUMS file."""
    return sha256(path / "SHA256SUMS")


def check_files(path: Path, expected_digest: str) -> str:
    """Step 1 and 2: the bundle is the one we pinned, and no file has changed."""
    sums = path / "SHA256SUMS"
    if not sums.is_file():
        raise BundleError(f"{path}: no SHA256SUMS file")
    digest = bundle_digest(path)
    if digest != expected_digest:
        raise BundleError(
            f"{path}: the bundle digest is {digest[:12]}..., "
            f"but MODEL_SHA256 expects {expected_digest[:12]}..."
        )
    for line in sums.read_text(encoding="utf-8").splitlines():
        expected, name = line.split()
        if sha256(path / name) != expected:
            raise BundleError(f"{path / name}: the file changed after the bundle was made")
    return digest


def check_versions(metadata: dict) -> None:
    """Step 3: the packages here are the versions the bundle was tested with."""
    for package in PACKAGES:
        tested = metadata["versions"][package]
        installed = version(package)
        if installed != tested:
            raise BundleError(
                f"{package} {installed} is installed, "
                f"but the bundle was tested with {tested}"
            )


def self_check(bundle: Bundle) -> float:
    """Step 5: score the parity cases and compare with the training scores."""
    cases = pd.read_csv(
        bundle.path / "parity_cases.csv", keep_default_na=False, na_values=[""]
    )
    scores = bundle.pipeline.predict_proba(cases[FEATURES])[:, 1]
    largest = float(np.abs(scores - cases["expected_score"]).max())
    if largest > TOLERANCE:
        raise BundleError(
            f"self-check failed: scores differ from the bundle's by up to {largest:.1e}"
        )
    return largest


def load_bundle(path: str | Path, expected_digest: str) -> Bundle:
    path = Path(path)
    digest = check_files(path, expected_digest)
    contract = json.loads((path / "contract.json").read_text(encoding="utf-8"))
    metadata = json.loads((path / "metadata.json").read_text(encoding="utf-8"))
    check_versions(metadata)

    pipeline = joblib.load(path / "model.joblib")  # step 4: only now

    fitted = list(pipeline.feature_names_in_)
    if fitted != contract["features"] or fitted != FEATURES:
        raise BundleError("the pipeline, the contract and this service list different features")
    bundle = Bundle(path, digest, pipeline, contract, metadata)
    self_check(bundle)
    return bundle

The tolerance is 1e-9, not 0. A score computed on another computer can differ in the last of its 16 digits, because the numbers are added in a slightly different order. Module 3 measures this. A real change, like the changed byte above, moves a score by 0.15.

Make the bundle again, with its digest

make_bundle.py must now write SHA256SUMS and print the digest. Make three changes in it:

1. In the docstring, change the end of the paragraph that starts with “Reads” to:

   make_bundle.py

   bundles/escalation-<version>/ with the model, its contract, its metadata, the
   parity cases and SHA256SUMS. It prints the bundle digest: give it to the
   service as MODEL_SHA256.

2. Change the import from escalation.bundle:

   make_bundle.py

   from escalation.bundle import FILES, PACKAGES, bundle_digest, sha256

3. At the end of main(), after the line that writes metadata.json, write SHA256SUMS and print the digest. The end of main() is now:

   make_bundle.py

       (out / "metadata.json").write_text(json.dumps(metadata, indent=2) + "\n", encoding="utf-8")
   
       lines = [f"{sha256(out / name)}  {name}" for name in FILES]
       (out / "SHA256SUMS").write_text("\n".join(lines) + "\n", encoding="utf-8")
   
       print(f"Made {out}: {len(cases)} parity cases, threshold {info['threshold']}.")
       print(f"Bundle digest (MODEL_SHA256): {bundle_digest(out)}")

Version 1.0.0 already exists, and a version is never changed. Here you may make it again, once, because 1.0.0 was never released: no service, setting or log points at it yet. After a release, this would be a new version.

1. Delete the old folder.

   macOS / Linux

   rm -r bundles/escalation-1.0.0

   Windows (PowerShell)

   Remove-Item -Recurse bundles\escalation-1.0.0

2. Make the bundle.

   Terminal

   python make_bundle.py ../ticket-model --version 1.0.0

   The output, for the course’s bundle:

   Made bundles/escalation-1.0.0: 14 parity cases, threshold 0.19.
   Bundle digest (MODEL_SHA256): 8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74

   Your digest is different. Your metadata.json has your own trained_at, so its hash, SHA256SUMS and the digest differ. Only the course’s bundle in the labs has the digest 8f233029…. Copy your digest; the next steps use it.

3. Look at bundles/escalation-1.0.0/SHA256SUMS. It has four lines, one per file. Your metadata.json line is different from this one:

   bundles/escalation-1.0.0/SHA256SUMS

   a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736  model.joblib
   eff05c0f94b86a95d4f610d397d2f7914e1407de3695532d105f71bfbfd9c9d8  contract.json
   9b281b1c04b90567b621bedfc152956600a4ab66bd24bfe0fc803c94b8d719ca  metadata.json
   6aad77934a0f4bcabb0bd450ef419528519646bec9b2879229b3a646a5f5f748  parity_cases.csv

4. Load the bundle in python, with your digest in place of the course’s:

   Python

   from escalation.bundle import load_bundle
   
   bundle = load_bundle("bundles/escalation-1.0.0", "8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74")
   print(bundle.version, bundle.threshold, bundle.digest[:12])

   The output:

   1.0.0 0.19 8f233029aa4c

   Your line shows the first 12 characters of your own digest.

5. Now give it the digest of another version, 1.1.0, from Module 5:

   Python (continued)

   load_bundle("bundles/escalation-1.0.0", "382077148b6cad58238e4832f0951c6556f82832bc6e7de1d3a2335bd6d36af0")

   The output (the last line; the first digest is yours):

   escalation.bundle.BundleError: bundles/escalation-1.0.0: the bundle digest is 8f233029aa4c..., but MODEL_SHA256 expects 382077148b6c...

   Gate 1 stopped it. Leave Python with exit().

Try the gates with a harmful-looking file

The last demonstration of the danger: a file that calls print while it loads. It is harmless on purpose: the only code in it is print. Put it into a copy of the bundle, and try to load the copy three ways. Make surprise.py in the project folder:

surprise.py

"""Harmless: the only code in the file is a call to print."""

import hashlib
import shutil
from pathlib import Path

import joblib

from escalation.bundle import FILES, BundleError, bundle_digest, load_bundle


class Surprise:
    def __reduce__(self):  # pickle calls this function at load time
        return (print, ("   >>> This line ran while the file was loading.",))


PINNED = bundle_digest(Path("bundles/escalation-1.0.0"))  # the digest you wrote down
copy = Path("surprise-bundle")
shutil.copytree("bundles/escalation-1.0.0", copy)
joblib.dump(Surprise(), copy / "model.joblib")

print("1. joblib.load on its own:")
joblib.load(copy / "model.joblib")

print("2. load_bundle, only model.joblib changed:")
try:
    load_bundle(copy, PINNED)
except BundleError as error:
    print("   Refused:", error)

print("3. load_bundle, SHA256SUMS rewritten to match:")
lines = [f"{hashlib.sha256((copy / name).read_bytes()).hexdigest()}  {name}" for name in FILES]
(copy / "SHA256SUMS").write_text("\n".join(lines) + "\n", encoding="utf-8")
try:
    load_bundle(copy, PINNED)
except BundleError as error:
    print("   Refused:", error)

shutil.rmtree(copy)

Predict before you run it: in which of the three tries does the line >>> This line ran… appear? Then run python surprise.py. The output (your digests differ):

1. joblib.load on its own:
   >>> This line ran while the file was loading.
2. load_bundle, only model.joblib changed:
   Refused: surprise-bundle/model.joblib: the file changed after the bundle was made
3. load_bundle, SHA256SUMS rewritten to match:
   Refused: surprise-bundle: the bundle digest is 9a3bb70e16ba..., but MODEL_SHA256 expects 8f233029aa4c...

Only the bare joblib.load ran the file’s code. In try 2, gate 2 found the changed file. In try 3, the “attacker” made the bundle agree with itself. Gate 1 still refused it, because the pinned digest came from outside the bundle. In neither try did load_bundle reach joblib.load. Delete surprise.py and one_byte.py when you are done; they are not part of the project.

Test the loader

The tests use the bundle in the project. Add two lines to tests/conftest.py, after import pytest, and the import that they need at the top:

tests/conftest.py

from pathlib import Path

import pytest

ROOT = Path(__file__).parent.parent
BUNDLE = ROOT / "bundles" / "escalation-1.0.0"

Then make tests/test_bundle.py. Each test breaks one gate in a copy of the bundle:

tests/test_bundle.py

import json
import shutil

import pytest

from escalation.bundle import BundleError, bundle_digest, load_bundle
from tests.conftest import BUNDLE


@pytest.fixture
def copy(tmp_path):
    target = tmp_path / "bundle"
    shutil.copytree(BUNDLE, target)
    return target


def test_the_pinned_bundle_loads():
    bundle = load_bundle(BUNDLE, bundle_digest(BUNDLE))
    assert bundle.version == "1.0.0"
    assert bundle.threshold == 0.19


def test_a_wrong_digest_stops_before_loading(copy, monkeypatch):
    loaded = []
    monkeypatch.setattr("escalation.bundle.joblib.load", lambda path: loaded.append(path))
    with pytest.raises(BundleError, match="MODEL_SHA256 expects"):
        load_bundle(copy, "0" * 64)
    assert loaded == []


def test_a_changed_file_is_found(copy):
    digest = bundle_digest(copy)
    with open(copy / "model.joblib", "ab") as model:
        model.write(b"\0")
    with pytest.raises(BundleError, match="changed after the bundle was made"):
        load_bundle(copy, digest)


def test_other_package_versions_are_refused(copy, monkeypatch):
    monkeypatch.setattr("escalation.bundle.version", lambda package: "0.0.1")
    with pytest.raises(BundleError, match="was tested with"):
        load_bundle(copy, bundle_digest(copy))


def test_a_self_check_failure_stops_the_bundle(copy):
    cases = (copy / "parity_cases.csv").read_text(encoding="utf-8").splitlines()
    header, first = cases[0], cases[1].rsplit(",", 1)
    cases[1] = f"{first[0]},0.5"
    (copy / "parity_cases.csv").write_text("\n".join(cases) + "\n", encoding="utf-8")
    sums = [line for line in (copy / "SHA256SUMS").read_text().splitlines()]
    import hashlib

    new = hashlib.sha256((copy / "parity_cases.csv").read_bytes()).hexdigest()
    sums = [f"{new}  parity_cases.csv" if line.endswith("parity_cases.csv") else line for line in sums]
    (copy / "SHA256SUMS").write_text("\n".join(sums) + "\n")
    with pytest.raises(BundleError, match="self-check failed"):
        load_bundle(copy, bundle_digest(copy))


def test_the_contract_lists_the_pipeline_features():
    contract = json.loads((BUNDLE / "contract.json").read_text())
    assert len(contract["features"]) == 11

test_a_wrong_digest_stops_before_loading replaces joblib.load with a function that only records its call. The list stays empty: with a wrong digest, nothing is unpickled. The tests take the expected digest from the bundle itself, because they test the loader, not the release. The service in Module 2 takes it from MODEL_SHA256.

Run every test:

Terminal

python -m pytest

The output (shortened):

tests/test_bundle.py ......                                              [ 33%]
tests/test_contract.py ............                                      [100%]

============================== 18 passed in 1.72s ==============================

Check a file that you downloaded

The same idea protects a file that you download, such as the course’s model in C17/start/model. Compute its hash before you load it, and compare it with the hash that the course publishes in that folder’s README.md:

macOS / Linux

shasum -a 256 ../ticket-model/model/escalation_model.joblib

Windows (PowerShell)

(Get-FileHash ..\ticket-model\model\escalation_model.joblib).Hash.ToLower()

The output on macOS:

a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736  ../ticket-model/model/escalation_model.joblib

If the hashes are different, do not load the file. Download it again. If it is still different, report it. If you trained the model yourself with the machine learning course’s train.py, your hash can be different, and that is not an error. Training gave the same bytes in two runs on the same computer with the same versions. On another computer or operating system, nobody promises that.

Safer formats, and why this course keeps joblib

Some formats cannot run code when they load. skops (skops.io) saves scikit-learn models in a format that lists the types inside the file. skops.io.load loads only the types that you name as trusted, and it refuses others. ONNX stores a model as a graph of math operations that a separate runtime executes. It is not Python code, but not every scikit-learn pipeline converts to ONNX. These formats reduce the risk of loading; they do not prove that the file is the one you expect.

This course keeps joblib, because the machine learning course saved the model with it and most scikit-learn examples use it. For a model that your own team made, the order of the gates makes joblib safe enough. The hash check proves the bytes before anything runs, and the digest comes from a reviewed place. If a model ever comes from outside your team, prefer a format like skops, and check the hash too.

Common mistake

Loading a model file to look at it

Symptom: a colleague shares escalation_model_v2.joblib in a chat. Tomás wants to see what it is, so he runs joblib.load("escalation_model_v2.joblib") in a Python session. There is no symptom: the file loads and looks like a normal pipeline. If the file held code, it already ran, with Tomás’s rights, before he saw anything.

Cause: loading a joblib or pickle file runs the instructions in it. “Looking at” a file this way is executing it. The demonstration above showed it with print; a real attacker’s file would read secrets or install something.

Fix: never load a model file that you cannot check. Ask where it came from. Get its hash from a trusted place, such as the release record or the person who trained it, by another channel. Compare before loading. Better: ask for a bundle, and load it only through load_bundle with a pinned digest. To see what a file is without loading it, read metadata.json and contract.json: they are plain text.

For a real AI service: the risk grows with automation. A service that loads whatever file is in a storage folder runs whatever anybody with write access put there. Pin the digest in reviewed settings, so that a new model needs a reviewed change.

Check your understanding

Why must the expected digest come from outside the bundle? What would a check against SHA256SUMS inside the bundle miss?

Show the answer

A person who changes model.joblib can also compute its new hash and rewrite SHA256SUMS. A check against the bundle’s own SHA256SUMS then passes, because the bundle agrees with itself (try 3 of surprise.py). The digest pinned in MODEL_SHA256, in reviewed settings, does not change with the bundle, so gate 1 refuses the rewritten bundle before anything is loaded.

Check your understanding

Tomás suggests moving joblib.load to the start of load_bundle, then checking the hashes: the result is the same, he says. Is it?

Show the answer

No. The service would refuse a bad bundle in both orders, but loading first runs any code in the file before the check. The harm is done by then. The hashes and versions need only bytes and plain text, so they can safely come first; joblib.load comes only after the bytes are proven.

Check your understanding

One byte of model.joblib changed on a disk. Which gate stops the bundle, and what message does the service log?

Show the answer

Gate 2, the file hashes. SHA256SUMS is unchanged, so gate 1 passes, but the hash of model.joblib no longer matches its line. The message is bundles/escalation-1.0.0/model.joblib: the file changed after the bundle was made, and the service does not start.

Module practice

Guided practice (done in this module). You wrote the contract and its 12 tests. You made the bundle escalation-1.0.0 with its four files and SHA256SUMS. You loaded it through the six gates, with 18 tests. Check that your project is complete: python -m pytest reports 18 passed.

Independent task: create a model bundle and validate its input schema. Write check_ticket.py in the project folder. It reads one ticket from a JSON file. It checks the ticket against the contract first. If the ticket does not match, it prints the names of the problem fields and exits with code 1, without loading the bundle. If the ticket matches, it loads the bundle with load_bundle and the digest from the environment variable MODEL_SHA256. Then it prints the score, the flag, the model version and any warnings.

Test it with three files in a folder practice/:

• good.json: the example ticket of contract.py.

• social.json: the same ticket with "channel": "social".

• from_csv.json: a July ticket as it is in the CSV file, with its extra columns and without prior_escalations_90d:

  practice/from_csv.json

  {"ticket_id": "T-119233", "customer_id": "C-00029", "created_at": "2026-07-01 07:15:55",
   "channel": "chat", "team": "returns", "priority": 3, "segment": "trade", "region": "unknown",
   "order_value": 54.38, "word_count": 77, "customer_tenure_days": 748, "prior_tickets_90d": 2,
   "created_hour": 7}

Set the digest of your bundle first:

macOS / Linux

export MODEL_SHA256=$(shasum -a 256 bundles/escalation-1.0.0/SHA256SUMS | cut -c1-64)

Windows (PowerShell)

$env:MODEL_SHA256 = (Get-FileHash bundles\escalation-1.0.0\SHA256SUMS).Hash.ToLower()

Here you compute the digest from your own bundle, which you just made and trust. In a deployment, the value comes from the release record instead.

Show a model answer

check_ticket.py

"""Score one ticket from a JSON file, or reject it before the model runs.

    python check_ticket.py practice/good.json
"""

import json
import os
import sys
from pathlib import Path

from pydantic import ValidationError

from escalation.bundle import load_bundle
from escalation.contract import TicketIn, to_frame, unknown_categories

body = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
try:
    ticket = TicketIn(**body)  # the contract first: nothing is loaded or scored yet
except ValidationError as error:
    fields = sorted({str(e["loc"][0]) for e in error.errors()})
    sys.exit(f"Rejected before inference: {', '.join(fields)}")

bundle = load_bundle("bundles/escalation-1.0.0", os.environ["MODEL_SHA256"])
score = bundle.pipeline.predict_proba(to_frame([ticket]))[0, 1]
print(f"{ticket.ticket_id}: score {score:.4f}, flag {bool(score >= bundle.threshold)}, model {bundle.version}")
for warning in unknown_categories(ticket, bundle.known):
    print("Warning:", warning)

The outputs of python check_ticket.py practice/good.json, practice/social.json and practice/from_csv.json:

T-120001: score 0.3507, flag True, model 1.0.0

T-120001: score 0.4215, flag True, model 1.0.0
Warning: channel: 'social' is a value the model never saw

Rejected before inference: created_at, customer_id, prior_escalations_90d

The last one exits with code 1 (echo $? on macOS and Linux, $LASTEXITCODE in PowerShell). Two problems are extra fields from the CSV file; one is a feature that the machine learning course added later and that this file does not have. The bundle was never loaded for it. With a wrong MODEL_SHA256, the good ticket stops at gate 1 with BundleError.

Module check

Check your understanding

Name the files of the bundle escalation-1.0.0, and the job of each in one line.

Show the answer

model.joblib: the fitted pipeline, preprocessing and model. contract.json: the features in order, their rules, the known categories, the missing-value rules and the output with the threshold. metadata.json: the version, the training date and data hashes, the validation results and the tested package versions. parity_cases.csv: 14 fixed tickets with their expected scores. SHA256SUMS: the hash of each of the four files; its own hash is the bundle digest.

Check your understanding

Predict: Priya trains a new model on newer data, and copies its file over model.joblib in bundles/escalation-1.0.0. The service starts with the old MODEL_SHA256. What happens, and what should she have done?

Show the answer

The service does not start. SHA256SUMS is unchanged, so gate 1 passes, but gate 2 finds that model.joblib changed after the bundle was made, and load_bundle raises BundleError before loading anything. She should have made a new version with make_bundle.py, for example --version 1.1.0. It gets a new folder, its own threshold, parity cases and digest. Then she releases that digest through the reviewed settings.

Check your understanding

Completion check: show that an incompatible feature payload is rejected before inference. What evidence proves that the model never ran?

Show the answer

Run python check_ticket.py practice/from_csv.json: it prints Rejected before inference: created_at, customer_id, prior_escalations_90d and exits with code 1. The evidence that the model never ran: check_ticket.py validates with TicketIn before it calls load_bundle. For a rejected ticket, it never loads the bundle and prints no score. In the service, the same contract returns 422 invalid_ticket with the fields list, as the explorer in lesson 1 showed.

Recap

• A hash is a fingerprint of bytes: one changed byte changed the score from 0.3507 to 0.2007, and the hash from a4acd259… to d7940912….
• The bundle digest is the hash of SHA256SUMS. It names the whole bundle in 64 characters.
• Provenance is who made an artifact, from what, and how you got it. The expected digest comes from a trusted, reviewed place outside the bundle.
• Loading a joblib file runs its code. load_bundle checks the digest, the file hashes and the versions before joblib.load, then the features and the 14 parity cases after it.

Module 1 in one paragraph. You wrote the inference contract, so that a wrong ticket is refused with the field’s name before the model runs. You bundled the whole pipeline with its contract, metadata, tested versions and parity cases, in a folder whose version never changes. And you load it only through six gates, with a digest from outside the bundle. Your project has escalation/contract.py, escalation/bundle.py, make_bundle.py, bundles/escalation-1.0.0/ and 18 passing tests.

Reset to a known state. If your project is broken, copy the snapshot end-of-m01 over it (Use a snapshot), then run python -m pytest: 18 passed. The snapshot has the course’s bundle, so its digest is 8f233029…. Module 1 made no containers or images, so there is nothing to remove in Docker.

Next, you use the bundle for the first job: scoring a whole file of tickets every night.

Next: Batch inference