Package artifacts

What you will be able to do

Turn the model in your notebook into files that another person can load and run. Then prove that the loaded model gives exactly the same scores.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook writes train.py and predict.py with %%writefile, runs them as commands, and runs the fresh-process check. It also shows every failure on this page. It needs no local setup. In this module the local project is the main route, because a handoff is files and commands. Use the notebook if you do not have the project, or to check your results.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• What a pipeline is, and what fit learns from the train set, from Preprocess with a pipeline.
• The final model: logistic regression on 11 features, valid average precision 0.391, from Select useful features: Change ticket_model.py.
• The threshold rule (the lowest threshold that flags at most 20% of valid: 0.19), from Thresholds and calibration.
• Exact version pins and a fresh environment, from Isolate dependencies: Record the versions in Python for Practical AI Engineering. Exit codes and argparse, from Command-line interface in the Python course. Checksums (SHA-256), from Deliver a reusable dataset in SQL and Data Preparation for AI.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv, active in a terminal. You made both in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows).
• The data in data/, from python get_data.py in the same lesson. The last line must be Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157). If it is not, run python get_data.py --reset.
• ticket_model.py with 7 entries in NUMBERS. You wrote it in Preprocess with a pipeline: Write ticket_model.py and added prior_escalations_90d in Select useful features: Change ticket_model.py. If yours is different or broken, download the course’s file: curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py (curl.exe on Windows).
• requirements.txt, from Define a baseline: Install the packages. You change it in Pin every version.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1, numpy 2.5.3 and joblib 1.6.0. The Windows and Linux commands are not tested. The notebook was tested in Jupyter.

A recipe without the oven temperature

You bake a cake, and your friends like it. One friend asks for the recipe, so you write it down: the flour, the sugar, the eggs, the butter. Your friend follows the note exactly, and the cake comes out flat and dry. What went wrong?

Nothing in the note was wrong, but it was not complete. You forgot the oven temperature, because your oven is always at the right setting. “One cup of sugar” meant your own cup, which is bigger than most. These facts were in your kitchen and in your head, so they did not travel with the note. Your friend had no way to know.

A model has the same problem. In this module, Larkfield’s escalation model leaves Priya’s notebook and goes to Grace’s team. They will use it every day, on another computer, long after the training. Everything the model needs must travel with it, and the receiver must be able to check that the result is the same as Priya’s.

The problem: the model lives in a notebook

Until now, the model lived in the memory of a notebook. When the notebook closes, the fitted pipeline is gone. To use it again, you must run every cell again, in the right order, with the same data files on the same computer.

Omar asks a fair question: “Why not send Grace’s team the notebook?” There are three reasons. First, the receiver must train the model again, and a small difference in the data or the packages gives a different model. Second, a notebook is not a command: a program that runs every morning cannot click through cells. Third, nobody can check that the receiver’s model is the same as Priya’s, because nothing records what Priya’s model gave.

So Priya saves the model in files. A file that training makes, and that another program uses later, is an artifact. (In everyday English, an artifact is an object that people made. Here, the training process made it.) The rest of this lesson answers two questions: what must the artifact hold, and how does the receiver prove that it works?

What must travel with the model

Start with the smallest possible artifact. In Fit transformations correctly in the SQL course, a scaler learned the mean and the standard deviation (SD) of five order values: 20, 40, 50, 60 and 100. Then it changed the value 400 to (400 − 54) ÷ 26.53 = 13.04. Save that fitted scaler, load it again, and use it:

tiny_artifact.py

import joblib
import pandas as pd
from sklearn.preprocessing import StandardScaler

train = pd.DataFrame({"order_value": [20.0, 40.0, 50.0, 60.0, 100.0]})
scaler = StandardScaler().fit(train)
joblib.dump(scaler, "tiny_scaler.joblib")

again = joblib.load("tiny_scaler.joblib")
print("mean and SD:", again.mean_, again.scale_.round(2))
print("400 ->", again.transform(pd.DataFrame({"order_value": [400.0]})).round(2).ravel())

The output:

Output

mean and SD: [54.] [26.53]
400 -> [13.04]

joblib.dump writes the fitted object to a file. joblib.load reads it back, with every number that it learned. The loaded scaler gives 13.04, the same as before. joblib is a small package that scikit-learn uses to save and load its objects.

Tomás has an idea to save space: “The model is the coefficients. Save only the model, and prepare the data again when we score.” This is the forgotten oven temperature. Suppose the scoring program builds a new scaler on each batch of new tickets. Today’s batch has two orders: 400 and 380.

tiny_artifact.py (continued)

batch = pd.DataFrame({"order_value": [400.0, 380.0]})
rebuilt = StandardScaler().fit(batch)
print("mean and SD:", rebuilt.mean_, rebuilt.scale_)
print("400 ->", rebuilt.transform(batch.head(1)).round(2).ravel())

Output

mean and SD: [390.] [10.]
400 -> [1.]

Check it by hand. The mean of 400 and 380 is 390, and both values are 10 from it, so the SD is 10. Then (400 − 390) ÷ 10 = 1.0. The same order gives 13.04 with the trained scaler and 1.0 with the rebuilt one. The model learned what 13.04 means, not what 1.0 means. No error appears: the score is quietly wrong.

The rule: save everything that fit learned, in one object. In this course that object is the pipeline. It holds the medians, the means and SDs, the list of categories, and the coefficients of the model. This is why Preprocess with a pipeline put every step into one object: now it is also one file.

The handoff contract

The pipeline file is necessary, but it is not enough. Go back to the cake. Your friend also needs the oven temperature, the size of the cup, and a photo of the finished cake, to compare with her own. Priya’s artifact is a folder, model/, with three files that do these three jobs.

The first file, escalation_model.joblib, is the fitted pipeline: the preprocessing and the model together. With it, the receiver scores new tickets exactly as in training. This is the cake mix with every learned amount already in it.

The second file, model_info.json, describes the model in plain text. Its most important part is the feature schema. A schema is a description of the data that a program expects: the column names, their types and the allowed values. The file also holds the threshold 0.19 and the versions of the packages. It has the checksums of the data and of the model file, and the validation results. These are the oven temperature and the size of the cup: the facts that the receiver cannot guess.

The third file, check_sample.csv, holds 20 validation tickets with the scores that the model gave them at training time. This is the photo of the cake. The receiver scores the same 20 tickets and compares. If the scores are the same, the model arrived complete.

File	What it holds	What the receiver uses it for
escalation_model.joblib	The fitted pipeline: preprocessing and model	Scoring new tickets exactly as in training
model_info.json	The feature schema, the threshold, the versions, the checksums, the validation results	Checking new input, turning scores into flags, knowing what made the model
check_sample.csv	20 validation tickets with their saved scores	Proving that a loaded model gives the same scores

Two scripts use these files. train.py makes the three files, once. predict.py reads them every time it scores a batch of new tickets. Together, the files and the scripts are a contract between training and scoring: each side knows exactly what the other provides.

The training artifact and the scoring contract. train.py writes three files once; predict.py reads them for every batch. Input that breaks the schema stops the run before anything is scored.

Text description of the diagram

python train.py fits on train and chooses the threshold on valid. It writes the folder model/, the artifact, with three files: escalation_model.joblib (the fitted pipeline), model_info.json (feature schema, threshold, versions, checksums) and check_sample.csv (20 tickets and their saved scores). predict.py reads the artifact and a CSV file of new tickets. First, the input checks compare the tickets with the schema. A missing column, a wrong type or a value out of range stops the run with exit code 1, and nothing is scored. An unknown category is allowed, but it is counted and reported. Then the pipeline gives one score per ticket, and a ticket with a score of at least 0.19 is flagged. The result is predictions.csv. Separately, predict.py --check reloads the artifact in a fresh process and compares the 20 saved scores.

This lesson builds train.py and the --check part of predict.py. The input checks come in Plan production checks, the last lesson of this module.

Worked example: train.py

Priya writes one script that does the whole training, from the data files to the artifact. Then anyone can repeat the training with one command, and get the same files. Create train.py in your project folder, next to ticket_model.py. The file has four parts; type them one after the other into the same file.

The first part has the imports, the review capacity and two small functions. choose_threshold() is the rule from Thresholds and calibration: the lowest threshold that fits the capacity. sha256() computes the checksum of a file, as in the SQL course.

train.py (part 1 of 4)

"""Train the escalation model, choose its threshold, and save the artifact.

Nextia Learning, C05, Module 8, lesson 1 (Package artifacts).

    python train.py

Fits the pipeline on train only, chooses the threshold on valid for the
review capacity, and writes into model/:

    escalation_model.joblib   the fitted preprocessing-plus-model pipeline
    model_info.json           feature schema, threshold, versions, checksums, valid results
    check_sample.csv          20 valid tickets and their scores, to check a reload

It never reads the test set: evaluate.py does that, once.
"""

import hashlib
import json
import platform
from datetime import datetime, timezone
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.metrics import average_precision_score, roc_auc_score

from ticket_model import CATEGORIES, DATA, FEATURES, NUMBERS, TARGET, build_pipeline, load

OUT = Path("model")
REVIEW_CAPACITY = 0.20  # the senior team can review at most 1 in 5 new tickets


def choose_threshold(scores, capacity):
    """The lowest threshold (in steps of 0.01) that flags no more than
    `capacity` of the tickets. Lower thresholds flag more tickets and catch
    more escalations, so the lowest one that fits catches the most."""
    for step in range(1, 100):
        threshold = step / 100
        if (scores >= threshold).mean() <= capacity:
            return threshold
    return 1.0


def sha256(path):
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()

The second part fits the pipeline on train, scores valid, chooses the threshold and computes the validation results. You computed every one of these numbers in Module 6. The difference is that a script now computes them, the same way each time.

train.py (part 2 of 4)

def main():
    train, valid = load("train"), load("valid")
    pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])

    scores = pipeline.predict_proba(valid[FEATURES])[:, 1]
    threshold = choose_threshold(scores, REVIEW_CAPACITY)
    flagged = scores >= threshold
    caught = int((flagged & (valid[TARGET] == 1)).sum())
    results = {
        "rows": len(valid), "escalated": int(valid[TARGET].sum()),
        "roc_auc": round(roc_auc_score(valid[TARGET], scores), 3),
        "average_precision": round(average_precision_score(valid[TARGET], scores), 3),
        "flagged": int(flagged.sum()), "flag_rate": round(flagged.mean(), 3),
        "precision": round(caught / flagged.sum(), 3),
        "recall": round(caught / valid[TARGET].sum(), 3),
    }

The third part saves the artifact. joblib.dump saves the whole pipeline, not only pipeline.named_steps["model"]. Then the script keeps the first 20 validation tickets, with their columns and their scores, in check_sample.csv.

train.py (part 3 of 4)

    OUT.mkdir(exist_ok=True)
    joblib.dump(pipeline, OUT / "escalation_model.joblib")
    sample = valid[["ticket_id", "created_at"] + FEATURES].head(20).copy()
    sample["score"] = scores[:20]
    sample.to_csv(OUT / "check_sample.csv", index=False)

The fourth part writes model_info.json and prints a summary. The schema takes the categories from the fitted encoder, so it lists exactly the categories that the model knows. A list typed by hand could differ from what the model learned.

train.py (part 4 of 4)

    categories = pipeline.named_steps["prepare"].named_transformers_["categories"].categories_
    info = {
        "model": "Larkfield ticket escalation: logistic regression in a scikit-learn pipeline",
        "trained_at": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC"),
        "target": f"{TARGET}: 1 if the ticket is escalated within 72 hours of creation",
        "prediction_time": "when the ticket is created",
        "features": {
            "categories": {name: [str(v) for v in values] for name, values in zip(CATEGORIES, categories)},
            "numbers": NUMBERS,
        },
        "threshold": threshold,
        "review_capacity": REVIEW_CAPACITY,
        "valid": results,
        "train_rows": len(train),
        "data_sha256": {name: sha256(DATA / name) for name in ("train.csv", "valid.csv", "extra_features.csv")},
        "artifact_sha256": sha256(OUT / "escalation_model.joblib"),
        "versions": {"python": platform.python_version(), "pandas": pd.__version__,
                     "scikit-learn": sklearn.__version__, "numpy": np.__version__, "joblib": joblib.__version__},
    }
    (OUT / "model_info.json").write_text(json.dumps(info, indent=2) + "\n", encoding="utf-8")

    print(f"Trained on {len(train)} tickets. Threshold {threshold} for a review capacity of {REVIEW_CAPACITY:.0%}.")
    print("Valid: ROC AUC {roc_auc}, average precision {average_precision}, flagged {flagged} of {rows} "
          "({flag_rate:.1%}), precision {precision}, recall {recall}.".format(**results))
    print(f"Saved {OUT / 'escalation_model.joblib'}, model_info.json and check_sample.csv.")


if __name__ == "__main__":
    main()

Most of the file is bookkeeping. Six lines carry the ideas of this lesson, and the table lists them with the reason for each:

Lines	Why
fit(train[FEATURES], train[TARGET])	The only fit. It gets the train rows only.
choose_threshold(scores, REVIEW_CAPACITY)	The threshold is chosen on valid and saved, so predict.py never uses the default 0.5.
joblib.dump(pipeline, ...)	The whole pipeline: preprocessing and model together.
"features"	The schema. predict.py checks new input against it in Plan production checks.
"data_sha256", "artifact_sha256"	Which exact data trained this model, and which exact model file this is.
"versions"	The versions that made the file. A different version can read it differently.

Run it:

Terminal

python train.py

The output:

Output

Trained on 16939 tickets. Threshold 0.19 for a review capacity of 20%.
Valid: ROC AUC 0.752, average precision 0.391, flagged 227 of 1179 (19.3%), precision 0.379, recall 0.47.
Saved model/escalation_model.joblib, model_info.json and check_sample.csv.

These are the numbers of Module 6: ROC AUC 0.752, average precision 0.391, 227 tickets flagged (19.3%), precision 0.379 and recall 0.470 (the script prints 0.47). The script takes about one second.

Open model/model_info.json. The middle part is the schema:

model/model_info.json (part)

  "features": {
    "categories": {
      "channel": ["chat", "email", "phone", "unknown", "web_form"],
      "team": ["account", "delivery", "payment", "returns", "warranty"],
      "segment": ["business", "guest", "home", "trade"],
      "region": ["east", "north", "south", "unknown", "west"]
    },
    "numbers": ["priority", "order_value", "word_count", "customer_tenure_days",
                "prior_tickets_90d", "created_hour", "prior_escalations_90d"]
  },
  "threshold": 0.19,

(The real file has one value per line. It is shortened here.) At the end of the file you see the checksums and the versions:

model/model_info.json (end)

  "artifact_sha256": "a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736",
  "versions": {
    "python": "3.14.6",
    "pandas": "3.0.6",
    "scikit-learn": "1.9.1",
    "numpy": "2.5.3",
    "joblib": "1.6.0"
  }

Your trained_at time is different. On macOS with the same versions, the artifact_sha256 was the same in two runs: training is deterministic here. The same data and code give the same file. On another system, the checksum can differ, even when the scores are the same.

Pin every version: requirements.txt

Back to the recipe: “one cup” is only clear when everybody uses the same cup. Packages are the cups of a model. The artifact is a picture of Python objects, made by one version of each package. A different version can read the same file in a different way. So the receiver must install the same versions.

In Define a baseline, you pinned pandas, scikit-learn and matplotlib. A pin is an exact version, written with ==. The artifact also depends on two packages that scikit-learn installed for you: numpy (the arrays inside the pipeline) and joblib (the file format). You never installed them by name, so nothing pins them yet. Check their versions:

Terminal

python -c "import numpy, joblib; print(numpy.__version__, joblib.__version__)"

Output

2.5.3 1.6.0

Change requirements.txt so that it has these five lines, each with an exact version:

requirements.txt

pandas==3.0.6
scikit-learn==1.9.1
numpy==2.5.3
joblib==1.6.0
matplotlib==3.11.2

Remove the ipykernel line if your file has one. It lets your editor run notebooks, but train.py and predict.py do not need it. It stays installed in your .venv. In a new environment, install it by hand if you want notebooks there (python -m pip install ipykernel). This file is the same as the course’s C05/project/requirements.txt.

A fresh process: predict.py –check

How does the receiver know that the files are enough? A test in the notebook where you trained proves little. That process still has the pipeline in memory, the imports done and your variables set. It is like tasting the cake in your own kitchen: of course it works there. The real test is your friend’s kitchen, where nothing of yours is present.

A fresh process is a new run of Python that starts with nothing in memory: a new terminal, or a new Colab runtime. If the loaded model gives the same scores there, the files alone are enough.

Create predict.py next to train.py. This is its first version. It loads the artifact, and it has two jobs: --check compares the 20 saved scores, and an input file gets scored. In Plan production checks you add the input checks.

predict.py (first version)

"""Score new tickets with the saved escalation model.

Nextia Learning, C05, Module 8 (Package artifacts; Plan production checks).

    python predict.py data/july_tickets.csv predictions.csv
    python predict.py --check     reload the model and compare the fixed sample
"""

import argparse
import json
import sys
from pathlib import Path

import joblib
import numpy as np
import pandas as pd

from ticket_model import FEATURES, read

MODEL = Path("model")


def load_model():
    info = json.loads((MODEL / "model_info.json").read_text(encoding="utf-8"))
    return joblib.load(MODEL / "escalation_model.joblib"), info


def check_sample(pipeline):
    sample = read(MODEL / "check_sample.csv")
    scores = pipeline.predict_proba(sample[FEATURES])[:, 1]
    largest = float(np.abs(scores - sample["score"]).max())
    if largest > 1e-9:
        sys.exit(f"Check failed: the reloaded model's scores differ from the saved ones by up to {largest:.2e}.")
    print(f"Check passed: {len(sample)} scores are identical after reloading (largest difference {largest:.1e}).")


def main():
    parser = argparse.ArgumentParser(description="Score new tickets with the saved escalation model.")
    parser.add_argument("input", nargs="?", help="CSV file with one row per new ticket")
    parser.add_argument("output", nargs="?", default="predictions.csv")
    parser.add_argument("--check", action="store_true", help="compare the fixed sample after reloading")
    args = parser.parse_args()
    pipeline, info = load_model()
    if args.check:
        check_sample(pipeline)
        return
    if not args.input:
        parser.error("give an input CSV file, or --check")

    df = read(args.input)
    scores = pipeline.predict_proba(df[FEATURES])[:, 1]
    out = pd.DataFrame({"ticket_id": df["ticket_id"], "score": scores.round(4),
                        "flag": (scores >= info["threshold"]).astype(int)})
    out.to_csv(args.output, index=False)
    print(f"Scored {len(out)} tickets with threshold {info['threshold']}: {out['flag'].sum()} flagged "
          f"({out['flag'].mean():.1%}). Wrote {args.output}.")


if __name__ == "__main__":
    main()

check_sample() reads the 20 tickets with read() from ticket_model.py, the same function that reads the training data. So the columns get the same types. It scores them with the loaded pipeline and finds the largest difference from the saved scores. A difference above 1e-9 (0.000000001) stops the program with exit code 1, so a script or a colleague sees the failure.

Open a new terminal. Go to the project folder and activate .venv. Then run the check:

New terminal

python predict.py --check

The output:

Output

Check passed: 20 scores are identical after reloading (largest difference 1.1e-16).

Why is the difference not exactly 0? The saved scores went through a CSV file as text. Reading the text back can change the last of about 16 digits. A difference of 1.1e-16 is far below anything that matters, and far below the limit of 1e-9. A different preprocessing gives differences of about 0.01, as the failure case below shows. So the limit separates “the same model” from “another model” with a wide margin.

Load only files you trust

A joblib file is not like a photo or a table of numbers. It is closer to a list of instructions: when you load it, Python follows them to rebuild the objects. Usually the instructions only rebuild a pipeline. But a file can hold any instruction.

A joblib file can run code

joblib.load (and Python’s pickle, which joblib uses) runs instructions that are stored in the file. A changed or malicious file can run any code on your computer, with your permissions, the moment you load it. Load only artifacts that you made or got from a source you trust, and never from an email attachment or an unknown download.

The notebook shows this with a harmless file. A small class tells pickle to call print when the object is loaded:

surprise.py

import joblib

class Surprise:
    def __reduce__(self):
        return (print, ("This line ran while the file was loading.",))

joblib.dump(Surprise(), "surprise.joblib")
joblib.load("surprise.joblib")

Output

This line ran while the file was loading.

Here the instruction only prints. A real attack could delete files or send your data away. The artifact_sha256 in model_info.json tells you if the model file changed by accident after training. It does not protect you from a person who changes both files. Protecting artifacts in a real service (access rights, signed files, a model registry) is a topic for Deploying AI Models for Real Users.

When versions differ

model_info.json records the versions that made the file. What happens if the receiver has other versions? Suppose a colleague has scikit-learn 1.9.1, but the file was made with 1.8.0. The notebook shows this without an old installation. It changes the version text inside a copy of the file, then loads the copy:

old_version.py

import warnings
from pathlib import Path

import joblib

raw = Path("model/escalation_model.joblib").read_bytes()
Path("old_version.joblib").write_bytes(raw.replace(b"1.9.1", b"1.8.0"))
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    joblib.load("old_version.joblib")
print(len(caught), "warnings. The first one:")
print(type(caught[0].message).__name__ + ":", caught[0].message)

Output

12 warnings. The first one:
InconsistentVersionWarning: Trying to unpickle estimator OneHotEncoder from version 1.8.0 when using version 1.9.1. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

One warning comes for each estimator inside the pipeline: the encoder, the imputers, the scaler, the model and the pipelines that hold them. The file still loads, and it may even give the right scores. But scikit-learn does not promise it. An estimator’s inner data can change between versions. Then the scores can be wrong without an error.

Fix: do not ignore the warning. Install the versions of requirements.txt (python -m pip install -r requirements.txt), or train again with the new versions and make a new artifact. Then run python predict.py --check again.

Guided practice: build and check the artifact

1. In a terminal in ~/projects/ticket-model, activate .venv.
2. Write the four parts of train.py into one file. Save it.
3. Run python train.py. You should see Threshold 0.19 and flagged 227 of 1179 (19.3%).
4. List the folder model/ (ls model, or dir model on Windows). You should see three files.
5. Open model/model_info.json. Find the five channels and the threshold 0.19.
6. Update requirements.txt to the five pinned lines.
7. Write predict.py (first version). Save it.
8. Open a new terminal, go to the project and activate .venv.
9. Run python predict.py --check. You should see Check passed: 20 scores are identical after reloading.
10. Run python predict.py data/july_tickets.csv predictions.csv. You should see Scored 1400 tickets with threshold 0.19: 282 flagged (20.1%). Wrote predictions.csv.

Step 10 scores the July tickets, which arrive after the training. It works, and it says nothing more. In Plan production checks, you see what it does not tell you.

If a step fails

• ModuleNotFoundError: No module named 'ticket_model': you are not in the project folder. Run cd ~/projects/ticket-model.
• FileNotFoundError: ... model/model_info.json: run python train.py first.
• KeyError: "['prior_escalations_90d'] not in index": your ticket_model.py is the Module 4 version, without the feature of Module 6. Download the course’s file (see Before you start).
• Check failed: ...: the model file and check_sample.csv come from different runs. Run python train.py again; it writes both.

Your turn: a smaller review team

Grace says that in August the senior team is smaller. They can review only 15% of new tickets. Change one line in train.py: REVIEW_CAPACITY = 0.15. Before you run it, predict:

1. Will the threshold go up or down?
2. Will precision go up or down? And recall?
3. Will predict.py --check still pass?

Then run python train.py and python predict.py --check. At the end, set REVIEW_CAPACITY = 0.20 again and run python train.py, so that the artifact is the one the course uses.

Show the answer

Output

Trained on 16939 tickets. Threshold 0.22 for a review capacity of 15%.
Valid: ROC AUC 0.752, average precision 0.391, flagged 174 of 1179 (14.8%), precision 0.431, recall 0.41.
Saved model/escalation_model.joblib, model_info.json and check_sample.csv.

Output

Check passed: 20 scores are identical after reloading (largest difference 1.1e-16).

1. The threshold goes up, from 0.19 to 0.22. Fewer tickets may be flagged, so a ticket needs a higher score.
2. Precision goes up (0.379 to 0.431): the flagged tickets are the ones with the highest scores. Recall goes down (0.470 to 0.410): fewer escalations are caught. This is the trade-off of The precision–recall trade-off. Recall 0.41 still meets the target of 40%, but only just.
3. Yes. The pipeline and the scores did not change; only the threshold changed. ROC AUC and average precision are the same, because they do not depend on the threshold. The threshold is part of the artifact (in model_info.json), so a capacity change needs a new artifact, even with the same model.

Common mistake

Save only the classifier

Symptom: Tomás still wants a smaller file. He saves only the model, joblib.dump(pipeline.named_steps["model"], ...), and plans to prepare the data again in the scoring program. The first try gives the raw ticket columns to the loaded model:

Output

ValueError: could not convert string to float: 'chat'

A warning comes before it: UserWarning: X has feature names, but LogisticRegression was fitted without feature names. The model alone never saw ticket columns, only the numbers that the preprocessing made.

So he builds the preprocessing again with build_pipeline() and fits it on the new tickets. On the 20 check tickets:

Output

ValueError: X has 22 features, but LogisticRegression is expecting 28 features as input.

On the 1,400 July tickets:

Output

ValueError: X has 29 features, but LogisticRegression is expecting 28 features as input.

On the 1,179 validation tickets, the same code runs with no error. But it flags 234 tickets instead of 227. Seven tickets change their flag, and scores differ by up to 0.025.

Cause: the model learned 28 numbers per ticket: 19 one-hot columns for the categories, 7 scaled numbers and 2 missing-value indicators. Only the fitted preprocessing knows how to make exactly those 28. Twenty tickets do not have every category, so the new encoder makes only 22 columns. July has a sixth channel, social, so it makes 29. In May, every category happens to be present. So the code runs, but the medians, means and SDs now come from May, not from train: the 1.0-instead-of-13.04 mistake of the tiny example.

Fix: save the whole pipeline with joblib.dump(pipeline, ...), as train.py does. Never fit anything in the scoring program. Then run python predict.py --check in a fresh process.

Consequence in AI work: the error cases are the lucky ones. The quiet case gives a different model every day, because each batch of tickets changes the preprocessing. The flag rate moves with the batch, and nobody sees an error.

Knowledge check

Check your understanding

Name the three files in model/ and say, for each, what the next person would lose without it.

Show the answer

• escalation_model.joblib, the fitted pipeline. Without it, nobody can score tickets the way the model was trained. Rebuilding the preprocessing gives other columns or other scaling.
• model_info.json: the feature schema, the threshold, the versions and the checksums. Without it, nobody knows which columns and categories to expect, that the threshold is 0.19 and not 0.5, or which versions and data made the model.
• check_sample.csv: 20 tickets with their saved scores. Without it, nobody can prove that the loaded model gives the same scores as at training time.

Check your understanding

Omar runs predict.py --check on a new laptop and sees InconsistentVersionWarning, then Check passed. He says the warning does not matter, because the check passed. Is he right? Explain.

Show the answer

Not completely. The warning says that the file was made with another scikit-learn version than the one installed. The check passed on 20 tickets, which is good evidence for those tickets, but scikit-learn does not promise that the rest behaves the same. Another version can change how an estimator reads its stored data. The safe fix is to install the versions of requirements.txt (the ones in model_info.json), or to train again with the new versions and make a new artifact. Then run the check again, without the warning.

Check your understanding

Why must the identical-scores check run in a fresh process, and not in the notebook where you trained the model?

Show the answer

The notebook still has the fitted pipeline, the imports and your variables in memory. A check there can use those by mistake and pass, even if a file is missing or wrong. A fresh process starts with nothing, as the person who receives the files does. If the scores are identical there, the files alone are enough to reproduce the model.

Recap

You can now hand over a model as files, not as a notebook:

• python train.py fits on train, chooses the threshold 0.19 on valid, and writes model/: the whole pipeline (escalation_model.joblib), its schema, threshold, versions and checksums (model_info.json), and 20 tickets with their scores (check_sample.csv).
• requirements.txt pins every package that the artifact needs, numpy and joblib too.
• python predict.py --check in a fresh process proves that the loaded model gives the same scores (largest difference 1.1e-16).
• Load only joblib files that you trust, and treat InconsistentVersionWarning as a problem to fix.

The files say how to run the model. Next, you write down what it is for, and what it must not be used for: the model card.

Next: Document use boundaries