Plan production checks

What you will be able to do

Plan the checks that tell you when the model stops fitting its job. Input checks stop bad data, monitoring signals show change, and a written rule says when to retrain.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook adds the input checks to predict.py and breaks the input in five ways. Then it scores the July tickets, computes every monitoring signal of this page and joins the labels. It ends with the module practice: the fresh-copy run. It needs no local setup, but the local project is the main route in this module.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 75 minutes

What you need to know

• The artifact and the first version of predict.py, from Package artifacts.
• The model card and its limits (calibration, new categories), from Document use boundaries.
• Precision, recall and the flag rate, from Classification metrics. Slices, from Slice analysis.
• handle_unknown="ignore": a category that train never saw gets 0 in every column, from Fit transformations correctly in SQL and Data Preparation for AI.
• Input checks that stop a pipeline before it loads bad data, from Deliver a reusable dataset in the SQL course.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project ~/projects/ticket-model with .venv active, from Define a baseline: Set up your project.
• model/ and train.py, from Package artifacts: Worked example: train.py. If model/ is missing, run python train.py.
• predict.py (first version), from Package artifacts: A fresh process. You change it in this lesson.
• data/july_tickets.csv and data/july_labels.csv, from python get_data.py. If they are missing, run python get_data.py --reset.
• MODEL_CARD.md, from Document use boundaries: Guided practice, and evaluate.py, from Thresholds and calibration: Test once with evaluate.py, for the module practice.
• If a project file is missing or broken, copy the course’s version from C05/project, for example curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/predict.py (curl.exe on Windows).

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows and Linux commands are not tested. The notebook was tested in Jupyter.

Warning lights on a dashboard

A car does not tell you everything all the time. It has a few warning lights, and each one watches one thing. A red oil light means “stop now”: driving on breaks the engine. An amber fuel light means “act soon”: you can still drive, but not for long. And the service book says “service every 15,000 km, or earlier if a light stays on”. The lights do not drive the car. They tell the driver when something is going wrong, before it becomes expensive.

A model in daily use needs the same. From 1 July, Larkfield’s model scores every new ticket. Nobody can read 50 scores a day and see that something is wrong. So Priya plans a small set of checks in advance. Some stop the run at once, some warn, and a written rule says when to service the model.

The problem: three tickets, five inputs

What does the first version of predict.py do when the input is not what the model expects? Each day, an export by Mei, the data engineer, sends the new tickets to predict.py as a CSV file with the same columns as july_tickets.csv. Exports change, colleagues rename columns, and new kinds of ticket appear.

Take the first three July tickets, and change them in four ways. Then run the first version on the five files. Two of the five give an error, with a long traceback that only a Python programmer can read. Two give no message at all: a ticket from a new channel, and order values in cents instead of dollars. The cents mistake turns one flag into three, and nothing warns anybody.

Input file	What changed	The first version of predict.py
tiny_ok.csv	Nothing	Scored 3 tickets with threshold 0.19: 1 flagged (33.3%).
tiny_social.csv	Ticket 3 has channel = social	Scored 3 tickets with threshold 0.19: 1 flagged (33.3%).
tiny_missing.csv	The column prior_escalations_90d is missing	KeyError: "['prior_escalations_90d'] not in index"
tiny_text.csv	Ticket 2 has priority = high	ValueError: Cannot use median strategy with non-numeric data: could not convert string to float: 'high'
tiny_cents.csv	order_value is in cents: 5438 instead of 54.38	Scored 3 tickets with threshold 0.19: 3 flagged (100.0%).

The quiet cases are the dangerous ones. A model scores whatever it gets. It never says “this input looks strange”, so the program around it must say it.

Two kinds of checks

Go back to the dashboard. Some problems are visible in one moment: a broken file, a missing column. Others build up slowly over days. So a program that uses a model needs two kinds of checks.

Input checks are the red lights. They run before scoring, on every batch. They compare the batch with the schema in model_info.json, the description of the expected columns, types and values from Package artifacts. A batch that the model cannot score correctly stops the run, with a message that says what is wrong.

Monitoring is the amber lights. It watches the results over time, and compares numbers like the share of flagged tickets with what you expect. It finds slow changes that no single batch shows. A change in the data over time is called drift. The name comes from a boat with nobody steering. Each minute it moves a little with the current, and after an hour it is far from where it was.

Both kinds of check feed one decision, written in advance: when to stop and look at the model again. That rule is the retraining trigger, the service book of the model. The diagram shows the whole daily loop.

The daily loop of checks. Input checks stop a bad batch before scoring; signals without labels warn on the same day; precision and recall follow when the labels arrive; a written trigger starts a review by people.

Text description of the diagram

New tickets go through the input checks every day. A failed check stops the run with exit code 1, and nothing is scored. A batch that passes, or only has an unknown category, gets scores and flags in predictions.csv. On the same day, four signals need no labels: the flag rate, the mean score, the share of unknown categories and the mean of each number feature. Seventy-two hours later the labels arrive, and precision and recall can be computed. If any trigger condition is true, Priya and Grace review the model. They find the cause, then fix the data, retrain, or update the card. Otherwise, the model stays, and the checks run again the next day.

Input checks in predict.py

Mei and Priya decide what each problem means, and what the program does. The rule they follow: stop when the scores would be wrong, and warn when the scores are still usable but the input has changed. A missing column and a text value in a number column make scoring impossible. A number far outside its usual range is almost always a unit or export error, so the scores would be quietly wrong. A category that the model never saw is different: the pipeline can still score the ticket, with less information.

Problem	Example	Action	Why
A required column is missing	prior_escalations_90d not in the file	Stop	The model cannot score without it.
A number column has text	priority = high	Stop	The value has no meaning for the model.
A number is outside its range	order_value = 5438 for a $54.38 order	Stop	A unit or export error. The scores would be quietly wrong.
An unknown category	channel = social	Warn and count	The pipeline can score it (handle_unknown="ignore"). Stopping would stop the routing of every ticket. But many of them mean that the input has changed.

“Stop” means: score nothing, print what is wrong, and exit with code 1. A scheduler or a colleague sees the failure at once. But why not just score the good rows? Tomás suggests it, and it looks kinder. But then the run looks successful, the bad rows wait in no queue, and nobody notices the problem until a customer complains. A loud stop is the fastest way to a fix.

Change predict.py in four places. First, the docstring now describes the checks:

predict.py (the docstring)

"""Score new tickets with the saved escalation model.

Nextia Learning, C05, Module 8 (Package artifacts; Plan production checks).

    python predict.py data/july_tickets.csv predictions.csv
    python predict.py --check     reload the model and compare the fixed sample

Checks the input before it scores: the required columns, numbers that are
numbers, values in their allowed range, and categories that the model knows.
A missing column or a wrong type stops the run (exit code 1). An unknown
category is allowed (the model treats it as "none of the known ones") but is
counted and reported, because many of them mean that the input has changed.
"""

Second, under MODEL = Path("model"), add the allowed range of each number. The ranges are wide on purpose: they catch unit errors and broken exports, not unusual tickets. A ticket with 3,000 words is unusual but real; an order of $543,800 is an export error.

predict.py (under MODEL)

RANGES = {"priority": (1, 3), "word_count": (0, 10000), "order_value": (0, 5000), "customer_tenure_days": (0, 20000),
          "prior_tickets_90d": (0, 500), "created_hour": (0, 23), "prior_escalations_90d": (0, 500)}

Third, add check_input() after load_model(). It reads the schema from model_info.json, so the checks always match the artifact:

predict.py (after load_model)

def check_input(df, info):
    """Return a list of problems that stop the run, and a list of warnings."""
    errors, warnings = [], []
    missing = [c for c in FEATURES if c not in df.columns]
    if missing:
        return [f"missing columns: {', '.join(missing)}"], warnings
    for name in info["features"]["numbers"]:
        values = pd.to_numeric(df[name], errors="coerce")
        bad = values.isna() & df[name].notna()
        if bad.any():
            errors.append(f"{name}: {bad.sum()} values are not numbers, for example {df.loc[bad, name].iloc[0]!r}")
            continue
        low, high = RANGES[name]
        outside = values.notna() & ((values < low) | (values > high))
        if outside.any():
            errors.append(f"{name}: {outside.sum()} values outside {low} to {high}")
    for name, known in info["features"]["categories"].items():
        unknown = df[name].notna() & ~df[name].isin(known)
        if unknown.any():
            values = ", ".join(sorted(df.loc[unknown, name].astype(str).unique())[:5])
            warnings.append(f"{name}: {unknown.sum()} rows have a category the model never saw ({values})")
    return errors, warnings

Three lines in it need a word of explanation:

Lines	Why
pd.to_numeric(..., errors="coerce")	Text that is not a number becomes missing. A value that was not missing before, and is missing now, was text. Real gaps (an account ticket’s order_value) stay allowed: the pipeline fills them.
RANGES[name]	A value outside its range is an error, not a warning.
~df[name].isin(known)	known is the list of categories in model_info.json, the categories that the encoder learned.

Fourth, in main(), check the input after reading it and before scoring. This is the complete main():

predict.py (main)

def main():
    parser = argparse.ArgumentParser(description="Score new tickets with the saved escalation model.")
    parser.add_argument("input", nargs="?", help="CSV file with one row per new ticket")
    parser.add_argument("output", nargs="?", default="predictions.csv")
    parser.add_argument("--check", action="store_true", help="compare the fixed sample after reloading")
    args = parser.parse_args()
    pipeline, info = load_model()
    if args.check:
        check_sample(pipeline)
        return
    if not args.input:
        parser.error("give an input CSV file, or --check")

    df = read(args.input)
    errors, warnings = check_input(df, info)
    for warning in warnings:
        print(f"Warning: {warning}")
    if errors:
        for error in errors:
            print(f"Error: {error}")
        sys.exit("Stopped: the input does not match the model's feature schema. Nothing was scored.")

    scores = pipeline.predict_proba(df[FEATURES])[:, 1]
    out = pd.DataFrame({"ticket_id": df["ticket_id"], "score": scores.round(4),
                        "flag": (scores >= info["threshold"]).astype(int)})
    out.to_csv(args.output, index=False)
    print(f"Scored {len(out)} tickets with threshold {info['threshold']}: {out['flag'].sum()} flagged "
          f"({out['flag'].mean():.1%}). Wrote {args.output}.")

Your predict.py is now the same as the course’s C05/project/predict.py. Run it on the five tiny files again (the notebook makes them):

Output: tiny_social.csv

Warning: channel: 1 rows have a category the model never saw (social)
Scored 3 tickets with threshold 0.19: 1 flagged (33.3%). Wrote tiny_out.csv.

Output: tiny_missing.csv (exit code 1)

Error: missing columns: prior_escalations_90d
Stopped: the input does not match the model's feature schema. Nothing was scored.

Output: tiny_text.csv (exit code 1)

Error: priority: 1 values are not numbers, for example 'high'
Stopped: the input does not match the model's feature schema. Nothing was scored.

Output: tiny_cents.csv (exit code 1)

Error: order_value: 3 values outside 0 to 5000
Stopped: the input does not match the model's feature schema. Nothing was scored.

tiny_ok.csv gives the same line as before. Every problem now has a message that Mei can act on, and the quiet cases are no longer quiet. These are the gates of the contract from Package artifacts:

The input checks are the gate between new data and the model. A broken batch stops before scoring; an unknown category passes, with a warning and a count.

Text description of the diagram

New tickets reach the pipeline only through the input checks, which use the schema in model_info.json. A missing column, a wrong type or a value out of range stops the run with exit code 1, and nothing is scored. An unknown category passes, with a warning and a count. Then the pipeline scores each ticket, and a score of at least 0.19 sets the flag in predictions.csv.

Worked example: the July tickets

The July export has 1,400 tickets created from 1 to 31 July 2026, after the training data ends. Their targets are not known yet. Priya scores them:

Terminal

python predict.py data/july_tickets.csv predictions.csv

Output

Warning: channel: 91 rows have a category the model never saw (social)
Scored 1400 tickets with threshold 0.19: 282 flagged (20.1%). Wrote predictions.csv.

From 15 July, Larkfield answers messages on social media, and 91 tickets come from the new channel social. The model scores them as “none of the channels I know”. The run continues, and the warning counts them. 20.1% flagged is just above the capacity of 20%. But one number for the month hides when it changed. Monitoring looks at the numbers over time.

Monitoring signals

A monitoring signal is a number that you compute for each period and compare with what you expect. The best signals are ready on the same day, because they need no labels. Priya chooses four of them, and for each she knows the normal value from May, the validation month:

• The flag rate, against the capacity of 20%. This is Grace’s constraint. In May it was 19.3%.
• The mean score. If it moves, the inputs moved, even if the flag rate has not changed yet. In May it was 0.120.
• The share of unknown categories. It shows new input that the model never learned. In May it was 0.
• The mean of each number feature, against the validation set. It shows drift in one input. Here the clearest is prior_tickets_90d, with a mean of 9.00 in May.

signals.py

import json

import pandas as pd

from ticket_model import read

info = json.load(open("model/model_info.json"))
july = read("data/july_tickets.csv").merge(pd.read_csv("predictions.csv"), on="ticket_id")
july["days"] = pd.cut(july["created_at"].dt.day, [0, 7, 14, 21, 28, 31],
                      labels=["1-7", "8-14", "15-21", "22-28", "29-31"])
july["unknown"] = ~july["channel"].isin(info["features"]["categories"]["channel"])
signals = july.groupby("days", observed=True).agg(
    tickets=("flag", "size"), flag_rate=("flag", "mean"), mean_score=("score", "mean"),
    unknown=("unknown", "mean"), prior_tickets=("prior_tickets_90d", "mean"))
print(signals.round(3))

The output:

Output

       tickets  flag_rate  mean_score  unknown  prior_tickets
days
1-7        311      0.174       0.113    0.000          5.322
8-14       314      0.185       0.111    0.000          5.245
15-21      302      0.262       0.138    0.126          3.894
22-28      321      0.209       0.137    0.100          3.592
29-31      152      0.158       0.125    0.138          3.033

Read the table against May. For the first two weeks of July, the flag rate is 17.4% and 18.5%, under the capacity. From 15 July, everything moves at once: 26.2% flagged, a mean score of 0.138, and 12.6% unknown channels. The last row has only three days and 152 tickets, so it is less reliable.

Weeks are coarse. The chart shows the flag rate over the last 7 days, computed again for each day from 7 July:

• Flag rate, last 7 days
• Capacity 20%

The share of tickets flagged in the last 7 days rises above the senior team's capacity from 15 July, when the social channel starts, and stays above it until 30 July.

Text description of the chart

One line for each day from 7 to 31 July: the share of the tickets of the last 7 days that the model flagged. From 7 to 14 July it is between 15.7% and 18.9%, under the dashed capacity line at 20%. On 15 July it reaches 21.0%. It rises to 26.2% on 21 July, then stays between 20.4% and 22.4% until 30 July. On 31 July it is 18.6%.

What changed? Priya looks at the slices, as in Slice analysis:

• The 91 social tickets are flagged at 23.1%. Other tickets from 15 July are flagged at 21.8%. So the new channel is not the only cause.
• The mean of prior_tickets_90d falls from 5.3 to 3.0 during the month. In May it was 9.00. Mei checks the export. She finds that the July export counts only the customer’s tickets before 1 July. A ticket on 25 July does not count the same customer’s tickets of early July. So the feature is computed differently from training, and it gets smaller each day.

Why does a smaller prior_tickets_90d raise the scores? In Explain predictions you saw its coefficient: −0.29, so fewer earlier tickets mean a higher score. Priya can read this in one line, because the model is a logistic regression. This is one of the reasons in When a simple model is better: a simple model is easier to debug when its inputs change.

This is the value of the signals: they show that something changed, and where to look. The cause needs a person who knows the data.

When the labels arrive

The target escalated_72h of a July ticket is known 72 hours after the ticket is created. Labels that arrive later than the predictions are normal in production. Mei sends them in july_labels.csv. Now Priya can compute the numbers that matter most for Grace’s decision. Recall: precision is the share of flagged tickets that escalated, and recall is the share of escalations that were flagged.

labels.py

import pandas as pd

from ticket_model import read

july = read("data/july_tickets.csv").merge(pd.read_csv("predictions.csv"), on="ticket_id")
july = july.merge(pd.read_csv("data/july_labels.csv"), on="ticket_id")
day = july["created_at"].dt.day
for name, part in (("1-14 July", day < 15), ("15-31 July", day >= 15), ("All July", day > 0)):
    d = july[part]
    caught = (d["flag"] & d["escalated_72h"]).sum()
    print(f"{name}: {len(d)} tickets, escalated {d['escalated_72h'].mean():.3f}, mean score {d['score'].mean():.3f},"
          f" precision {caught / d['flag'].sum():.3f}, recall {caught / d['escalated_72h'].sum():.3f}")

Output

1-14 July: 625 tickets, escalated 0.149, mean score 0.112, precision 0.366, recall 0.441
15-31 July: 775 tickets, escalated 0.133, mean score 0.135, precision 0.312, recall 0.515
All July: 1400 tickets, escalated 0.140, mean score 0.125, precision 0.333, recall 0.480

For the month: 14.0% escalated, precision 0.333 and recall 0.480. The model still catches more than 40% of escalations. But from 15 July, it flags more tickets while fewer escalate. Precision falls to 0.312: the senior team reviews more tickets that did not need it. The mean score (0.125) is still under the real rate (0.140), as the card says.

But why not wait for the labels, and watch only precision and recall? Omar asks this, and it is a good question: they are the numbers that matter. The answer is time. The labels of 31 July arrive on 3 August. The signals without labels moved on 15 July. They are the amber light that comes on before the damage is known.

A written retraining trigger

Retraining means fitting the model again, with newer data, to make a new artifact. When should Priya do it? “When the model gets worse” is too vague: nobody knows who decides, or when, or what “worse” means. A retraining trigger is a rule, written in advance, that starts a review. It is like the service book of the car: a number of kilometres, or a light that stays on. Priya writes it in the model card, under a new section:

MODEL_CARD.md (new section)

## Monitoring and retraining

Every working day, compute over the last 7 days: the flag rate, the mean score, the share of
unknown categories, and the mean of each number feature. When the labels arrive (72 hours
later), compute precision and recall for each month.

Start a retraining review when one of these happens:

1. The flag rate is above 20% on 5 working days in a row.
2. Unknown categories are more than 2% of the tickets of the last 7 days.
3. Recall is below 0.40 on a month with labels.

The review is done by Priya and Grace within 5 working days. Until it ends, the senior team
reviews flagged tickets in score order, highest first, up to its capacity.

Each condition comes from something you already know. Condition 1 is Grace’s capacity, with “5 days in a row” so that one busy day does not start a review. Condition 2 is a small share, because the model learned nothing about a new category. Condition 3 is the improvement target from Define a baseline: catch at least 40% of escalations.

But why not retrain automatically every month? Tomás likes the idea: no meetings, always fresh data. The trigger starts a review, not an automatic retraining, for one reason: a person checks the cause first. If the cause is a broken export, retraining on broken data teaches the model the mistake. The July case shows it: an automatic retraining would learn the wrong prior_tickets_90d.

In July:

Condition	July	Fires?
1. Flag rate above 20% on 5 working days in a row	Above 20% on 15, 16, 17, 20 and 21 July	Yes, on 21 July
2. Unknown categories above 2% of the last 7 days	7 social tickets of 319 on 15 July (2.2%)	Yes, on 15 July
3. Recall below 0.40 on a labelled month	Recall 0.480	No

The review finds two causes: the new channel, and the export error in prior_tickets_90d. Grace and Priya decide:

1. Mei fixes the export of prior_tickets_90d, and adds a check that compares the mean of each number feature with May.
2. Priya retrains with May and June included. The new train set ends on 31 May, so the model learns from tickets after the policy change. June chooses the threshold. July, with its labels, is the new test set, used once.
3. social stays an unknown category until there are enough labelled social tickets to learn from. The warning continues, and the model card says so.
4. Priya updates MODEL_CARD.md, with the new numbers, the new artifact SHA-256 and the new review date.

The second step also helps with calibration: the new model learns from tickets with the higher escalation rate after 1 May.

How to run these checks on a schedule, store the predictions, send alerts and roll back to an older artifact is deployment. Deploying AI Models for Real Users teaches it. This course stops at the plan and the scripts.

Guided practice: checks and signals

1. In ~/projects/ticket-model, activate .venv.
2. Change predict.py in the four places of Input checks in predict.py.
3. Run python predict.py --check. You should see Check passed.
4. Run python predict.py data/july_tickets.csv predictions.csv. You should see the warning about 91 social rows, then 282 flagged (20.1%).
5. Make a broken copy of the input. Open data/july_tickets.csv in a text editor, save it as broken.csv, and in broken.csv rename the column priority to prio.
6. Run python predict.py broken.csv out.csv. You should see Error: missing columns: priority, then Stopped:.
7. Save signals.py, and run it. Compare your table with the output above.
8. Save labels.py, and run it. You should see precision 0.333, recall 0.480 for all of July.
9. Add the section Monitoring and retraining to your MODEL_CARD.md.

If a step fails

• Step 3 or 4 says NameError: name 'RANGES' is not defined: add RANGES under MODEL = Path("model").
• Step 6 shows a KeyError traceback instead of Error:: main() does not call check_input() before pipeline.predict_proba. Compare main() with the code above.
• Step 7 says FileNotFoundError: predictions.csv: run step 4 first.

Your turn: the review on 18 July

Change one condition: it is now 18 July, not the end of the month. The labels of 1 to 14 July have arrived (72 hours after the last of those tickets). Use the outputs above to answer:

1. Which of the three conditions of the trigger have fired by 18 July?
2. What is the recall for 1 to 14 July? Does condition 3 fire for those two weeks?
3. Should Priya and Grace start the review on 18 July?

Show the answer

1. Condition 2 fired on 15 July: 7 of 319 tickets in the last 7 days (2.2%) came from social. Condition 1 has not fired: by 17 July (the last working day before 18 July), the 7-day flag rate was above 20% on 3 working days in a row (15, 16 and 17 July: 21.0%, 21.5% and 22.8%). It needs 5. (On 18 July the rate is 24.3%, but 18 July is a Saturday.)
2. Recall is 0.441 for 1 to 14 July, above 0.40. Condition 3 does not fire. It is defined on a labelled month, and two weeks before the change says little about after it.
3. Yes. One condition is enough to start a review. The review on 18 July can already ask Mei about the new channel. It does not wait for the flag rate to cross the line for 5 days.

Common mistake

Order values in cents, and no range check

Symptom: in August, Mei’s colleague changes the export. order_value is now in cents. The first version of predict.py runs on the July file in this form with no error and no warning. It flags 83.6% of the tickets, and the mean score is 0.799. The senior team gets more than four times its capacity.

Cause: the model learned that a large order means more risk. An order of $54.38 is now 5438, and every order looks huge. The column still has numbers, so a check of types alone passes. In the SQL course you met the same problem in the old web form, which sent cents: Handle messy values.

Fix: a range for each number, as in RANGES. The finished predict.py stops the run:

Output (exit code 1)

Warning: channel: 91 rows have a category the model never saw (social)
Error: order_value: 879 values outside 0 to 5000
Stopped: the input does not match the model's feature schema. Nothing was scored.

In cents, 879 of the 1,400 order values are above 5,000: every order above $50. The other orders are $50 or less, or missing, so they stay in the range. That is why a range check catches a unit error on most rows, not on every row.

Consequence in AI work: a model scores whatever it gets. It never says “this input looks strange”. The schema and the ranges are your only protection against a quiet change upstream. Keep them in the scoring program, next to the model, not in a document.

See it end to endRank a call listCase study

The call-list case study shows a model that drifts as the economy changes, and the checks that catch it.

Knowledge check

Check your understanding

Why does predict.py stop on a missing column, but only warn about an unknown category?

Show the answer

Without the column, the pipeline cannot compute the features, or it fails with an error that does not explain the problem. Nothing can be scored correctly, so the run stops with a clear message and exit code 1. An unknown category is different. The pipeline can score it, because OneHotEncoder(handle_unknown="ignore") gives it 0 in every channel column. Stopping would stop the routing of every ticket because of a few. But many unknown values mean that the input changed, so the program counts and reports them, and the monitoring has a trigger for their share.

Check your understanding

In July the flag rate is 20.1% for the month, almost exactly the capacity. Tomás says monitoring found nothing. Use the 7-day signals to explain why he is wrong.

Show the answer

The month hides the change. From 1 to 14 July, the flag rate is 17.4% and 18.5% per week. From 15 July it is 26.2%, then 20.9%: above the capacity of 20%. The 7-day rate stays above 20% from 15 to 30 July. The mean score goes up from about 0.112 to 0.138. Unknown channels go from 0 to 12.6%, and the mean prior_tickets_90d falls. Two triggers fire: unknown categories on 15 July, and the flag rate on 21 July. When the labels arrive, precision from 15 July is 0.312, lower than the 0.366 of the first two weeks.

Module practice

This module turned a notebook model into a delivery: files, commands and documents that another person can run and check. The module practice asks you to deliver the classifier and an evaluation report, so that a colleague can reproduce every number with the commands alone.

Your task: deliver the classifier

Deliver the project folder with:

1. The code: ticket_model.py, train.py, evaluate.py, predict.py, requirements.txt, get_data.py.
2. MODEL_CARD.md, with the section Monitoring and retraining.
3. A short evaluation report, reports/evaluation.md: the prediction time and target, the baseline comparison, the threshold and how it was chosen, the test results (once), the main limits, and the exact commands that reproduce every number in it, with their expected output.
4. Proof that it works from nothing: run the commands in a fresh copy of the project, in a new terminal, with a new .venv. The artifact must give identical scores there.

Do not send .venv, data/ or model/: the commands make them again. That is the point of the test.

Show the model answer

reports/evaluation.md:

reports/evaluation.md

# Evaluation report: ticket escalation model

Prediction time: when a ticket is created. Target: escalated_72h (escalated to a senior
agent within 72 hours). Decision: a flagged ticket goes straight to the senior team, which
can review at most 20% of new tickets.

Data: train 16,939 tickets (Jul 2024 to Apr 2026), validation 1,179 (May 2026),
test 1,002 (1 to 27 June 2026). Synthetic data, no real people.

Model: logistic regression in one pipeline (one-hot categories, median imputation with
indicators, scaling), 11 features, fitted on train only. Chosen over gradient boosting by
the one-standard-error rule (CV average precision 0.3393 against 0.3424, standard error
0.008). Threshold 0.19: the lowest that flags at most 20% of the validation set.

| At threshold 0.19 | Validation | Test (once) | Rule "priority 1" (test) |
|---|---|---|---|
| Flagged | 227 (19.3%) | 196 (19.6%) | 150 (15.0%) |
| Precision | 0.379 | 0.357 | 0.367 |
| Recall | 0.470 | 0.446 | 0.350 |
| Average precision | 0.391 | 0.345 | |

The model meets the target (at most 20% flagged, at least 40% caught) on validation and
test. The rule catches 35%. Nothing was changed after the test.

Limits: scores are too low since the policy change of 2026-05-01 (do not show them as a
chance); weak on account and priority 3 tickets; new channels are scored with less
information. See MODEL_CARD.md.

## Reproduce

    python -m pip install -r requirements.txt
    python get_data.py          -> Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
    python train.py             -> Threshold 0.19 ... flagged 227 of 1179 (19.3%), precision 0.379, recall 0.47.
    python predict.py --check   -> Check passed: 20 scores are identical after reloading
    python evaluate.py          -> reports/test_report.json: precision 0.357, recall 0.446

Artifact SHA-256 (macOS, the versions in requirements.txt):
a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736

The fresh-copy run. Make a new folder, copy only the delivered files into it, and run the commands:

macOS and Linux

New terminal

mkdir -p ~/projects/ticket-model-fresh/reports
cd ~/projects/ticket-model
cp ticket_model.py train.py evaluate.py predict.py requirements.txt get_data.py MODEL_CARD.md ~/projects/ticket-model-fresh/
cp reports/evaluation.md ~/projects/ticket-model-fresh/reports/
cd ~/projects/ticket-model-fresh
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python get_data.py
python train.py
python predict.py --check
python evaluate.py
python predict.py data/july_tickets.csv predictions.csv
shasum -a 256 model/escalation_model.joblib predictions.csv ../ticket-model/predictions.csv

Windows

New terminal (PowerShell)

New-Item -ItemType Directory -Force ~\projects\ticket-model-fresh\reports
cd ~\projects\ticket-model
Copy-Item ticket_model.py, train.py, evaluate.py, predict.py, requirements.txt, get_data.py, MODEL_CARD.md ~\projects\ticket-model-fresh\
Copy-Item reports\evaluation.md ~\projects\ticket-model-fresh\reports\
cd ~\projects\ticket-model-fresh
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python get_data.py
python train.py
python predict.py --check
python evaluate.py
python predict.py data/july_tickets.csv predictions.csv
Get-FileHash model\escalation_model.joblib, predictions.csv, ..\ticket-model\predictions.csv

The output of the last five commands on macOS (the JSON of evaluate.py is shortened):

Output

Trained on 16939 tickets. Threshold 0.19 for a review capacity of 20%.
Valid: ROC AUC 0.752, average precision 0.391, flagged 227 of 1179 (19.3%), precision 0.379, recall 0.47.
Saved model/escalation_model.joblib, model_info.json and check_sample.csv.
Check passed: 20 scores are identical after reloading (largest difference 1.1e-16).
{
  "rows": 1002,
  "escalated": 157,
  "threshold": 0.19,
  "roc_auc": 0.749,
  "average_precision": 0.345,
  ...
  "precision": 0.357,
  "recall": 0.446,
  ...
}
Warning: channel: 91 rows have a category the model never saw (social)
Scored 1400 tickets with threshold 0.19: 282 flagged (20.1%). Wrote predictions.csv.
a4acd259a398e7c84517b98bf50d811197f90b0f2f1ea15f1f9e168ba3aff736  model/escalation_model.joblib
91936b4eff2c28a1b0c3382b8b88a05f2f7e677fc8395b2ab4486830a3dd9ff1  predictions.csv
91936b4eff2c28a1b0c3382b8b88a05f2f7e677fc8395b2ab4486830a3dd9ff1  ../ticket-model/predictions.csv

The two predictions.csv files have the same SHA-256: the fresh copy gives exactly the same score and flag for all 1,400 July tickets. evaluate.py runs in the fresh copy because it has no report yet. That run reproduces the same model; it does not choose anything. Its numbers must be identical, and they are.

On Windows, pandas writes CSV files with other line endings, so the checksums differ from these. Compare the two files of the same computer: they must be equal.

A complete delivery has: every number in the report reproduced by a listed command; predict.py --check passing in the fresh copy; identical predictions in the two copies; and a card and a report that name the same threshold and artifact.

Module check

Check your understanding

Retrieval: list what a handoff of the escalation model contains, and what each part is for.

Show the answer

• model/escalation_model.joblib: the whole fitted pipeline (preprocessing and model), saved with joblib.dump.
• model/model_info.json: the feature schema, the threshold 0.19, the package versions, the checksums of the data and of the model file, and the validation results.
• model/check_sample.csv: 20 tickets with their saved scores, for predict.py --check.
• requirements.txt: exact pins of pandas, scikit-learn, numpy, joblib and matplotlib.
• train.py, predict.py (with input checks), evaluate.py and ticket_model.py: the commands that make, check and use the artifact.
• MODEL_CARD.md: intended use, not-for uses, data, why this model, evaluation, slices, limits, owners, review date, monitoring and the retraining trigger.
• An evaluation report with the commands that reproduce its numbers.

Check your understanding

Predict: a new export has the channel whatsapp in 40 of 1,000 tickets and priority written as P1, P2 and P3. What does predict.py print, and does it score the batch?

Show the answer

It prints a warning for the channel: channel: 40 rows have a category the model never saw (whatsapp). Then it prints an error for the priority: priority: 1000 values are not numbers, for example 'P1' (or the first value in the file). Then Stopped: the input does not match the model's feature schema. Nothing was scored., with exit code 1. The batch is not scored, because of the priority. The channel alone would be scored, with a warning. 40 of 1,000 is 4%, above the 2% of the trigger, so it would also start a retraining review.

Check your understanding

Completion check: load the artifact in a fresh process and verify identical predictions on a fixed sample. Give the commands, the output that proves it, and why a fresh process matters.

Show the answer

In a new terminal (or a new copy of the project with a new .venv), in the project folder:

python predict.py --check
python predict.py data/july_tickets.csv predictions.csv

--check loads model/escalation_model.joblib, scores the 20 tickets of model/check_sample.csv, and compares them with the scores saved at training time: Check passed: 20 scores are identical after reloading (largest difference 1.1e-16). The difference is far below the limit of 1e-9; it comes only from saving the scores as text. For a larger fixed sample, the SHA-256 of predictions.csv for the 1,400 July tickets is the same in two runs and in a fresh copy (91936b4e… on macOS).

A fresh process matters because it starts with nothing in memory: no fitted pipeline, no variables, no imports from the notebook. If the scores are identical there, the files alone reproduce the model. A complete answer shows the command, the passed check with its largest difference, and says why the check is not done in the training notebook.

Reset to a known state

If your project is broken or you want to start Module 8 again:

1. Delete the outputs: the folders model/ and reports/, and the files predictions.csv, out.csv and broken.csv.
2. Run python get_data.py --reset. You should see the Ready: line with 16939, 1983, 1179, 183, 1002 and 157.
3. If a code file is broken, copy the course’s version from C05/project: curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/<file> (curl.exe on Windows).
4. Run python train.py, then python predict.py --check. You should see Threshold 0.19 and Check passed.
5. Run python evaluate.py only if you need reports/test_report.json again. It gives the same numbers, because the model is the same.

Recap

In this module you prepared the model for handoff:

• Package (Package artifacts): train.py saves the whole pipeline with its schema, threshold 0.19, versions and checksums, and a check sample. requirements.txt pins every version. predict.py --check proves identical scores in a fresh process.
• Document (Document use boundaries): the model card says what the model is for and not for, with the evidence: test recall 0.446 at 19.6% flagged, against 0.350 for the rule; the weak slices with their counts; the calibration gap; and why a simple model was chosen.
• Check (this lesson): predict.py stops on a missing column, a wrong type or a value out of range, and counts unknown categories. Monitoring compares the flag rate, the mean score, unknown categories and the inputs with what you expect. In July, two of three trigger conditions fired: the new social channel and a 7-day flag rate up to 26.2%. With the labels, July recall is 0.480 and precision 0.333.
• Retrain by a written rule and a review, not by feeling. Deployment mechanics come in the deployment course.

Across the course, you learned what machine learning is and how to judge a model before you build one. You framed a prediction problem and beat simple baselines. You built a pipeline, tried model families and recorded each experiment. You tuned without touching the test set, chose a threshold for the capacity, explained the errors, and chose a simple model for good reasons. Now you delivered it with its limits. Next come the case studies: real projects on public data, from the question to a recommendation. Complete at least one for the certificate. The final assignment then asks you to do it all yourself, from the baseline to the evaluation report.

Next: Forecast bike-share demand