Prediction inputs and outputs

What you will be able to do

Write down exactly what the escalation model accepts and what it returns, so that a wrong ticket is refused before the model scores it.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The model and its artifact, model/ with escalation_model.joblib and model_info.json, from Package artifacts in Machine Learning: From Problem to Reliable Model.
• The threshold 0.19, and why a score is not a calibrated probability, from Thresholds and calibration in the machine learning course.
• Pydantic models, extra="forbid" and the 422 answer, from Validation and error responses in Build and Deploy Your First API.

What you need on your computer

• Python 3.12 or later.
• The machine learning course’s project folder ~/projects/ticket-model, with its .venv and the folder model/ that python train.py made in Package artifacts: Worked example.
• If you do not have it, download the folder C17/start of the labs as ~/projects/ticket-model. It has model/, with the three model files, and data/, but no .venv. Check the model file before you load it. The command and the expected hash are in Artifact trust and integrity: Check a file that you downloaded.

Tested with Python 3.14.6 on macOS, Apple silicon. The Windows and Linux steps are not tested.

A new service for one model

In the API course and the Docker course you built ticket-api, which gives each ticket a category. In this course you build a second, small service, escalation-service. It answers one question: how likely is this new ticket to be escalated?

Why not add the model to ticket-api? With one model per service, each model has its own image, its own releases and its own rollbacks. A new escalation model never needs a new release of the category API. The help desk calls ticket-api for the category and escalation-service for the risk. The data is the Larkfield data from the machine learning course, which is synthetic.

A form that the clerk can refuse

At a passport office, a clerk checks your form before anybody works on it. A date of birth in the future, or an empty name, and the form comes back to you at the counter. Nobody prints a passport from a wrong form, and nobody guesses what you meant.

A model has no clerk. It takes any numbers that you give it and returns a score.

The problem: the model answers anything

Priya loads the pipeline from the machine learning course and scores one example ticket: priority 1, an order of $89.50, 112 words. Then she changes the priority to values that the help desk never uses, 0 and 4. Run this in ~/projects/ticket-model, with its .venv active. (If you downloaded C17/start, it has no .venv. Run it later, after step 3 of Write the contract, with the .venv of escalation-service active.)

Python, in ~/projects/ticket-model

import joblib
import pandas as pd

pipeline = joblib.load("model/escalation_model.joblib")
ticket = dict(channel="email", team="payment", segment="home", region="west", priority=1,
              order_value=89.5, word_count=112, customer_tenure_days=400,
              prior_tickets_90d=2, created_hour=10, prior_escalations_90d=1)
for priority in [0, 1, 2, 3, 4]:
    row = pd.DataFrame([{**ticket, "priority": priority}])
    print(priority, round(pipeline.predict_proba(row)[0, 1], 4))

The output:

0 0.5702
1 0.3507
2 0.1802
3 0.0821
4 0.0351

• priority 1 to 3: values the help desk uses
• priority 0 and 4: values that do not exist

The raw pipeline scores priority 0 and 4 without complaint. Priority 0 gets the highest score of all, and is flagged.

Text description of the chart

The example ticket, scored by the pipeline from the machine learning course with no checks in front of it. Priority 1 gives 0.351, priority 2 gives 0.180 and priority 3 gives 0.082. Priority 0, which does not exist, gives 0.570, the highest score, and the ticket is flagged. Priority 4, which does not exist either, gives 0.035.

No error, no warning. The pipeline continued the line that it learned: a lower number means a more urgent ticket, so 0 looks “more urgent than urgent”. Priya tried more mistakes on the same ticket:

Change	Score	What the pipeline did
created_hour 25	0.3213	Scored an hour that does not exist
word_count −50	0.1775	Scored a negative count; the flag changed from true to false
channel "Email"	0.4215	Scored a channel that it never saw, as “none of the known ones”
order_value 0	0.3240	Scored an order of $0
order_value empty (NaN)	0.3597	Filled the training median and set the “missing” indicator
priority "high"	error	ValueError: Cannot use median strategy with non-numeric data

Only the last mistake stops the pipeline, and its message does not even name the field. Every other mistake becomes a score that looks normal, and the help desk would act on it.

Using a trained model to score a new case is called inference. Training happens once. Inference happens for every new ticket, long after training, with input that another program sends. So the service must check the input at the moment of inference, before the model sees it.

The idea: an inference contract

The clerk’s form has its rules printed on it. The service needs the same: a written list of what a ticket must look like, and of what the answer looks like. This is the inference contract. Both sides rely on it: the client knows what to send, and the model knows what it gets.

For the escalation model, the contract says six things:

1. Names and order. The 11 features that the pipeline was trained on, in the training order, and no other key.
2. Types. Text for the four categories (channel, team, segment, region), whole numbers for counts and hours, and a number for order_value.
3. Ranges. For example, priority 1 to 3, created_hour 0 to 23, order_value 0 to 5,000.
4. Missing values. Which features may be null, and what null means.
5. Unknown categories. A category value that the model never saw is accepted and reported as a warning. The service counts them, because many of them mean that the input has changed (Plan production checks in the machine learning course).
6. The output. What the service returns, and what each value means.

Tomás asks: “The machine learning course’s predict.py already checked the CSV file. Why again?” Because predict.py checked a file that Priya’s own job wrote. A service gets requests from programs that other people write and change. After one wrong client release, every ticket can have priority 0.

A contract has a cost. It is strict on purpose: a client with an old format gets an error until its team fixes it. And a change to the model’s inputs is now a change to the contract, which both sides must agree. For a quick test in a notebook, you do not need one. For a service that other programs call, you do.

Missing is not the same as null

Rule 4 needs care. order_value can be empty for a real reason: the ticket has no order. Compare four requests that differ in one part:

The request has	Meaning	The contract says
"order_value": 89.5	An order of $89.50	Accepted
"order_value": null	There is no order	Accepted: the pipeline fills the training median, as in training
"order_value": 0	An order of $0	Accepted, but it is not the same as no order
No order_value key	The client forgot it, or sends an old format	Rejected

The measurement above showed why this matters: null gives 0.3597 and 0 gives 0.3240 for the same ticket. A client that writes 0 for “no order” changes every score a little, and nobody sees an error. So the contract requires every key. A null says “I know that this value is empty”. A missing key says nothing, and the service must not guess.

Only two features may be null: order_value (no order) and customer_tenure_days (a guest has no first order). The other nine may never be null.

The output: a score, a flag and a version

A good answer of the finished service looks like this (a real response):

{"ticket_id":"T-120001","score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[]}

Each value has one meaning:

• score, 0 to 1. A higher score means a higher risk of escalation. The score ranks tickets; it is not a calibrated chance. The machine learning course showed in Thresholds and calibration that these scores do not match the real escalation rates. So nobody may read 0.35 as “a 35% chance”.
• flag: true when the score is at least the threshold. The ticket goes to the senior team.
• threshold: 0.19. It belongs to the model version, because it was chosen so that this model flags about 20% of tickets, Grace’s review capacity. A new model needs its own threshold.
• model_version, so that every answer can be traced to the model that gave it.
• warnings, for example an unknown category. The score is less certain.

You write the input part of the contract now. The service and its answers come in Module 3.

Write the contract

Make the project and its virtual environment, as in the API course.

1. Make the folders and go into the project folder.

   macOS / Linux

   mkdir -p ~/projects/escalation-service/escalation ~/projects/escalation-service/tests
   cd ~/projects/escalation-service

   Windows (PowerShell)

   New-Item -ItemType Directory -Force $HOME\projects\escalation-service\escalation, $HOME\projects\escalation-service\tests
   cd $HOME\projects\escalation-service

2. Make requirements.txt. The four model packages have the same versions as in the machine learning course, because the model must run with the versions that trained it.

   requirements.txt

   # Needed to run the service
   fastapi[standard]==0.142.2
   scikit-learn==1.9.1
   pandas==3.0.6
   numpy==2.5.3
   joblib==1.6.0
   
   # Needed to work on the service
   pytest==9.1.1
   httpx2==2.13.1

3. Make the virtual environment, activate it and install the packages.

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   Windows (PowerShell)

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt

4. Make .gitignore. Some of its lines are for files that later modules make.

   .gitignore

   .venv/
   __pycache__/
   .env
   out/
   *.log
   bench/results.jsonl

5. Make two empty files: escalation/__init__.py and tests/__init__.py. They make the two folders into Python packages, so that the tests can import escalation.

Now write the contract. Pydantic checks the names, types and ranges, as in the API course. The first part of the file lists the features and their rules:

escalation/contract.py

"""The inference contract: what a ticket must look like before the model scores it.

The model learned from 11 features. This file says, for each one, its name,
its type, the values it may take, and whether it may be missing. The API and
the batch job both check tickets with these classes, so the two paths always
apply the same rules.
"""

import math

import pandas as pd
from pydantic import BaseModel, ConfigDict, Field

# The order of the columns that the pipeline was fitted on (C05 ticket_model.py).
FEATURES = [
    "channel",
    "team",
    "segment",
    "region",
    "priority",
    "order_value",
    "word_count",
    "customer_tenure_days",
    "prior_tickets_90d",
    "created_hour",
    "prior_escalations_90d",
]
CATEGORIES = ["channel", "team", "segment", "region"]


class TicketFeatures(BaseModel):
    """The 11 features of one ticket, as they are when the ticket is created."""

    model_config = ConfigDict(extra="forbid")

    channel: str = Field(min_length=1, max_length=30, description="How the customer contacted us.")
    team: str = Field(min_length=1, max_length=30, description="The team that owns the ticket.")
    segment: str = Field(min_length=1, max_length=30, description="The kind of customer.")
    region: str = Field(min_length=1, max_length=30, description="The customer's region.")
    priority: int = Field(ge=1, le=3, description="1 is the most urgent, 3 the least.")
    order_value: float | None = Field(
        ge=0, le=5000, description="The order's value in dollars. Null if there is no order."
    )
    word_count: int = Field(ge=0, le=10000, description="Words in the first message.")
    customer_tenure_days: int | None = Field(
        ge=0, le=20000, description="Days since the customer's first order. Null for a guest."
    )
    prior_tickets_90d: int = Field(ge=0, le=500, description="The customer's tickets in the last 90 days.")
    created_hour: int = Field(ge=0, le=23, description="The hour the ticket was created (0 to 23).")
    prior_escalations_90d: int = Field(ge=0, le=500, description="The customer's escalations in the last 90 days.")

Three details carry the rules. extra="forbid" refuses any other key. No field has a default value, so every key is required, even order_value: float | None allows null, not a missing key. The categories are plain text, not a fixed list, because an unknown value must pass with a warning.

The second part goes at the end of the same file. It adds the ticket ID and three helpers:

escalation/contract.py (continued)

class TicketIn(TicketFeatures):
    """A ticket to score: its ID and its features."""

    model_config = ConfigDict(
        extra="forbid",
        json_schema_extra={
            "examples": [
                {
                    "ticket_id": "T-120001",
                    "channel": "email",
                    "team": "payment",
                    "segment": "home",
                    "region": "west",
                    "priority": 1,
                    "order_value": 89.5,
                    "word_count": 112,
                    "customer_tenure_days": 400,
                    "prior_tickets_90d": 2,
                    "created_hour": 10,
                    "prior_escalations_90d": 1,
                }
            ]
        },
    )

    ticket_id: str = Field(pattern=r"^T-\d{6}$", description="The ticket's ID, for example T-120001.")


def unknown_categories(ticket: TicketFeatures, known: dict[str, list[str]]) -> list[str]:
    """Warnings for category values that the model never saw in training.

    They are allowed: the one-hot encoder gives them no column, so the model
    scores them as "none of the known values". Many of them mean the input
    has changed, so the service reports and counts them.
    """
    warnings = []
    for name in CATEGORIES:
        value = getattr(ticket, name)
        if value not in known[name]:
            warnings.append(f"{name}: {value!r} is a value the model never saw")
    return warnings


def to_frame(tickets: list[TicketFeatures]) -> pd.DataFrame:
    """The tickets as a table with the training columns, in the training order.

    A missing number (None) becomes NaN, so the pipeline's imputer fills it
    with the training median, exactly as in training. Do not fill it here.
    """
    rows = [t.model_dump(include=set(FEATURES)) for t in tickets]
    frame = pd.DataFrame(rows, columns=FEATURES)
    for name in FEATURES:
        if name not in CATEGORIES:
            frame[name] = frame[name].astype("float64")
    return frame


def is_missing(value) -> bool:
    return value is None or (isinstance(value, float) and math.isnan(value))

to_frame builds the columns in the order of FEATURES. scikit-learn also checks the column names that it saw in training, so a table with other names fails loudly. A missing number stays NaN, and the pipeline’s imputer fills it exactly as in training. If the service filled it with 0, it would make the quiet mistake of the table above. is_missing is a small helper for later modules.

Try the contract

Start python in ~/projects/escalation-service and send two mistakes at once:

Python, in ~/projects/escalation-service

from escalation.contract import TicketIn

good = TicketIn.model_config["json_schema_extra"]["examples"][0]
TicketIn(**{**good, "priority": "high", "created_hour": 24})

The output (shortened):

pydantic_core._pydantic_core.ValidationError: 2 validation errors for TicketIn
priority
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='high', input_type=str]
...
created_hour
  Input should be less than or equal to 23 [type=less_than_equal, input_value=24, input_type=int]
...

Both fields are named, with the reason. Now leave out order_value:

Python (continued)

missing = {k: v for k, v in good.items() if k != "order_value"}
TicketIn(**missing)

The output (shortened):

pydantic_core._pydantic_core.ValidationError: 1 validation error for TicketIn
order_value
  Field required [type=missing, input_value={'ticket_id': 'T-120001',...ior_escalations_90d': 1}, input_type=dict]
...

Last, an unknown channel. The known categories are in the machine learning course’s model_info.json:

Python (continued)

import json
from escalation.contract import unknown_categories

known = json.load(open("../ticket-model/model/model_info.json"))["features"]["categories"]
unknown_categories(TicketIn(**{**good, "channel": "social"}), known)

The output:

["channel: 'social' is a value the model never saw"]

Leave Python with exit().

Explore: accepted, warned or rejected?

The explorer shows real answers of the finished service from Module 3, for one change at a time. That service uses exactly the contract.py that you just wrote.

Explore

The contract: accepted, accepted with a warning, or rejected?

Result

Accepted

HTTP status

200

Score

0.3507

No change: the service scores the ticket and answers 200. Every key is present, every value has the right type and is inside its range, and every category is one the model knows.

• fails: rejected with 422
• warning: scored, and reported
• the rule that lets it pass

The contract's rules for this request

Rule	What it says	Result
Names	Only ticket_id and the 11 features. No other key.	passes
Every key present	A missing key is an error, even for a value that may be null.	passes
Types	Text for the 4 categories, whole numbers for the counts, a number for order_value.	passes
Ranges	For example priority 1 to 3, created_hour 0 to 23, order_value 0 to 5000.	passes
Null	Allowed only for order_value (no order) and customer_tenure_days (a guest).	passes
Known categories	A value the model never saw is accepted, with a warning.	passes

Change to the example ticket: none: the example ticket as it is

The service's answer: 200

{
  "ticket_id": "T-120001",
  "score": 0.35065699401024064,
  "flag": true,
  "threshold": 0.19,
  "model_version": "1.0.0",
  "warnings": []
}

Real responses of the service with the bundle escalation-1.0.0. The request ID changes on every call, so it shows as "...".

One change to the example ticket at a time. The table marks the rule of the contract that decides. Below it is the real response of POST /v1/score.

Text description of the explorer

The explorer starts with the example ticket and no change: every rule passes, the service answers 200, and the score is 0.3507.

The other changes and the real answers: priority “high” fails the type rule: 422 with fields: ["priority"]. created_hour 24 fails the range rule: 422, fields: ["created_hour"]. The key order_value removed fails the rule “every key present”: 422, fields: ["order_value"]. order_value null passes, because null is allowed for this feature: 200, score 0.3597. team null fails the null rule: 422, fields: ["team"]. An extra field colour fails the names rule: 422, fields: ["colour"]. channel “social” passes every rule but gets a warning, because the model never saw this channel: 200, score 0.4215, warnings: ["channel: 'social' is a value the model never saw"]. Every 422 answer has the code invalid_ticket and the message “The ticket does not match the model’s contract.”

Two of the seven changes were accepted: null where null is allowed, and an unknown category with a warning. The service refused the other five, with the name of the field, before the model ran.

Test the contract

Tests keep the rules from changing by accident. The example ticket goes into a fixture, so that each test changes one thing in a fresh copy:

tests/conftest.py

import pytest

# The example ticket of contract.py. Each test changes one thing in a copy.
TICKET = {
    "ticket_id": "T-120001",
    "channel": "email",
    "team": "payment",
    "segment": "home",
    "region": "west",
    "priority": 1,
    "order_value": 89.5,
    "word_count": 112,
    "customer_tenure_days": 400,
    "prior_tickets_90d": 2,
    "created_hour": 10,
    "prior_escalations_90d": 1,
}


@pytest.fixture
def ticket():
    return dict(TICKET)

tests/test_contract.py

import pytest
from pydantic import ValidationError

from escalation.contract import FEATURES, TicketIn, to_frame, unknown_categories


def test_a_good_ticket_passes(ticket):
    assert TicketIn(**ticket).priority == 1


@pytest.mark.parametrize(
    "change, field",
    [
        ({"priority": "high"}, "priority"),
        ({"priority": 4}, "priority"),
        ({"created_hour": 24}, "created_hour"),
        ({"word_count": -1}, "word_count"),
        ({"channel": ""}, "channel"),
        ({"team": None}, "team"),
        ({"colour": "red"}, "colour"),
    ],
)
def test_a_bad_value_is_rejected(ticket, change, field):
    with pytest.raises(ValidationError) as error:
        TicketIn(**{**ticket, **change})
    assert field in str(error.value)


def test_null_is_allowed_only_where_the_contract_says(ticket):
    TicketIn(**{**ticket, "order_value": None, "customer_tenure_days": None})
    with pytest.raises(ValidationError):
        TicketIn(**{**ticket, "word_count": None})


def test_a_missing_field_is_not_the_same_as_null(ticket):
    del ticket["order_value"]
    with pytest.raises(ValidationError):
        TicketIn(**ticket)


def test_an_unknown_category_is_a_warning_not_an_error(ticket):
    known = {"channel": ["email"], "team": ["payment"], "segment": ["home"], "region": ["west"]}
    warnings = unknown_categories(TicketIn(**{**ticket, "channel": "social"}), known)
    assert warnings == ["channel: 'social' is a value the model never saw"]


def test_the_frame_has_the_training_columns_in_order(ticket):
    frame = to_frame([TicketIn(**{**ticket, "order_value": None})])
    assert list(frame.columns) == FEATURES
    assert frame["order_value"].isna().all()  # NaN for the imputer, not 0

Run the tests in ~/projects/escalation-service, with .venv active:

Terminal

python -m pytest

The output (shortened):

collected 12 items

tests/test_contract.py ............                                      [100%]

============================== 12 passed in 0.26s ==============================

The parametrized test counts as seven tests, one for each bad value. The first run can take a few seconds, while Python imports pandas.

Your turn: priority 2.5 and priority 2.0

Grace’s team asks what happens if a client sends priority 2.5, or 2.0. Write down your prediction for each. Then try both in Python with TicketIn, and add the value that is rejected to the list in test_a_bad_value_is_rejected.

Show the answer

2.5 is rejected: Input should be a valid integer, got a number with a fractional part [type=int_from_float, input_value=2.5, input_type=float]. 2.0 is accepted and becomes the whole number 2, because Pydantic converts a number without a fractional part. The text "2" is accepted too, for the same reason. This is safe for the model: 2.0 and 2 are the same priority.

Add one line to the list:

tests/test_contract.py

        ({"priority": 2.5}, "priority"),

python -m pytest then reports 13 passed. The course’s snapshot has 12 contract tests, so the counts in later lessons are one lower than yours if you keep the line.

Common mistake

A capital letter changes the score

Symptom: a new release of the help desk sends "channel": "Email". The service answers 200, but the scores are higher than before, and every answer has a warning. A real response of the finished service:

{"ticket_id":"T-120001","score":0.42145836840785755,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":["channel: 'Email' is a value the model never saw"]}

With "email", the same ticket gets 0.3507.

Cause: the model learned the channel email, in lower case. Email is a different text, so the one-hot encoder gives it no column, and the model scores it as “none of the known channels”. That score is the same as for social, a channel that the model never saw.

Fix: the contract did its part: it accepted the ticket and reported the unknown value in warnings. Tell the client team to send the exact values from model_info.json, and to test them. Do not change the value inside the service (for example with .lower()) without the model’s owner. The contract must say the same thing on both sides.

For a real AI service: warnings help only if somebody reads them, so count them. One unknown value is noise. A sudden rise means that a client or the data changed. Module 5 adds this count to the service’s health signals.

Check your understanding

A client sends a ticket without the key order_value. Another client sends order_value null. Why does the contract reject the first and accept the second?

Show the answer

Null says that the value is known to be empty: the ticket has no order. The pipeline handles that case as in training, with the training median and a missing-value indicator. A missing key says nothing: the client may have forgotten the field, or it may send an old format. If the service guessed (null? 0?), the score could change without any error: 0 gives 0.3240 where null gives 0.3597. So every key is required, and only two features may be null.

Check your understanding

Omar reads score 0.35 in an answer and says: so this ticket has a 35% chance of escalation. What do you tell him?

Show the answer

The score ranks tickets: a higher score means a higher risk. It is not calibrated, so 0.35 does not mean a 35% chance (see Thresholds and calibration in the machine learning course). What the help desk uses is the flag. The score is at least 0.19, the threshold of model 1.0.0, so the ticket goes to the senior team. The threshold belongs to that model version.

Check your understanding

Predict: a client sends priority 0. What does the raw pipeline do, and what does the service with the contract do?

Show the answer

The raw pipeline scores it without complaint: 0.5702 for the example ticket, the highest score of all priorities, so the ticket is flagged. The service with the contract rejects it with 422 and fields: ["priority"], because priority must be 1 to 3. The model never sees the ticket.

This course ends with three case studies. Each one serves another model on real data, with a contract like this one.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Score credit default in batch and onlineServe one credit-default artifact two ways, a monthly batch-scoring job and an online endpoint, and prove they give the same scores.
• Release a new house-price modelRelease a retrained house-price model behind the house-price API from the API course: shadow it, compare it on real traffic, promote it, and roll it back.
• Score hospital readmissions asynchronouslyAccept a large discharge file as a queued job, score it in the background, and let the caller fetch the results later.

Recap

• Inference is using a trained model on a new case. The model scores anything, even values that do not exist: priority 0 got 0.5702.
• The inference contract says what a ticket must look like: names and order, types, ranges, missing values and unknown categories. It also says what the answer means: a score that ranks, a flag, the threshold, the model version and warnings.
• Missing is not the same as null. Every key is required, and only order_value and customer_tenure_days may be null.
• An unknown category is accepted with a warning, not refused.
• contract.py holds the rules, and 12 tests keep them.

Next, you package the model with everything that it needs to run: its preprocessing, this contract, its versions and a set of fixed test cases.

Next: Bundle model dependencies