Monitor model health

What you will be able to do

Know that the model has a problem before Grace’s team feels it. The service counts its own health signals, and a short script checks the real results when they arrive three days later.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The monitoring signals and the retraining trigger, from Plan production checks: Monitoring signals in Machine Learning: From Problem to Reliable Model.
• Precision and recall, from Classification metrics.
• The shadow run and RELEASES.md, from Version and rollout.
• p50 and p95, from Benchmark the service. Workers, from Container resource settings.

What you need on your computer

• ~/projects/escalation-service with the shadow code and the three bundles, from Version and rollout: Guided practice.
• ~/projects/ticket-model/data/july_labels.csv from the machine learning course.
• If your project is missing or broken, copy the snapshot nextia-learning-labs/C17/snapshots/end-of-m04 and do the guided practice of Version and rollout. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

Signs now, results later

In a hospital, a nurse checks each patient’s temperature and pulse every few hours. The blood test results come back two days later. The nurse does not wait for the laboratory: a fever tonight is a reason to act tonight. The laboratory then confirms or corrects the picture.

A model in production is in the same position. The true answer for a ticket, escalated or not, arrives 72 hours after the score. Until then, the service can watch only the signs that it sees at once.

The problem: a log warns nobody

In July, model 1.0.0 flagged 17.9% of the tickets from 1 to 14 July and 21.9% from 15 to 31 July. From 15 July, Grace’s team got more flagged tickets than it can review. In the machine learning course, Priya found it afterwards, in the July predictions. The review found two causes: the new social channel and a broken export of prior_tickets_90d (Plan production checks).

In the last lesson, Amira found rc1’s 86% with grep, after the run. A log file warns nobody until a person reads it. The service must count the signals itself, all the time, and say when a number crosses a line.

The health signals of a model service

Model health is whether the model still does its job on today’s inputs. A service cannot measure it directly, so it watches signals: numbers that it can compute at once, and that move when something goes wrong. The diagram shows each signal, and when it is ready.

Two kinds of signals. The service counts the first kind at once. The second kind needs the true answer, which comes 72 hours later.

Text description of the diagram

A new ticket arrives. The contract checks it first. If it does not match, the service answers 422, and the count contract_errors goes up for each wrong field. If it matches, the ticket is scored, and the service writes one prediction log line.

At once, GET /v1/monitor shows four signals. They are the contract errors, the unknown categories, the flag rate and mean score, and the latency. They need no labels.

72 hours later, the label arrives: was the ticket escalated or not? Then monitoring/outcomes.py joins the log lines with the labels and computes precision and recall.

Each signal answers one question:

• Contract errors: does a client send tickets that the model cannot score? A sudden rise usually means that a client changed.
• Unknown categories: does the model see values that it never learned, such as social? These tickets get a score, but a less reliable one.
• Latency: is the service still fast enough for the help desk?
• Flag rate, against the review capacity of 20%: Grace’s constraint, in one number.
• Mean score: if it moves, the inputs or the model changed, even when the flag rate has not moved yet.

The flag rate and the mean score are drift indicators. Drift means that the data in production moves away from the data that the model learned from. The machine learning course computed the same two signals afterwards, from the July file. Here the service counts them for every ticket, as it scores it.

A tiny example: why a window and a minimum

Here are the flags of the first ten tickets after a release (1 = flagged):

0 1 0 0 1 0 0 0 1 1    →  4 of 10 flagged = 40%

40% is twice the capacity. Is the model broken? With 10 tickets, one more flag moves the rate by 10 points, so 40% can be bad luck. With 1,000 tickets, one flag moves it by 0.1 point. So the monitor does two things:

• It computes the flag rate and the mean score over a window: the last 1,000 scored tickets. Old tickets leave the window, so the numbers follow today’s traffic.
• It sends an alert only when the window has at least 200 tickets.

The July chart later in this lesson shows how much one day can vary.

monitor.py

escalation/monitor.py keeps the signals in memory. The first part stores each scored ticket and counts the errors:

escalation/monitor.py

"""Model health signals, kept in memory for the last WINDOW scored tickets.

The service cannot know at once whether a score was right: the true answer
(escalated or not) arrives 72 hours later. So it watches signals that it can
see now:

    contract errors     tickets rejected before scoring, by field
    unknown categories  values the model never saw, by feature
    flag rate           the share of tickets flagged, against the review capacity
    mean score          a change means the inputs or the model changed
    latency             p50 and p95 of the scoring time

Each worker process has its own monitor. With two workers, ask each one, or
send the prediction log to one place (C25 does this properly).
"""

import threading
from collections import Counter, deque
from dataclasses import dataclass

import numpy as np

WINDOW = 1000
MIN_FOR_ALERTS = 200


@dataclass
class Scored:
    score: float
    flag: bool
    ms: float
    shadow_score: float | None = None
    shadow_flag: bool | None = None


class Monitor:
    def __init__(self, capacity: float):
        self.capacity = capacity
        self.recent: deque[Scored] = deque(maxlen=WINDOW)
        self.contract_errors: Counter[str] = Counter()
        self.unknown: Counter[str] = Counter()
        self.scored_total = 0
        self.lock = threading.Lock()

    def contract_error(self, fields: list[str]) -> None:
        with self.lock:
            self.contract_errors.update(fields)

    def scored(self, item: Scored, warnings: list[str]) -> None:
        with self.lock:
            self.recent.append(item)
            self.scored_total += 1
            self.unknown.update(w.split(":")[0] for w in warnings)

deque(maxlen=WINDOW) is a list that keeps only the last 1,000 items: when a new item comes in, the oldest one drops out. The lock stops two requests from changing the counts at the same time. The second part computes the report:

escalation/monitor.py (continued)

    def report(self) -> dict:
        with self.lock:
            items = list(self.recent)
            errors, unknown, total = dict(self.contract_errors), dict(self.unknown), self.scored_total
        report = {
            "scored_total": total,
            "window": len(items),
            "contract_errors": errors,
            "unknown_categories": unknown,
            "flag_rate": None,
            "mean_score": None,
            "latency_ms": None,
            "shadow": None,
            "alerts": [],
        }
        if not items:
            return report
        scores = np.array([i.score for i in items])
        flags = np.array([i.flag for i in items])
        ms = np.array([i.ms for i in items])
        report["flag_rate"] = round(float(flags.mean()), 3)
        report["mean_score"] = round(float(scores.mean()), 4)
        report["latency_ms"] = {
            "p50": round(float(np.percentile(ms, 50)), 2),
            "p95": round(float(np.percentile(ms, 95)), 2),
        }
        shadowed = [i for i in items if i.shadow_score is not None]
        if shadowed:
            shadow_flags = np.array([i.shadow_flag for i in shadowed])
            live_flags = np.array([i.flag for i in shadowed])
            report["shadow"] = {
                "compared": len(shadowed),
                "flag_rate": round(float(shadow_flags.mean()), 3),
                "mean_score": round(float(np.mean([i.shadow_score for i in shadowed])), 4),
                "same_flag": round(float((shadow_flags == live_flags).mean()), 3),
            }
        if len(items) >= MIN_FOR_ALERTS and report["flag_rate"] > self.capacity:
            report["alerts"].append(
                f"flag rate {report['flag_rate']:.3f} is above the review capacity {self.capacity:.2f}"
            )
        if report["shadow"] and len(shadowed) >= MIN_FOR_ALERTS and report["shadow"]["flag_rate"] > self.capacity:
            report["alerts"].append(
                f"shadow flag rate {report['shadow']['flag_rate']:.3f} is above the review capacity {self.capacity:.2f}"
            )
        return report

Now connect it to the service, in three files.

escalation/main.py makes one Monitor per process. Add the import, then create it after the bundles are loaded, with the capacity from the bundle’s metadata:

escalation/main.py

from escalation.monitor import Monitor
...
            logger.error("Start-up error: %s", error)
            raise
        app.state.monitor = Monitor(app.state.bundle.metadata["review_capacity"])

Also add one line in create_app(), after app.state.settings = settings. A contract error can happen in a test before start-up, and it needs a monitor to count in:

escalation/main.py

    app.state.settings = settings
    app.state.monitor = Monitor(0.2)  # replaced at start-up; here so that errors can count

escalation/errors.py counts the fields of each contract error, in invalid_request:

escalation/errors.py

        fields = sorted({field_name(e) for e in exc.errors()})
        request.app.state.monitor.contract_error(fields)

escalation/routes.py gives each scored ticket to the monitor. Add import time and from escalation.monitor import Scored at the top. Then change score():

escalation/routes.py

def score(ticket: TicketIn, request: Request, response: Response) -> ScoreOut:
    started = time.perf_counter()
    state = request.app.state
    [result], timings = score_tickets(state.bundle, [ticket])
    response.headers["Server-Timing"] = timings.header()
    item = Scored(result.score, result.flag, (time.perf_counter() - started) * 1000)
    record = { ... }  # as before
    if state.shadow is not None:  # score with the candidate too, but never answer with it
        [shadow], _ = score_tickets(state.shadow, [ticket])
        item.shadow_score, item.shadow_flag = shadow.score, shadow.flag
        record |= {"shadow_version": state.shadow.version, "shadow_score": shadow.score, "shadow_flag": shadow.flag}
    state.monitor.scored(item, result.warnings)
    predictions.info(json.dumps(record))

Then add the route, after GET /v1/model. It uses the same API key as the other /v1 routes:

escalation/routes.py

@router.get("/monitor", tags=["model"], summary="Model health signals for the recent tickets")
def monitor(request: Request) -> dict:
    report = request.app.state.monitor.report()
    report["model_version"] = request.app.state.bundle.version
    return report

Finally, two tests in tests/test_api.py check the counts. Add one line at the end of each test:

tests/test_api.py

def test_a_contract_error_has_the_error_shape(client, ticket):
    ...
    assert client.get("/v1/monitor").json()["contract_errors"] == {"created_hour": 1, "priority": 1}

def test_an_unknown_category_is_scored_with_a_warning(client, ticket):
    ...
    assert client.get("/v1/monitor").json()["unknown_categories"] == {"channel": 1}

Worked example: the shadow run, watched by the service

Amira builds the image again. She repeats the shadow run of the last lesson: 1.0.0 live, rc1 in shadow, the 1,400 July tickets in order. Then she asks the service:

Terminal

curl http://127.0.0.1:8000/v1/monitor

Predict: the window holds the last 1,000 tickets, which are mostly from the second half of July. Is the live flag rate above or below the 20.1% of the whole month?

The output, formatted:

GET /v1/monitor

{
  "scored_total": 1400,
  "window": 1000,
  "contract_errors": {},
  "unknown_categories": {"channel": 91},
  "flag_rate": 0.212,
  "mean_score": 0.1299,
  "latency_ms": {"p50": 2.17, "p95": 3.07},
  "shadow": {"compared": 1000, "flag_rate": 0.87, "mean_score": 0.4371, "same_flag": 0.342},
  "alerts": [
    "flag rate 0.212 is above the review capacity 0.20",
    "shadow flag rate 0.870 is above the review capacity 0.20"
  ],
  "model_version": "1.0.0"
}

Read it from the top:

• contract_errors is empty: every July ticket matched the contract.
• unknown_categories: 91 tickets came from channel social. This count is for all 1,400 tickets, not only the window. It is the same 91 that the machine learning course found.
• flag_rate 0.212: the live model is above the capacity in the last 1,000 tickets. That is higher than the month’s 20.1%, because the window holds mostly late July. The first alert says so.
• mean_score 0.1299: compare it with 0.120, the mean score of May in the machine learning course. It moved up.
• latency_ms: the time to score the live answer inside the service, without HTTP. It does not include the shadow’s work.
• shadow: rc1 would flag 87%, and it agrees with the live flag on only 34.2% of the tickets. The second alert says so.

Now one wrong ticket with priority "high" and created_hour 24: the curl command of Connect the API, with those two values changed. On a new container, the monitor then shows:

"scored_total":0, ... "contract_errors":{"created_hour":1,"priority":1}

The ticket was never scored, but the monitor counted each field that was wrong.

The July flag rate, day by day

Predict: each July day has between 13 and 67 tickets. Does 1.0.0 stay close to 20% every day?

The chart shows the share of each day’s tickets that each version flags. The scores are the real scores of the three bundles, with the same code as the service.

• 1.0.0 (live)
• 1.1.0-rc1
• 1.1.0
• Capacity 20%

The daily flag rate of three versions on the 1,400 July tickets. rc1 is far above the capacity on every day. 1.0.0 and 1.1.0 move together around it, and are above it on most days from 15 July.

Text description of the chart

One point per day of July 2026 for each version, with a dashed line at the capacity of 20%.

Candidate 1.1.0-rc1 flags between 70.8% and 94.5% of each day’s tickets: far above the line on every day.

Version 1.0.0, the live model, moves between 4.5% (4 July, 22 tickets) and 35.3% (24 July). From 1 to 14 July it is above 20% on 5 days of 14. From 15 to 21 July it is above 20% on 6 of 7 days, between 27.3% and 30.3%. The exception is 20 July, with 17.2%.

Version 1.1.0 follows 1.0.0 closely, a little lower on most days. It is also above 20% on most days from 15 July. Over the month, 1.0.0 flags 20.1%, 1.1.0 flags 19.4% and rc1 flags 86.1%.

Three things stand out:

1. rc1 is wrong on every single day. No window or minimum is needed to see it. A shadow run of one day would have been enough.
2. One day is noisy for 1.0.0. On 4 July it flagged 4.5% of 22 tickets; on 24 July, 35.3%. A rule on one day would raise false alarms. This is why the monitor uses a window of 1,000 tickets, and the machine learning course’s trigger waited for 5 working days in a row.
3. 1.1.0 does not fix late July. It moves with 1.0.0 because the cause is in the inputs, not in the model. Retraining alone does not repair a broken export.

When the labels arrive

72 hours after each ticket, the help desk knows whether it was escalated. Mei delivers the labels as july_labels.csv, as in the machine learning course. Copy it into the project:

Terminal, in ~/projects/escalation-service

cp ../ticket-model/data/july_labels.csv data/

monitoring/outcomes.py reads the prediction lines from a log and joins them with the labels by ticket_id. Then it computes precision and recall for each model version, and for the shadow.

Show monitoring/outcomes.py

monitoring/outcomes.py

"""When the true answers arrive, check how the served model really did.

    docker compose logs escalation --no-log-prefix > predictions.log
    python monitoring/outcomes.py predictions.log data/july_labels.csv

Reads the prediction lines that the service writes (one JSON object per
scored ticket, from the logger escalation.predictions), joins them with the
labels by ticket_id, and prints precision and recall for each model version
(and for the shadow model, if there was one). Tickets without a label yet are
counted, not guessed.
"""

import argparse
import json

import pandas as pd

MARK = "escalation.predictions: "


def read_predictions(path) -> pd.DataFrame:
    rows = []
    with open(path, encoding="utf-8") as log:
        for line in log:
            if MARK in line:
                rows.append(json.loads(line.split(MARK, 1)[1]))
    return pd.DataFrame(rows).drop_duplicates("ticket_id", keep="last")


def report(name: str, flags: pd.Series, truth: pd.Series) -> str:
    caught = int((flags & truth).sum())
    precision = caught / flags.sum() if flags.sum() else float("nan")
    recall = caught / truth.sum() if truth.sum() else float("nan")
    return (f"{name}: flagged {int(flags.sum())} of {len(flags)} ({flags.mean():.1%}), "
            f"precision {precision:.3f}, recall {recall:.3f}")


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("predictions")
    parser.add_argument("labels")
    args = parser.parse_args()

    predictions = read_predictions(args.predictions)
    labels = pd.read_csv(args.labels)
    joined = predictions.merge(labels, on="ticket_id", how="left")
    waiting = joined["escalated_72h"].isna().sum()
    known = joined.dropna(subset=["escalated_72h"])
    truth = known["escalated_72h"].astype(int) == 1
    print(f"{len(predictions)} predictions, {len(known)} with a label, {waiting} still waiting for one.")
    print(f"Escalated: {int(truth.sum())} ({truth.mean():.1%}).")
    for version, group in known.groupby("model_version"):
        print(report(f"Model {version}", group["flag"].astype(bool), truth[group.index]))
    if "shadow_version" in known:
        for version, group in known.dropna(subset=["shadow_version"]).groupby("shadow_version"):
            print(report(f"Shadow {version}", group["shadow_flag"].astype(bool), truth[group.index]))


if __name__ == "__main__":
    main()

Run it with the log of the shadow run:

Terminal, with .venv active

python monitoring/outcomes.py predictions.log data/july_labels.csv

The output:

1400 predictions, 1400 with a label, 0 still waiting for one.
Escalated: 196 (14.0%).
Model 1.0.0: flagged 282 of 1400 (20.1%), precision 0.333, recall 0.480
Shadow 1.1.0-rc1: flagged 1205 of 1400 (86.1%), precision 0.159, recall 0.974

These are the numbers that matter most for Grace. 1.0.0 catches 48% of the escalations, and one flagged ticket in three is a real escalation. rc1 catches 97% of them, but only one flagged ticket in six is real. It “finds” more escalations because it flags almost everything.

Omar: “Then why not watch only precision and recall?” Because they arrive late. The labels of 31 July arrive on 3 August. The flag rate crossed 20% on 15 July, at once.

What the monitor cannot see

The monitor is small on purpose. Know its limits before you trust it:

• It lives in one process’s memory. Each worker has its own monitor, and a restart sets every count to zero. The failure case below shows two workers.
• The window is the last 1,000 tickets. At about 45 tickets a day, that is about three weeks, and the minimum of 200 tickets is four or five days. A sudden change shows slowly: on its first day, it is less than 5% of the window.
• It alerts only above the capacity. A version that flags too few tickets gets no alert. Its recall falls, and only outcomes.py shows it, three days later.
• A signal shows that something changed, not why. The cause of late July was in the data export. A person had to find it.

A team with many services sends the prediction lines and the signals to one central place, with dashboards and alerts. The course MLOps and LLMOps: Operating AI in Production does that. The idea stays the same.

Guided practice: watch the shadow run

1. Add escalation/monitor.py and change main.py, errors.py, routes.py and tests/test_api.py, as in monitor.py.
2. Run python -m pytest. You should see 42 passed.
3. Add monitoring/outcomes.py, and copy july_labels.csv into data/.
4. Build the image with docker compose build.
5. Start the shadow container and send the July tickets, as in steps 3 and 4 of Version and rollout: Worked example.
6. Call GET /v1/monitor. Compare flag_rate, shadow and alerts with the output above.
7. Send the example ticket with created_hour 24, with the curl command of Connect the API. Call GET /v1/monitor again, and find the count in contract_errors.
8. Save the log to predictions.log, then run outcomes.py.
9. Remove the container with docker rm -f escalation-shadow.

Show the expected results

Step 6 gives the JSON of the worked example: "flag_rate": 0.212, the shadow’s "flag_rate": 0.87, and two alerts. Your latency_ms values are different: they depend on your computer.

Step 7 adds "contract_errors": {"created_hour": 1}, and scored_total stays 1400.

Step 8 gives the four lines of When the labels arrive.

Your turn: not every label has arrived

On 23 July, the labels exist only for the tickets of 1 to 20 July. Make a copy of july_labels.csv with only those tickets, as labels_to_20_july.csv. It has 887 rows. Predict what the first line of outcomes.py says, then run it with this file.

Show the answer

One way to make the file, in a Python session with pandas:

import pandas as pd
tickets = pd.read_csv("data/july_tickets.csv", parse_dates=["created_at"])
labels = pd.read_csv("data/july_labels.csv")
early = tickets.loc[tickets["created_at"].dt.day <= 20, "ticket_id"]
labels[labels["ticket_id"].isin(early)].to_csv("labels_to_20_july.csv", index=False)

The output of python monitoring/outcomes.py predictions.log labels_to_20_july.csv:

1400 predictions, 887 with a label, 513 still waiting for one.
Escalated: 129 (14.5%).
Model 1.0.0: flagged 179 of 887 (20.2%), precision 0.341, recall 0.473
Shadow 1.1.0-rc1: flagged 761 of 887 (85.8%), precision 0.164, recall 0.969

The script counts the 513 tickets without a label. It does not guess them, and it does not count them as “not escalated”. If it did, recall would look better and precision worse than they are.

Common mistake

Two workers, two different answers

Symptom: with WORKERS=2 and CPUS=2.0 in .env, Amira sends the 1,400 July tickets, then calls GET /v1/monitor six times. The answers do not agree:

scored_total 935, flag_rate 0.206, unknown {'channel': 62}, alerts: ['flag rate 0.206 is above the review capacity 0.20']   (4 times)
scored_total 465, flag_rate 0.191, unknown {'channel': 29}, alerts: []                                                     (2 times)

Cause: each worker is a separate process with its own Monitor in its own memory. Each request goes to one of the two workers. 935 + 465 = 1,400: each worker saw a part of the traffic, and each one answers for its part only. One of them alerts, the other does not.

Fix: for this course, run one worker, as Module 4 chose for Larkfield’s load. With more workers or more containers, compute the signals from the prediction log of all of them together, as outcomes.py does. A central monitoring system does this for you (see the MLOps course).

For a real AI service: an alert that depends on which worker answers is worse than no alert, because it looks like a decision. Know where your counts live before you act on them.

Check your understanding

The monitor shows flag_rate 0.40 after 10 tickets, and no alert. Is that a bug? Explain.

Show the answer

No. With 10 tickets, one flag moves the rate by 10 points, so 40% can be chance. The monitor alerts only when its window holds at least 200 tickets. The daily July chart shows the same effect. On single days, 1.0.0 flagged between 4.5% and 35.3%, but over the month it flagged 20.1%.

Check your understanding

Which health signals can the service compute at once, and which need the labels? Why does the difference matter?

Show the answer

At once: contract errors, unknown categories, latency, the flag rate and the mean score. They need no labels. Precision and recall need to know whether each ticket was escalated, and that is known 72 hours later. The early signals are the warning; precision and recall are the confirmation. In July, the flag rate crossed 20% on 15 July, but the labels of 31 July arrived on 3 August.

Check your understanding

A new version flags 9% of tickets: far fewer than before. Does the monitor alert? What would show the problem?

Show the answer

No. monitor.py alerts only when the flag rate is above the review capacity. The mean score falls, which a person can see in /v1/monitor, but no alert says so. When the labels arrive, outcomes.py shows the low recall: the model misses escalations. A team could add a lower limit to the alerts, for example 10%.

Recap

• Model health is watched through signals. At once: contract errors, unknown categories, latency, the flag rate against the capacity, and the mean score. Later, with labels: precision and recall.
• The flag rate and the mean score are drift indicators: they move when production data moves away from the training data.
• monitor.py counts the signals over the last 1,000 tickets and alerts only after 200, because small samples are noisy. GET /v1/monitor shows them.
• In the shadow run, the monitor raised two alerts: 1.0.0 at 21.2% and rc1 at 87%.
• outcomes.py joins the prediction log with the labels: 1.0.0 had precision 0.333 and recall 0.480 in July.
• The monitor lives in one process: two workers give two different answers.

Next, rc1 goes live on purpose, in a drill, and you take it back.

Next: Rollback and retirement