Version and rollout

What you will be able to do

Release a new model version without risk to the help desk. You give each release a name and a digest, and you test the candidate on real tickets while the old model still answers.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The retraining plan of Machine Learning: From Problem to Reliable Model: retrain with the tickets up to May, choose the threshold on June. See Plan production checks: A written retraining trigger.
• The threshold 0.19 and the review capacity of 20%, from Thresholds and calibration.
• Bundles and their digests, from Bundle model dependencies and Artifact trust and integrity.
• The prediction log line and GET /v1/model, from Connect the API. Parity, from Check parity.
• compose.yaml, .env and bench/bench.py, from Container resource settings and Benchmark the service.

What you need on your computer

• ~/projects/ticket-model from the machine learning course, with model/ trained by python train.py (Package artifacts).
• ~/projects/escalation-service at the end of Module 4, with its .venv, .env and the image escalation-service:1.0.0. 42 tests pass.
• In .env, WORKERS=1 and CPUS=1.0. The Module 4 practice ended with CPUS=0.5: set it back, because this module’s times were measured with one CPU.
• If your project is missing or broken, copy the snapshot nextia-learning-labs/C17/snapshots/end-of-m04. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

A new pilot flies with a captain first

A new airline pilot has passed every exam. Even so, the first flights are with an experienced captain in the next seat. The new pilot does the work, and the captain can take control at any moment. The airline learns how the new pilot flies real routes before any passenger depends on them alone.

A new model version needs the same thing. Its tests on old data are its exams. The question is how it behaves on today’s tickets.

The candidate passes every test

The machine learning course planned a retraining: fit the same pipeline on the tickets up to May, and choose the threshold on June (Plan production checks). The script retrain.py does it. It goes in ~/projects/ticket-model, next to train.py, because it uses the data and functions of the machine learning course. (The snapshot end-of-m05 keeps a copy in training/retrain.py.)

Show retrain.py (save it in ~/projects/ticket-model)

retrain.py

"""Retrain the escalation model with the newer tickets, for a new version.

Nextia Learning, C17, Module 5. Put this file in your C05 ticket-model folder,
next to train.py, and run it there:

    python retrain.py --out model-1.1.0

C05 decided (Plan production checks) to retrain with the May and June tickets
after the warranty policy changed. This script fits the same pipeline on
train + valid (July 2024 to May 2026), and chooses the threshold for the 20%
review capacity on the June tickets (test), which the new model has not seen.
It writes the same three files as train.py, so make_bundle.py can read them.

    --balanced         weight the rare escalations more (class_weight="balanced")
    --threshold 0.19   keep a fixed threshold instead of choosing a new one

Release candidate 1.1.0-rc1 was made with both options. Together they are a
mistake: balanced weights move every score up, so the old threshold flags
far more tickets than the senior team can review.
"""

import argparse
import json
import platform
from datetime import datetime, timezone
from pathlib import Path

import joblib
import numpy as np
import pandas as pd
import sklearn
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score

from ticket_model import CATEGORIES, DATA, FEATURES, NUMBERS, TARGET, build_pipeline, load
from train import REVIEW_CAPACITY, choose_threshold, sha256


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--out", type=Path, required=True)
    parser.add_argument("--balanced", action="store_true", help='use class_weight="balanced"')
    parser.add_argument("--threshold", type=float, help="keep this threshold (not recommended)")
    args = parser.parse_args()

    fit_on = pd.concat([load("train"), load("valid")], ignore_index=True)
    june = load("test")
    model = LogisticRegression(max_iter=1000, class_weight="balanced") if args.balanced else None
    pipeline = build_pipeline(model).fit(fit_on[FEATURES], fit_on[TARGET])
    scores = pipeline.predict_proba(june[FEATURES])[:, 1]
    threshold = args.threshold if args.threshold is not None else choose_threshold(scores, REVIEW_CAPACITY)
    flagged = scores >= threshold
    caught = int((flagged & (june[TARGET] == 1)).sum())
    results = {
        "rows": len(june), "escalated": int(june[TARGET].sum()),
        "roc_auc": round(roc_auc_score(june[TARGET], scores), 3),
        "average_precision": round(average_precision_score(june[TARGET], scores), 3),
        "flagged": int(flagged.sum()), "flag_rate": round(flagged.mean(), 3),
        "precision": round(caught / max(flagged.sum(), 1), 3),
        "recall": round(caught / june[TARGET].sum(), 3),
    }

    args.out.mkdir(exist_ok=False)
    joblib.dump(pipeline, args.out / "escalation_model.joblib")
    categories = pipeline.named_steps["prepare"].named_transformers_["categories"].categories_
    info = {
        "model": "Larkfield ticket escalation: logistic regression in a scikit-learn pipeline, "
                 "retrained with the tickets up to May 2026",
        "trained_at": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC"),
        "target": f"{TARGET}: 1 if the ticket is escalated within 72 hours of creation",
        "prediction_time": "when the ticket is created",
        "features": {
            "categories": {name: [str(v) for v in values] for name, values in zip(CATEGORIES, categories)},
            "numbers": NUMBERS,
        },
        "threshold": threshold,
        "threshold_chosen_on": "fixed by hand" if args.threshold is not None else "test (June 2026)",
        "review_capacity": REVIEW_CAPACITY,
        "valid": results,
        "evaluated_on": "test (June 2026)",
        "train_rows": len(fit_on),
        "data_sha256": {name: sha256(DATA / name) for name in
                        ("train.csv", "valid.csv", "test.csv", "extra_features.csv")},
        "artifact_sha256": sha256(args.out / "escalation_model.joblib"),
        "versions": {"python": platform.python_version(), "pandas": pd.__version__,
                     "scikit-learn": sklearn.__version__, "numpy": np.__version__, "joblib": joblib.__version__},
    }
    (args.out / "model_info.json").write_text(json.dumps(info, indent=2) + "\n", encoding="utf-8")
    print(f"Trained on {len(fit_on)} tickets (to the end of May). Threshold {threshold}"
          f"{' (fixed by hand)' if args.threshold is not None else ' for a review capacity of 20%'}.")
    print("June: ROC AUC {roc_auc}, average precision {average_precision}, flagged {flagged} of {rows} "
          "({flag_rate:.1%}), precision {precision}, recall {recall}.".format(**results))
    print(f"Saved {args.out}/escalation_model.joblib and model_info.json.")


if __name__ == "__main__":
    main()

The new version is 1.1.0: the middle number grows because the model changed, while the contract stays the same. Tomás tried one more idea: weight the rare escalations more, with class_weight="balanced". He kept the old threshold, 0.19. He calls his version 1.1.0-rc1. “rc” means release candidate: a version that may become the release. Priya made the planned version, 1.1.0, without these two options:

Terminal, in ~/projects/ticket-model

python retrain.py --out model-1.1.0
python retrain.py --out model-1.1.0-rc1 --balanced --threshold 0.19

The output:

Trained on 18118 tickets (to the end of May). Threshold 0.2 for a review capacity of 20%.
June: ROC AUC 0.749, average precision 0.346, flagged 193 of 1002 (19.3%), precision 0.363, recall 0.446.
Saved model-1.1.0/escalation_model.joblib and model_info.json.
Trained on 18118 tickets (to the end of May). Threshold 0.19 (fixed by hand).
June: ROC AUC 0.751, average precision 0.343, flagged 851 of 1002 (84.9%), precision 0.177, recall 0.962.
Saved model-1.1.0-rc1/escalation_model.joblib and model_info.json.

Tomás: “The AUC is 0.751, the best of the two. The contract and the parity checks pass, and every test passes. Ship it.”

Look at the flagged tickets in June: 851 of 1,002 (84.9%). Grace’s senior team can review 20%. With rc1 live, the team would get four times its capacity.

Why the AUC did not catch it

Predict: ROC AUC measures how well a model puts escalated tickets above the others. Can a model have the same AUC and flag four times as many tickets?

The chart shows six real July tickets, scored by both models. The tickets are in the order of the 1.0.0 score.

• 1.0.0
• 1.1.0-rc1
• Threshold 0.19

Six July tickets. Both models put them in the same order, so the ranking is as good. But rc1 moved every score up, and four of the six are now above 0.19.

Text description of the chart

Six July tickets, numbered 1 to 6 in the order of their 1.0.0 score. The 1.0.0 scores are 0.020, 0.026, 0.038, 0.069, 0.106 and 0.255. The rc1 scores of the same tickets are 0.124, 0.173, 0.224, 0.344, 0.502 and 0.734. Both lines go up from left to right, so both models rank the tickets in the same order. A dashed line marks the threshold 0.19. Model 1.0.0 flags one ticket of six (ticket 6). Candidate rc1 flags four (tickets 3 to 6).

AUC compares only the order, so it stays the same. But balanced class weights tell the model to treat escalations as if they were as common as the other tickets. So every score moves up. The threshold 0.19 was chosen for the old scores, and now it cuts far too low.

The lesson is short: the threshold belongs to the model version. A new model needs a threshold chosen on its own scores, as retrain.py does without --threshold. A check of the flag rate against the capacity catches rc1. A check of the AUC cannot.

A release is a version and a digest

Before any test on live tickets, each candidate becomes a bundle, in its own folder:

Terminal, in ~/projects/escalation-service

python make_bundle.py ../ticket-model --version 1.1.0-rc1 --model-dir model-1.1.0-rc1
python make_bundle.py ../ticket-model --version 1.1.0 --model-dir model-1.1.0

The output (the course’s bundles in labs):

Made bundles/escalation-1.1.0-rc1: 14 parity cases, threshold 0.19.
Bundle digest (MODEL_SHA256): fa04bc40bea99723095ec4db600b7060fbfce0f2ec461534bb06c1829b47fe06
Made bundles/escalation-1.1.0: 14 parity cases, threshold 0.2.
Bundle digest (MODEL_SHA256): 382077148b6cad58238e4832f0951c6556f82832bc6e7de1d3a2335bd6d36af0

Your digests are different, because metadata.json holds your own trained_at. Use your own digests everywhere below.

Now there are three versions. Omar asks: “Which one answers the help desk today, and who said yes?” After a month, nobody can answer from memory. A car has the same problem, and a service book solves it: each service, its date, what was done, and who signed. So Amira writes it down.

A model registry is the service book of a model. It lists each version: what it is, where it is, its status, and the evidence for that status. Tools such as MLflow and Azure Machine Learning keep a registry for a team, and MLOps and LLMOps: Operating AI in Production uses one. For one model, a file in the project is enough. Amira makes RELEASES.md:

RELEASES.md

# Releases of the escalation model

| Version | Bundle digest (MODEL_SHA256) | Status | Since | Evidence | Approved by |
|---|---|---|---|---|---|
| 1.0.0 | `8f233029aa4c...3b74` | live | 2026-10-08 | C05 validation: ROC AUC 0.752, flagged 19.3% of May. Parity 14 of 14. | Grace, Priya |
| 1.1.0-rc1 | `fa04bc40bea9...fe06` | candidate | 2026-10-08 | June: ROC AUC 0.751, flagged 84.9%. | not yet |
| 1.1.0 | `382077148b6c...6af0` | candidate | 2026-10-08 | June: ROC AUC 0.749, flagged 19.3%. | not yet |

Write the full 64-character digests in your file; this page shortens them. Each row names the model by its version and its digest. The version is for people. The digest proves which bytes ran, as the image digest did in the Docker course. A version is never changed: a fix is a new row.

The Status column follows a fixed path. The diagram shows every status, and the check that moves a version to the next one.

The release states of a model version. Each arrow is a check or a decision, and each one goes into RELEASES.md.

Text description of the diagram

A version starts as a candidate when make_bundle.py makes its bundle. If the tests and the parity checks pass, it goes to shadow. If a check fails, it is retired.

In shadow, the version scores real tickets, but it never answers. If its signals are good and Grace approves, it goes live. If its signals fail, it is retired.

When a newer version goes live, the old live version becomes known good: it stays ready, so the team can go back to it. A rollback makes it live again. When the newer version has proved itself, the known-good version is retired.

If a live version causes an alert, it is rolled back. It is retired later, when the cause is known. Lesson 3 practises this branch.

Shadow: score with both, answer with one

How can rc1 meet real tickets without any risk to Grace’s team? Do what the airline does: let it do the work next to the experienced model, and keep the experienced model in control.

A shadow deployment runs the candidate next to the live model. For each request, the service scores the ticket with both models. It answers with the live model only, and it writes both scores in the prediction log. The help desk sees no change. The team sees how the candidate behaves on real traffic. The managed endpoints of Module 4 offer the same idea as traffic mirroring.

One shadow request from the real run: one ticket, two models, one answer. The candidate's score goes only into the log.

Text description of the diagram

Ticket T-120631 arrives at the service. The live model 1.0.0 scores it 0.082, under the threshold. The shadow model 1.1.0-rc1 scores the same ticket 0.373, over the threshold. The service writes one log line with both scores. The help desk receives 0.082 and “not flagged”, from model 1.0.0. The shadow’s opinion changes nothing for the help desk.

The service needs three small changes. Each one names a file and the reason.

escalation/settings.py reads two new, optional settings. Add the two fields to Settings, after model_sha256:

escalation/settings.py

    model_bundle: Path
    model_sha256: str
    shadow_bundle: Path | None
    shadow_sha256: str | None

In load_settings(), read them after the check of MODEL_SHA256, and pass them to Settings:

escalation/settings.py

    shadow = env.get("SHADOW_BUNDLE") or None
    shadow_sha = env.get("SHADOW_SHA256", "").strip().lower() or None
    if shadow and (shadow_sha is None or len(shadow_sha) != 64):
        raise SettingsError("SHADOW_BUNDLE is set, so SHADOW_SHA256 must be its digest")
    settings = Settings(
        model_bundle=Path(env.get("MODEL_BUNDLE", "bundles/escalation-1.0.0")),
        model_sha256=sha,
        shadow_bundle=Path(shadow) if shadow else None,
        shadow_sha256=shadow_sha,

The shadow bundle goes through the same six gates as the live bundle, with its own digest. A candidate does not get more trust because it only shadows: loading it still runs code in the live service.

escalation/main.py loads the shadow at start-up, inside the same try, and names it in the log:

escalation/main.py

        try:
            app.state.bundle = load_bundle(settings.model_bundle, settings.model_sha256)
            app.state.shadow = None
            if settings.shadow_bundle:
                app.state.shadow = load_bundle(settings.shadow_bundle, settings.shadow_sha256)
        except BundleError as error:
            logger.error("Start-up error: %s", error)
            raise
        # ... app.state.jobs = JobRunner(...) as before ...
        shadow = f", shadow {app.state.shadow.version}" if app.state.shadow else ""
        logger.info(
            "model %s loaded (bundle %s...)%s in %.0f ms",
            app.state.bundle.version,
            app.state.bundle.digest[:12],
            shadow,
            (time.perf_counter() - started) * 1000,
        )

escalation/routes.py scores with the shadow in POST /v1/score, after the live score and before the log line. Add this block after the record = {...} dictionary:

escalation/routes.py

    if state.shadow is not None:  # score with the candidate too, but never answer with it
        [shadow], _ = score_tickets(state.shadow, [ticket])
        record |= {"shadow_version": state.shadow.version, "shadow_score": shadow.score, "shadow_flag": shadow.flag}

The answer after the block does not change: it uses result, the live score. In the same file, ModelInfo gets one field, shadow_version: str | None. GET /v1/model fills it with shadow.version if shadow else None, where shadow is request.app.state.shadow. Then anybody can ask the service which candidate it shadows.

Worked example: rc1 in shadow on the July tickets

Amira sends the 1,400 July tickets to the service, in order, with 1.0.0 live and rc1 in shadow. The July tickets are synthetic, like all of Larkfield’s data. compose.yaml does not pass the shadow settings, so she starts the container with docker run for this test.

1. Stop the Compose service, because it uses port 8000:

   Terminal, in ~/projects/escalation-service

   docker compose down

2. Build the image again. It now has the three bundles and the new code:

   Terminal

   docker compose build

   The tag stays escalation-service:1.0.0, because compose.yaml sets it. From now on, the image holds more than one bundle, and the settings choose one at start. In Compose, MODEL_VERSION picks the bundle folder, and MODEL_SHA256 must be its digest. In Module 4 the tag named the code and its one model. Now the tag names the code, and the digest names the model. Lesson 3 uses this to go back to an older model without a new build.

3. Start the service with rc1 in shadow. Use your own digests:

   macOS/Linux

   Terminal

   docker run -d --name escalation-shadow -p 127.0.0.1:8000:8000 --cpus 1 --memory 512m \
     -e MODEL_SHA256=8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74 \
     -e SHADOW_BUNDLE=bundles/escalation-1.1.0-rc1 \
     -e SHADOW_SHA256=fa04bc40bea99723095ec4db600b7060fbfce0f2ec461534bb06c1829b47fe06 \
     escalation-service:1.0.0

   Windows

   PowerShell

   docker run -d --name escalation-shadow -p 127.0.0.1:8000:8000 --cpus 1 --memory 512m `
     -e MODEL_SHA256=8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74 `
     -e SHADOW_BUNDLE=bundles/escalation-1.1.0-rc1 `
     -e SHADOW_SHA256=fa04bc40bea99723095ec4db600b7060fbfce0f2ec461534bb06c1829b47fe06 `
     escalation-service:1.0.0

4. Send each July ticket once, in file order. --warmup 0 makes every request count:

   Terminal, with .venv active

   python bench/bench.py http://127.0.0.1:8000 --requests 1400 --warmup 0

5. Save the container’s log, with the prediction lines, to a file:

   macOS/Linux

   Terminal

   docker logs escalation-shadow > predictions.log 2>&1

   Windows

   PowerShell

   docker logs escalation-shadow 2>&1 | Out-File -Encoding utf8 predictions.log

The start-up line in the log names both models:

INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...), shadow 1.1.0-rc1 in 937 ms

The service now checks and loads two bundles, so its start-up takes longer. One prediction line from the log:

INFO escalation.predictions: {"request_id": "657a0d114f22", "ticket_id": "T-120631", "model_version": "1.0.0", "score": 0.08204913759363192, "flag": false, "shadow_version": "1.1.0-rc1", "shadow_score": 0.37341856790793543, "shadow_flag": true}

Predict: 1.0.0 flagged 20.1% of July in the batch job of Module 2. How many of the 1,400 tickets did each model flag here?

Count the lines with a flag:

macOS/Linux

Terminal

grep -c '"flag": true' predictions.log
grep -c '"shadow_flag": true' predictions.log

Windows

PowerShell

(Select-String '"flag": true' predictions.log).Count
(Select-String '"shadow_flag": true' predictions.log).Count

The output:

282
1205

The live model flagged 282 tickets (20.1%), the same as the batch job. rc1 would have flagged 1,205 (86.1%). Every ticket that 1.0.0 flagged, rc1 flagged too, and 923 more. Grace’s team saw none of them: every answer came from 1.0.0. Amira writes the result in RELEASES.md, and rc1 does not go live.

Caution

A shadow doubles the scoring work. On the test computer, the benchmark lines for 1,400 requests, one at a time, were:

1.0.0 only:          p50 3.5 ms, p95 7.3 ms, ... 233.3 requests/s, 0 errors.
1.0.0 + rc1 shadow:  p50 5.2 ms, p95 7.1 ms, ... 181.2 requests/s, 0 errors.

Five pairs of runs gave between 1.6 and 3.1 ms more at p50, because the computer was busy with other work. For Larkfield’s 45 tickets a day, this does not matter. For a large model, a shadow can double the cost of the service. Turn it off when the test ends.

Remove the test container. The next lessons use Compose again:

Terminal

docker rm -f escalation-shadow

Canary: a small share of real answers

A shadow never answers, so it cannot show what the candidate’s answers do. The next step for many teams is a canary. The new version answers a small share of the real requests, for example 1 in 10, and the old version answers the rest. If the canary’s numbers go wrong, few users are affected. The name comes from the birds that miners took into mines: a problem shows in the bird before it reaches the people.

What would a 10% canary of rc1 do to Grace’s team? Work it out with the July rates. 90% of the tickets get the 20.1% of 1.0.0, and 10% get the 86.1% of rc1:

0.9 × 20.1% + 0.1 × 86.1% = 18.1% + 8.6% = 26.7% of all tickets flagged

Even at 10%, rc1 pushes the team above its capacity. And each of those flags is a real ticket that a senior person must open.

	Shadow	Canary
Who gets the candidate’s answer	Nobody (log only)	A small share of real requests
Risk to users	None	Small, but real
What you learn	How it scores real inputs	How it scores, and what its answers do
Cost	Double scoring work for every request	A second version to run and watch

Use a shadow first, when a wrong answer has a cost for a person, as a flag does here. Use a canary after the shadow looks good, to learn what a log cannot show. A canary needs a way to split the traffic. If you took the CI/CD course, its lesson Progressive release does this with revisions and traffic weights. This course does not need it.

Guided practice: two candidates, one registry, one shadow

1. Save retrain.py in ~/projects/ticket-model, next to train.py. It is in The candidate passes every test.
2. In ~/projects/ticket-model, activate its .venv. Run both retrain.py commands. Compare your two June lines with the output above.
3. In ~/projects/escalation-service, make both bundles, as in A release is a version and a digest. Copy the two digests.
4. Make RELEASES.md with three rows. Use your own three digests. The digest of 1.0.0 is the MODEL_SHA256 in your .env.
5. Change settings.py, main.py and routes.py, as in Shadow: score with both, answer with one.
6. Run python -m pytest. You should see 42 passed: the tests run without a shadow.
7. Do the five steps of the worked example. Then count the flags.
8. Before you remove the container, run curl http://127.0.0.1:8000/v1/model, as in Connect the API. Check that shadow_version is "1.1.0-rc1".
9. In RELEASES.md, add the shadow result to the evidence of rc1. Then remove the container.

Show the expected results

Step 2 gives the two June lines of the output above: 19.3% flagged for 1.1.0, 84.9% for rc1. On the same computer and versions, your models are the same, byte for byte, as the course’s models. Only the bundle digests differ.

Step 7 gives 282 and 1205. Step 8 gives (shortened):

{"model_version":"1.0.0","bundle_sha256":"8f233029aa4c...","threshold":0.19,...,"shadow_version":"1.1.0-rc1"}

The evidence of rc1 in step 9 can be: “Shadow run, July: 1,205 of 1,400 flagged (86.1%). Not made live.”

Your turn: shadow 1.1.0

Start the container again with 1.1.0 in shadow instead of rc1, and send the July tickets. Before you count, predict: does 1.1.0 flag more or fewer tickets than 1.0.0?

Show the answer

INFO escalation: model 1.0.0 loaded (bundle 8f233029aa4c...), shadow 1.1.0 in 726 ms

The counts are 282 and 272. Version 1.1.0 flags 272 tickets (19.4%), a few fewer than the live model. Priya chose its threshold, 0.20, on its own June scores, so its flag rate stays close to 20%. This is the honest result: 1.1.0 is only a little different from 1.0.0. The next lesson shows that it also goes above 20% in late July.

Common mistake

The service does not start: SHADOW_SHA256 is missing

Symptom: you start the container with SHADOW_BUNDLE, but without -e SHADOW_SHA256=.... The container stops at once, with exit code 1. docker logs shows:

Start-up error: SHADOW_BUNDLE is set, so SHADOW_SHA256 must be its digest

Cause: load_settings() refuses a shadow bundle without its digest. Without the digest, the service cannot check the candidate’s files before it loads them.

Fix: add -e SHADOW_SHA256= with the digest from RELEASES.md, and start again. Copy the digest from the registry, not from the bundle folder: a folder cannot vouch for itself.

For a real AI service: a candidate in shadow runs inside the live service. A bad or changed candidate file can stop the live service, or run code in it. So a shadow needs the same trust checks as a live model.

Check your understanding

rc1 has a ROC AUC of 0.751 on June and passes every test. Why was it still wrong for Larkfield?

Show the answer

AUC measures only the order of the scores. Balanced class weights moved every score up but kept the order, so the AUC stayed the same. The old threshold 0.19 fits the old scores. So rc1 flagged 84.9% of June and 86.1% of July: four times the senior team’s capacity of 20%. The threshold belongs to the model version, and the release check must compare the flag rate with the capacity.

Check your understanding

Omar asks why RELEASES.md stores the digest when the version name is already there. What do you answer?

Show the answer

The version is a name that people choose. Two different bundles can carry the same name, for example a bundle made again on another computer. The digest comes from the files themselves, so it proves exactly which bytes ran. The service checks the digest at start-up, so you can compare the registry with the running service.

Check your understanding

Grace says: no candidate may send a single extra ticket to my team before we see its numbers. Shadow or canary? Why?

Show the answer

Shadow. In a shadow deployment the candidate scores every ticket, but only the live model answers, so the candidate’s flags go only into the log. A canary answers a share of real tickets: a 10% canary of rc1 would flag about 26.7% of all tickets.

Recap

• retrain.py made two candidates. rc1 ranked as well as 1.0.0 (AUC 0.751), but it flagged 86% of July: the threshold belongs to the model version.
• Each release is a version and a digest. RELEASES.md is your model registry: version, digest, status, date, evidence and who approved.
• A version moves through fixed states: candidate, shadow, live, known good, retired, or rolled back.
• A shadow scores every request with the candidate and answers with the live model. Here it cost about 2 ms more per request.
• A canary answers a small share of real requests. Use it after a good shadow, when a wrong answer is acceptable for a few users.

Next, the service watches its own numbers, so that nobody needs grep to see that something is wrong.

Next: Monitor model health