Final assignment

What you will be able to do

Hand the escalation service to the team that will run it, with evidence. You release a new model version, prove that the batch job and the online service give the same scores for the same tickets, measure what the service needs, and prove that you can go back to the previous version.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

• You completed Modules 1 to 5. Each part below names the lessons that it uses.
• The case studies of Module 6 are not needed for this assignment. The certificate asks for at least one of them as well.

What you need on your computer

• Your escalation-service project as at the end of Rollback and retirement: bundles escalation-1.0.0 and escalation-1.1.0, RELEASES.md, compose.yaml, .env, bench/bench.py, parity/check_parity.py, data/july_tickets.csv.
• Docker, as in Container resource settings.
• If your project is missing or broken, copy the snapshot end-of-m05. Use a snapshot has the steps. A snapshot has no .env: make it again from .env.example.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

The situation

Grace’s help desk will depend on the escalation score every day. Before the platform team accepts the service, they ask for four things:

1. Compatible artifacts. The bundle that runs is the one that was reviewed, and it fits the service’s contract.
2. Parity. The nightly batch job and the online service give the same score and the same flag for the same ticket.
3. Measured latency and resources. Numbers from a stated workload, not guesses, and a conclusion that does not go beyond them.
4. A working rollback plan. A drill that goes back to the previous version and checks that everything works after it.

Rules for every part

• Make a file evidence.md outside the project folder, for example ~/projects/c17-evidence.md. After each part, copy into it the commands and outputs that the part asks for.
• Never copy an API key into the evidence.
• Each number in your evidence comes from a command that you ran. Write down the conditions: the computer, the CPU and memory limits, and what else was running.

Part 1: release 1.1.0 with compatible artifacts

Uses Bundle model dependencies, Artifact trust and integrity and Version and rollout.

1. Show that the 1.1.0 bundle has not changed since it was made. Compute its digest and compare it with the one in RELEASES.md.
2. Show that 1.1.0 and 1.0.0 have the same contract: the same features in the same order and the same schema. Only the version and the threshold may differ. Write a short Python command that loads both contract.json files and prints what differs.
3. Make 1.1.0 the live version: set MODEL_VERSION and MODEL_SHA256 in .env, then start the service with docker compose up -d --build --wait.
4. Show GET /v1/model. Its model_version and bundle_sha256 must equal your RELEASES.md entry.
5. Update RELEASES.md: 1.1.0 is live, 1.0.0 is the known-good version to go back to.

What you should see

The digests are equal. The contracts differ only in model_version and in output.threshold (0.2 against 0.19). /v1/model answers with version 1.1.0 and the digest from RELEASES.md. The container is healthy.

Part 2: online against batch, for the same tickets

Uses Batch inference, Online inference and Check parity.

A help desk day has about 45 tickets. Use the first 200 July tickets as a small, fixed workload: about four days.

1. Make the file: the header and the first 200 tickets of data/july_tickets.csv, in data/day.csv.

   head -n 201 data/july_tickets.csv > data/day.csv

   On Windows (PowerShell): Get-Content data\july_tickets.csv -TotalCount 201 | Set-Content -Encoding utf8 data\day.csv.

2. Run the batch job on it with the same bundle as the service (the same MODEL_BUNDLE and MODEL_SHA256 as in .env), into out/day.

3. Write compare_paths.py in the project folder. It must send every ticket of data/day.csv to POST /v1/score, one at a time, compare each online score with the batch score of the same ticket in out/day/scores.csv, and print: the number of tickets, the model versions it saw, the largest score difference, how many flags differ, and the time that each path took.

4. Run it against the running service. Then run parity/check_parity.py against it too.

Tip

Read scores.csv with pd.read_csv(..., float_precision="round_trip"). The default reader can drop the last digit of a long number, and you would measure the reader instead of the service.

What you should see

With 1.0.0, the course’s run printed:

200 tickets. Model versions: 1.0.0.
Largest score difference: 1.1e-16 (tolerance 1e-09). Flags that differ: 0.
Online, one request at a time: 0.79 s. Batch job: 1.52 s.
The two paths agree.

Your times will differ. Expect one model version only, a largest difference far below 1e-9, and no flag that differs.

Explain the times. Here the online path was faster, although Module 2 showed that scoring all tickets in one call is about 350 times faster per ticket. Run the batch job a few times on both files: on the course’s computer it took 0.67–1.05 s for the 200 tickets and 0.60–0.77 s for all 1,400 (the first run of the day is the slowest). The size of the file hardly matters, because almost all of the time is start-up: importing scikit-learn, then loading and checking the bundle. The scoring itself takes milliseconds. The online service paid its start-up once, when it started, so each request pays only its own few milliseconds: about 0.8 s for 200 tickets, but 5 to 6 s for 1,400. Your evidence must explain your own numbers in this way.

Part 3: measured latency and resources

Uses Container resource settings and Benchmark the service.

1. Use the configuration from your docs/infrastructure.md (workers, CPUs, memory). Write it in your evidence.
2. Run bench/bench.py twice with the same settings: 1,000 requests at concurrency 1, then at concurrency 4. Copy both result lines.
3. While the second run works, run docker stats --no-stream and copy the memory and CPU columns.
4. Measure a cold start: stop the container, then time docker compose up -d --wait.
5. Write four sentences: is the p95 under the 50 ms budget at Larkfield’s load, where does the time go, what is the bottleneck under load, and what you did not measure (for example the network to a cloud, or a second computer).

What you should see

On the course’s computer, one worker with 1 CPU gave a p95 of about 6 ms at concurrency 1, and used about 150 MiB of memory. At concurrency 4 the p50 grows because requests wait for the one CPU, while the throughput stays about the same. A local cold start takes a few seconds. The live cloud run in Benchmark the service took about 24 to 26 seconds from zero replicas: do not claim your laptop’s numbers for a cloud.

Part 4: a rollback drill

Uses Monitor model health and Rollback and retirement.

1. Before the drill, save the service’s log: docker compose logs --no-log-prefix > predictions-1.1.0.log. Replacing the container deletes it.
2. Go back to 1.0.0: set its MODEL_VERSION and MODEL_SHA256 from RELEASES.md in .env, then time docker compose up -d --wait.
3. Verify after the rollback: /v1/model (version and digest), parity/check_parity.py for the 1.0.0 bundle (14 of 14), a good ticket (200) and a bad ticket (422 with its fields).
4. Run the batch job with 1.0.0 into a new folder, then compare_paths.py again. The two paths must agree again.
5. Write the rollback plan in RELEASES.md: when you roll back (the signals from Module 5 and their limits), the exact steps, how long it took, and how you check it.

What you should see

The drill takes a few seconds on your computer (the course measured 6 to 7 s to healthy, with about 2 s without any answer). After it, /v1/model shows 1.0.0 and its digest, parity is 14 of 14, the good ticket gets 200 and the bad ticket 422. compare_paths.py prints “The two paths agree.” with version 1.0.0 only.

Check your work

Review your evidence with this table, or ask a colleague to use it. Mark each item as demonstrated, partially demonstrated or not yet demonstrated, with a reason. The course is complete when every item is demonstrated. You can improve an item and review it again as often as you need.

Required evidence	Demonstrated when
Compatible model and preprocessing artifacts	The running bundle’s digest equals the reviewed digest; the two contracts differ only in version and threshold; /v1/model shows the version and digest of RELEASES.md.
Training-serving parity tests	check_parity.py gives 14 of 14; compare_paths.py shows one model version, a largest difference below 1e-9 and no flag that differs, for 200 tickets.
Measured latency and resource use	Two benchmark lines with their settings, memory and CPU from docker stats, a timed cold start, and four sentences that do not go beyond what you measured.
Working rollback plan	The drill is timed; after it, the version, the digest, parity and the contract (a good and a bad ticket) are checked; RELEASES.md has the plan and the record.

A working happy path is not enough. If the batch job and the service use different bundles, if a check passes only because it was not run against the running service, or if the rollback is described but not done, the item is not yet demonstrated.

Common mistake

The two paths do not agree

Symptom: compare_paths.py prints two model versions, or flags that differ, and “The two paths do NOT agree.”

Cause: the batch job and the service used different bundles. Most often, .env names 1.1.0 for the service, but the shell that ran the batch job still had MODEL_SHA256 of 1.0.0, or the other way round. The batch job’s summary.json shows which bundle it used.

Fix: run the batch job with the same MODEL_BUNDLE and MODEL_SHA256 as the service, then compare again. In a real system, one release record sets both, so they cannot drift apart.

Recap

• A release is a version and a digest. Check both where the model runs.
• The same tickets through two paths must give the same scores within a stated tolerance, and the same flags.
• A batch job pays its start-up for every run; a service pays it once. Which path is faster depends on the size of the file.
• A benchmark is evidence only with its workload and conditions, and only for what it measured.
• A rollback plan is proven only by a drill and the checks after it.

The case studies of Module 6 practise the course on new models. The certificate asks for at least one of them: Score credit default in batch and online, Release a new house-price model or Score hospital readmissions asynchronously.