Skip to content
NextiaNextia Learning

Rollback and retirement

  • Lesson 15
  • 45 min
  • Module 5: Release, monitor and recover
Lesson 15 of 15 · Deploying AI Models for Real UsersRollback and retirement
Course overview

1. Define the inference contract

  1. Prediction inputs and outputs
  2. Bundle model dependencies
  3. Artifact trust and integrity

2. Choose a serving pattern

  1. Batch inference
  2. Online inference
  3. Asynchronous inference

3. Build the prediction service

  1. Load and initialize
  2. Connect the API
  3. Check parity

4. Choose infrastructure and measure

  1. CPU, GPU and managed endpoints
  2. Container resource settings
  3. Benchmark the service

5. Release, monitor and recover

  1. Version and rollout
  2. Monitor model health
  3. Rollback and retirement

6. Case studies

  1. Score credit default in batch and online
  2. Release a new house-price model
  3. Score hospital readmissions asynchronously

What you will be able to do

When a release goes wrong, go back to the known-good model in seconds and prove that it answers correctly. Then remove the faulty version, so that nobody can start it again by mistake.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • ~/projects/escalation-service with the monitor, the three bundles, RELEASES.md and data/july_labels.csv, from Monitor model health: Guided practice. 42 tests pass.
  • Your .env with MODEL_VERSION=1.0.0 and the digest of 1.0.0.
  • If your project is missing or broken, copy the snapshot nextia-learning-labs/C17/snapshots/end-of-m05. It has every file of this module, but not the bundle of rc1: this lesson retires it. For the drill, make it again with the commands of A release is a version and a digest, and use your own digest. Use a snapshot has the steps.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

Keep the old bridge open

Section titled “Keep the old bridge open”

When a city builds a new bridge, it keeps the old bridge open until the new one has passed its load test. If the new bridge shows a crack on the first day, traffic goes back to the old bridge at once. Nobody repairs the new bridge while the cars wait.

A model release works the same way. The new version may fail with real traffic, even after a good shadow run. So you keep the old version ready, and you know exactly how to switch back.

The problem: an alert, and every minute counts

Section titled “The problem: an alert, and every minute counts”

To practise, Amira makes rc1 live on purpose, on her own computer. This is a simulation: no real ticket reaches it. She sets .env to rc1 and starts it:

.env (the drill)
MODEL_VERSION=1.1.0-rc1
MODEL_SHA256=fa04bc40bea99723095ec4db600b7060fbfce0f2ec461534bb06c1829b47fe06
Terminal, in ~/projects/escalation-service
docker compose up -d --wait
python bench/bench.py http://127.0.0.1:8000 --requests 300 --warmup 0
curl http://127.0.0.1:8000/v1/monitor

After 300 July tickets, the monitor shows (shortened):

{"scored_total":300,"window":300,...,"flag_rate":0.837,"mean_score":0.4147,...,
 "alerts":["flag rate 0.837 is above the review capacity 0.20"],"model_version":"1.1.0-rc1"}

Larkfield gets about 45 tickets a day. At 83.7%, rc1 flags about 38 of them every day, and the senior team can review 9. Each day adds about 29 flagged tickets that the team cannot handle.

Tomás: “Let me find the bug and fix it. I need a day, maybe two.” Each day costs Grace’s team 29 tickets. And a fix made in a hurry is a new version that nobody has tested.

Roll back first, find the cause later

Section titled “Roll back first, find the cause later”

A rollback returns the service to the last version that was known to work, without any change to that version. In the release states, it is the branch from Live to Rolled back, while Known good becomes Live again:

The statuses of a model version: Candidate, Shadow, Live, Known good, Rolled back and Retired. The rollback branch: an alert sends the live version to Rolled back, and the rollback arrow takes the Known good version back to Live. Rolled back and Known good versions are retired later.
The rollback branch. The faulty version goes to Rolled back, and the known-good version goes back to Live. Nothing is rebuilt.
Text description of the diagram

The same diagram as in the first lesson of this module. This lesson uses two of its arrows. An alert after release moves the live version, here rc1 in the drill, to rolled back. The arrow rollback moves the known-good version, 1.0.0, back to live. Later, when the cause is known, the rolled-back version is retired.

A rollback is fast only if three things are true before the problem starts:

  1. The known-good version is still there. Its bundle is in the image, next to the new one. Nothing needs to be built or downloaded.
  2. Its digest is written down. RELEASES.md has it. You copy it, you do not compute it in a hurry.
  3. The steps were tested. A rollback that nobody has tried is a plan, not a rollback. The practice run is called a rollback drill.

Amira’s rollback is two lines in .env and one command. First, save the prediction log of rc1: the old container’s log disappears when Compose replaces the container.

Terminal
docker compose logs escalation --no-log-prefix > predictions-rc1.log

Then set .env back to 1.0.0, with the digest from RELEASES.md:

.env (rollback)
MODEL_VERSION=1.0.0
MODEL_SHA256=8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74

Predict: the image is already on the computer, and the cold start in Module 4 was about 2 s with one CPU. How long until Compose says that the service is healthy?

Terminal
time docker compose up -d --wait

The output:

 Container escalation-service-escalation-1 Recreate
 Container escalation-service-escalation-1 Recreated
 Container escalation-service-escalation-1 Starting
 Container escalation-service-escalation-1 Started
 Container escalation-service-escalation-1 Waiting
 Container escalation-service-escalation-1 Healthy
docker compose up -d --wait  0.05s user 0.05s system 1% cpu 6.959 total

7.0 seconds to Healthy. Five more drills took 5.9 to 6.5 seconds. The service answers sooner than that: Compose waits for the health check, which runs every 5 seconds. Amira measured the gap with a script that called /ready every 0.05 s during a rollback. rc1 answered until the old container stopped. Then nothing answered for about 2 seconds. Then 1.0.0 answered.

After the rollback: check the contract and the predictions

Section titled “After the rollback: check the contract and the predictions”

Healthy means that /ready answers. It does not prove that the right model answers correctly. Amira checks three things, in this order.

Which model answers? curl http://127.0.0.1:8000/v1/model:

{"model_version":"1.0.0","bundle_sha256":"8f233029aa4c896dce1a5b3dfa970cd085088386bbe7784519f15a5b1a433b74","threshold":0.19,...,"shadow_version":null}

The digest is the one in RELEASES.md. Does it score as in training? The parity check of Module 3:

Terminal, with .venv active
python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0

The output (shortened):

case     ticket      expected     served difference
normal   T-116940    0.187598   0.187598    2.8e-17
...
unknown  T-900008    0.111725   0.111725    5.6e-17
14 of 14 cases match within 1e-09.

Does the contract still hold? Send one good ticket and one bad ticket with the curl commands of Connect the API. The two answers:

POST /v1/score, the example ticket T-120001 →
200 {"ticket_id":"T-120001","score":0.35065699401024064,"flag":true,"threshold":0.19,"model_version":"1.0.0","warnings":[]}


POST /v1/score, priority "high" and created_hour 24 →
422 {"error":{"code":"invalid_ticket","message":"The ticket does not match the model's contract.","request_id":"f1fc9c0f7335","fields":["created_hour","priority"]}}

The score is the same as in Module 3, to the last digit, and the error shape is the same. Amira adds a line to the history in RELEASES.md: the drill, the 7.0 s, parity 14 of 14, the contract checks.

The trap: one container and a wrong digest

Section titled “The trap: one container and a wrong digest”

Compose has one container for the service. docker compose up -d stops the old container first, then starts the new one. That is the 2-second gap. If the new container cannot start, there is no old container to fall back to.

Amira tried it on purpose. rc1 was live, and she rolled back with the right version but the digest of 1.1.0, from the wrong row of RELEASES.md:

 Container escalation-service-escalation-1 Recreate
 Container escalation-service-escalation-1 Recreated
 Container escalation-service-escalation-1 Starting
 Container escalation-service-escalation-1 Started
 Container escalation-service-escalation-1 Waiting
container escalation-service-escalation-1 exited (3)

The log says why:

ERROR escalation: Start-up error: bundles/escalation-1.0.0: the bundle digest is 8f233029aa4c..., but MODEL_SHA256 expects 382077148b6c...
ERROR:    Application startup failed. Exiting.

The digest check did its job: it refused a bundle that did not match. But rc1 was already stopped. Before the rollback, the help desk had a bad model. Now it has no model at all, until somebody fixes .env. A rollback made the situation worse.

Two ways to roll back. One container with docker compose up -d: rc1 answers, rc1 stops, there is no service for about 2 s, then 1.0.0 answers. With a wrong digest, rc1 stops and the new container exits with code 3: no service at all. Revisions on a platform (Docker and CI/CD courses): rc1 answers, 1.0.0 starts next to rc1, when 1.0.0 is ready the traffic moves, then rc1 stops. With a wrong digest, 1.0.0 fails and rc1 still answers.
With one container, the old version stops before the new one is ready. With revisions, the old version answers until the new one is ready, so a bad rollback changes nothing.
Text description of the diagram

The top row is this course’s setup: one container, replaced by docker compose up -d. rc1 answers, then rc1 stops, then there is no service for about 2 seconds, then 1.0.0 answers. With a wrong digest, the new container exits with code 3 after rc1 has stopped, and nothing answers.

The bottom row is a platform with revisions, as in the Docker and CI/CD courses. 1.0.0 starts next to rc1. The platform moves the traffic only when 1.0.0 is ready, and then rc1 stops. With a wrong digest, 1.0.0 never becomes ready, and rc1 keeps answering. That is still wrong, but no worse than before.

Two things protect you on a single container:

  • Copy, then check. Before up, compare the digest in .env with the row of the version in RELEASES.md, character by character, or with a short command such as grep 8f233029 RELEASES.md.
  • Prefer a platform with revisions for a real service. Azure Container Apps keeps the old revision until the new one is ready (Operational configuration in the Docker course). If you took the CI/CD course, Rollback strategy rolls back only the model with a setting, in the same way.

Retire the faulty version

Section titled “Retire the faulty version”

After the drill, Grace and Priya decide: rc1 will never go live. Its threshold mistake is understood, and 1.1.0 is the corrected candidate. Leaving rc1 in the project has a cost: somebody can type 1.1.0-rc1 in .env one day, and the image will start it without complaint. Its digest is valid.

Retirement removes a version from everything that can run it, and keeps the record of why. For rc1:

  1. The registry. In RELEASES.md, the status of rc1 becomes retired, with the reason and the date. The row stays: the history of what was tried is evidence.
  2. The image. Delete the folder bundles/escalation-1.1.0-rc1. If you use Git, its history keeps the files.
  3. The running resources. Remove any container that ran it, such as escalation-shadow from the first lesson, and any SHADOW_ setting.

Rebuild, and look inside the new image:

Terminal
docker compose build
docker run --rm escalation-service:1.0.0 ls bundles

The output:

escalation-1.0.0
escalation-1.1.0

Now try the old mistake: MODEL_VERSION=1.1.0-rc1 and its digest in .env, then docker compose up -d --wait. The service refuses:

ERROR escalation: Start-up error: bundles/escalation-1.1.0-rc1: no SHA256SUMS file

That is the point of retirement: a retired version cannot come back by mistake. Set .env back to 1.0.0, and start again.

The same rule applies to everything that a release leaves behind. In the cloud, a canary revision, a test endpoint or a second container costs money, even with no traffic. It is also one more thing that can answer a request. Remove it when its test ends. Verify and remove in the Docker course shows how for a cloud deployment.

The final RELEASES.md of the course has 1.0.0 live (known good), rc1 retired and 1.1.0 a candidate, with a history table. The snapshot end-of-m05 has it.

Guided practice: the rollback drill

Section titled “Guided practice: the rollback drill”
  1. In .env, set MODEL_VERSION=1.1.0-rc1 and the digest of rc1 from RELEASES.md.
  2. Run docker compose up -d --wait.
  3. Send 300 July tickets with python bench/bench.py http://127.0.0.1:8000 --requests 300 --warmup 0.
  4. Call GET /v1/monitor. Check that alerts has one alert.
  5. Save the log to predictions-rc1.log.
  6. In .env, set MODEL_VERSION=1.0.0 and the digest of 1.0.0. Compare the digest with RELEASES.md.
  7. Run time docker compose up -d --wait (Windows: Measure-Command { docker compose up -d --wait }). Write down the time.
  8. Call GET /v1/model. Check the version and the digest.
  9. Run check_parity.py. You should see 14 of 14 cases match.
  10. Send the good ticket and the bad ticket, as in After the rollback. Check 200 with model 1.0.0, then 422 with two fields.
  11. Add the drill to RELEASES.md: the date, the time, and the results of steps 8 to 10.
Show the expected results

Step 4: "flag_rate":0.837 and the alert flag rate 0.837 is above the review capacity 0.20. The same 300 tickets give the same rate on every computer.

Step 7: about 6 to 7 seconds on the test computer. Your time is different.

Steps 8 to 10: the outputs of After the rollback.

A history line for step 11:

| 2026-10-08 | Rollback drill: 1.1.0-rc1 live, 300 July tickets, alert (flag rate 0.837). Back to 1.0.0: healthy after 7.0 s, parity 14 of 14, contract checks pass. | Amira |

Your turn: retire rc1

Section titled “Your turn: retire rc1”

Retire rc1 in the three places of Retire the faulty version. Then prove it: the image lists only two bundles, and MODEL_VERSION=1.1.0-rc1 stops at start-up. Run python -m pytest at the end. Predict first: does any test need rc1?

Show the answer

docker run --rm escalation-service:1.0.0 ls bundles lists escalation-1.0.0 and escalation-1.1.0. The start-up with rc1 fails with no SHA256SUMS file. The tests give 42 passed: every test uses the bundle 1.0.0, so no test needs rc1. The row of rc1 in RELEASES.md:

| 1.1.0-rc1 | `fa04bc40bea9...fe06` | retired | 2026-10-08 | Shadow, July: flagged 87.0% of the last 1,000 tickets. Balanced weights with the old threshold 0.19. Never answered a real request. Bundle removed from the image. | Grace, Priya |

Common mistake

The rollback stops the service: a wrong digest

Symptom: during a rollback, docker compose up -d --wait ends with container ... exited (3). curl http://127.0.0.1:8000/ready cannot connect. The log shows Start-up error: bundles/escalation-1.0.0: the bundle digest is 8f233029aa4c..., but MODEL_SHA256 expects 382077148b6c....

Cause: .env has the right version but the digest of another version. The service correctly refuses to load the bundle. With one container, Compose stopped the old version first, so nothing answers now.

Fix: copy the digest of 1.0.0 from its row in RELEASES.md into .env, and run docker compose up -d --wait again. In the drill this took 5.9 s. Then do the three checks of After the rollback.

For a real AI service: the moment of a rollback is the moment when people are in a hurry. Make it a copy from a written record, practised in a drill. Use a platform that keeps the old version answering until the new one is ready.

Check your understanding

Tomás wants to fix rc1 in place instead of rolling back. Give two reasons to roll back first.

Show the answer

First, time: rc1 flags about 38 of the 45 tickets a day while the team can review 9, and a rollback takes seconds. A fix takes a day or more. Second, safety: a fix made in a hurry is a new, untested version. 1.0.0 is known good, with its digest, parity and history. Roll back, then find the cause without pressure, and make a new candidate.

Check your understanding

After a rollback, Compose says Healthy. Why is that not enough? What do you check?

Show the answer

Healthy only means that /ready answered. Check three things. Which model answers: in GET /v1/model, the version and the digest equal the row in RELEASES.md. Does it score as in training: check_parity.py gives 14 of 14. Does the contract hold: a good ticket gets 200 and the expected score, and a bad ticket gets 422 with the fields.

Module practice

Section titled “Module practice”

Guided practice → independent task. The guided practice was the drill with rc1, a model whose scores moved. Now make your own faulty candidate with a different mistake, release it in your local simulation, and restore 1.0.0.

Your task: a candidate with a wrong threshold

Section titled “Your task: a candidate with a wrong threshold”
  1. In ~/projects/ticket-model, make a candidate with a threshold that was copied wrongly: python retrain.py --out model-1.1.0-rc2 --threshold 0.12.
  2. Bundle it as version 1.1.0-rc2. Add its row to RELEASES.md.
  3. Set .env to rc2 and its digest. Make it live with docker compose up -d --build --wait: the image needs the new bundle. Send 300 July tickets, and read /v1/monitor.
  4. Roll back to 1.0.0. Time it, and do the three checks.
  5. Retire rc2.
  6. Write two sentences: how the monitor of rc2 differs from the monitor of rc1, and what the difference tells you about the cause.
Show a model answer

Steps 1 and 2:

Trained on 18118 tickets (to the end of May). Threshold 0.12 (fixed by hand).
June: ROC AUC 0.749, average precision 0.346, flagged 352 of 1002 (35.1%), precision 0.307, recall 0.688.
Made bundles/escalation-1.1.0-rc2: 14 parity cases, threshold 0.12.
Bundle digest (MODEL_SHA256): d6fad059f7dbdb1ff8603a3d62e4c56af5b1cab5d185344c6a97d3f6b7a6a8f8

Step 3, /v1/monitor:

{"scored_total":300,"window":300,"contract_errors":{},"unknown_categories":{},"flag_rate":0.33,"mean_score":0.1173,
 "latency_ms":{"p50":2.12,"p95":3.22},"shadow":null,"alerts":["flag rate 0.330 is above the review capacity 0.20"],"model_version":"1.1.0-rc2"}

Step 4: healthy after 6.2 s. /v1/model gives 1.0.0 with its digest, and parity gives 14 of 14 cases match within 1e-09. The good ticket gives 200 with the score 0.35065699401024064. The bad ticket gives 422 with created_hour and priority.

Step 5: rc2’s row says retired, its folder is deleted, the image lists two bundles.

Step 6: rc1 had a mean score of 0.4147 on the same 300 tickets, so its scores moved: the model itself changed. rc2 has a normal mean score, 0.1173, but a flag rate of 33%: the scores are fine, and the threshold is wrong. The mean score tells you where to look.

Module check

Section titled “Module check”

Check your understanding

Retrieval: what does each column of RELEASES.md record, and which column does the service check at start-up?

Show the answer

Version: the name for people. Bundle digest: the value for MODEL_SHA256. Status: candidate, shadow, live, known good, rolled back or retired. Since: the date of the status. Evidence: the numbers that justify the status. Approved by: who decided. The service checks the digest: load_bundle compares the digest of SHA256SUMS with MODEL_SHA256 before it loads anything.

Check your understanding

Predict: 1.1.0 runs in shadow next to 1.0.0 on the July tickets. Will the monitor raise an alert for the shadow? Should 1.1.0 then be retired like rc1?

Show the answer

Yes, an alert. In the last 1,000 tickets, the shadow flag rate is 0.206, just above 0.20. The live model is at 0.212. But it should not be retired like rc1. Its threshold comes from its own scores. Over the month it flags 19.4%, with precision 0.342 and recall 0.474, close to 1.0.0. Both versions go above 20% in late July because of the inputs, not because of the model: the social channel and the broken export. The decision is Grace’s and Priya’s, with this evidence.

Check your understanding

Completion check: after a rollback, show that the API contract and the predictions are correct. Which commands and which results?

Show the answer

GET /v1/model: model_version 1.0.0, and bundle_sha256 equal to the digest of 1.0.0 in RELEASES.md. python parity/check_parity.py http://127.0.0.1:8000 bundles/escalation-1.0.0: 14 of 14 cases match within 1e-09, which covers normal, missing, extreme and unknown inputs. POST /v1/score with the example ticket: 200, score 0.35065699401024064, model_version 1.0.0. POST /v1/score with priority “high” and created_hour 24: 422 with fields ["created_hour","priority"].

Recap

Module 5 in short

  • A release is a version and a digest, recorded in RELEASES.md, your model registry. The threshold belongs to the version: rc1 kept the AUC and flagged 86%.
  • A shadow scores real traffic with the candidate and answers with the live model. A canary answers a small share. Shadow first when a wrong answer costs a person time.
  • The service counts its health signals: contract errors, unknown categories, latency, the flag rate against the capacity and the mean score. outcomes.py adds precision and recall when the labels arrive.
  • A rollback goes back to the known-good version without a new build. It takes two lines in .env and about 7 seconds. Then check the model, parity and the contract.
  • With one container, a wrong digest during a rollback leaves no service. Platforms with revisions keep the old version answering.
  • Retirement removes a version from the image and the running resources, and keeps its row as evidence.

Reset to a known state

  1. Stop and remove the service: docker compose down.
  2. Remove the image: docker image rm escalation-service:1.0.0.
  3. If you made a test container that still exists, remove it: docker rm -f escalation-shadow.
  4. Copy the snapshot nextia-learning-labs/C17/snapshots/end-of-m05 to a new folder, as in Use a snapshot. Make its .venv and its .env with MODEL_VERSION=1.0.0 and the digest of the course’s 1.0.0 bundle.
  5. Run python -m pytest. You should see 42 passed. Then docker compose up -d --build --wait.

Next, a case study practises a release and a rollback on a regression model, where there is no threshold to check.

Next: Release a new house-price model