Skip to content
NextiaNextia Learning

Rollback strategy

  • Lesson 13
  • 45 min
  • Module 5: Recover and maintain the release process
Lesson 13 of 15 · CI/CD, Testing and Safe ReleasesRollback strategy
Course overview

1. Understand the delivery path

  1. From commit to release
  2. Choose a branch workflow
  3. Define checks

2. Create a repeatable CI pipeline

  1. Runner setup
  2. Build and test
  3. Publish evidence

3. Manage environments and secrets

  1. Configuration separation
  2. Credentials and access
  3. Promote to staging

4. Release AI and database changes

  1. AI evaluation gate
  2. Migration safety
  3. Progressive release

5. Recover and maintain the release process

  1. Rollback strategy
  2. Hotfix procedure
  3. Release communication

What you will be able to do

“Can we go back?” has three different answers. The code, the model and the data go back in different ways and at different speeds. One of them never goes back by itself. Choose the right way before you need it.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

Run with GitHub Actions (ubuntu-24.04 runners) in the public repository Nextia-Intelligence/ticket-api-ci, Docker Desktop 4.81.0 on macOS (Apple silicon), and Azure CLI 2.88.0 in canadacentral. The Windows and Linux steps are not tested.

Three things can go back

Section titled “Three things can go back”

At 18:41 on 2026-10-07, Amira found a bug in version 1.2.0: GET /v1/history answered 500 (the next lesson shows it). The help desk lead asked the usual question: “Can we go back?”

Look at what production ran at that time. One answer from POST /v1/classify shows three separate parts:

One answer from production, 1.2.x with the flag on
{"category":"shipping","priority":1,"confidence":0.9,"score":0.9,"model_version":"keywords-1.1"}
Part Where it lives What it gave this answer
The application (the code) The image, named by its digest The field score exists only in 1.2.x
The model The setting CLASSIFIER_VERSION=1.1 of the app keywords-1.1, shipping and priority 1
The data The rows in the database Each classification is saved, with or without score

A rollback returns one part to an earlier state that was known to work. Each part has its own way back:

  • Application rollback: deploy the previous image, by its digest. Nothing is built.
  • Model rollback: change only the flag. The image stays.
  • Data: a deployment never changes rows that are already written. To correct them, you repair forward: a new migration changes the data to the correct state. A restore (load a database backup) is the last option, because every row written since the backup is lost.
What went wrong? Three columns. The new code is wrong: roll back the application, deploy the previous digest with the Deploy workflow; live 120 s to production through staging and approval; it also takes away score and keywords-1.1, and the database stays as it is. The new model is wrong: roll back only the flag, CLASSIFIER_VERSION=1.0 on the same image; live 43 s; it keeps the image, score and the fix. Rows were written wrongly: repair the data forward with a new migration, 003_backfill_score, after no 1.1.0 is left; one migration, 1 empty score, then 0; a rollback never changes rows, restore a backup only when data is lost. Under all three: a rollback works only because migration 002 was expand-first.
The rollback decision: what went wrong decides what goes back. The times are from the drill in the reference repository on 2026-10-07.
Text description of the diagram

Start from the question “What went wrong?”. If the new code is wrong, roll back the application: deploy the previous digest with the Deploy workflow. In the drill this took 120 seconds to production, through staging and the approval. It also takes away the new field score and the new model, and it does not change the database. If only the new model is wrong, roll back only the flag: set CLASSIFIER_VERSION=1.0 on the same image. This took 43 seconds and keeps everything else. If rows were written wrongly, nothing rolls them back: write a new migration that corrects them, such as 003_backfill_score. Restore a backup only when data is lost. All of this works only because migration 002 added a column that version 1.1.0 can ignore.

So Amira had three choices at 18:41. A rollback to 1.1.0 would stop the 500, because 1.1.0 does not read score. But clients would lose score, which the 1.2.0 release notes had promised. The help desk would lose keywords-1.1. A flag rollback would not help: the bug was in the code, not in the model. She chose a hotfix (Lesson 2): a small new version that fixes only the bug. The bug affected one endpoint, and production stayed up for everything else.

Then, with 1.2.1 in production, the team ran a rollback drill: a planned rollback that measures the time and checks what changes. Then the real rollback is not the first one. The rest of this lesson is that drill.

Before a rollback: what does it take away?

Section titled “Before a rollback: what does it take away?”

A rollback to an older version takes away everything that came after it. Check it before you start, not after. For the ticket API:

1.1.0 1.2.0 1.2.1
score in the answer no yes yes
Knows CLASSIFIER_VERSION no: always keywords-1.0 yes yes
Runs the migrations no: makes the table at start-up yes, 001 and 002 yes, 001 and 002
GET /v1/history with a row from 1.1.0 200 500 (the bug) 200

Two questions decide whether a rollback is safe:

  1. Does the old version work with the database as it is now? Yes here: migration 002 only added a column, and 1.1.0 ignores it. This is why Migration safety: Expand now, contract later made every migration expand-first. If 002 had renamed confidence, 1.1.0 would fail on every request, and the only way back would be a restore.
  2. Do clients already use something new? A client that reads score breaks when the field is gone. Tell them before, or choose a smaller rollback.

Worked example: the application rollback (live)

Section titled “Worked example: the application rollback (live)”

Find the previous digest. A rollback deploys an image that CI built and tested before. Never build the old code again: a new build is a new artifact, with new packages, and nobody has tested it. The digest is on the release page (the last line of each release), or in the registry:

Terminal
docker buildx imagetools inspect ghcr.io/usmanahmed99/ticket-api-ci:1.1.0

The output (shortened):

Name:      ghcr.io/usmanahmed99/ticket-api-ci:1.1.0
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a

Use your own owner name instead of usmanahmed99. The tag 1.1.0 can move if someone tags again. The digest cannot: it is the release that you tested.

Start the Deploy workflow by hand. In your repository, open Actions → Deploy → Run workflow. Keep the branch main. Paste the digest into digest, and write the reason into reason. Then select Run workflow. With the GitHub CLI, the same:

Terminal (live)
gh workflow run deploy.yml \
  -f digest=sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a \
  -f reason="Rollback drill: production back to 1.1.0"

A rollback takes the same path as every release: staging first, with the smoke check, then the approval, then production. This is slower than a direct change in Azure. But it is the path that you tested, it leaves a record, and a broken old image stops in staging. When the staging job is done, the production job waits. The reviewer selects Review deployments, then production, writes a comment and approves. In the drill, the comment was “Rollback drill: production back to 1.1.0”.

The result: run 37671631615. The jobs (live):

Find the image                      19:03:55 → 19:04:00   success
staging / Deploy to staging         19:04:04 → 19:04:56   success
production / Deploy to production   19:05:10 → 19:06:01   success

The run started at 19:03:50. Production ran 1.1.0 about 120 seconds later. The 14 seconds between the two jobs were the approval. The smoke checks passed in both environments, including ok history (503) (the cloud has no database).

Check what changed. Amira sent one ticket to production. The output (live):

{"category":"other","priority":3,"confidence":0.2,"model_version":"keywords-1.0"}

And the app’s settings and image:

Terminal (live)
az containerapp show -g rg-ticket-api-ci -n ticket-api-production \
  --query "{image: properties.template.containers[0].image, env: properties.template.containers[0].env[].name}"

The output (live):

{
  "env": [
    "API_KEY",
    "REQUIRE_API_KEY",
    "SHOW_DOCS",
    "CLASSIFIER_VERSION"
  ],
  "image": "ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a"
}

Read the two outputs together. CLASSIFIER_VERSION is still set (to 1.1), but the answer says keywords-1.0, and score is gone. Version 1.1.0 does not know the flag, so it ignores it. The application rollback took away three things at once: the bug fix, the API field and the model.

Roll forward. A drill ends in the state where it started. Amira deployed the 1.2.1 digest the same way: run 37671961512, 116 seconds. The output (live):

{"category":"shipping","priority":1,"confidence":0.9,"score":0.9,"model_version":"keywords-1.1"}

The flag worked again at once, because it was still set on the app.

Worked example: the model-only rollback (live)

Section titled “Worked example: the model-only rollback (live)”

Now assume that keywords-1.1 is wrong for a type of ticket, and the code is fine. Change only the flag:

Terminal (live)
az containerapp update -g rg-ticket-api-ci -n ticket-api-production \
  --set-env-vars CLASSIFIER_VERSION=1.0 --query properties.latestRevisionName -o tsv

The output (live):

ticket-api-production--0000010

A changed setting makes a new revision with the same image. After 43 seconds, the answer was (live):

{"category":"other","priority":3,"confidence":0.2,"score":0.2,"model_version":"keywords-1.0"}

score stays, the fix stays, only the model is back. Amira turned it on again with --set-env-vars CLASSIFIER_VERSION=1.1, and the answer said keywords-1.1 again.

This change does not go through the pipeline: it is a direct change of production. The canary in Progressive release: Worked example: the live canary was one too. So the production approval does not apply. In an emergency, that is acceptable for a flag. The flag exists for this purpose, and the evaluation gate tested both values. Write it down (who, when, why), because the pipeline has no record of it. In a team, put the command in a workflow job with environment: production, as for the canary: then a reviewer approves it, and the run records it. The --set-env-vars value stays on the app for later deployments too.

The simulated route

Section titled “The simulated route”

The same drill on your computer. The local production environment runs on http://127.0.0.1:8082, with its own database. Run the commands in the project folder.

Start with 1.2.1 and the flag on. In deploy/local/production.env, set CLASSIFIER_VERSION=1.1. Then deploy the 1.2.1 image:

Terminal (simulated)
scripts/promote-local.sh production ghcr.io/usmanahmed99/ticket-api-ci@sha256:84937455beb63744755622d539748c9adb1772f379c9eb7e4f961c5b7f1bc924
curl -s -X POST http://127.0.0.1:8082/v1/classify -H 'Content-Type: application/json' \
  -H "X-API-Key: $(cat secrets/production_api_key.txt)" -d '{"subject":"Package never arrived"}'

The output (simulated, shortened):

ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)
ok    history (200)
production now runs ghcr.io/usmanahmed99/ticket-api-ci@sha256:84937455beb63744755622d539748c9adb1772f379c9eb7e4f961c5b7f1bc924
{"category":"shipping","priority":1,"confidence":0.9,"score":0.9,"model_version":"keywords-1.1"}

Model-only rollback. In deploy/local/production.env, set CLASSIFIER_VERSION=1.0. Run the same promote-local.sh command with the same digest. Compose sees a changed setting, and makes the api container again with the same image. The output (simulated, shortened):

 Container ticket-api-production-api-1 Recreate
 Container ticket-api-production-api-1 Recreated
ok    health (200)
...
{"category":"other","priority":3,"confidence":0.2,"score":0.2,"model_version":"keywords-1.0"}

It took 8 seconds. The answers are the same as the live ones. Set the flag back to 1.1, and run the command again.

Application rollback. The local environment cannot use promote-local.sh for this rollback (the common mistake at the end of this lesson shows why). Start only the api service with the old image, and leave the database and the migrations alone:

Terminal (simulated)
export IMAGE=ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a
docker compose -f deploy/local/compose.yaml --env-file deploy/local/production.env --env-file .env \
  up -d --wait --no-deps api
scripts/smoke.sh http://127.0.0.1:8082 "$(cat secrets/production_api_key.txt)"

--no-deps starts only the service that you name. The output (simulated):

 Container ticket-api-production-db-1 Running
 Container ticket-api-production-api-1 Starting
 Container ticket-api-production-api-1 Started
 Container ticket-api-production-api-1 Waiting
 Container ticket-api-production-api-1 Healthy
ok    health (200)
ok    ready (200)
ok    classify with the key (200)
ok    classify without a key (401)
ok    history (200)

It took 6 seconds. A classification now has no score and says keywords-1.0, as in the live drill. Roll forward with the promote-local.sh command and the 1.2.1 digest.

Data does not roll back

Section titled “Data does not roll back”

Look at the database after the local rollback. 1.1.0 saved two rows while it ran:

Terminal (simulated)
docker exec ticket-api-production-db-1 psql -U tickets -d tickets \
  -c "SELECT id, request_id, model_version, confidence, score FROM classifications ORDER BY id"

The output (simulated):

 id |  request_id  | model_version | confidence | score
----+--------------+---------------+------------+-------
  1 | 4d10436731e8 | keywords-1.0  |        0.7 |   0.7
  2 | a49f44a4effc | keywords-1.0  |        0.2 |   0.2
  3 | ef19e98fb391 | keywords-1.0  |        0.2 |
  4 | 119fbf2734a9 | keywords-1.0  |        0.7 |   0.7
  5 | d7dc11adb8bf | keywords-1.0  |        0.2 |   0.2
  6 | cd2ce474a397 | keywords-1.0  |        0.7 |
  7 | a8276a5e60ee | keywords-1.0  |        0.2 |
(7 rows)

Rows 6 and 7 came from 1.1.0 during the rollback. Row 3 came from 1.1.0 earlier, during the rollout of 1.2.0. When production went forward to 1.2.1, these rows stayed empty. The rollback did not remove the score column, and going forward did not fill it. The same is true in the other direction. Rows that 1.2.x wrote keep their score after a rollback to 1.1.0. A rollback changes the code that runs. It never changes the data that is already written.

1.2.1 reads these rows correctly (it uses confidence when score is empty: that is the hotfix of Lesson 2). But other readers of the table, such as a report, would see empty values. Correct the data forward, with a new migration:

migrations/003_backfill_score.sql
-- Forward repair: fill score for the rows that version 1.1.0 wrote during the
-- rollout of 1.2.0. Run it after no 1.1.0 instance is left, so no new empty
-- rows appear. It changes no structure, so every version keeps working.
UPDATE classifications SET score = confidence WHERE score IS NULL;

A migration that fills a new column from old data is a backfill. Three rules make this one safe:

  • Run it after no 1.1.0 is left. Otherwise 1.1.0 writes new empty rows after the backfill.
  • It changes no structure. 1.1.0, 1.2.0 and 1.2.1 all keep working, so it does not block a later rollback.
  • Make a backup first, because it changes rows. An ADD COLUMN (002) is safe to keep after a rollback. An UPDATE changes data that you cannot get back without a copy.

In the reference repository, 003 came in PR #8 (feature/backfill-score into main). Its build is the image of commit 30e0c7e. The local run, in order:

Terminal (simulated)
docker rm -f ticket-api-old
docker exec ticket-api-production-db-1 psql -U tickets -d tickets -At \
  -c "SELECT count(*) FROM classifications WHERE score IS NULL"
docker exec ticket-api-production-db-1 pg_dump -U tickets -d tickets -t classifications --data-only -f /tmp/before-003.sql
scripts/promote-local.sh production ghcr.io/usmanahmed99/ticket-api-ci@sha256:1ec838b17cdeefb95620a308b99f31799f4fa10c5ddd9d46a17a2506a7e10947
docker logs ticket-api-production-migrate-1
docker exec ticket-api-production-db-1 psql -U tickets -d tickets -At \
  -c "SELECT count(*) FROM classifications WHERE score IS NULL"

ticket-api-old is the 1.1.0 container from the next lesson’s bug test. If you did not start it, docker rm -f says “No such container”: that is correct. The backup goes into the database container, so it is deleted with the environment. For a real database, keep it somewhere else. The output (simulated, shortened):

3
ok    health (200)
...
ok    history (200)
production now runs ghcr.io/usmanahmed99/ticket-api-ci@sha256:1ec838b17cdeefb95620a308b99f31799f4fa10c5ddd9d46a17a2506a7e10947
applied  003_backfill_score
Database is up to date (3 migrations).
0

Three empty rows before, none after. The migrate service ran 003 before the new api started, as for every release. In the reference run (with one empty row), the result was the same: 1, then 0. Run the migrations again, and only the last line comes: Database is up to date (3 migrations).

When is a restore right? Only when data is lost or destroyed, for example after a migration that deleted a column by mistake. A restore loads the backup and loses every row written since the backup. Say how many before you start. Here, nothing was lost: every empty score can be calculated from confidence. So a forward repair is right, and no row is lost.

Guided practice: a rollback drill on your computer

Section titled “Guided practice: a rollback drill on your computer”
  1. Write the starting state: run scripts/promote-local.sh production with the 1.2.1 digest and CLASSIFIER_VERSION=1.1. Classify “Package never arrived”. Write down the answer.
  2. Predict: after a rollback of only the flag, which fields change in the answer?
  3. Set CLASSIFIER_VERSION=1.0 in deploy/local/production.env. Run promote-local.sh again with the same digest.
  4. Classify the same ticket. Compare with your prediction. Then set the flag back to 1.1, and deploy again.
  5. Predict: after an application rollback to 1.1.0, which fields change? Does the flag still work?
  6. Roll back the application with up -d --wait --no-deps api and the 1.1.0 digest. Run scripts/smoke.sh.
  7. Classify the ticket twice. Then count the rows with an empty score.
  8. Roll forward: run promote-local.sh with the 1.2.1 digest. Count the empty rows again.
  9. Make a backup with pg_dump. Deploy the image with 003. Count the empty rows a third time.
Show the expected results
  1. "score":0.9,"model_version":"keywords-1.1", shipping, priority 1.
  2. to 4. Only the model changes: "score":0.2,"model_version":"keywords-1.0", other. score stays, because the image is the same.
  3. to 7. score is gone and the answer says keywords-1.0, although the flag is 1.1: 1.1.0 does not know the flag. All five smoke checks pass (1.1.0 history works). The count of empty rows goes up by 3: your two tickets and the smoke check’s ticket.
  4. The count does not change. Going forward does not fill the rows.
  5. applied 003_backfill_score, Database is up to date (3 migrations)., then 0.

Your turn: add 003 to your repository

Section titled “Your turn: add 003 to your repository”

Add the forward repair to your own repository with a pull request feature/backfill-score into main: the file migrations/003_backfill_score.sql. Before you push, predict which CI check fails, and why. (In the reference repository this was PR #8, after the hotfix of Lesson 2. You can do it now or after Lesson 2.)

Show the answer

The database test test_migrations_apply_once fails, because it expects the line Database is up to date (2 migrations). Now there are 3. Change the expected line in tests/test_database.py to (3 migrations): a new migration changes this test on purpose. The other tests pass: 003 changes no structure. In the PR description, say that 003 runs only after no 1.1.0 instance is left. After the merge, the Deploy workflow deploys the new build, and its migrate step applies 003. In the live route there is no database, so 003 does nothing there. Compare with PR #8, which also adds the Rollback section of docs/release-process.md.

Common mistake

The local rollback stopped production

Symptom (simulated): Amira rolls back the local production with the usual script and the 1.1.0 digest:

Terminal (simulated)
scripts/promote-local.sh production ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a

The output (simulated, shortened):

 Container ticket-api-production-migrate-1 Recreate
 Container ticket-api-production-api-1 Recreate
 Container ticket-api-production-api-1 Recreated
 Container ticket-api-production-migrate-1 Started
 Container ticket-api-production-migrate-1 Error service "migrate" didn't complete successfully: exit 1
service "migrate" didn't complete successfully: exit 1

After 2 seconds, the script stops with exit code 1. curl http://127.0.0.1:8082/health gets no answer. The 1.2.1 container is gone, and the new api container is only “Created”. The rollback did not go back: it stopped production.

Cause: the deploy step of 1.2.x runs the migrations first, with the image that it deploys. docker logs ticket-api-production-migrate-1 shows:

/usr/local/bin/python: No module named ticket_api.migrate

1.1.0 has no migration runner: it is older than the migrations. Compose removed the old containers before the new ones were ready, so there was nothing to fall back to.

Fix:

  1. Start only the application with the old image: docker compose … up -d --wait --no-deps api, as in the simulated route. The database already has the schema that 1.1.0 needs.
  2. Check with scripts/smoke.sh, as after every deployment.
  3. Before the next real rollback, run it once as a drill. A drill finds this problem at a quiet time.

Why the live route did not fail: the cloud apps have no database, so the Deploy workflow has no migration step. And Container Apps keeps the old revision until the new one is ready. A local Compose deployment has no such safety.

For a real AI service: a rollback runs the old version with the new deployment steps. Some steps came later than the old version, for example a migration, a model download or a new secret. If the old version cannot do such a step, the rollback fails exactly when you need it. Keep the deployment steps backwards-compatible, and practise the rollback before the incident.

Check your understanding

After the application rollback to 1.1.0, CLASSIFIER_VERSION=1.1 was still set, but the answer said keywords-1.0. Why? What does this mean for a rollback plan?

Show the answer

Version 1.1.0 does not know the setting CLASSIFIER_VERSION: it is older than the flag, so it ignores it and always uses keywords-1.0. An application rollback takes away everything that came after the old version: here the bug fix, the field score and the new model. A rollback plan must list these losses before the rollback starts. Tell the clients and the help desk. If only the model is wrong, a rollback of the flag is smaller and faster (43 s), and keeps the rest.

Check your understanding

After a rollback from 1.2.1 to 1.1.0 and back, some rows have an empty score. Amira says: the rollback forward to 1.2.1 will fill them. Is she right? What do you do?

Show the answer

No. A deployment changes the code that runs, never the rows that are already written. The rows that 1.1.0 wrote stay without score, also after the return to 1.2.1. (1.2.1 shows them correctly, because it uses confidence for an empty score, but the rows are still empty.) Repair them forward with a migration that fills the column: 003_backfill_score. Run it after no 1.1.0 instance is left, and make a backup first, because it changes rows. A restore from the backup is only for data that is lost: it would also delete every row written since the backup.

Check your understanding

Why must you deploy the old digest, and not build the old commit again, for an application rollback?

Show the answer

The digest names the exact image that CI tested and that ran in production before. A new build of the same commit is a different artifact: the base image and the packages can be newer, and no check ran on it. Then the rollback would deploy something untested, at the worst moment. The digest also makes the rollback fast, because nothing is built. And it is traceable: the release page and build-info.json show which commit and run made it.

Recap

  • Three parts can go back, each in its own way: the application (deploy the previous digest, about 2 minutes through staging and approval), the model (change only the flag, 43 s), and the data (never by a rollback).
  • Before a rollback, list what it takes away. From 1.2.x to 1.1.0: the fix, score and keywords-1.1, because 1.1.0 does not know the flag.
  • A rollback works only if the old version works with today’s database. Expand-first migrations (002 added a column) keep it possible.
  • Correct wrong data forward, with a migration such as 003_backfill_score: after no old instance is left, and after a backup. Restore a backup only when data is lost, and say what is lost since the backup.
  • Run a drill before you need the real rollback. Locally, the drill found that the old version cannot run the new migration step.

Next: Hotfix procedure