Skip to content
NextiaNextia Learning

Configuration separation

  • Lesson 7
  • 40 min
  • Module 3: Manage environments and secrets
Lesson 7 of 15 · CI/CD, Testing and Safe ReleasesConfiguration separation
Course overview

1. Understand the delivery path

  1. From commit to release
  2. Choose a branch workflow
  3. Define checks

2. Create a repeatable CI pipeline

  1. Runner setup
  2. Build and test
  3. Publish evidence

3. Manage environments and secrets

  1. Configuration separation
  2. Credentials and access
  3. Promote to staging

4. Release AI and database changes

  1. AI evaluation gate
  2. Migration safety
  3. Progressive release

5. Recover and maintain the release process

  1. Rollback strategy
  2. Hotfix procedure
  3. Release communication

What you will be able to do

Run the image that CI tested in two places, staging and production, with different settings. Then nobody can say “it worked in staging” about a different build.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

Run with GitHub Actions (ubuntu-24.04 runners) in the public repository Nextia-Intelligence/ticket-api-ci, Docker Desktop 4.81.0 (Engine 29.6.1, Compose 5.2.0) on macOS (Apple silicon), and Azure CLI 2.88.0 in canadacentral. The Windows and Linux steps are not tested.

Two routes

Section titled “Two routes”

This module has two routes, as in the Docker course. Choose one, and use it in all three lessons. You can also do both.

Live Simulated
Where staging and production run Two Azure container apps, ticket-api-staging and ticket-api-production Two Compose projects on your computer, ticket-api-staging and ticket-api-production
Who deploys The Deploy workflow in GitHub Actions You, with scripts/promote-local.sh
Cost Practically zero, if you delete it on the same day Free
You need An Azure subscription Only Docker

Every output has a label. Live outputs come from the real runs in Nextia-Intelligence/ticket-api-ci. Simulated outputs come from Docker on a Mac on the same day. The Azure resources of the test run no longer exist, so their addresses do not answer now.

One image, many environments

Section titled “One image, many environments”

Amira’s CI run on main built one image and tested it. Its digest is sha256:4eb3db8d…. The help desk wants to try the API before customers get it. The team wants /docs there, so that testers can send requests in the browser. Production must hide /docs.

Sam asks: “Can CI build a second image for staging, with the docs on?” Amira says no. A second build is a second artifact. Nobody tested it, and nobody can prove that it has the same code and the same packages. So she keeps one artifact (the built and tested result of the pipeline, here the image) and changes only its settings.

An environment is one place where the application runs, with its own settings, secrets, access and data. This course has two:

  • Staging is a copy of production for checks. The team tries a new version there before users see it.
  • Production is the environment that users and other systems call.

The image in staging is a release candidate: a build that can become the release, if every check passes. When it reaches production, nothing in it changes. Only the environment around it is different.

Must be the same in every environment May differ, and must be outside the image
The code The settings: SHOW_DOCS, CLASSIFIER_VERSION, the app name, the address
The packages (requirements-run.txt) The secrets: each environment has its own API key
The base image and the Dockerfile The identity that deploys it, and what that identity may change
The image digest The data store: each environment has its own database, or none

The rule is simple: if a difference is in the right column, put it in a setting. If someone wants a difference in the left column, it is a new version, and it goes through CI again.

At the top, one image built and tested once by CI: ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d..., which has the tags sha-92fff66 and 1.1.0. An arrow goes to the GitHub environment staging (branch policy main, no reviewer, so it starts at once; variables APP_NAME, APP_URL, AZURE_CLIENT_ID and RESOURCE_GROUP; secret SMOKE_API_KEY with the staging key). A second arrow goes to the GitHub environment production (branch policy main, a required reviewer who must approve; the same variable names with other values; secret SMOKE_API_KEY with the production key). From each GitHub environment, an arrow says: sign in with OIDC, no password. It goes into the Azure resource group rg-ticket-api-ci, with the Container Apps environment cae-ticket-api-ci in canadacentral. There, each environment has its own managed identity, id-deploy-staging or id-deploy-production, with one federated credential and the role Container Apps Contributor on its own app only. Each identity updates its own container app. ticket-api-staging runs the image at sha256:4eb3db8d with SHOW_DOCS=true, so /docs answers 200, its own api-key secret, and REQUIRE_API_KEY=true. ticket-api-production runs the same digest with SHOW_DOCS=false, so /docs answers 404, its own api-key secret, and REQUIRE_API_KEY=true. Neither app has a database: history is off.
The environment boundaries of the live route. The digest is the same in both columns. Everything else in a column belongs to that environment only: its settings, its key, its identity and its permission.
Text description of the diagram

The digest at the top is the only thing that the two environments share. Each column is a boundary. Staging’s settings, key and identity cannot reach production’s app, and the other way round. Production has one more rule than staging: a person must approve before its job starts. This lesson makes the settings. Credentials and access makes the identities and the GitHub environments. Promote to staging moves the digest from left to right.

Where the settings live

Section titled “Where the settings live”

The settings must live somewhere that the image does not. Each route has its own place for them.

Live: the container app holds its settings and its secret. You set them when you create the app, and the pipeline changes only the image. A deployment with az containerapp update --image keeps every other setting of the app.

Simulated: the project holds one settings file per environment, and one git-ignored key file per environment:

deploy/local/staging.env
# Settings of the local staging environment. No secrets here.
ENVIRONMENT=staging
PORT=8081
SHOW_DOCS=true
CLASSIFIER_VERSION=1.1
deploy/local/production.env
# Settings of the local production environment. No secrets here.
ENVIRONMENT=production
PORT=8082
SHOW_DOCS=false
CLASSIFIER_VERSION=1.0

The settings files are in Git: they hold no secret, and a reviewer must see every change to them. The keys are in secrets/staging_api_key.txt and secrets/production_api_key.txt, and secrets/ is in .gitignore. The database password is in .env, which is git-ignored too.

CLASSIFIER_VERSION is there for Module 4. Version 1.1.0 of the API does not read it yet. You see what that means in the worked example.

One Compose file describes one environment. The environment’s name comes from the settings file:

deploy/local/compose.yaml
# One environment (staging or production) on your computer: the API and its
# own database. The image is given by digest, so every
# environment runs exactly the same build. Start it with scripts/promote-local.sh.
name: ticket-api-${ENVIRONMENT:?Set ENVIRONMENT}


services:
  api:
    image: ${IMAGE:?Set IMAGE to the image name with its digest}
    ports:
      - "127.0.0.1:${PORT:?Set PORT}:8000"
    environment:
      API_KEY_FILE: /run/secrets/api_key
      REQUIRE_API_KEY: "true"
      DATABASE_URL: postgresql://tickets:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}@db:5432/tickets
      SHOW_DOCS: ${SHOW_DOCS:-false}
      CLASSIFIER_VERSION: ${CLASSIFIER_VERSION:-1.0}
    secrets:
      - api_key
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2)"]
      interval: 5s
      timeout: 3s
      retries: 5
    depends_on:
      db:
        condition: service_healthy


  db:
    image: postgres:18.6-trixie
    environment:
      POSTGRES_USER: tickets
      POSTGRES_DB: tickets
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "tickets", "-d", "tickets"]
      interval: 2s
      timeout: 3s
      retries: 15


secrets:
  api_key:
    file: ../../secrets/${ENVIRONMENT}_api_key.txt


volumes:
  pgdata:

Four lines do the separation:

Line What it does
name: ticket-api-${ENVIRONMENT…} Each environment is its own Compose project. Its containers, network and volume names start with ticket-api-staging or ticket-api-production.
image: ${IMAGE…} The image comes from outside, by digest. The file has no image of its own, so it cannot run a different build by accident.
file: ../../secrets/${ENVIRONMENT}_api_key.txt Each environment reads its own key.
pgdata: Each project gets its own volume, so each environment has its own database.

:? makes a missing value an error, not an empty string. If you forget the settings file, Compose stops with Set ENVIRONMENT, and nothing starts.

This is the file of the reference repository’s pull request #1. It has one gap on Apple silicon, which Promote to staging shows and fixes.

Separate data stores

Section titled “Separate data stores”

Staging and production never share a database. A tester’s tickets in staging must not appear in production’s history. A migration that fails in staging must not touch production’s rows. In Module 4, each environment runs the migrations on its own database for this reason.

  • Simulated: each Compose project has its own PostgreSQL container and its own volume, ticket-api-staging_pgdata and ticket-api-production_pgdata.
  • Live: the course has no database in the cloud, to keep the cost at zero. Both apps run without DATABASE_URL, so history is off, and /v1/history answers 503 by design. If you add a database later, give each environment its own server or its own database, with its own password.
At the top, the same image as the cloud, pulled by digest from GHCR: ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d.... It goes to two commands. scripts/promote-local.sh staging, which you run first, reads deploy/local/staging.env (PORT=8081, SHOW_DOCS=true, CLASSIFIER_VERSION=1.1), the git-ignored secret secrets/staging_api_key.txt and the git-ignored password in .env. scripts/promote-local.sh production, which you run after staging passes, as your approval, reads deploy/local/production.env (PORT=8082, SHOW_DOCS=false, CLASSIFIER_VERSION=1.0), secrets/production_api_key.txt and .env. Each command runs docker compose up -d --wait and then smoke.sh, and makes one Compose project. The project ticket-api-staging has the api container with the same digest on 127.0.0.1:8081, where /docs answers 200, with platform linux/amd64, emulated on arm64, and the key at /run/secrets/api_key; a db container, postgres:18.6-trixie, with its own database; and the volume ticket-api-staging_pgdata, which stop-local.sh --volumes removes. The project ticket-api-production has the same parts on 127.0.0.1:8082, where /docs answers 404, with the volume ticket-api-production_pgdata. The two projects share no container, network, volume or key.
The local route: the same picture as the cloud, with two Compose projects. Your own command for production takes the place of the reviewer's approval.
Text description of the diagram

The simulated route keeps every boundary that you can have on one computer. Each environment has its own project, port, settings file, key and database. The image is the same digest as in the cloud. What it cannot simulate: the identities, the HTTPS address, and the approval by another person. In the simulation, you are the reviewer, and your second command is the approval.

Worked example (live): two apps, one image

Section titled “Worked example (live): two apps, one image”

Amira makes the resource group and the environment, as in the Docker course, with the course’s names:

Terminal (live route)
az group create --name rg-ticket-api-ci --location canadacentral --tags purpose=practice course=C19 -o table
az containerapp env create --resource-group rg-ticket-api-ci --name cae-ticket-api-ci --location canadacentral -o table

The output (live, shortened):

Location       Name
-------------  ----------------
canadacentral  rg-ticket-api-ci


WARNING: No Log Analytics workspace provided.
WARNING: Generating a Log Analytics workspace with name "workspace-rgticketapiciPwmn"
Container Apps environment created. To deploy a container app, use: az containerapp create --help


Location        Name               ResourceGroup
--------------  -----------------  ----------------
Canada Central  cae-ticket-api-ci  rg-ticket-api-ci

Then she makes the two apps from the same image. Only --name, the key and SHOW_DOCS differ. The image is the digest of the first green CI run on main (commit c862935). There is no registry in Azure: the image is in GitHub Container Registry, and the package is public, so Azure pulls it with no password.

Terminal (live route)
IMAGE=ghcr.io/usmanahmed99/ticket-api-ci@sha256:6c9bb4ed7c7cffc27f0c5489ba867d4d5db16def1c59c8bcbaeb3d26c45ce2e1
STAGING_KEY=$(python3 -c "import secrets; print(secrets.token_urlsafe(24))")
az containerapp create --resource-group rg-ticket-api-ci --name ticket-api-staging --environment cae-ticket-api-ci \
  --image "$IMAGE" --ingress external --target-port 8000 \
  --cpu 0.25 --memory 0.5Gi --min-replicas 0 --max-replicas 2 \
  --secrets api-key="$STAGING_KEY" \
  --env-vars API_KEY=secretref:api-key REQUIRE_API_KEY=true SHOW_DOCS=true

Use your own image name and digest. The output (live, shortened):

Container app created. Access your app at https://ticket-api-staging.blackhill-8c0e6b69.canadacentral.azurecontainerapps.io/


Fqdn                                                                       Revision
-------------------------------------------------------------------------  ---------------------------
ticket-api-staging.blackhill-8c0e6b69.canadacentral.azurecontainerapps.io  ticket-api-staging--cbaey1q

For production, she runs the same command with three changes: --name ticket-api-production, a new key in PRODUCTION_KEY, and SHOW_DOCS=false. The output (live, shortened):

Container app created. Access your app at https://ticket-api-production.blackhill-8c0e6b69.canadacentral.azurecontainerapps.io/


Fqdn                                                                          Revision
----------------------------------------------------------------------------  ------------------------------
ticket-api-production.blackhill-8c0e6b69.canadacentral.azurecontainerapps.io  ticket-api-production--hx6e7q9

Keep the two keys in the terminal (or in a password manager) until Credentials and access. There, each key goes into its GitHub environment as SMOKE_API_KEY, so that the pipeline’s smoke check can call the app.

The proof came later in the day. After release 1.1.0, both apps ran the image ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d…. The deploy log of run 37665743749 shows the same image line in both jobs (live):

staging / Deploy to staging       | IMAGE: ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a
production / Deploy to production | IMAGE: ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a

And /docs answered 200 in staging and 404 in production (live). Same bytes, different behaviour, because of one setting.

Worked example (simulated): compare the two environments

Section titled “Worked example (simulated): compare the two environments”

Before she starts anything, Amira asks Compose what each environment would run. docker compose config reads the Compose file and the settings files, and prints the result with every variable filled in. It starts nothing.

Terminal (simulated route)
export IMAGE=ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a
docker compose -f deploy/local/compose.yaml --env-file deploy/local/staging.env --env-file .env config > reports/staging-config.yaml
docker compose -f deploy/local/compose.yaml --env-file deploy/local/production.env --env-file .env config > reports/production-config.yaml
diff reports/staging-config.yaml reports/production-config.yaml

Predict first: which lines will diff print? Will the image: line be one of them?

The output (simulated; the secret file paths shortened):

1c1
< name: ticket-api-staging
---
> name: ticket-api-production
10c10
<       CLASSIFIER_VERSION: "1.1"
---
>       CLASSIFIER_VERSION: "1.0"
13c13
<       SHOW_DOCS: "true"
---
>       SHOW_DOCS: "false"
30c30
<         published: "8081"
---
>         published: "8082"
61c61
<     name: ticket-api-staging_default
---
>     name: ticket-api-production_default
64c64
<     name: ticket-api-staging_pgdata
---
>     name: ticket-api-production_pgdata
67,68c67,68
<     name: ticket-api-staging_api_key
<     file: .../ticket-api-ci/secrets/staging_api_key.txt
---
>     name: ticket-api-production_api_key
>     file: .../ticket-api-ci/secrets/production_api_key.txt

Every difference is in the right column of the table: the name, two settings, the port, the network, the volume and the key. The image: line is not in the list, because it is the same in both files:

    image: ghcr.io/usmanahmed99/ticket-api-ci@sha256:4eb3db8d4ea249fb9b44b27d96deb68aba6da3e3fbc9cf32311a9cbadf4d857a

This is a check that you can repeat before every deployment: compare what will run, and look for a difference in the left column.

Later, in Promote to staging, Amira starts both projects and calls them. Two results matter here (simulated):

  • /docs answered 200 on port 8081 and 404 on port 8082, as in the cloud.
  • POST /v1/classify with “Package never arrived” gave {"category":"other","priority":3,"confidence":0.2,"model_version":"keywords-1.0"} in both environments, although staging has CLASSIFIER_VERSION=1.1.

The second result is not a mistake in the setup. Version 1.1.0 of the code does not read CLASSIFIER_VERSION. A setting that the code ignores changes nothing, and no error tells you. That is why you check a setting by its effect (here /docs), not by its presence in a file. Module 4 adds the code that reads the flag, and then the same files give keywords-1.1 in staging.

Guided practice: the local environment files

Section titled “Guided practice: the local environment files”

Work in your project folder, on a new branch: git switch -c feature/deploy. The branch merges into main by pull request in Promote to staging: Guided practice: promote your own digest.

  1. Make the folder deploy/local, and write deploy/local/compose.yaml, staging.env and production.env as above.

  2. If .env does not exist, copy .env.example to .env, and change the password in it.

  3. Make the two keys. Each one is new, and only for that environment:

    Terminal
    mkdir -p secrets reports
    python3 -c "import secrets; print(secrets.token_urlsafe(24), end='')" > secrets/staging_api_key.txt
    python3 -c "import secrets; print(secrets.token_urlsafe(24), end='')" > secrets/production_api_key.txt

  4. Check that Git does not see the keys: git status --short. Expected: Git shows deploy/, and secrets/, reports/ and .env are not.

  5. Set IMAGE to your own image with the digest from build-info.json, as in the worked example.

  6. Write both configurations into reports/, and compare them with diff (or Compare-Object).

  7. Find the image: line in both files. Expected: the same digest, letter for letter.

  8. Commit the three files in deploy/local/: git add deploy/local and git commit -m "Local staging and production environments".

Show the expected result

diff lists only the project name, CLASSIFIER_VERSION, SHOW_DOCS, the port (8081 and 8082), the network, the volume and the key file, as in the worked example. Both files have the same image: line with your digest. git status --short shows ?? deploy/ before the commit, and nothing about secrets/, reports/ or .env. After step 8, git log --oneline -1 shows your commit on feature/deploy.

Your turn: a third environment

Section titled “Your turn: a third environment”

The help desk lead asks for a training environment, training, where new staff can try the API with the docs on. Write deploy/local/training.env with PORT=8083. Do not change compose.yaml.

Before you run docker compose config with the new file, predict:

  1. the project name,
  2. the name of its volume,
  3. the key file that it needs.

Then run it, and check your answers. What else must you make before it can start?

Show the answer

ENVIRONMENT=training gives the project ticket-api-training, the volume ticket-api-training_pgdata and the key file secrets/training_api_key.txt. A test with a third file printed the matching names, and published: "8083". docker compose config does not check that the key file exists, so you must make secrets/training_api_key.txt yourself, with a new key. The new environment needed no change to the code, the image or the Compose file: only a settings file and a key. That is the sign of good configuration separation.

A second build for the release

Section titled “A second build for the release”

Common mistake

The tag that started a new build

Symptom: the first version of the reference repository’s ci.yml also ran on version tags. A push of the tag v1.1.0 would start the whole CI workflow again, and the image job would build a new image for the tag. Staging had checked the image of the main run, not this one. Nothing in the pipeline could prove that the two builds were the same: a base image or a package can change between two builds.

Cause: the pipeline treated “make a release” as “build”. This is the most common mistake in delivery pipelines: a new artifact for each environment or each step.

Fix: a tag must not build. Pull request #2 removed the tag trigger from ci.yml:

.github/workflows/ci.yml (pull request #2, part)
push:
  branches: [main, "hotfix/**"]
  tags: ["v*"]

and added release.yml, which gives the existing image its version tag. The test run measured the result: the tag 1.1.0 and the tag sha-92fff66 have the same digest, sha256:4eb3db8d…. Promote to staging shows the workflow.

For a real AI service: an image often holds a model file or downloads one at build time. A second build can pick up a different model, and its predictions differ from the ones that staging checked. Promote one artifact by digest, and the predictions in production are the ones that you tested.

Check your understanding

Sam wants /docs in staging and asks CI to build a staging image with SHOW_DOCS set to true inside it. What do you answer, and what do you do instead?

Show the answer

A second image is a second artifact. Nobody tested it, and nobody can prove that it has the same code and packages. Keep one image, and put the difference in a setting outside the image: SHOW_DOCS=true in staging (the container app’s environment variable, or deploy/local/staging.env) and false in production. Then both environments run the same digest, and the check in staging is a check of the production build.

Check your understanding

Predict: you set CLASSIFIER_VERSION=1.1 in staging.env and start version 1.1.0 of the API. Which model_version does staging report, and why?

Show the answer

keywords-1.0, the same as production. Version 1.1.0 of the code does not read CLASSIFIER_VERSION, so the setting has no effect, and nothing reports an error. Check a setting by its effect, for example /docs (200 in staging, 404 in production), not by its presence in a file. The flag works from the version that reads it (Module 4).

Check your understanding

Name two things that must be the same in staging and production, and two that must differ. Why must the data store differ?

Show the answer

The same: the image digest and the code (also the packages and the Dockerfile, which are inside the image). Different: the settings (such as SHOW_DOCS) and the secrets (each environment’s own key), also the identity and the data store. The data store must differ, so that test data from staging never appears in production. Also, a migration or a mistake in staging then cannot change production’s records.

Recap

  • One artifact, many environments: staging and production run the same image digest. Only settings, secrets, identities and data stores differ.
  • Live: each container app holds its own settings and secret. Simulated: deploy/local/<environment>.env and the git-ignored secrets/<environment>_api_key.txt.
  • One Compose file with name: ticket-api-${ENVIRONMENT} gives each environment its own project, port, key and database.
  • docker compose config and diff show what differs before anything runs. The image: line must be the same.
  • Check a setting by its effect. Version 1.1.0 ignores CLASSIFIER_VERSION.
  • A release must not build again: a tag that triggers a second build makes an artifact that nobody tested.

The apps exist, but nobody can deploy to them yet without your own Azure sign-in. The next lesson gives each environment an identity that needs no password and can change only its own app.

Next: Credentials and access