Build pipeline

What you will be able to do

Turn the project into one artifact that runs the same way on any computer. Make it only after a fixed list of automated checks has passed on a clean computer. Without this, “it works” means “it works on my laptop, today”.

Before you startWhat to know and to have readyAbout 90 minutes

What you need to know

• What continuous integration is, and why checks run on a clean runner, from From commit to release and Define checks in CI/CD, Testing and Safe Releases.
• Why you build an image once and ship that build, from Build and test: Build once, then keep that build in the CI/CD course.
• A Dockerfile, the layer cache and .dockerignore, from Dockerfile essentials, Build efficiently and Runtime safety in Docker and Cloud Essentials for AI Developers.
• An evaluation that blocks a release, from AI evaluation gate in the CI/CD course.
• In this course: the held-out set and its lock, from Plan the evidence; the acceptance thresholds, from Acceptance criteria; the isolation checks and the held-out evaluation, from Run held-out tests.

What you need on your computer

• Your project at the end of Module 5. For the guided practice, a fresh copy of the snapshot end-of-m06, as in Use a snapshot.
• Docker Desktop running (docker version shows a server), and about 5 GB of free disk for the image and its build cache.
• A terminal in the project folder, with the virtual environment active.

Tested on macOS (Apple silicon) with Python 3.12.13 and Docker Engine 29.6.1. The Windows and Linux commands are written, not tested by the course team. The GitHub Actions workflow is written, not run by the course team: its steps were run one by one on the same computer.

A cake for a party

Ana bakes a cake for a friend’s party. She does not bake it in her friend’s kitchen, where the oven, the flour and the time are all different. She bakes it at home. Before she leaves, she does the same three things every time: she tastes a slice, she checks that no nuts went in, and she writes the date on the box. Then she carries that cake, in its box, to the party.

Two ideas hide in this habit. First, the checks are a fixed list, done every time, not only when she remembers. Second, the cake that she checked is the cake that arrives. If she baked a second cake at the party, nobody would have tasted that one.

The problem: it works on Sam’s laptop

Sam’s assistant works. On Sam’s laptop, the page answers, the citations open, and the isolation checks say 0 leaks. Then Omar asks for a demo on Friday, from the computer in the meeting room. Amira asks to review Sam’s last change on her own laptop.

What does Sam’s laptop have that another computer does not? More than Sam thinks:

• the virtual environment, about 990 MB with the CPU build of torch;
• the two search models in the Hugging Face cache, about 1 GB, downloaded at the first question;
• .env with three secrets, and .idp/ with the practice provider’s signing key;
• a database with 51 documents already loaded;
• and Sam’s memory of the commands, in the right order.

None of these is in Git. So “it works” is a fact about one laptop, on one day.

Changes add a second problem. On Friday afternoon, Tomás makes a “small fix” (a constructed change, for this lesson). Q37, the French barbecue question, is the held-out question that hybrid search broke in Document trade-offs. Tomás says that a table question belongs in the tuning set, and he moves it out of heldout.jsonl into dev.jsonl. The held-out score would then rise. Sam runs the check from Module 1 on that copy:

Terminal

python -m scripts.holdout check

heldout.jsonl changed after it was reserved. A changed held-out set cannot show that you did not tune on it: restore it (git checkout evaluation/heldout.jsonl).

The check works, and it exits with code 1. But it helps only if somebody runs it. At 6 p.m. on a Friday, nobody remembers to.

The idea: a build pipeline

The answer is Ana’s habit, done by a computer. A build pipeline is a fixed list of automated checks that a clean computer runs on every change, in a fixed order. If every check passes, the pipeline packages the project into one artifact. The artifact that passed the checks is the one that you ship.

Three names come with this idea:

• Continuous integration (CI) is the first half: the checks run on every change, on a clean computer that has only what is in Git. That computer is called a runner. “Continuous”, because it happens on every push. “Integration”, because a change joins the main branch only after it passes.
• An artifact is the packaged result that you ship. Here it is a container image: the code, the Python packages, the two models and the data pack, in one file system that runs the same on every computer with Docker.
• A tag is the image’s version name, for example doc-assistant:1.0.0. Each release gets a new tag. The old tags stay, so you can go back to one.

The checks are Ana’s tasting. The image is the box. “Carry that cake” means: do not build again for the demo. Run the image that passed the checks.

A tiny example: six checks, in order

The test job of Sam’s pipeline has six steps. Each step answers one question, and each one catches a mistake that the others miss:

Step	Command	It catches	Time
Lint	ruff check .	an unused import, a name that does not exist	0.3 s
Held-out set unchanged	python -m scripts.holdout check	Tomás’s move of Q37	under 0.1 s
Tests	python -m pytest	a broken endpoint, a missing access check, a migration error	40.8 s
Load the pack	python -m scripts.load_pack	a document that no longer parses, a model that does not download	30.4 s
Isolation	python -m scripts.isolation	one shop’s passage in the other shop’s results	6.4 s
Evaluation smoke test	python -m scripts.evaluate --set heldout --min-correct 0.75	a change that makes the answers worse while every test passes	4.8 s

The times come from one run on the course team’s laptop, on a fresh copy of the snapshot, with other work running. Your times will differ.

The order is deliberate. The two fastest checks come first, so a typo fails in under a second, not after a minute of tests. The checks that need the database come next. The checks that load the models come last. If any step fails, the job stops, and no image is built.

The time of each step of the test job, run one by one on a fresh copy of the end-of-m06 snapshot. The tests take the longest; lint and the held-out lock take less than a second.

Text description of the chart

A column chart with six columns, one per step of the test job, in the order of the job. Lint takes 0.3 s and the held-out lock under 0.1 s, so their columns are almost flat. The tests take 40.8 s, loading the pack 30.4 s, the isolation checks 6.4 s and the evaluation smoke test 4.8 s. The whole job takes about 83 s on this laptop.

The whole flow of this module is in one picture. This lesson builds the first group and the image. The next lesson starts the image and runs the smoke test. The last lesson of the module rolls back.

The deployment flow of this module: checks first, one build, a start that waits for readiness, a smoke test, and a rollback to the previous tag when the start or the smoke test fails.

Text description of the diagram

The first group is the test job. Six checks run in order: lint, the held-out lock, the tests with PostgreSQL, loading the 51 documents of the pack, the isolation checks with 0 leaks, and the evaluation smoke test with a floor of 0.75 correct. A failed step stops the job, and no image is built. When all pass, the second group starts. The image is built once and tagged doc-assistant:1.0.0. The stack starts, and Compose waits until /ready answers. The data is loaded inside the container, and the smoke test runs its 7 checks. If all 7 pass, the release is done, and the previous tag is kept. Two red paths lead to the rollback: the API never gets ready, or a smoke check fails. The rollback starts the previous tag with its settings and keeps the data. Then the smoke test runs again.

The evaluation smoke test is a gate

A smoke test is a short, quick check that the most important thing still works; the next lesson explains the name. Tests check that the code does what Sam wrote. They cannot check that the answers are still good. A new prompt or a new retrieval setting can pass all the tests and still answer worse. So the last check runs the held-out questions and fails the job below a floor:

Terminal

python -m scripts.evaluate --set heldout --min-correct 0.75

hybrid, answer_v2, recorded: 41 questions | correct 0.878 | retrieval hit 0.886 | citations valid 1.000 | abstained when it should 0.833 | injection obeyed 0 of 2 | median 9 ms | US$0.000167 per question
  answerable       12  correct 0.833  hit 0.833
  contradiction     3  correct 1.0  hit 1.0
  exact-code        4  correct 1.0  hit 1.0
  injection-bait    2  correct 1.0  hit 1.0
  other-shop        2  correct 0.5  hit None
  staff-only        5  correct 1.0  hit 1.0
  table             5  correct 0.8  hit 0.8
  unanswerable      4  correct 1.0  hit None
  version           4  correct 0.75  hit 0.75

The floor, 0.75, comes from criterion Q1, which Sam and Grace wrote in Module 1, before any measurement. The run replays the recorded answers. So it needs no key, costs nothing, and gives the same numbers on every run: 0.878 correct, 36 of 41.

To see what a red gate looks like, raise the floor above the result. This is not a real threshold. It only shows the failure:

Terminal

python -m scripts.evaluate --set heldout --min-correct 0.9

hybrid, answer_v2, recorded: 41 questions | correct 0.878 | retrieval hit 0.886 | ...
  ...
FAILED: correct 0.878 is below 0.9

The command exits with code 1, and CI marks the job as failed.

“But Module 1 said: look at the held-out answers once, at the end. Does CI use them up?” Priya answers: the gate prints only the totals and the counts per kind, never a question or an answer. Nobody tunes on a pass or a fail. If a real change fails the gate, you look for the cause on the tuning set, as in Inspect error categories.

Package once: the image

The image is the box. Sam’s Dockerfile reuses the Docker course’s patterns, and each comment says why a line is there:

Dockerfile

# The document assistant's image: the API, the worker thread, the practice identity provider,
# the scripts and the data pack, and the two local models (downloaded at build time, so the
# container needs no internet when it runs).
FROM python:3.12.13-slim-trixie

ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 HF_HOME=/opt/hf PIP_NO_CACHE_DIR=1
WORKDIR /app

# The CPU build of torch first (the default build on Linux is the much larger CUDA one).
RUN pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu
COPY requirements.txt .
RUN grep -v -E '^(pytest|ruff)==' requirements.txt > requirements-run.txt && pip install -r requirements-run.txt

# The models, pinned to the revisions in doc_assistant/embed.py and rerank.py
COPY doc_assistant/embed.py doc_assistant/rerank.py /tmp/models/
RUN python -c "import sys; sys.path.insert(0, '/tmp/models'); import embed, rerank; \
from sentence_transformers import SentenceTransformer, CrossEncoder; \
SentenceTransformer(embed.LOCAL_MODEL, revision=embed.LOCAL_REVISION); \
CrossEncoder(rerank.RERANK_MODEL, revision=rerank.RERANK_REVISION)" && rm -rf /tmp/models

COPY doc_assistant doc_assistant
COPY idp idp
COPY migrations migrations
COPY recordings recordings
COPY scripts scripts
COPY data data
COPY evaluation evaluation
ARG APP_VERSION=1.0.0
ENV APP_VERSION=${APP_VERSION} HF_HUB_OFFLINE=1

RUN useradd --create-home --uid 10001 app && mkdir -p traces && chown app traces
USER app
EXPOSE 8000
CMD ["fastapi", "run", "doc_assistant/main.py", "--host", "0.0.0.0", "--port", "8000"]

Five choices in this file matter for a capstone:

1. Pinned versions. The base image has its exact Python version (3.12.13-slim-trixie), torch has its exact version, and the models have their exact revisions. The same file builds the same image next month.
2. The CPU build of torch first. On Linux, pip install torch installs the build for NVIDIA graphics cards by default. That build is several times larger, and the assistant does not use a graphics card.
3. The models are inside the image. They download during the build. HF_HUB_OFFLINE=1 then forbids any download while the container runs. A demo computer without internet, or with a slow connection, still answers.
4. No test tools, no secrets. requirements-run.txt drops pytest and ruff. .dockerignore keeps .env, .idp, .venv, tests and .git out of the build, so no secret can enter a layer.
5. A user that is not root. The container runs as app, so a mistake in the code cannot change the system files of the container.

The cost of choice 3 is size. The image has about 1.1 GB of compressed content, and about 3.7 GB on disk once unpacked. Most of it is torch and the two models. The course team’s first build took about 107 s. A build that changes only the code takes much less, because the slow layers come first and stay in the cache, as in Build efficiently. The other way is a small image that downloads the models when it starts. That image is faster to push, but every start needs the internet and is slow. For a demo on an unknown computer, the bigger image is the safer choice.

The workflow file

GitHub Actions reads .github/workflows/ci.yml. It has two jobs. test runs the six checks, with PostgreSQL as a service container. image runs only if test passed (needs: test). It builds the image, starts the whole stack, loads the data and runs the smoke test against it. The next lesson explains those steps. Here are the important parts:

.github/workflows/ci.yml (parts)

jobs:
  test:
    runs-on: ubuntu-24.04
    services:
      db:
        image: pgvector/pgvector:0.8.7-pg18-trixie
        # ... a health check, so the steps start only when PostgreSQL answers
    steps:
      # ... check out, Python 3.12, CPU torch first, then requirements.txt
      - name: Lint
        run: ruff check .
      - name: The held-out set has not changed
        run: python -m scripts.holdout check
      - name: Tests
        run: python -m pytest
      - name: Load the data pack (downloads the two models)
        run: python -m scripts.load_pack
      - name: Isolation checks (must be 0 leaks)
        run: python -m scripts.isolation
      - name: Evaluation smoke test (recorded answers, held-out set)
        run: python -m scripts.evaluate --set heldout --min-correct 0.75

  image:
    needs: test
    # ... builds the image with the commit as its version, starts the stack,
    #     loads the data, runs scripts/smoke.py, prints the logs on failure, stops

The database password in the workflow is ci-only-not-secret. It protects a database that exists for a few minutes on a runner. So it is not a secret, and its name says so.

A pipeline file that nobody runs on their own computer drifts away from the project. So one test keeps them together. test_the_release_files_agree in tests/test_operations.py checks that the Dockerfile has USER app and HF_HUB_OFFLINE=1, and that Compose and CI use the same PostgreSQL image. It also checks that CI still has every step of the table above, and that .dockerignore names .env and .idp. If somebody deletes the isolation step, a test fails on their own laptop.

Guided practice: run the pipeline on a clean copy

You run the six checks and the build on a fresh copy of the snapshot, as a CI runner would. The copy is your “clean computer”.

1. Copy the snapshot end-of-m06 to a new folder, for example ~/projects/doc-assistant-ci. Keep your own project where it is.

2. Make the environment, the settings and the practice provider’s keys:

   macOS / Linux

   cd ~/projects/doc-assistant-ci
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   cp .env.example .env
   python -m idp init

   Windows (PowerShell)

   cd ~\projects\doc-assistant-ci
   py -3.12 -m venv .venv
   .venv\Scripts\Activate.ps1
   pip install -r requirements.txt
   copy .env.example .env
   python -m idp init

   On Linux without a graphics card, first run pip install torch==2.14.1 --index-url https://download.pytorch.org/whl/cpu. If another project uses port 5432, set DB_PORT in .env, and change the port in the two database URLs.

3. Start the database: docker compose up -d --wait db. The last line ends with Healthy.

4. Run the two fast checks: ruff check ., then python -m scripts.holdout check. You see:

   All checks passed!
   Held-out set: 41 questions (bramble 14, larkfield 27). SHA-256 matches the lock (e8eda11340d6...).
   Tuning set (dev.jsonl): 62 questions (bramble 20, larkfield 42).

5. Run the tests: python -m pytest. The project’s settings already make the output short: one dot per test, then a last line that starts with 108 passed. The tests take about 40 s on the course team’s laptop.

6. Load the pack: python -m scripts.load_pack. If the two models are not in your cache yet, this downloads them first. The end of the output:

   Indexed 51 documents in 29.5 s (failed 0).
     bramble: 15 documents, 60 passages, 0 failed
     larkfield: 36 documents, 185 passages, 0 failed

7. Run the isolation checks: python -m scripts.isolation. The last line is Isolation checks: 414 probes, 0 leaks..

8. Run the gate: python -m scripts.evaluate --set heldout --min-correct 0.75. The first line says correct 0.878, and the exit code is 0.

9. Build the image. The first build takes a few minutes:

   macOS / Linux

   APP_VERSION=1.0.0 docker compose --profile app build api
   docker image ls doc-assistant

   Windows (PowerShell)

   $env:APP_VERSION = "1.0.0"
   docker compose --profile app build api
   docker image ls doc-assistant

   You see one image, doc-assistant with the tag 1.0.0. Its disk usage is about 3.7 GB, and its content size about 1.1 GB. The content size is the compressed layers that a registry stores; the disk usage is the unpacked image on your computer.

Keep the database and the image: the next lesson uses them. If you need the memory now, docker compose down stops the database and keeps its data.

If you add the release files to your own Module 5 project instead, your test count is 104. The last lesson of this module adds the four tests of tests/test_operations.py.

Your turn: break one check on purpose

Change one condition, and predict the result before you run it. In your clean copy, open doc_assistant/limits.py. Just above the line from .errors import ApiError, add the line import os and an empty line, and do not use os. Which step of the table fails first, and how long does it take?

Show the answer

The lint step fails first, in well under a second:

F401 [*] `os` imported but unused
  --> doc_assistant/limits.py:8:8
   |
 6 | """
 7 |
 8 | import os
   |        ^^
 9 |
10 | from .errors import ApiError
   |
help: Remove unused import: `os`
...
Found 1 error.
[*] 1 fixable with `--fix`.

ruff reports the unused import (rule F401) and exits with code 1. In CI, the job stops here, and the tests never start. That is why the fast checks come first: a mistake this small should not cost a minute of tests. Remove the line, or run ruff check --fix ., and the step passes again.

Common mistake

A check that is not in the pipeline

Symptom: a constructed case. Sam’s pull request is green. Two days later, Priya runs the held-out evaluation by hand. The share of correct answers is lower than 0.878. Nobody knows which change caused it.

Cause: in this case, the workflow had lint and tests, but no evaluation step. A retrieval change passed all the tests, because the tests use a small hash embedder and check behaviour, not answer quality. Nothing checked the answers on every change.

Diagnosis: compare the steps of ci.yml with the evidence plan. Every claim that a change can break needs a step. “The answers are good enough” had none.

Fix: add the gate step, with the floor from the acceptance criteria. test_the_release_files_agree lists the required steps, so the step cannot disappear again without a failing test.

Consequence in AI work: in an AI product, most regressions are in the answers, not in the code. A pipeline with only tests protects the part that breaks least often.

Your project

Do the same for your own product, on the standard or the own-idea track.

1. Write the Dockerfile. Start from Sam’s. If your models are different, change the download step, and keep the pinned revisions. If your product calls a hosted model only, you can drop torch and get a much smaller image.
2. Write .dockerignore. Check that .env, your key files and any data that is not public are in it.
3. Write the workflow with the six steps, in the same order. Use your own acceptance criterion from Module 1 as the floor of the gate. Do not choose a floor after you see the result.
4. Run every step on a clean copy of your repository, as in the guided practice. A step that passes only in your working folder is not a check yet.
5. If your repository is on GitHub, push it and let Actions run it. If you do not use GitHub, say so in your README, and keep the output of the local run.

Evidence to keep: Dockerfile, .dockerignore, .github/workflows/ci.yml, the output of the six steps on a clean copy (or the link to a green run), and the image tag and size.

Checklist

• The six steps run in the order fast to slow, and a failed step stops the job.
• The evaluation floor is your own acceptance criterion, written before you measured.
• The image has pinned versions, runs as a user that is not root, and contains no secret.
• A test checks that the release files agree.
• You ran the whole job on a clean copy, not only in your working folder.

See it end to endReview a weak capstoneCase study

In the case study, a submission’s pipeline is green because two steps were removed from its workflow: the isolation checks and the held-out evaluation. A green pipeline proves only what it checks.

Knowledge check

Check your understanding

Ana tastes a slice of her cake at home, then carries that cake to the party. Which two parts of a build pipeline does this habit match, and what would 'baking a second cake at the party' be?

Show the answer

The tasting is the fixed list of checks that runs on every change (continuous integration). Carrying the same cake is shipping the artifact that passed the checks: one image, with one tag. Baking a second cake at the party is building the image again for the demo or for production. Nobody checked that second build, and it can differ, for example if a package or the base image changed in between.

Check your understanding

Predict: a pull request changes the answer prompt. All 108 tests pass. Which step of the test job can still fail, and why do the tests not see the problem?

Show the answer

The evaluation smoke test can fail: it runs the held-out questions and refuses a result below 0.75 correct. The tests check behaviour (status codes, citations that exist, access rules) with a small hash embedder and short fixed answers. They do not measure whether the answers are right. With a changed prompt, no recorded answer matches any more, so the mock answers instead, and the share of correct answers falls. The gate sees it; the tests do not.

Check your understanding

Sam's image is about 1.1 GB because the two models are inside it. Tomás wants a small image that downloads the models when it starts. Give one reason for each choice, and say which one you would pick for a demo on an unknown computer.

Show the answer

Models inside: the image starts with no internet, always with the same pinned model files, and no download slows the first question. Models downloaded at start: a smaller image, faster to push and pull, and a model update needs no new build. For a demo on an unknown computer, keep the models inside. A missing or slow connection in a meeting room would break the demo, and the pinned files keep the answers the same as in the evaluation.

Recap

• “It works on my laptop” depends on files that are not in Git: the environment, the models, the secrets, a loaded database and your memory of the commands.
• A build pipeline runs a fixed list of checks on a clean computer on every change. Only if all pass does it build one artifact, an image with a tag. You ship that image, and you do not build again.
• Sam’s test job has six steps, fast to slow: lint, the held-out lock, the tests, loading the pack, the isolation checks, and an evaluation gate with the floor from criterion Q1.
• The evaluation gate catches what tests cannot: answers that get worse while the code still works.
• The image pins every version, keeps the models inside, runs as app, and contains no secret. It has about 1.1 GB of compressed content, mostly torch and the models.

Next, you start this image with its database and its identity provider. You check that it is ready, and you prove the core journey with a smoke test.

Next: Release the demo