Evaluation gates

What you will be able to do

Turn the evidence about a candidate release into one decision that a script can enforce: ship it to a canary, or do not ship it. The decision combines offline quality, critical failures, latency and cost. Any one failed check stops the release.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Release manifests and experiment records, from Artifact lineage and Experiment records. EXP-001 says “do not ship” for prompt v3.
• The SLOs and the cost budget of US$0.40 per 1,000 tickets, from Service objectives.
• The release decision of the evaluation course: blockers, a paired difference with its 95% interval, and a limit. See Trade-offs and regressions.

What you need on your computer

• Your project ticket-api at the end of Module 4, or a copy of the snapshot end-of-m04. python -m pytest prints 207 passed.
• Docker running PostgreSQL and Valkey (docker compose up -d db cache), for the migration and the tests.

No account, no key and no money. The gate reads files only.

The pilot’s checklist

Before every flight, a pilot reads a checklist: fuel, flaps, instruments, doors. Each line is pass or fail. If one line fails, the plane does not take off. A clear sky and a full cabin do not change that.

The checklist has three useful properties:

• It is written before the flight. Nobody argues about the fuel limit while passengers wait.
• Some lines are absolute. A door that does not close is never “mostly fine”.
• Passing it is not the end. After take-off, the crew still watches the instruments.

A release of the assistant needs the same thing.

The problem: a decision that nobody had to read

Remember “last week”. The evaluation course had run prompt v3 on its frozen holdout tickets, and its decision was DO NOT SHIP. Priya wrote it in the experiment record EXP-001. Then someone switched prompt v3 on for every ticket as a settings change. Nothing in the path of that change asked for the decision.

In the practice replay of the incident, 186 of the 1,798 tickets that v3 handled got no answer. 10 tickets that needed a person reached nobody. The answers are real recordings from the evaluation course; the switch is constructed.

The evidence existed. It was just not in the way of the change. A decision that lives in a document can be skipped. A decision that a script checks before every rollout cannot.

The idea: one gate, three kinds of evidence

An evaluation gate, or release gate, is a check that runs before a candidate release reaches customers. It reads the candidate’s evidence and a file of limits, and it gives one decision. Every line must pass.

The gate combines three kinds of evidence:

1. Blockers: critical failures. A list of tickets where a failure is never acceptable, such as a safety ticket without an answer. One failure on one of them stops the release, whatever the averages say.
2. Quality, with its uncertainty. The candidate is compared with the live release on the same tickets. The gate uses the lower end of the 95% interval of the difference, not the average. The question is “could it plausibly be worse than we accept?”, not “is it better on average?”.
3. Operational limits. Cost per 1,000 tickets and the 95th percentile of the triage time. A correct answer that breaks the budget or the time objective still hurts the service.

The gate is the first step of a release path. A pass does not mean “every customer gets it”. It means “it may now go to a small group of customers”. The next lesson, Controlled rollouts, covers that part.

The release path of this module. The gate decides with offline evidence before any customer sees the candidate. The shadow and the canary then test it on live tickets.

Text description of the diagram

A candidate release starts as a new release manifest. The gate checks its offline evidence against written limits. If any check fails, the active release stays, and the reason goes into the experiment record. If every check passes, the candidate can run as a shadow, which sees every ticket but answers no customer. Then it runs as a canary for a share of the tickets. If a rollback rule is met, the canary goes back to the active release, and the team proves that the service recovered. If enough tickets pass and no rule is met, the canary is promoted.

A tiny example: one blocker beats good averages

Here is a constructed candidate. It is cheaper and faster than prompt v2, and its decisions are about as good. But one safety ticket in the evaluation got no answer. Run the gate on it:

Run Python in your browser

A release gate in a few lines

The box prints one line for each check, then the decision. Five checks pass, and critical tickets failed fails, so the decision is DO NOT SHIP (1 failed). Set the blocker to 0 and run it again: SHIP to a canary. Then set the lower end to −0.05: DO NOT SHIP (1 failed) again.

Python code (you can change it)# A constructed candidate: cheaper and faster than prompt v2, but one # safety ticket got no answer. candidate = { "critical tickets failed": 1, "answers without text": 0, "decisions, lower end of new - old": -0.02, "needs a person, share caught": 24 / 24, "cost per 1,000 tickets (US$)": 0.30, "triage time p95 (s)": 4.1, } # The limits: "<=" means "at most", ">=" means "at least". limits = { "critical tickets failed": ("<=", 0), "answers without text": ("<=", 0), "decisions, lower end of new - old": (">=", -0.03), "needs a person, share caught": (">=", 1.0), "cost per 1,000 tickets (US$)": ("<=", 0.40), "triage time p95 (s)": ("<=", 8.0), } failed = 0 for check, value in candidate.items(): op, limit = limits[check] ok = value <= limit if op == "<=" else value >= limit failed += not ok print(f"{check:<35} {value:>6.2f} {op} {limit:<5} {'pass' if ok else 'FAIL'}") print("Decision:", "SHIP to a canary" if failed == 0 else f"DO NOT SHIP ({failed} failed)") # Try: set "critical tickets failed" to 0. Then set the lower end to -0.05.

The first run prints:

critical tickets failed               1.00 <= 0     FAIL
answers without text                  0.00 <= 0     pass
decisions, lower end of new - old    -0.02 >= -0.03 pass
needs a person, share caught          1.00 >= 1.0   pass
cost per 1,000 tickets (US$)          0.30 <= 0.4   pass
triage time p95 (s)                   4.10 <= 8.0   pass
Decision: DO NOT SHIP (1 failed)

Two things to notice:

• Nothing is averaged. A cheap, fast release cannot “earn back” a safety ticket without an answer.
• The quality check uses the interval. Suppose a candidate is better on average by 0.01, with an interval from −0.05 to +0.07. The true difference could then be 5 points worse: more than the 3 points that the team accepts. So the gate says no. This limit is a non-inferiority margin: “not worse by more than 3 points”. To pass, the candidate needs more evaluation tickets, or a real improvement.

Add the module’s files to your project

This module adds files for all three of its lessons: the gate, the rollout tools and the improvement table. Copy them now, once. Each lesson explains its own files.

1. In a terminal, go to your project and activate the virtual environment.

   macOS / Linux

   cd ~/projects/ticket-api
   source .venv/bin/activate

   Windows (PowerShell)

   cd ~\projects\ticket-api
   .venv\Scripts\Activate.ps1

2. Copy the module’s files from the snapshot end-of-m05. Twelve files are new, and four change.

   macOS / Linux

   S=../nextia-learning-labs/C25/snapshots/end-of-m05
   mkdir -p operations/evidence
   for f in migrations/020_release_state.sql ticket_api/rollout.py ticket_api/worker.py \
            ticket_api/releases.py ops/cmd_gate.py ops/cmd_rollout.py ops/cmd_canary.py \
            operations/gates.toml operations/rollout.toml operations/evidence/assistant-v2.json \
            operations/evidence/assistant-v3.json scripts/scenario.py scripts/practice.py \
            docs/improve.md tests/test_rollout.py README.md; do
     cp "$S/$f" "$f"
   done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C25\snapshots\end-of-m05"
   New-Item -ItemType Directory -Force operations\evidence | Out-Null
   $files = 'migrations\020_release_state.sql', 'ticket_api\rollout.py', 'ticket_api\worker.py',
            'ticket_api\releases.py', 'ops\cmd_gate.py', 'ops\cmd_rollout.py', 'ops\cmd_canary.py',
            'operations\gates.toml', 'operations\rollout.toml', 'operations\evidence\assistant-v2.json',
            'operations\evidence\assistant-v3.json', 'scripts\scenario.py', 'scripts\practice.py',
            'docs\improve.md', 'tests\test_rollout.py', 'README.md'
   foreach ($f in $files) { Copy-Item "$S\$f" $f }

3. Apply the new migration. It adds two tables for the next lesson and keeps your data.

   python -m ticket_api.migrate

   applied  020_release_state
   Database is up to date (20 migrations).

4. The code and the last migration changed, so your old release manifests no longer describe this service. Build new ones for prompt v2 and prompt v3:

   python -m ops release build --prompt assistant-v2 --evidence baseline-v2-chat-small-r1-b6a2b9 --write
   python -m ops release build --prompt assistant-v3 --evidence candidate-v3-chat-small-r1-31b45f --write

   The last line of each:

   Saved operations/releases/1.5.0-5a4f87.toml. Run the service with RELEASE_ID=1.5.0-5a4f87
   Saved operations/releases/1.5.0-422985.toml. Run the service with RELEASE_ID=1.5.0-422985

   Your IDs are the same if your files are exactly the snapshot’s files and your project is not a Git repository. Otherwise they differ, and that is correct.

5. Open .env. If it has a RELEASE_ID line, set it to your new v2 ID, for example RELEASE_ID=1.5.0-5a4f87. Building the v3 manifest does not deploy it.

6. Run the tests. Expect 213 passed, 6 more than before:

   python -m pytest

   213 passed, 1 deselected in 21.45s

Files	What they add	Lesson
operations/evidence/, operations/gates.toml, ops/cmd_gate.py	the evidence of prompt v2 and v3, the limits, python -m ops gate	this one
migrations/020_release_state.sql, ticket_api/rollout.py, ticket_api/worker.py, ops/cmd_rollout.py, ops/cmd_canary.py, operations/rollout.toml, scripts/scenario.py, scripts/practice.py	active, shadow and canary releases; rollback rules; the practice scenarios	Controlled rollouts
docs/improve.md	from a symptom to the change that fixes its cause	Retrain or revise
ticket_api/releases.py, tests/test_rollout.py, README.md	a clearer index line in new manifests; tests; the commands	all three

Run the gate on prompt v3

The gate’s evidence for each prompt is a small file in operations/evidence/. It holds the evaluation course’s results on its 98 frozen holdout tickets: tickets that nobody used to write or tune the prompts. The file names the run and the evaluation set, d4c96eaa10c0, so you can trace every number back.

Run the gate for the candidate:

python -m ops gate assistant-v3

Release gate for assistant-v3 (baseline assistant-v2), evidence: candidate-v3-chat-small-r1-31b45f on the holdout split of d4c96eaa10c0
check                                            value  limit     result  detail
----------------------------------------------  ------  --------  ------  --------------------------
blockers (critical cases failed)                     1  <= 0      FAIL    T-65663 no_answer_critical
answers without text (holdout)                       8  <= 0      FAIL    of 98 tickets
decisions: lower end of (candidate - baseline)  -0.184  >= -0.03  FAIL    -0.112 [-0.184, -0.041]
needs a person: share caught                     0.958  >= 1.0    FAIL    23 of 24
cost per 1,000 tickets (US$)                     0.480  <= 0.4    FAIL    baseline 0.339
triage time p95 (s)                              8.534  <= 8.0    FAIL    baseline 4.896
Decision: DO NOT SHIP (6 failed: blockers (critical cases failed), answers without text (holdout), decisions: lower end of (candidate - baseline), needs a person: share caught, cost per 1,000 tickets (US$), triage time p95 (s))

Read it row by row:

• Blocker: T-65663, a safety ticket, got no answer. That alone is enough.
• Answers without text: 8 of 98 holdout tickets. The model used all its tokens to reason and wrote nothing.
• Decisions: v3 is worse by 0.112, and the interval reaches −0.184. At the plausible worst, that is 18 points worse, against a margin of 3.
• Needs a person: 23 of the 24 tickets that needed a person went to one. One customer would wait for nobody.
• Cost: US$0.480 per 1,000 tickets, over the budget of US$0.40. Omar wanted v3 to be cheaper; v2 costs US$0.339 on the same tickets.
• Time: the 95th percentile of the triage call is 8.534 s, over the limit of 8.0 s.

The command also ends with exit code 1. A release script or a CI job reads that code and stops. That is what puts the decision in the way of the change:

macOS / Linux

python -m ops gate assistant-v3 > /dev/null; echo $?

Windows (PowerShell)

python -m ops gate assistant-v3 | Out-Null; $LASTEXITCODE

1

The gate for the two prompts, with the values of python -m ops gate. Prompt v3 fails every check. Prompt v2 passes every check and may go to a canary.

Text description of the diagram

The gate reads two inputs: the offline evidence from the evaluation course, and the limits file. It runs six checks in three groups. The blockers: no critical ticket may fail, and no answer may be without text. The quality: the lower end of the difference must be at least −0.03, and every ticket that needs a person must be caught. The operations: at most US$0.40 per 1,000 tickets, and a triage p95 of at most 8 seconds. Prompt v3 fails all six. Prompt v2 passes all six, so it may go to a canary.

The gate also passes the live prompt

A gate must be able to say yes, or nobody trusts it. Run it on prompt v2, the live prompt:

python -m ops gate assistant-v2

Release gate for assistant-v2 (baseline assistant-v2), evidence: baseline-v2-chat-small-r1-b6a2b9 on the holdout split of d4c96eaa10c0
check                                           value  limit     result  detail
----------------------------------------------  -----  --------  ------  -----------------------
blockers (critical cases failed)                    0  <= 0      pass    none
answers without text (holdout)                      0  <= 0      pass    of 98 tickets
decisions: lower end of (candidate - baseline)  0.000  >= -0.03  pass    +0.000 [+0.000, +0.000]
needs a person: share caught                    1.000  >= 1.0    pass    24 of 24
cost per 1,000 tickets (US$)                    0.339  <= 0.4    pass    baseline 0.339
triage time p95 (s)                             4.896  <= 8.0    pass    baseline 4.896
Decision: SHIP to a canary (the canary's own checks come next: python -m ops canary)

Here the candidate and the baseline are the same evaluation run, so the difference is exactly 0. The other checks are real: v2 has no blocker, catches 24 of 24, and stays inside the cost and time limits.

Look at the last words: SHIP to a canary. The offline evidence cannot see everything. The holdout has 98 tickets; Larkfield gets 822 a day. A rare failure can hide in the others. So the gate opens the next door, not the last one.

Where the limits come from

Open operations/gates.toml. Each limit comes from a decision that the team already made, and each has an owner:

Limit	Value	Comes from	Owner
blockers_max, answers_without_text_max	0	the safety SLO: every ticket that needs a person reaches one	Grace
decisions_difference_min	−0.03	the margin that Priya accepts: not worse by more than 3 points	Priya
needs_human_recall_min	1.0	the safety SLO again	Grace
cost_per_1000_usd_max	0.40	the cost budget in operations/slos.toml	Kwame
latency_p95_s_max	8.0	the triage-slow alert: p95 over 8 s	Amira

Write the limits before you see the candidate’s numbers. If you choose a limit after you see the result, you can make any candidate pass. A change to gates.toml is a change like any other: it needs a reason and a review.

“The judge preferred v3, 126 to 48”

Tomás reminds the team about the evaluation course. There, a model judge compared the replies and preferred v3’s warmer replies 126 times to 48. “Doesn’t that count?”

It counts as evidence about the tone of the replies. But it is an average preference, and it says nothing about the safety ticket without an answer. A gate does not weigh good news against bad news. It asks whether every line passes. The judge’s preference can be a line in a future gate, with its own limit. It can never cancel a blocker.

When to use a gate, and what it costs

Sam asks, “Do we need the gate for every change?”

Amira answers: “For every change that can change an answer.”

• Use the gate for a new prompt, model or model version, policy text, output schema, token limit or retrieval index. Each of these gets a new release manifest, and each needs fresh evidence.
• The evidence must match the release. Evidence from another prompt, another model or another evaluation set does not describe this release. The gate’s first line names the run, the split and the set: read them.
• A gate does not replace the canary. 98 holdout tickets catch big and frequent failures. They cannot prove that a failure in 1 of 1,000 tickets is absent.

What it costs: an evaluation run per candidate. The evaluation course’s five runs of prompt v3 cost about US$0.33 in model calls (prices change: check the provider’s current page). It also costs care. Someone must keep the holdout tickets frozen, the critical tickets listed and the limits current. That is cheap next to two days of unanswered safety tickets.

Common mistakes

Common mistake

ops gate: no evidence

Symptom: python -m ops gate assistant-v4 exits with code 2:

ops gate: no evidence operations/evidence/assistant-v4.json: evaluate assistant-v4 first

Cause: no evaluation has run for that prompt, so there is nothing to judge.

Fix: run the evaluation. Then save its results as operations/evidence/assistant-v4.json, with the same fields as assistant-v3.json. A missing file is a “no”, never a “yes”.

Common mistake

The tests or the API stop with a settings error

Symptom: after the migration, python -m pytest and the API stop at once with a message like this one (your ID may differ):

Settings error: RELEASE_ID 1.5.0-c7b614 does not match: migrations: the manifest ends with 019_answer_index.sql, the project with 020_release_state.sql

Cause: .env still names a manifest that you built before this module. The service refuses to run a release that does not describe its files.

Fix: do steps 4 and 5 above: build the new manifests, and set RELEASE_ID to the new v2 ID.

Reset. If your project is broken, copy the snapshot end-of-m05, keep your .env, and run python -m scripts.load --reset, as the snapshots README shows. Then do steps 4 and 5 again.

Knowledge check

Check your understanding

Name the three kinds of evidence that the release gate combines, and give one check of each from ops gate.

Show the answer

Blockers, or critical failures: no critical ticket may fail (prompt v3 failed T-65663). Quality with its uncertainty: the lower end of the 95% interval of the difference must be at least −0.03 (v3: −0.184). Operational limits: at most US$0.40 per 1,000 tickets (v3: 0.480), or a triage p95 of at most 8.0 s (v3: 8.534 s).

Check your understanding

Predict: a candidate is better than v2 on average by 0.02, with a 95% interval from −0.04 to +0.08. Every other check passes. What does the gate decide, and why?

Show the answer

DO NOT SHIP. The gate reads the lower end, −0.04, which is below the margin of −0.03. The true difference could be 4 points worse, more than the team accepts. More evaluation tickets would make the interval narrower, and then the candidate might pass.

Check your understanding

Practical check: show that your gate stops prompt v3 and passes prompt v2. Which outputs and exit codes prove it?

Show the answer

python -m ops gate assistant-v3 prints six FAIL lines and Decision: DO NOT SHIP (6 failed: …), with exit code 1. python -m ops gate assistant-v2 prints six pass lines and Decision: SHIP to a canary …, with exit code 0. python -m pytest prints 213 passed.

Recap

• An evaluation gate (release gate) turns the evidence about a candidate into one decision. A script enforces it through the exit code.
• It combines blockers, quality and operational limits. One critical failure stops the release. Quality uses the lower end of the 95% interval against a margin. The limits cover cost and latency.
• Prompt v3 fails all six checks; prompt v2 passes all six. A pass means “ship to a canary”, not “ship to everyone”.
• Limits come from the SLOs and the budget, have owners, and are written before anyone sees the candidate’s numbers.
• Your project has the module’s files and new release manifests for v2 and v3, and python -m pytest prints 213 passed.

Next, the canary: prompt v3 on 10% of the tickets, the rule that catches it, and the proof that the rollback worked.

Next: Controlled rollouts