AI evaluation gate

What you will be able to do

A unit test proves that the code does what it says. It cannot prove that a new model is better for the people who use it. This lesson adds a check that can. It scores a fixed set of labelled tickets on every pull request, and it blocks a model change that makes the important cases worse.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The CI workflow, its job “Lint, test, audit”, and scripts/check.sh as its local copy, from Build and test: Checks that mean something and The same checks on your computer.
• The reports artifact and the ruleset with its required checks, from Publish evidence: The test report, even when the tests fail and Protect main with a ruleset.
• One image with different settings per environment, from Configuration separation: Where the settings live. Your deploy/local/staging.env already has CLASSIFIER_VERSION=1.1. Version 1.1.0 ignores it. After this lesson, the code reads it.
• Accuracy and recall, if you want a longer explanation than the one below: Classification metrics: Precision and recall. A test set that you use only once: Train, validation and test sets.

What you need on your computer

• Your repository github.com/<you>/ticket-api-ci at version 1.1.0, with .github/workflows/ci.yml, scripts/check.sh and the virtual environment from requirements-lock.txt. They come from Build and test: Guided practice: add the checks and the image job. The tag v1.1.0 comes from Promote to staging: Guided practice: promote your own digest.
• The ruleset on main with the required checks “Lint, test, audit” and “Build, smoke-test and push the image”, from Publish evidence: Protect main with a ruleset.
• If your project is missing or broken, copy the snapshot end-of-m03, as in Use a snapshot. A snapshot holds files, not GitHub settings: make the ruleset again if your repository is new.

Run with GitHub Actions (ubuntu-24.04 runners, Python 3.14.8) in the public repository Nextia-Intelligence/ticket-api-ci. Also run on macOS (Apple silicon) with Python 3.14.6. The Windows and Linux steps are not tested.

A test is not an evaluation

The help desk lead wants a better classifier. Amira has one: keywords-1.1, the same method as keywords-1.0 with more words for each category. Before it reaches users, she must answer one question: is it better, and is it better for the tickets that matter most?

The 25 tests cannot answer that. A test checks one input and one expected output that the developer chose: “Refund” gives billing. A classifier is wrong for some tickets by design. The question is how often, and for which tickets. So Amira needs a different kind of check, an evaluation: run the classifier on many tickets whose right answer a person wrote down, and count.

Three tickets show the idea:

Ticket	Right answer (a person wrote it)	keywords-1.0 says	keywords-1.1 says
“Package not delivered”	shipping	other	shipping
“Charged twice”	billing, priority 1	billing, priority 1	billing, priority 1
“Two more seats”	account	other	other

On these three, 1.0 is right once and 1.1 twice. Each word in that sentence needs a definition:

• A labelled ticket is a ticket with the right answer next to it: the category and the priority.
• The evaluation set is the list of labelled tickets. Here it is evaluation/tickets.jsonl, 51 tickets.
• Accuracy is the share of tickets with the right category: 2 of 3 is 0.667.
• Recall of a category is the share of the tickets in that category that the classifier finds. If there are 7 account tickets and it finds 6, the recall of account is 0.857. Accuracy can look good while one small category is almost never found. Recall per category shows it.
• A must-pass ticket is a ticket that must always be right, with the right priority, whatever the averages say. “Charged twice” is one: a customer who lost money must reach a person first. The output of the gate calls them “must-pass cases”.

Dev and holdout: why some tickets stay unseen

Amira chose the new words of keywords-1.1 by reading tickets. If she measures 1.1 on the same tickets, it looks better than it is: she fitted it to them. So the set has two parts:

• 18 dev tickets (“development”): she read them to choose the words.
• 33 holdout tickets: nobody used them to choose anything. They stand for the tickets of next month.

The difference is large. On the dev tickets, keywords-1.1 has an accuracy of 1.0: it is right on every ticket that chose its words. On the holdout tickets, it has 0.909. The second number is the honest one, so the gate uses only the holdout tickets. If you change the keywords because of a holdout ticket, that ticket is no longer unseen: move it to dev, and add new holdout tickets.

Where such a set comes from

Here the set is supplied, as the outline asks: 51 tickets, a script, and three settings files. Building a good evaluation set is its own skill: choosing the tickets, labelling them with two people, finding the hard cases. The course Evaluating and Testing AI Systems teaches it (not published yet). The command python -m evaluation.run works the same way with any classifier that has a predict method, so a set from the evaluation course can replace this one.

The gate: three questions

A gate is a check that stops the pipeline when its answer is no. The evaluation gate asks three questions. All three must say yes.

The three parts of the gate, with the real numbers of pull request #4. Two parts passed. One failed, and that was enough to block the merge.

Text description of the diagram

The gate runs the classifier on the 33 holdout tickets and checks three things. Floors: no release may go below an accuracy of 0.6 or a recall of 0.5 in any category. No regression: the new version must not be worse than the version in production, the baseline (keywords-1.0, accuracy 0.636). Accuracy may drop by at most 0.02, and the recall of each category by at most 0.1. Must-pass: the three must-pass tickets must have the right category and the right priority. In pull request #4, parts 1 and 2 passed and part 3 failed on E033, so the gate failed.

The settings are in evaluation/thresholds.json:

evaluation/thresholds.json

{
  "split": "holdout",
  "baseline": "evaluation/baseline.json",
  "min_accuracy": 0.6,
  "min_recall_per_category": 0.5,
  "max_accuracy_drop": 0.02,
  "max_recall_drop": 0.1,
  "must_pass_all": true
}

baseline.json holds the results of keywords-1.0, the classifier in production. The thresholds are task-specific: they come from this task and this team, not from a textbook. The help desk can live with a classifier that is wrong for one ticket in three (the floor). It cannot live with a release that is worse than today (no regression). It cannot live with a lost parcel in the normal queue (must-pass).

The first draft of this gate failed in production

Amira’s first draft had one rule: accuracy at least 0.85. Then she ran it on the classifier that was already in production. The output (simulated, with the draft’s thresholds):

keywords-1.0 on 33 holdout tickets (baseline keywords-1.0)
accuracy 0.636
...
FAIL: accuracy 0.636 < 0.85
Gate failed.

A gate that the current production version fails is not useful. Every release would need an exception, and soon nobody looks at it. A fixed number says “good enough in general”. The baseline says “not worse than what users have today”. So the gate compares with the baseline, and keeps only low floors for the cases that no release may reach.

Worked example: keywords-1.1 behind a flag

Amira works on a branch, feature/classifier-1.1, made from main. Her change has three parts.

1. The classifier. keywords-1.1 is a subclass with more words. Version 1.0 stays the same, so production does not change:

ticket_api/classifier.py (the new part)

# keywords-1.1: more words for each category, found in the evaluation set's
# development cases. Login words are checked before "account", so "I cannot
# log in to my account" is a login ticket.
KEYWORDS_1_1 = {
    "billing": KEYWORDS["billing"] + ("bill", "overcharged", "receipt", "subscription"),
    "login": KEYWORDS["login"] + ("reset link", "verification code", "2fa", "can't sign"),
    "shipping": KEYWORDS["shipping"] + ("package", "arrived", "delivered", "shipment"),
    "account": KEYWORDS["account"] + ("email address", "profile", "delete my"),
}
URGENT_WORDS_1_1 = URGENT_WORDS + ("can't sign", "never arrived", "urgent")


class KeywordClassifier11(KeywordClassifier):
    """keywords-1.1: the same method with more words."""

    version = "keywords-1.1"
    keywords = KEYWORDS_1_1
    urgent_words = URGENT_WORDS_1_1

make_classifier("keywords", version) returns the class for "1.0" or "1.1", and stops with unknown CLASSIFIER_VERSION for any other value. The full file is ticket_api/classifier.py at 19d172b.

2. The flag. A feature flag is a setting that turns a feature on or off without a new build. Here it is CLASSIFIER_VERSION, read in config.py, with 1.0 as the default:

ticket_api/config.py (one new line in load_settings)

        classifier_version=os.environ.get("CLASSIFIER_VERSION", "1.0"),

main.py passes it on: make_classifier(settings.classifier_mode, settings.classifier_version). Two new tests check that 1.1 gives keywords-1.1 and that the default is keywords-1.0. The new code can go to production with the flag off, and Progressive release turns it on step by step.

3. The evaluation. Amira adds the supplied evaluation/ folder and runs it for both versions. First the version in production:

Terminal (in the project, with the virtual environment active)

python -m evaluation.run --classifier 1.0

The output (simulated):

keywords-1.0 on 33 holdout tickets (baseline keywords-1.0)
accuracy 0.636
recall   account  0.571
recall   billing  0.625
recall   login    0.625
recall   other    1.0
recall   shipping 0.571
must-pass failures: 0
Gate passed.

The production version passes its own gate: it is the baseline. Now the candidate:

Terminal

python -m evaluation.run --classifier 1.1

The output (simulated; the same lines as in CI):

keywords-1.1 on 33 holdout tickets (baseline keywords-1.0)
accuracy 0.909
recall   account  0.857
recall   billing  0.875
recall   login    0.875
recall   other    1.0
recall   shipping 1.0
must-pass failures: 0
Gate passed.

Accuracy goes from 0.636 to 0.909: 30 of 33 tickets right instead of 21. Every recall goes up. The three tickets that 1.1 still gets wrong are hard on purpose. For example, E051 “Two more seats” (account) has none of the account words. The exit code is 0 when the gate passes and 1 when it fails, so a pipeline step can use it directly.

The command also writes reports/evaluation.json, the evidence for the review:

reports/evaluation.json (shortened)

{
  "classifier": "keywords-1.1",
  "split": "holdout",
  "cases": 33,
  "accuracy": 0.909,
  "must_pass_failures": [],
  "baseline": "keywords-1.0",
  "problems": [],
  "passed": true
}

The CI job already uploads reports/ as the reports artifact. So the evaluation report goes next to the JUnit report, with no new step.

The gate in CI

Amira adds one step to the checks job, after the security check, and adds the same line to scripts/check.sh:

.github/workflows/ci.yml (the checks job, changed lines)

  checks:
    name: Lint, test, audit, evaluate
    # ...
      - name: AI evaluation gate
        run: python -m evaluation.run --classifier 1.1

Two decisions are in these lines:

• The gate tests 1.1, the candidate, although production still runs 1.0. The question of the gate is “may this change reach users?”, so it tests what the change brings. When 1.1 becomes the baseline later, baseline.json changes in its own pull request.
• The job’s name changes to “Lint, test, audit, evaluate”, because the name is what people read in the checks list. This rename has a cost, as the common mistake at the end of the lesson shows.

She pushes and opens pull request #3, feature/classifier-1.1 into main. Its CI run, run 37666141730, passed: the checks job took 25 seconds, gate included. The full file is ci.yml at 19d172b.

Worked example: the change that the tests liked

After the merge, Sam tries keywords-1.1 in staging. Sam’s report from Define checks: 25 tests passed, and the urgent ticket was not urgent is still true with the new classifier: “I lost my password” gets priority 1. Sam is right. It is a normal login ticket, and priority 1 is for the urgent queue. The word “lost” is in the urgent words.

Amira opens pull request #4, feature/fewer-urgent-tickets into main. It removes "lost" from the urgent words of keywords-1.1. “All tests pass,” she writes. Predict before you read on: which parts of the gate will pass?

The first run, run 37666448935, failed on something smaller. Her one-line list was too long for the linter:

E501 Line too long (101 > 100)
  --> ticket_api/classifier.py:22:101
...
Found 1 error.

She wrapped the list, one word per line, and pushed again. The second run, run 37666571062, failed at the gate:

The public job page of run 37666571062 (logged out). Lint, tests and the security check passed. Only the AI evaluation gate failed, so the image job did not run.

The log of that step (live, from the run; the log needs a GitHub sign-in to read on the page):

keywords-1.1 on 33 holdout tickets (baseline keywords-1.0)
accuracy 0.909
recall   account  0.857
recall   billing  0.875
recall   login    0.875
recall   other    1.0
recall   shipping 1.0
must-pass failures: 1
FAIL: must-pass cases failed: E033
Gate failed.
##[error]Process completed with exit code 1.

Every number is the same as before the change. The averages did not see anything. But E033, “Parcel lost in transit: The tracking stopped and the parcel is lost”, now gets priority 2. Without the word “lost”, nothing in that ticket is urgent. A customer whose parcel is lost waits in the normal queue.

keywords-1.1 on the 33 holdout tickets	Accuracy	account	billing	login	other	shipping	Must-pass failures	Gate
keywords-1.0 (baseline)	0.636	0.571	0.625	0.625	1.0	0.571	0	passed
keywords-1.1 (PR #3)	0.909	0.857	0.875	0.875	1.0	1.0	0	passed
keywords-1.1 without “lost” (PR #4)	0.909	0.857	0.875	0.875	1.0	1.0	1 (E033)	failed

This is the point of the must-pass part. The change fixed Sam’s ticket, which is not in the set, and broke the most important ticket that is. Accuracy counts categories, not priorities, so it did not move. Only a check that names the ticket saw it. Amira closed the pull request with the reason:

Pull request #4, closed with a comment that says which ticket failed and why the change cannot merge. The next person who tries the same change can read it.

The gate did not decide what to do about Sam’s ticket. It made one bad answer impossible to merge by accident. The right fix, a rule that tells a lost parcel from a lost password, is a new change with its own evaluation.

Guided practice: add the gate to your repository

Do this in your own repository. Each step names the branch.

1. On main, pull the latest changes. Then make the branch: git switch -c feature/classifier-1.1.
2. Copy the supplied evaluation files into a new evaluation/ folder. The commands are below the steps. Check that you have five files: __init__.py, run.py, thresholds.json, baseline.json, tickets.jsonl (51 lines).
3. Add keywords-1.1 and the flag: change ticket_api/classifier.py, ticket_api/config.py and ticket_api/main.py as in the worked example. Copy them from commit 19d172b if you prefer.
4. Add the two tests of the flag to tests/test_classify.py (from the same commit). Run python -m pytest -q. Expect 27 passed.
5. Run python -m evaluation.run --classifier 1.0, then --classifier 1.1. Compare with the outputs in the worked example.
6. Add the step “AI evaluation gate” to ci.yml, and rename the checks job to “Lint, test, audit, evaluate”. Add the step to scripts/check.sh.
7. Run scripts/check.sh. Expect “Gate passed.” in the gate step, and “All checks passed.” at the end.
8. Commit, push the branch, and open a pull request: feature/classifier-1.1 into main.
9. Wait for the checks. Both pass. Look at the merge box: it still says that a required check is missing. Fix it as in the common mistake, then merge.

The commands for step 2 download the files from the reference repository, at the commit that added them:

macOS and Linux

Terminal (in the project)

mkdir -p evaluation
base=https://raw.githubusercontent.com/Nextia-Intelligence/ticket-api-ci/19d172b/evaluation
for f in __init__.py run.py thresholds.json baseline.json tickets.jsonl; do
  curl -fsSL -o "evaluation/$f" "$base/$f"
done
wc -l evaluation/tickets.jsonl

Windows (PowerShell)

PowerShell (in the project)

New-Item -ItemType Directory -Force evaluation | Out-Null
$base = "https://raw.githubusercontent.com/Nextia-Intelligence/ticket-api-ci/19d172b/evaluation"
foreach ($f in "__init__.py", "run.py", "thresholds.json", "baseline.json", "tickets.jsonl") {
  Invoke-WebRequest -Uri "$base/$f" -OutFile "evaluation/$f"
}
(Get-Content evaluation/tickets.jsonl).Count

Show the expected result

wc -l says 51 evaluation/tickets.jsonl. The two evaluation runs give the two outputs of the worked example, with exit code 0 (check with echo $?). pytest -q says 27 passed: the 25 tests from the Docker course and the two new ones. In your pull request, the check “Lint, test, audit, evaluate” passes and its log ends with Gate passed. The run’s reports artifact now holds evaluation.json next to junit.xml. Before you change the ruleset, the merge box waits for “Lint, test, audit”, a check that no workflow runs any more.

Your turn: make Sam’s ticket a must-pass ticket

Sam’s ticket matters too. Try it as a fourth must-pass ticket, without changing the classifier. This is a throwaway experiment: add the line on your computer, and do not commit it. Add it at the end of the file:

evaluation/tickets.jsonl (a new last line)

{"id": "E052", "split": "holdout", "subject": "I lost my password", "body": "", "category": "login", "priority": 2, "must_pass": true}

Predict first: what does the gate say for keywords-1.1? And for keywords-1.0, the version in production? Then run both.

Show the answer

Both fail (simulated):

keywords-1.0 on 34 holdout tickets (baseline keywords-1.0)
accuracy 0.647
...
must-pass failures: 1
FAIL: must-pass cases failed: E052
Gate failed.
keywords-1.1 on 34 holdout tickets (baseline keywords-1.0)
accuracy 0.912
...
must-pass failures: 1
FAIL: must-pass cases failed: E052
Gate failed.

Both versions put “lost” in the urgent words, so both give E052 priority 1. A must-pass ticket is a requirement. If you add one that production already fails, every pull request fails, also the ones that have nothing to do with it. So a new must-pass ticket goes in the same pull request as the fix that makes it pass. The gate then proves the fix. Here, that fix must keep E033 urgent too: for example, treat “lost” as urgent only with a shipping word.

Undo the experiment before you go on: git restore evaluation/tickets.jsonl. Then wc -l evaluation/tickets.jsonl says 51 again, and git status shows no change. The rest of the course, and the snapshots, use these 51 tickets. Final assignment: Part 1: requirements and checks starts from them, and adds Sam’s ticket for real, together with the fix.

A renamed check

One more thing happened in pull request #3, before the merge.

Common mistake

The check was renamed, and the pull request cannot merge

Symptom (live): in pull request #3, both checks were green: “Lint, test, audit, evaluate” and “Build, smoke-test and push the image”. But the merge button stayed off. gh pr view 3 --json mergeStateStatus said BLOCKED. Nothing on the page was red.

Cause: the ruleset on main requires checks by name. It waited for “Lint, test, audit”, the old name of the job. No workflow runs a job with that name any more, so the required check stays “expected” for ever. A rename looks like a cosmetic change, but for the ruleset it removes one check and adds a new one that nobody requires.

Fix:

1. Open the repository’s Settings → Rules → Rulesets, and open the ruleset of main (“Protect main” in the reference repository).
2. Under Require status checks to pass, remove “Lint, test, audit”.
3. Add “Lint, test, audit, evaluate”. The list offers the names of checks that ran recently, so it is there after the first run with the new name.
4. Save. The merge button in the pull request becomes active at once, because the new required check already passed.

Do not fix it by removing the required check, or by switching the ruleset off “only for this merge”. That leaves main open for every merge after it.

For a real AI service: the gate is only a gate if the ruleset requires it. A pipeline can run the evaluation, show a red cross, and still let the merge through, if the check is not required. After you add or rename a check, open a pull request and look at the merge box: it must list the new name as required.

Check your understanding

Pull request #4 did not change the accuracy (0.909) or any recall. Why did the gate still fail, and why could the averages not see the problem?

Show the answer

The must-pass part failed: E033, “Parcel lost in transit”, got priority 2 instead of 1, because “lost” was the only urgent word in it. Accuracy and recall count only the category, and E033 was still shipping. A priority error on one ticket out of 33 also moves no average much. A must-pass ticket names the ticket and checks the category and the priority, so one broken important ticket is enough to fail.

Check your understanding

Amira wants to choose new keywords by reading the tickets that keywords-1.1 gets wrong in the holdout split. What is the problem, and what should she do?

Show the answer

The holdout tickets would then help to choose the words, so they are no longer unseen. The gate would measure the classifier on tickets that it was fitted to, and the score would look better than the score on next month’s tickets, as on the dev split (1.0 instead of 0.909). She can move the tickets that she reads to the dev split. Then she adds new labelled holdout tickets that nobody has read, so the gate keeps an honest test.

Check your understanding

Predict: Amira changes the gate in ci.yml to python -m evaluation.run --classifier 1.0. Production runs 1.0. Is the gate now safer?

Show the answer

No. The gate would always test the classifier that is already in production, so it passes on every pull request, whatever the pull request changes in keywords-1.1. Pull request #4 would have merged. The gate must test the version that the change brings (here 1.1, the candidate), and compare it with the version in production (the baseline).

Recap

• A test checks one chosen answer. An evaluation scores a classifier on a fixed set of labelled tickets: accuracy, recall per category and must-pass tickets.
• The gate uses the holdout tickets: nobody used them to build the classifier. On the dev tickets, 1.1 looked perfect (1.0). On the holdout, it had 0.909.
• The gate has three parts: floors, no regression against the baseline in production, and must-pass tickets with the right priority. A fixed high threshold failed the production version itself.
• Pull request #4 passed lint (after a fix), every test and the security check. Accuracy stayed at 0.909. It still failed: E033 lost its priority 1.
• A required check is required by name. After a rename, update the ruleset, or a green pull request stays BLOCKED.

Next: Migration safety