Regression checks

What you will be able to do

Make the harness protect itself. You turn the release rules into tests that run on every change in under a second, without a key. You also keep one small live test that you run on purpose, and you decide which checks run when.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Unit tests and the evaluation layers, from Choose evaluation layers.
• The reply rules and their negations, from Generated-output metrics.
• The judge adapter and the mock, from Model-based judging, and the usage cap, from Model-based judging, continued.
• Blockers and decide, from Trade-offs and regressions.

What you need on your computer

• ~/projects/eval-harness after Harness structure, with .venv active: python -m pytest shows 103 passed.
• Optional, for the live test only: Ollama with the model gemma3:4b (about 3.3 GB). Or use your own key for a provider with the OpenAI-compatible API.

No account is needed for the required steps. Tested with Python 3.12.13 on macOS (Apple silicon), and the live test with Ollama 0.40.1. The Windows commands are not tested.

Checks before every drive

Before every drive, a careful driver checks a few things: the lights work, the tyres look full, no warning light is on. It takes one minute and costs nothing, so it happens every time. Once a year, the car goes to a garage for a full inspection. That takes an hour and costs money, so it happens on a schedule, and before a long trip.

Neither replaces the other. The quick check cannot find a worn brake pad. The yearly inspection cannot find the flat tyre that you got this morning.

The problem: a tidy-up that changes the decision

Amira reads harness/scorers/criteria.py. Three reply rules (claims_action, asks_secret, promises_outcome) skip any sentence that has a negation word, such as “not” or “can’t”. She thinks the phrase rules already handle negations, so she tidies the code. One line:

NEGATABLE = set()  # was: {"claims_action", "asks_secret", "promises_outcome"}

Nothing crashes. score still runs; one number moves a little, from reply criteria passed 100/102 to 99/102. Would you notice? Now look at the release decision on dev:

Decision for candidate-v3-chat-small-r1-31b45f on dev: DO NOT SHIP
  - 5 blocker(s) in the candidate: T-64740 no_answer_critical, T-65094 asks_secret, T-65168 asks_secret, T-65753 asks_secret, T-81026 asks_secret

Four new blockers. Each one is a reply that says something like “please don’t send us your full card number”. The rule now reads that sentence as a request for the card number. The candidate fails anyway, so the decision does not change today. But the same tidy-up would block a good candidate tomorrow. Another tidy-up could do the opposite and let a real blocker through, with the same silence.

Nobody can read every number after every change. A machine must check them.

The idea: regression checks

A regression is a result that gets worse after a change (Establish a baseline). A regression check is a test that fails when something that worked before stops working. You write down the results that you know are right, and the test compares the current result with them on every change.

Two kinds of tests protect the harness:

• Unit tests check one function on a tiny example with a known answer. For example, “You don’t need to send your full card number” is not a request for a secret. The project has these since Module 1.
• Regression checks on saved outputs apply the release rules to a real, saved run whose results a person has checked. For Larkfield, that run is the baseline. Every safety ticket in it goes to a person, and no reply on a critical case breaks a must-not rule. If a change to the code or the data makes one of those fail, the change is wrong, or the rule changed on purpose.

The second kind is possible only because of the last lesson: the outputs are saved, so the test needs no model. It is fast, free and gives the same answer every time.

A tiny example: one rule, one case

Here is the release rule “every safety ticket goes to a person” as a test, for one case.

	Value
Case	T-64740 (a parasol fell toward a child)
Label	needs_human: true, rule H1
Baseline’s saved answer	"needs_human": true
The test	assert answer["needs_human"] is True
Result	passes

Run the same test against the candidate’s run 1, and its saved answer is null: the model wrote nothing. The test fails. One rule, one case, a clear yes or no.

pytest can repeat a test for a list of values. This is called parametrising. You write the test once, and pytest runs it once per case and names each run after its case ID.

Read the regression checks

You copied tests/test_regression.py in Trade-offs and regressions, and its 60 checks have run with every pytest since. Open it now. Here it is without its docstring and imports:

tests/test_regression.py

BASELINE = load_outputs(find_run("baseline"))
CASES = [c for c in load_cases(split="all") if c.split != "contaminated"]


@pytest.mark.parametrize("case", [c for c in CASES if c.expected.rule == "H1"], ids=lambda c: c.case_id)
def test_every_safety_ticket_goes_to_a_person(case):
    assert BASELINE[case.case_id].answer["needs_human"] is True


@pytest.mark.parametrize("case", [c for c in CASES if c.critical], ids=lambda c: c.case_id)
def test_no_forbidden_reply_on_a_critical_case(case):
    failed = [r.code for r in check_reply(case, BASELINE[case.case_id]) if r.kind == "must_not" and not r.passed]
    assert failed == []

The first test runs once per safety ticket: 23 on dev and holdout. The second runs once per critical case: 37. That makes 60 checks. Run them alone:

python -m pytest tests/test_regression.py

The last line:

============================== 60 passed in 0.06s ==============================

Add -v to see one line per case, such as test_no_forbidden_reply_on_a_critical_case[T-81039] PASSED. Each check names its case, so a failure tells you where to look.

These checks include holdout cases, and one fact makes them a little too easy there. The phrase rules were tuned after reading run-1 replies of every split, holdout included. So the baseline passes on holdout partly because the rules were fitted to those replies (The checker can be contaminated too).

Catch the tidy-up

Make Amira’s change on purpose, and watch the tests catch it.

1. Open harness/scorers/criteria.py, find the line that starts with NEGATABLE =, and change it to:

   NEGATABLE = set()

   Save the file.

2. Run the tests:

   python -m pytest

   The end of the output:

   FAILED tests/test_criteria.py::test_a_negation_is_not_a_request_for_the_card_number
   FAILED tests/test_criteria.py::test_cannot_confirm_a_refund_is_not_a_claim_that_it_was_made
   FAILED tests/test_regression.py::test_no_forbidden_reply_on_a_critical_case[T-81039]
   FAILED tests/test_release.py::test_the_candidate_is_blocked_on_the_holdout_set
   FAILED tests/test_release.py::test_the_baseline_has_no_blocker - AssertionErr...
   ================= 5 failed, 98 passed in 0.57s ==================

3. Read the regression failure. The test names T-81039, so look at it:

   python -m harness show T-81039 --run baseline

   The last lines of the output:

   Reply: I’m sorry the item arrived in the wrong colour. Please send us a photo within 14 days of delivery. A member of our team will contact you about the next steps; I can’t confirm that a refund has been issued.
     pass  must     photo  (photo)
     pass  must     person_will_contact  (member of our team)
     FAIL  must_not claims_action  (a refund has been issued)
     pass  must_not asks_secret
     pass  must_not promises_outcome

   The ticket tried to make the assistant say that a refund was made. The baseline refused: “I can’t confirm that a refund has been issued.” Without the negation check, the rule reads only “a refund has been issued”, and a correct refusal becomes a blocker.

4. Undo the change: put back NEGATABLE = {"claims_action", "asks_secret", "promises_outcome"}, or copy the file again from the snapshot:

   macOS and Linux

   cp $S/harness/scorers/criteria.py harness/scorers/

   Windows

   Copy-Item $S\harness\scorers\criteria.py harness\scorers\

   If $S is empty, set it again as in Harness structure.

5. Run python -m pytest again. The last line is 103 passed.

Three kinds of tests failed for one cause. A unit test on a tiny sentence, a regression check on a real saved reply, and a release test on the whole decision. Together they say what broke and where it matters.

Mocks: the judge without a model

Some parts of the harness call a model: the judges of Module 4. A test that calls a real model is slow and costs money. It also needs a key, and it gives another verdict each time. So the tests use a mock: a stand-in that behaves like the real thing in a fixed, known way. Here, MockProvider replays the verdicts that were recorded from the real judges.

The mock replays recorded verdicts by request key. A live judge always goes through the cap first.

Text description of the diagram

Every judge request gets a request key: 16 characters of the SHA-256 of the model, the messages, the schema and the token limit.

With JUDGE_PROVIDER=mock, the default and the only choice inside pytest, MockProvider looks up the key in recordings/judge_*.jsonl, 2,416 recorded verdicts. It gives the same verdict every time. A changed request has another key, and the mock answers RecordingNotFound.

With JUDGE_PROVIDER=openai_compatible, used by pytest -m live, CappedProvider first checks: fewer than 20 calls and 60,000 tokens so far? If yes, the request goes to the live model, on Ollama or with your key. If no, it raises UsageCapReached, and nothing is sent.

tests/conftest.py makes the mock the rule for every test. An autouse fixture is a piece of setup that pytest runs before each test without being asked. This one sets JUDGE_PROVIDER to mock and removes JUDGE_API_KEY, whatever your .env file says. So python -m pytest never sends a request and never reads a key, even on a computer where a live judge is set up.

A mock has a limit, and you must know it. It proves that your code builds the same requests and reads the answers correctly. It cannot tell you whether the real judge still answers the way it did when the verdicts were recorded. Models change.

The opt-in live test

For that question, the project has one small live test, tests/test_live.py. It asks the live judge to score three baseline replies and checks two things. Each verdict is a score from 1 to 5, or a reported problem. And exactly 3 calls went through the cap.

It is marked live, and pytest.ini leaves it out of normal runs:

pytest.ini

markers =
    live: calls a real judge (needs JUDGE_PROVIDER=openai_compatible); run with: pytest -m live
addopts = -m "not live"

1. Copy the test, and run the normal tests:

   macOS and Linux

   cp $S/tests/test_live.py tests/
   python -m pytest

   Windows

   Copy-Item $S\tests\test_live.py tests\
   python -m pytest

   The last line: 103 passed, 1 deselected. “Deselected” means that the marker kept the live test out.

2. Ask for it, without a live judge set up:

   python -m pytest -m live -rs

   The end of the output:

   SKIPPED [1] tests/test_live.py:27: Set JUDGE_PROVIDER=openai_compatible to run the live test.
   ====================== 1 skipped, 103 deselected in 0.08s ======================

   The test skips itself and says why. -rs prints the reason.

3. Optional, with Ollama. Start Ollama, run ollama pull gemma3:4b once, and create a file .env in the project folder with these three lines:

   .env

   JUDGE_PROVIDER=openai_compatible
   JUDGE_BASE_URL=http://localhost:11434/v1
   JUDGE_MODEL=gemma3:4b

   Then run python -m pytest -m live -v. On our Mac, it passed in 16 seconds:

   tests/test_live.py::test_live_judge_within_the_cap PASSED                [100%]
   
   ====================== 1 passed, 103 deselected in 16.17s ======================

4. Optional: see the cap stop the test. Set a cap of 2 calls for one run.

   macOS and Linux

   JUDGE_MAX_CALLS=2 python -m pytest -m live

   Windows

   $env:JUDGE_MAX_CALLS = "2"; python -m pytest -m live; Remove-Item Env:JUDGE_MAX_CALLS

   The end of the output:

   E           harness.providers.UsageCapReached: Usage cap reached: 2 calls, 3344 tokens (limits 2 calls, 60000 tokens). Nothing was sent.

   The third request was never sent. The cap checks before the call, not after.

5. Run python -m pytest once more. It still shows 103 passed, 1 deselected, with .env in place: the fixture keeps normal tests on the mock.

What did the live judge say? On the three replies, gemma3:4b gave the scores 5, 5 and 4. The recorded chat-strong verdicts for the same replies are 5, 5 and 3. The test passed anyway, on purpose. It checks the connection and the format: that the adapter, the cap and the parser still work with a real model. It does not check quality. Three verdicts from a small local model say nothing about quality; Model-based judging explains how much evidence a judge needs.

Never commit .env: it can hold a key. The project’s .gitignore already leaves it out.

What runs when

Now Priya writes the plan, like the driver’s quick check and yearly inspection.

Larkfield's test plan: what runs on every change, before a release, and on purpose.

Text description of the diagram

On every change: python -m pytest. At the end of this module, 106 tests in under a second, with no key and no network, because the mock replays the recorded judges. They cover functions on tiny examples, the 60 regression checks on the baseline’s saved replies, and a report that rebuilds the same. Cost: nothing.

Before a release: record the candidate on all cases. The 271 tickets took about 9 minutes and US$0.13. Repeat it 5 times on dev, with blockers per run. Then decide and report with --split holdout, once, for the decision. A person reads every blocker, and the report goes to Omar. Cost: minutes and cents.

Live, opt-in: python -m pytest -m live. Three judge calls through the cap. It checks the connection and the answer’s format, not quality. Run it when the provider, the model version or the adapter changes. With gemma3:4b on Ollama, it took 16 seconds. Cost: free locally, or cents with a key.

In a CI pipeline, the first column runs on every pull request; the second is the release gate.

The first column is cheap, so it runs all the time and stops bad changes early. The second column is the real evaluation of a new system. It costs model calls and a person’s time, so it runs when there is a candidate. The third column is rare and small.

A CI pipeline (continuous integration) is a service that runs your checks on a clean computer for every change. CI/CD, Testing and Safe Releases builds one step by step. Its lesson AI evaluation gate puts a fixed evaluation in the pipeline, like the second column here.

When to use regression checks, and what they cost

Write a regression check for every rule that must never break: the blockers, and any result that a person has checked and accepted. Write it on a saved run that a person has reviewed, so that “right” means something. Keep the checks fast, so that nobody is tempted to skip them.

They cost little to run but something to keep. When a rule changes on purpose, for example when Grace adds a blocker, the tests change in the same commit, with the reason. A regression check is also only as good as the saved run behind it. If the baseline were wrong on a case, the test would protect the wrong answer. And a check can only fail on a case that is in the set.

Do not turn quality judgements into exact tests. “The judge gives this reply a 4” is not a regression check: a live judge may give 3 tomorrow, and the reply is no worse. Test the rules and the plumbing exactly. Measure quality with the harness, with intervals.

Common mistake

The live test runs on every change

Symptom: someone removes addopts = -m "not live" from pytest.ini, so that “all the tests run”. On a computer with a key in .env, every python -m pytest now makes 3 paid calls and takes seconds longer. On a computer without a live judge, the test skips and nobody notices. Some days the provider is slow, and the run fails with a timeout that has nothing to do with the change.

Cause: a live test depends on the network, the provider and the model of the day. It does not belong in the checks that must give the same answer on every change.

Fix: keep the live marker and addopts = -m "not live". Run python -m pytest -m live on purpose, when the provider, the model version or the adapter changes, and keep the cap.

Guided practice: point the checks at the candidate

The regression checks pass on the baseline. Predict what they say about the candidate’s run 1, then check.

1. Copy tests/test_regression.py to tests/test_regression_candidate.py.
2. In the copy, change find_run("baseline") to find_run("candidate").
3. Predict: how many of the 60 checks fail, and which cases?
4. Run python -m pytest tests/test_regression_candidate.py.
5. Delete the copy, so that your project matches the snapshot again.

Show the answer

FAILED tests/test_regression_candidate.py::test_every_safety_ticket_goes_to_a_person[T-64740]
FAILED tests/test_regression_candidate.py::test_every_safety_ticket_goes_to_a_person[T-65663]
========================= 2 failed, 58 passed in 0.07s =========================

The two failures are the two safety tickets with no answer, the blockers of Module 5. Each one fails with TypeError: 'NoneType' object is not subscriptable: there is no answer to read needs_human from. A clearer test would check for a missing answer first and say so. No critical case of the candidate breaks a must-not rule.

Your turn: add a regression check

Grace says: “A French ticket must always get a French reply.” Turn it into a regression check on the baseline.

1. Create tests/test_my_rules.py. Use the two tests of test_regression.py as a model, and check_code("reply_in_french", "must", reply) from harness.scorers.criteria.
2. Parametrise it over the French cases of dev and holdout (c.language == "fr").
3. Run it. How many checks are there? Do they pass?
4. Independent variation: point it at the candidate’s run 1. What do you expect?

Show an example answer

tests/test_my_rules.py

import pytest

from harness.dataset import load_cases
from harness.run import find_run, load_outputs
from harness.scorers.criteria import check_code

BASELINE = load_outputs(find_run("baseline"))
CASES = [c for c in load_cases(split="all") if c.split != "contaminated"]


@pytest.mark.parametrize("case", [c for c in CASES if c.language == "fr"], ids=lambda c: c.case_id)
def test_every_french_ticket_gets_a_french_reply(case):
    assert check_code("reply_in_french", "must", BASELINE[case.case_id].reply).passed

20 passed: 10 French cases on dev and 10 on holdout. The candidate passes them all too: its French slice was fine. Your project now has 20 more tests than the snapshot. That is fine: keep the file, or delete it to match the snapshot.

Knowledge check

Check your understanding

What is the difference between a unit test and a regression check on saved outputs? Give one Larkfield example of each.

Show the answer

A unit test checks one function on a tiny example with a known answer. For example: “You don’t need to send your full card number” is not a request for a secret. A regression check applies a release rule to a real saved run that a person has checked. For example: every safety ticket in the baseline’s run has needs_human: true. Both run on every change, with no model, because the inputs are fixed.

Check your understanding

Why do the tests use a mock judge, and what can the mock not tell you?

Show the answer

The mock replays recorded verdicts by request key, so the tests are fast, free, need no key and give the same result every time. It proves that the code builds the same requests and reads the answers correctly. It cannot tell you whether the real judge still behaves as it did when the verdicts were recorded. The opt-in live test checks the connection and the format with a real model, and a new evaluation checks the quality.

Check your understanding

Predict: you set JUDGE_MAX_CALLS=2 and run the live test, which needs 3 calls. What happens to the third request?

Show the answer

It is never sent. CappedProvider checks the limits before each call. After 2 calls it raises UsageCapReached (“Nothing was sent”), and the test fails. The cap protects against a loop or a mistake that would otherwise send many paid requests.

Recap

• A regression check fails when something that worked before stops working. Larkfield’s 60 checks apply the release rules to the baseline’s saved replies.
• One tidy-up of the negation rule added 4 false blockers on dev; 5 tests caught it in under a second, and named the case.
• Tests use a mock judge that replays recorded verdicts. A fixture forces it, whatever .env says. The mock cannot tell you whether the real model has changed.
• The live test is opt-in (python -m pytest -m live), goes through the cap, and checks the connection and the format, not quality.
• The plan: tests on every change; recorded runs, decide and the report before a release; the live test when the provider or the model changes.
• Your project has test_live.py: 103 passed, 1 deselected.

Next, the assistant is in production: what the first month of use tells you, and how new cases join the set.

Next: Production feedback