Choose meaningful checks

What you will be able to do

Keep the tests that check what the user needs, and throw away the tests that only repeat the code. Prove each new test: it fails on the unfixed code and passes on the fixed code.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The regression test and git stash, from Ask for evidence.
• Normal, boundary and error cases, from Focused tests in the Python course.
• HEAD~1, “one commit before HEAD”, from Undo safely in the Git course.

What you need on your computer

• The project ~/projects/ticket-cleaner-assist with the fix committed: 35 passed, git status clean, the environment active.
• The copy ~/projects/ticket-cleaner-check from Reproduce first, for the module practice. If you deleted it, make it again with steps 6 and 7 of that lesson.

An answer key copied from the student

A teacher has no time to write the answer key for a maths test. So she takes the first student’s paper and copies its answers into the key. Then she marks every paper with that key.

The first student gets 100%, even for the two answers that were wrong. Every student who made the same mistakes also gets 100%. A student who answered those two questions correctly loses marks.

The key looks complete: it has an answer for every question. It is useless, because it came from the paper that it should check. A real key comes from the subject: from what the right answer is.

The problem: tests that agree with the bug

Tomás has a proposal: “The assistant fixed it. Let it write the tests too.” The course team recorded that request on the unfixed code, before anyone found the cause (transcript tests-mirror):

Prompt

Write tests for clean_category in ticket_cleaner/parsing.py.

What the assistant printed (part)

I'll add direct tests for alias normalization, whitespace trimming, preserving unrecognized category names, and converting non-string inputs to text. No
production changes are needed.

It added 11 test cases, and all of them passed: 35 passed. That looks like good work. Now read two of the cases:

tests/test_parsing.py (two cases from the session)

        (" Custom Category ", "Custom Category"),

@pytest.mark.parametrize(("value", "expected"), [(123, "123"), (None, "None")])

" Custom Category " becomes "Custom Category": the spaces go, the capital letters stay. That is Grace’s bug, written down as the expected result. The assistant even named it: “preserving unrecognized category names”. It read the code, saw what the code does, and wrote tests that say “do that”.

The course team ran all 11 cases on the unfixed and on the fixed code. None of them catches the bug: they all pass on the unfixed code. Two of them fail after the fix: these two. So the tests do not protect the fix. They fight it. With these tests in the project, the right fix looks like a mistake.

The idea: a check whose answer comes from the requirement

A test compares what the code does with an expected value. A test is meaningful when its expected value comes from the requirement. That is what the user needs, what the bug report says, or what the rest of the program promises. A test whose expected value comes from the code under test can only agree with that code: it repeats the implementation. Call it a mirror test, because it shows the code back to itself.

For a bug, three kinds of meaningful check work together:

1. The regression test. The minimal failing input, with the expected value of the bug report. It must fail on the unfixed code.
2. Edge cases. The near neighbours of the bug: other spellings, spaces, a capital letter in a web form name. Some fail on the unfixed code; some pass on both and keep old behaviour safe.
3. The requirement as the user sees it. Grace’s own command on her own file, through the program’s entry point. It checks the whole path, not one function.

Then prove each new test: run it on the unfixed code and on the fixed code. A test that fails before and passes after catches the bug. A test that passes on both keeps old behaviour, which is useful, but it proves nothing about the fix. A test that passes before and fails after repeats the bug: delete it, or correct its expected value from the requirement.

A tiny example: two sources for one expected value

Take one input, " Custom Category ", and ask two questions. What does the code return? And what should it return?

The code before the fix keeps the capital letters, and the code after the fix does not. The course team ran both versions:

clean_category(' Custom Category ') and clean_category(None)

before the fix: 'Custom Category' 'None'
after the fix:  'custom category' ''

The requirement is in the bug report: “Capital letters and spaces around a category do not make a new category, as for the status and the ID.” So the expected value is 'custom category'. It comes from the requirement, not from either version of the code.

And None? Ask the program, not the function. A row with no category never reaches clean_category():

python -c "from ticket_cleaner.parsing import split_records; print(split_records([{'id': 'T-1', 'status': 'open', 'category': None, 'priority': '1'}])[1])"

Output

[Rejected(row=1, problems=['missing category'])]

find_problems() rejects it first. So no requirement says what clean_category(None) returns. The test pinned a value that the program never uses, and it broke when an unrelated detail changed.

Ask where each expected value comes from. Only an expected value from the requirement can disagree with the code, and so catch a bug.

Text description of the diagram

The diagram is a table of three sources of an expected value. Each has real test cases from the two recorded sessions, run on the unfixed and on the fixed clean_category().

1. The code under test, what it returns today. Examples: clean_category(' Custom Category ') → 'Custom Category', and clean_category(None) → 'None'. Unfixed: pass; fixed: fail. The test repeats the bug: it fails when the bug is fixed.
2. A part of the code that already works. Examples: clean_category('Sign-in') → 'login', and clean_category(' billing ') → 'billing'. Pass on both. It keeps old behaviour, but it cannot show this bug.
3. The requirement, the bug report’s “Expected”. Example: to_ticket(…) with the category ' LOGIN ' → 'login'. Unfixed: fail; fixed: pass. It catches the bug, and it proves the fix.

At the bottom: a test whose expected value comes from the code can only agree with the code.

The third row is from a second session. The course team gave the same assistant Grace’s email and the two rows. It asked for “one test in tests/test_parsing.py that fails on the current code because of this bug, and passes when the bug is fixed” (transcript tests-failing-first). It wrote one test. It failed before the fix and passed after it. Same model, same function: the difference was the requirement in the prompt.

Now predict, then run the 12 cases of both sessions yourself:

Explore

Which generated tests catch Grace's bug?

The results of each test case on the unfixed and on the fixed code, measured by the course team.

The 12 test cases of the two recorded sessions

Test case and expected value	Prompt	Unfixed code	Fixed code	Verdict
clean_category('Sign-in') → 'login'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category('log-in') → 'login'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(' sign-in ') → 'login'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(' LOG-IN ') → 'login'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category('billing') → 'billing'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(' billing ') → 'billing'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(' Custom Category ') → 'Custom Category'“Write tests for clean_category”	“Write tests for clean_category”	pass	fail	repeats the bug
clean_category('') → ''“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(' \t\n ') → ''“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(123) → '123'“Write tests for clean_category”	“Write tests for clean_category”	pass	pass	passes on both
clean_category(None) → 'None'“Write tests for clean_category”	“Write tests for clean_category”	pass	fail	repeats the bug
to_ticket(… category ' LOGIN ' …).category → 'login'“a test that fails because of the bug”	“a test that fails because of the bug”	fail	pass	catches the bug

Eleven cases from the prompt “Write tests for clean_category”, one from the prompt for a test that fails because of the bug. Results measured by the course team on the unfixed and the fixed code.

Text description of the explorer

The explorer lists the 12 test cases of two recorded sessions. Each case shows the call and the value that the test expects, and the prompt that produced it. Two buttons run the cases on the unfixed code and on the fixed code. Once both have run, a verdict column appears.

The results, as the course team measured them:

• 11 cases from “Write tests for clean_category”. 9 of them pass on both versions: the four web form names ('Sign-in', 'log-in', ' sign-in ', ' LOG-IN ' → 'login'), 'billing' and ' billing ' → 'billing', '' and ' \t\n ' → '', and 123 → '123'. 2 of them pass on the unfixed code and fail on the fixed code: ' Custom Category ' → 'Custom Category' and None → 'None'. None of the 11 catches the bug.
• 1 case from the prompt for a test that fails because of the bug: to_ticket(…) with the category ' LOGIN ' → 'login'. It fails on the unfixed code and passes on the fixed code: it catches the bug.

Summary: 1 case catches the bug, 2 cases repeat the bug, 9 cases pass on both.

See it in your project

You apply the mirror tests for a moment, run them on both versions of the code, and throw them away. Then you add the module’s own tests and prove each one. git restore --source=HEAD~1 puts a file back as it was one commit ago, in the commit before your fix. It changes only the file in your folder, not the history.

1. Apply the session’s tests, from the labs folder:

   git apply ../nextia-learning-labs/C15/transcripts/tests-mirror.patch
   git diff --stat
   python -m pytest

   Output (part)

    tests/test_parsing.py | 31 ++++++++++++++++++++++++++++++-
    1 file changed, 30 insertions(+), 1 deletion(-)
   E       AssertionError: assert 'custom category' == 'Custom Category'
   E       AssertionError: assert '' == 'None'
   FAILED tests/test_parsing.py::test_clean_category_trims_text_without_changing_unrecognized_names[ Custom Category -Custom Category]
   FAILED tests/test_parsing.py::test_clean_category_converts_non_string_values_to_text[None-None]
   ========================= 2 failed, 44 passed in 0.05s =========================

   On your fixed code, two of the new cases fail. Your regression test passes.

2. Put back the unfixed parsing.py, and run the tests again:

   git restore --source=HEAD~1 ticket_cleaner/parsing.py
   python -m pytest
   python -m pytest -k clean_category

   Output (the last lines)

   FAILED tests/test_parsing.py::test_capital_letters_do_not_make_a_new_category
   ========================= 1 failed, 45 passed in 0.04s =========================
   ====================== 11 passed, 35 deselected in 0.03s =======================

   On the unfixed code, only your regression test fails. -k clean_category selects the 11 session cases, whose names contain clean_category: all 11 pass. None of them sees the bug.

3. Throw the session’s tests away, and put back the fixed code:

   git restore ticket_cleaner/parsing.py tests/test_parsing.py
   git status --short
   python -m pytest

   git status --short prints nothing, and the tests give 35 passed.

Now the module’s own tests. You already have the regression test. Add the edge cases and the requirement as Grace sees it.

4. Open tests/test_parsing.py. After test_capital_letters_do_not_make_a_new_category, add the edge cases:

   tests/test_parsing.py (new test)

   @pytest.mark.parametrize(
       ("name", "expected"),
       [("Billing ", "billing"), ("LOGIN", "login"), (" SIGN-IN", "login")],
   )
   def test_category_is_cleaned_like_the_other_fields(valid_raw, name, expected):
       valid_raw["category"] = name
       assert to_ticket(valid_raw).category == expected

   Each expected value comes from the bug report’s rule. "Billing " is the spelling of ticket T-3010 in Grace’s export. " SIGN-IN" is a web form name with capitals and a space: it worked before the fix, and it must still work.

5. Open tests/test_cli.py. At the end of the file, add Grace’s own command as a test:

   tests/test_cli.py (new test)

   def test_web_form_export_has_one_line_per_team(tmp_path):
       output = tmp_path / "week.json"
       assert main(["data/web-form-export.csv", "--output", str(output)]) == 0
       summary = json.loads(output.read_text(encoding="utf-8"))
       expected = {"account": 1, "billing": 3, "login": 4, "shipping": 2}
       assert summary["by_category"] == expected

   Leave two empty lines before it. The expected value is the one in Grace’s email, not a value copied from a run.

6. Run the tests on the fixed code:

   python -m pytest

   Output (the last line)

   ============================== 39 passed in 0.04s ==============================

7. Prove the new tests: run them on the unfixed code.

   git restore --source=HEAD~1 ticket_cleaner/parsing.py
   python -m pytest

   Output (part)

   FAILED tests/test_cli.py::test_web_form_export_has_one_line_per_team - Assert...
   FAILED tests/test_parsing.py::test_capital_letters_do_not_make_a_new_category
   FAILED tests/test_parsing.py::test_category_is_cleaned_like_the_other_fields[Billing -billing]
   FAILED tests/test_parsing.py::test_category_is_cleaned_like_the_other_fields[LOGIN-login]
   ========================= 4 failed, 35 passed in 0.05s =========================

   Four new cases fail on the unfixed code. [ SIGN-IN-login] passes, as expected: it guards old behaviour.

8. Put the fixed code back. Do not skip this step:

   git restore ticket_cleaner/parsing.py
   python -m pytest

   The tests give 39 passed again.

9. Check, and commit the two test files:

   ruff check .
   ruff format --check .
   git diff --stat
   git add tests/test_parsing.py tests/test_cli.py
   git commit -m "Test the category cleaning and Grace's export"

   Output

   All checks passed!
   21 files already formatted
    tests/test_cli.py     | 8 ++++++++
    tests/test_parsing.py | 9 +++++++++
    2 files changed, 17 insertions(+)
   [feature/category-option 2489c76] Test the category cleaning and Grace's export
    2 files changed, 17 insertions(+)

   Your commit hash differs.

The diagram has each new test, and what it did on both versions:

Every new test was run on both versions. Four catch the bug; one guards old behaviour; the old tests show that nothing else changed.

Text description of the diagram

The diagram is a table of the module’s new tests, run on the unfixed code (git restore --source=HEAD~1 ticket_cleaner/parsing.py) and on the fixed code.

1. test_capital_letters_do_not_make_a_new_category, Grace’s minimal input: failed on the unfixed code, passed on the fixed code.
2. test_category_is_cleaned_like_the_other_fields[Billing -billing], a capital letter and a space: failed, then passed.
3. [LOGIN-login], capital letters: failed, then passed.
4. [ SIGN-IN-login], a web form name in capitals with a space, which worked before: passed and passed. It keeps old behaviour.
5. test_web_form_export_has_one_line_per_team, Grace’s own command and file through main(): failed, then passed.
6. The 34 tests from before, with test_baseline.py (the sample file’s report): passed and passed.

At the bottom: unfixed, 4 failed, 35 passed; fixed, 39 passed.

The old tests matter too. test_baseline.py pins the report of the Python course’s sample file, from Repository preparation. It passes on both versions, so the fix changed nothing that it should not change.

When to write which test, and what it costs

Write the regression test for every bug, from the minimal input, before the fix. Add the edge cases that the bug report suggests, and one test of the requirement through the program’s entry point. That was 5 test cases here, written in a few minutes, and the whole suite still runs in a tenth of a second.

An assistant can write tests, and they are cheap: the mirror session cost US$0.012 in one call, and the failing-first session US$0.010. Prices change; with a free tier or no assistant, you pay nothing. The price is not the risk. The risk is a prompt without the requirement: then the only source of expected values is the code. Give the assistant the requirement, ask for tests that fail on the current code, and run every new test on both versions yourself.

Do not keep every test that passes. A test that pins an unused detail, such as clean_category(None), makes every future change harder and protects nobody. Test behaviour that someone depends on: the report, the categories, the entry point.

Common mistake

The unfixed code goes into the commit

Symptom: Tomás proves his new tests on the unfixed code, as in step 7. They fail as expected, so he commits with git add -A. Then python -m pytest on the branch gives 4 failed, 35 passed, and Grace’s report is split again.

Cause: he skipped step 8. git restore --source=HEAD~1 changed parsing.py in his folder, and git add -A took that change too. Before the commit, the signs were there:

$ git status --short
 M tests/test_cli.py
 M tests/test_parsing.py
 M ticket_cleaner/parsing.py
$ git diff --stat
 tests/test_cli.py         | 8 ++++++++
 tests/test_parsing.py     | 9 +++++++++
 ticket_cleaner/parsing.py | 9 +++------
 3 files changed, 20 insertions(+), 6 deletions(-)

The lesson changes only tests, but parsing.py is in the list.

Fix: before the commit, run git restore ticket_cleaner/parsing.py. After a commit, put the fixed file back from the fix commit with git restore --source=HEAD~1 ticket_cleaner/parsing.py, and commit it. Add files by name, and read git diff --stat first.

Module practice: fix a defect and prove the test

The module’s practice task: fix a seeded defect and add a test that catches the unfixed version. You work in ticket-cleaner-check, the copy of end-of-m03 from Reproduce first, which still has Grace’s bug. Your target is the second symptom: --category login on Grace’s export finds 2 of the 4 login tickets.

1. Go to the copy, activate its environment, and make it a Git repository, so that you can stash:

   macOS and Linux

   cd ~/projects/ticket-cleaner-check
   source .venv/bin/activate
   git init -q && git add -A && git commit -qm "Start"

   Windows

   cd ~\projects\ticket-cleaner-check
   .venv\Scripts\Activate.ps1
   git init -q; git add -A; git commit -qm "Start"

2. On your own: write one test in tests/test_cli.py for the second symptom. Take its expected values from the requirement, not from a run. Run it: it must fail.

3. On your own: fix the cause in ticket_cleaner/parsing.py, without copying from your project. Run all the tests.

4. Prove your test with git stash push ticket_cleaner/parsing.py, run it, and git stash pop.

Completion check: show that your fix addresses the cause, and that it keeps the existing behaviour. Name the evidence for each.

Show a worked solution

The test, at the end of tests/test_cli.py. The expected values come from Grace’s export: four login tickets, all open.

tests/test_cli.py (new test)

def test_category_option_finds_every_spelling_of_the_team(tmp_path):
    output = tmp_path / "login.json"
    argv = ["data/web-form-export.csv", "--category", "login"]
    assert main([*argv, "--output", str(output)]) == 0
    summary = json.loads(output.read_text(encoding="utf-8"))
    assert summary["selected"] == 4
    assert summary["by_category"] == {"login": 4}

On the unfixed code, the course team’s run:

python -m pytest tests/test_cli.py

E       assert 2 == 4
FAILED tests/test_cli.py::test_category_option_finds_every_spelling_of_the_team
========================= 1 failed, 4 passed in 0.04s ==========================

The fix, in clean_category(): two lines.

text = str(value).strip()
return CATEGORY_ALIASES.get(text.lower(), text)
text = clean_text(value)
return CATEGORY_ALIASES.get(text, text)

Then python -m pytest gives 35 passed. With the fix stashed, python -m pytest tests/test_cli.py gives 1 failed, 4 passed again; after git stash pop, it passes.

It addresses the cause. The test fails without the fix and passes with it. The fix is in clean_category(), where the capital letters stayed. It is not in the --category code, which was right: it compared with the wrong text. A fix in cli.py or report.py could make this test pass and leave Grace’s first symptom.

It keeps the existing behaviour. All 34 old tests still pass, among them test_baseline.py (the sample file’s report) and the web form names test. ruff format --check . prints 21 files already formatted.

When you have compared your work with the solution, delete the copy. Check the folder name before you press Enter:

macOS and Linux

deactivate
cd ~/projects
rm -rf ticket-cleaner-check

Windows

deactivate
cd ~\projects
Remove-Item -Recurse -Force ticket-cleaner-check

Module check

You have finished the module when you can show these on your own project, ~/projects/ticket-cleaner-assist:

• git log --oneline shows ten commits on feature/category-option. The last two are Test the category cleaning and Grace's export and Clean the category like the other fields (Grace's bug).

• python -m pytest prints 39 passed, and ruff format --check . prints 21 files already formatted.

• python -m ticket_cleaner data/web-form-export.csv gives "by_category": {"account": 1, "billing": 3, "login": 4, "shipping": 2}.

• The new tests catch the unfixed code. The fix is now two commits back. main still has the course’s start, with the unfixed parsing.py, so take the file from there. This also works after a reset:

  git restore --source=main ticket_cleaner/parsing.py
  python -m pytest
  git restore ticket_cleaner/parsing.py

  The first pytest prints 4 failed, 35 passed. After the last command, git status --short prints nothing, and the tests give 39 passed.

• On macOS and Linux, this command prints nothing, because your project matches the snapshot end-of-m04:

  diff -rq . ../nextia-learning-labs/C15/snapshots/end-of-m04 -x .git -x .venv -x reports -x __pycache__ -x .pytest_cache -x .ruff_cache

Reset. If your project is broken, follow the reset steps of Context and boundaries with the snapshot end-of-m04 in place of end-of-m01. The last line of python -m pytest is then 39 passed.

See it end to endFix a bug in an unfamiliar codebaseCase study

The case study writes the failing test first, then compares five fixes on one table of 12 postcodes. One fix passes every test and still gives a zone to a district that Larkfield does not serve.

See it end to endUpgrade a dependency safelyCase study

The case study saves today’s decisions on more inputs than the tests have. The assistant’s upgrade passed every test and still changed six of them.

Knowledge check

Check your understanding

The session 'Write tests for clean_category' produced 11 test cases that all passed on the buggy code. Explain in two sentences why none of them caught the bug.

Show the answer

The assistant took every expected value from what the code returned: it read clean_category() and wrote tests that say “do that”, including “preserving unrecognized category names”. A test whose expected value comes from the code under test can only agree with that code. So the tests passed on the bug, and two of them failed after the fix.

Check your understanding

Predict the results on the unfixed and on the fixed code for each case, and say whether to keep it: (a) clean_category('Invoice') → 'billing'; (b) clean_category(' Custom Category ') → 'Custom Category'; (c) the two rows login and LOGIN → ['login', 'login'].

Show the answer

• (a) Passes on both: Invoice is a web form name, and that part already worked. Keep it if no other test covers the web form names. It guards old behaviour, but it does not prove the fix.
• (b) Passes on the unfixed code, fails on the fixed code: it repeats the bug. Delete it, or correct the expected value from the requirement to 'custom category'.
• (c) Fails on the unfixed code, passes on the fixed code: it catches the bug. Keep it: it is the regression test.

Check your understanding

Practical: how do you prove that a new test catches the bug, when the fix is already committed? Give the commands for this project, and the output that proves it.

Show the answer

Put the unfixed file back from the commit before the fix, run the tests, then restore the fixed file:

git restore --source=main ticket_cleaner/parsing.py
python -m pytest
git restore ticket_cleaner/parsing.py

main has the course’s start, so its parsing.py is the unfixed one. HEAD~1 works too, but only when the fix is the last commit. The proof is that the new tests appear under FAILED on the unfixed code (here 4 failed, 35 passed) and pass after the restore (39 passed). Before you commit anything, git status --short must not list parsing.py.

Recap

This lesson

• A test is meaningful when its expected value comes from the requirement. A mirror test takes it from the code under test, and can only agree with the code.
• “Write tests for clean_category” gave 11 cases: none caught the bug, and 2 failed after the fix. A prompt with the requirement, asking for a test that fails first, gave 1 case that caught it.
• For a bug: a regression test, edge cases, and the requirement through the entry point. Prove each new test on the unfixed and the fixed code.
• Your project matches end-of-m04: 39 passed.

Module 4: debug and test with assistance

• Reproduce first: a minimal failing input, and a fresh environment to tell a code defect from an environment problem.
• Ask for evidence: a hypothesis, a targeted check, a test that fails, and only then the fix. In the course team’s runs, blind fixes worked but left no evidence.
• Choose meaningful checks: expected values from the requirement, and every new test proved on both versions.

Next, in Module 5, you make the project ready to hand over. You improve the code only where the change needs it, and you write down your decisions.

Next: Refactor selectively