Ask for evidence

Lesson 11
40 min
Module 4: Debug and test with assistance

Lesson 11 of 15 · AI-Assisted Software DevelopmentAsk for evidence

Course overview

1. Establish a safe working setup

2. Write actionable tasks

3. Review generated work

4. Debug and test with assistance

5. Deliver maintainable work

6. Case studies

What you will be able to do

Find the cause of a bug with a hypothesis and a check that can prove it wrong, and ask an assistant for the same. Then every fix comes with evidence: a test that failed before the fix and passes after it.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

Grace’s reproduction and the answer “code, not environment”, from Reproduce first.
How a ticket gets its category: to_ticket() and clean_category() in parsing.py, from Supply relevant context.
git stash, from Recover work in the Git course.

What you need on your computer

The project ~/projects/ticket-cleaner-assist: 34 passed, git status clean, the environment active.
Optional: a coding assistant. Without one, you read the recorded sessions, and you do every step yourself.

The knee that hurts on the stairs

You tell a doctor: “My knee hurts when I climb stairs.” One doctor says: “Take these pills. If it still hurts, come back.” You come back twice, and get different pills each time. Perhaps the third ones work. You never learn why.

Another doctor says: “I think it is the tendon below the kneecap. If I am right, it hurts when I press here.” She presses. It hurts. Now the treatment has a reason. Later, she presses the same place again. It does not hurt, and you both know that the treatment worked.

If the press had not hurt, she would not give the treatment anyway. She would think of another cause and test that one.

The problem: “Fix it.” again and again

The course team gave an assistant two files, report.py and cli.py, without a repository map, and one prompt (transcript bug-fix-blind-no-map):

Grace says the report is wrong for the new web form export. Fix it.

The assistant asked for evidence instead of changing anything:

What is wrong with the report for the new web form export? Please share:

- A small sample export that reproduces the issue (with sensitive data removed).
- The command used to generate the report.
- The actual report and the expected result.

The next two prompts gave it nothing new: “It is still wrong. Fix it.” In turn 2 it asked again, and also asked for three more files. The tool added them, and the model named a likely cause in clean_category(), but asked for an example to confirm it. In turn 3, it asked once more, and then changed the code:

Also provide a small anonymized export, the command you ran, and the actual versus expected report. The earlier suggestion about category capitalization was
unverified.

Its change was right: the course team’s 5 hidden checks of the bug passed. But look at what the session leaves you. It took 3 turns and 5 model calls. It ran no example. It added no test, so 24 passed before and after. And the assistant said itself that its idea was “unverified”. A right result that nobody can show is right is a lucky result.

So this lesson cannot tell you that blind rewrites fail. In the course team’s runs, they did not. It shows something else: a blind fix leaves no evidence. The diagram compares four recorded sessions of the same bug:

A table of four recorded sessions of Grace's bug. Fix it, larger model, repository map on: asked for 6 files, changed 3 files plus 44 minus 3 in one turn, 31 passed, 2 calls, 0.026 US dollars; evidence: 7 new test cases, 5 catch the bug, 2 pass on both, no test failed before the fix. Fix it, smaller model: asked for 2 files, changed one line, 24 passed, 2 calls, 0.0019 US dollars; evidence: no new test. Fix it, then twice It is still wrong, no map: turns 1 and 2 asked for an example and files, turn 3 changed 3 lines, 24 passed, 5 calls, 0.040 US dollars; evidence: no test, no example run, it called its earlier idea unverified. Evidence first: turn 1 a hypothesis with the line, turn 2 the test, 1 failed 24 passed, turn 3 one changed line, 25 passed, 3 calls, 0.028 US dollars; evidence: a test that failed before the fix and passes after it. At the bottom: a right result is not yet evidence; a test that fails first is. — All four sessions ended with the bug fixed. Only one of them shows why, with a test that failed first.

The idea: a hypothesis and a targeted check

A hypothesis is a guess about the cause that is exact enough to test. It names a place in the code, and it predicts what a check will show. “Something is wrong with categories” is not a hypothesis: no result can prove it wrong. “clean_category() returns the original text when the name is not a web form name” is one. It predicts that clean_category("LOGIN") returns "LOGIN".

A targeted check is the smallest run that can prove the hypothesis wrong: one function call, one command, one test. It is targeted because it looks at one place, not at the whole program. If the result disagrees with the prediction, the hypothesis is wrong, and you need a new one. If it agrees, you have evidence: an output that you can show to somebody else.

Put the steps in a loop, and you have the doctor’s method for code:

The debugging loop: each step makes the evidence that the next step uses. “Fix it.” jumps from the observation straight to a fix, and skips the evidence.

Text description of the diagram

The diagram is a loop of six steps, each with Grace’s bug as the example.

Observation: Grace’s report has Billing, LOGIN and Login next to billing and login.
Minimal input: two rows, login and LOGIN, give {"LOGIN": 1, "login": 1}, the same in a new environment.
Hypothesis: clean_category() keeps the original text when the name is not a web form name.
Targeted check: clean_category('LOGIN') returns 'LOGIN'; the two-row test gives 1 failed, 34 passed. An arrow goes back from step 4 to step 3: if the check disagrees, you need a new hypothesis.
Fix: text = clean_text(value), one cleaning rule, as for the status and the ID.
Regression test, in green: 35 passed; Grace’s export gives account 1, billing 3, login 4, shipping 2; without the fix, the test fails again.

A dashed red arrow, labelled “Fix it.”, goes from step 1 straight to step 5.

Beside the loop are the four measured sessions of the table above:

“Fix it.” with the larger model: 1 turn, 3 files, +44 −3, 5 of 5 checks, no test failed before the fix.
“Fix it.” with the smaller model: +1 −1, no new test.
“Fix it.” three times without a map: turns 1 and 2 asked for evidence; turn 3 changed 3 lines, no test.
Evidence first: 1 failed, 24 passed, then one changed line, 25 passed, 5 of 5 checks.

All four passed Grace’s 5 checks. Only one left a test that failed first.

You already did steps 1 and 2 in Reproduce first. This lesson does steps 3 to 6.

A tiny example: one function, three values

Read the function in ticket_cleaner/parsing.py:

def clean_category(value: object) -> str:
    """Return the category, with the web form's names changed to the report's names."""
    text = str(value).strip()
    return CATEGORY_ALIASES.get(text.lower(), text)

CATEGORY_ALIASES.get(key, default) looks the key up in the web form’s names. If the key is there, it returns the report’s name. If not, it returns the default, which is text, the original spelling. The hypothesis predicts three results: LOGIN stays LOGIN, Sign-in becomes login, and login becomes login. One command checks all three:

python -c "from ticket_cleaner.parsing import clean_category; print(repr(clean_category('LOGIN')), repr(clean_category('Sign-in')), repr(clean_category(' login ')))"

'LOGIN' 'login' 'login'

All three agree with the prediction. The check took one second, and it looked at one function, not at the whole program.

Test a rival hypothesis too: “report.py counts wrongly”. If that were true, two tickets with the same category could still give two lines. Give count_by_category() two login tickets:

python -c "from ticket_cleaner.records import Ticket; from ticket_cleaner.report import count_by_category; t = [Ticket('T-1', 'open', 'login', 1), Ticket('T-2', 'open', 'login', 1)]; print(count_by_category(t))"

{'login': 2}

The counting is right, so the rival hypothesis is wrong. The cause is earlier, in the text that the counting receives.

Ask the assistant for evidence

You can ask an assistant to follow the same loop. The course team recorded it (transcript bug-fix-evidence), with the same model as the blind sessions. The first prompt had Grace’s email and the two-row input, as in Supply relevant context. It ended with a request for a hypothesis, not for a fix:

Do not change any file yet. Give one hypothesis for the cause, with the line of code, and say which small test would show that the hypothesis is right.

**Hypothesis:** `clean_category()` preserves the original casing for categories that are not aliases:

```python
return CATEGORY_ALIASES.get(text.lower(), text)
```

The lookup uses lowercase, but its fallback is `text`. Since `"LOGIN"` is not an alias key, it stays `"LOGIN"` instead of becoming `"login"`, splitting the
report counts.

It named the line and proposed a three-line test. It also wrote: “With the current code, it fails”. But it had not run anything. That sentence is a prediction, so the next prompt checks it:

Add that test to tests/test_parsing.py and nothing else. It must fail on the current code.

After turn 2, the project’s tests gave 1 failed, 24 passed: the prediction was right. Only then did the last prompt ask for the change:

Now fix the cause with the smallest change, so that all the tests pass.

Lowercase the fallback category so non-alias names merge regardless of casing. Alias handling stays unchanged.
(...)
25 passed in 0.03s

Its change was one line: text became text.lower() in the default of CATEGORY_ALIASES.get(). The session took 3 calls and US$0.028. Each turn produced one piece of evidence: a hypothesis with its line, a test that failed, then a fix that made it pass.

The three prompts work with any assistant, also with a chat window where you move the code yourself:

“Do not change any file yet. Give one hypothesis, with the line of code, and a small test that would show it.”
“Add that test and nothing else. It must fail on the current code.” Then run it yourself.
“Now fix the cause with the smallest change, so that all the tests pass.” Then run them yourself.

Find the cause and fix it in your project

You now do the loop in your project. If you use an assistant, send it the three prompts, with the bug report and the files parsing.py, records.py and tests/test_parsing.py. Compare its hypothesis, test and fix with the steps below. Accept nothing until you have seen the test fail.

Run the targeted check of the tiny example above. It prints 'LOGIN' 'login' 'login'.
Look at how the other fields are cleaned:
- macOS and Linux
- Windows
Terminal window
grep -n "clean_text(raw" ticket_cleaner/parsing.py
Terminal window
Select-String -Pattern "clean_text\(raw" -Path ticket_cleaner\parsing.py
Output (macOS)
```
30:        if clean_text(raw.get(field)) == "":
33:    status = clean_text(raw.get("status"))
37:    priority = clean_text(raw.get("priority"))
46:        id=clean_text(raw["id"]).upper(),
47:        status=clean_text(raw["status"]),
49:        priority=int(clean_text(raw["priority"])),
```
Every field goes through clean_text(), which removes outer spaces and makes small letters. Only the category, on line 48, does not. That is the expected rule, and the bug report says it too: “as for the status and the ID”.

Write the regression test from the minimal input. Open tests/test_parsing.py. After the test test_web_form_names_become_report_categories, add:

def test_capital_letters_do_not_make_a_new_category():
    # Grace's bug report: "LOGIN" and "login" were counted as two categories.
    rows = [
        {"id": "T-1", "status": "open", "category": "login", "priority": "1"},
        {"id": "T-2", "status": "open", "category": "LOGIN", "priority": "1"},
    ]
    tickets, rejected = split_records(rows)
    assert rejected == []
    assert [ticket.category for ticket in tickets] == ["login", "login"]

Leave two empty lines before the next test. The test uses the two rows of the bug report, and the expected value of the bug report.

Run the tests. The new test must fail, for the reason that the hypothesis predicts:

python -m pytest

>       assert [ticket.category for ticket in tickets] == ["login", "login"]
E       AssertionError: assert ['login', 'LOGIN'] == ['login', 'login']
E
E         At index 1 diff: 'LOGIN' != 'login'
E         Use -v to get more diff

tests/test_parsing.py:37: AssertionError
=========================== short test summary info ============================
FAILED tests/test_parsing.py::test_capital_letters_do_not_make_a_new_category
========================= 1 failed, 34 passed in 0.05s =========================

The second ticket keeps 'LOGIN'. That is the targeted check, now kept as a test.

Fix the cause. In ticket_cleaner/parsing.py, replace the function clean_category() with this version:

def clean_category(value: object) -> str:
    """Return the category as the report names it: cleaned, in small letters.

    The web form's names are changed to the report's names (CATEGORY_ALIASES).
    """
    text = clean_text(value)
    return CATEGORY_ALIASES.get(text, text)

clean_text() already returns small letters, so the lookup needs no .lower(). The docstring now says what the function promises.

Run the tests again:

python -m pytest

============================== 35 passed in 0.03s ==============================

Why clean_text(value), and not the recorded session’s text.lower()? Both pass every test and all 5 hidden checks. But the session’s line adds a second cleaning rule, used only for categories. Amira prefers one rule for every field: if the team changes how text is cleaned, it changes in one place. That is a decision, not a test result, so she writes it in the commit message. It also explains why “the assistant’s fix works” is not the end of a review.

Now prove that the test depends on the fix. Put the fix aside for a moment, and only the fix:

Stash the change of parsing.py, run the tests, and bring the change back:

git stash push ticket_cleaner/parsing.py
python -m pytest
git stash pop
python -m pytest

FAILED tests/test_parsing.py::test_capital_letters_do_not_make_a_new_category
========================= 1 failed, 34 passed in 0.06s =========================
============================== 35 passed in 0.04s ==============================

Without the fix, the test fails; with it, the test passes. That is the evidence.

Run Grace’s command, and the second symptom of the last lesson:
Terminal window
```
python -m ticket_cleaner data/web-form-export.csv
python -m ticket_cleaner data/web-form-export.csv --category login
```
After the first command, by_category is {"account": 1, "billing": 3, "login": 4, "shipping": 2}: the result that Grace expected. After the second, the report has "selected": 4. The one fix repaired both symptoms, as the shared minimal input suggested.

Check the format and the size of the change, then commit the test and the fix together:

ruff check .
ruff format --check .
git diff --stat
git add ticket_cleaner/parsing.py tests/test_parsing.py
git commit -m "Clean the category like the other fields (Grace's bug)"

All checks passed!
21 files already formatted
 tests/test_parsing.py     | 11 +++++++++++
 ticket_cleaner/parsing.py |  9 ++++++---
 2 files changed, 17 insertions(+), 3 deletions(-)
[feature/category-option 211fca5] Clean the category like the other fields (Grace's bug)
 2 files changed, 17 insertions(+), 3 deletions(-)

Your commit hash differs.

When to ask for evidence, and what it costs

Ask for evidence for every bug that a user reported, and for every change that you cannot check by reading it. In money, the difference is small. The evidence session cost US$0.028 in 3 calls. The blind session with the map cost US$0.026 in 2 calls. The one that repeated “Fix it.” cost US$0.040 in 5 calls. Prices change, so check your provider’s current page. With a free tier or no assistant, you pay nothing.

The real cost is your time: a few minutes for one hypothesis and one test. You get it back the first time a hypothesis is wrong. Without the check, a wrong guess becomes a wrong change, and then a second change on top of it.

There are limits. A hypothesis can agree with one check and still be incomplete: the check looks at one place. That is why the loop ends with all the tests and Grace’s own command, not only the new test. And for a throwaway experiment, a quick guess is fine. You are not going to merge it.

Common mistake

The stash takes the test too

Symptom: Tomás wants to see the new test fail without the fix. He runs git stash and then python -m pytest:

Saved working directory and index state WIP on feature/category-option: 1c3a803 Start
============================== 34 passed in 0.04s ==============================

All tests pass, without the fix. (In your project, the stash line names your last commit.) He concludes that the test does not need the fix.

Cause: git stash with no file name stashes every change: the fix and the new test. The test is not in the folder any more, so it cannot fail. 34 passed is the old test suite. The count gives it away: 34, not 35.

Fix: git stash pop brings both changes back; git status --short then lists both files again. Stash only the fix: git stash push ticket_cleaner/parsing.py. Always read the number of tests, not only the word “passed”.

Your turn: a hypothesis for the second symptom

Before your fix, --category login found 2 of the 4 login tickets. Write a hypothesis that explains it, with the place in the code. Then name a targeted check, and say what it showed after your fix.

Show the answer

Hypothesis: the option compares each ticket’s category with the cleaned name login. Before the fix, the tickets T-3003 and T-3008 kept Login and LOGIN, so they did not match. The option was right; the categories that it compared were wrong. The cause is the same line of clean_category().

Targeted check: the same command after the fix. If the hypothesis is right, the report has "selected": 4 and {"login": 4}. Step 8 showed exactly that. A check that could prove it wrong: if the option itself compared wrongly, it would still miss tickets after the fix.

What you did not need: a change to report.py or cli.py. A blind “fix the –category option” request could have changed the option, and left the cause.

Knowledge check

Check your understanding

Which of these is a hypothesis that you can test? For each, say why or why not. (a) 'The category code is buggy.' (b) 'clean_category() returns the original text when the name is not a web form name, so clean_category('LOGIN') returns LOGIN.' (c) 'The assistant probably made a mistake.'

Show the answer

Only (b). It names a place, clean_category(), and predicts one exact result, 'LOGIN'. One command can prove it wrong. (a) names an area, but predicts nothing: every result agrees with it. (c) is about a person, not the code, and gives no check to run.

Check your understanding

In the evidence session, the assistant wrote in turn 1 that its test 'fails' on the current code. Why did the next prompt still say 'It must fail on the current code'? What would you do if it had passed?

Show the answer

In turn 1 the assistant had run nothing. “It fails” was a prediction, written as a fact. Turn 2 turned it into a result: 1 failed, 24 passed. If the test had passed, either the test did not test the bug, or the hypothesis was wrong. Then you do not ask for a fix. You go back to the hypothesis, or write a test from the minimal input that does fail.

Check your understanding

Predict: you stash only the fix with git stash push ticket_cleaner/parsing.py and run python -m pytest. What is the last line, and why is it the evidence that you need?

Show the answer

1 failed, 34 passed, with FAILED tests/test_parsing.py::test_capital_letters_do_not_make_a_new_category. The test is still in the folder, and the code is back to the unfixed version. So the test fails without the fix, and, after git stash pop, passes with it. That shows that the test catches the bug and that the fix is what repairs it.

Recap

A hypothesis names a place in the code and predicts a result. A targeted check is the smallest run that can prove it wrong.
The debugging loop: observation, minimal input, hypothesis, targeted check, fix, regression test. “Fix it.” jumps from the observation to the fix.
In the course team’s runs, blind fixes worked, but left no evidence: no test that failed first. Asking for a hypothesis, then a failing test, then the fix, gave all three for about the same cost.
The accepted fix uses clean_text(), one cleaning rule for every field. Stash only the fix to see the test fail again.
Your project has one new commit: the fix and its regression test, 35 passed.

Next, Tomás asks the assistant to write the rest of the tests. You find out which tests are worth keeping.

Next: Choose meaningful checks