Problem and acceptance criteria

What you will be able to do

Turn a request into a task that someone else can finish and check. Write what happens now, what should happen, the inputs, the limits, and the evidence of done. An assistant cannot read your mind, and neither can a colleague.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The clean starting point and the baseline test, from Repository preparation.
• Options and exit codes of a command-line program, from Command-line interface in Python for Practical AI Engineering.
• How a test checks one behaviour, from Focused tests.

What you need on your computer

• The project ~/projects/ticket-cleaner-assist from Module 1, on the branch feature/category-option, with its environment active. python -m pytest prints 25 passed, and git status says working tree clean.
• If the project is missing or broken, copy the snapshot end-of-m01, as the reset steps of Context and boundaries show.
• No account, no key and no money. You read two recorded sessions in the labs folder C15/transcripts/.

A cake for Saturday

You phone a bakery: “I need a cake for Saturday.” On Saturday, you get a lemon cake for six people, with “Congratulations” on top. It is a good cake. But your party has twelve guests, one of them cannot eat nuts, and the cake is for Sam’s birthday.

The baker did nothing wrong. Your order left many choices open, and the baker made each choice in a reasonable way. They were not your choices.

Now try again: “A chocolate cake for twelve people, with Happy birthday, Sam, and no nuts. I collect it on Saturday at 10.” You can check each part of this order when you collect the cake. The baker can check it before you arrive, too.

The problem: a reasonable change that is wrong

The course team gave an assistant Grace’s feature as one sentence (transcript feature-vague):

Prompt

Add a way to filter the report by category.

The assistant asked for four files, changed them, and added five tests. Its change was small: 4 files, 77 lines added and 3 removed. All tests passed: 29 passed. Before its edits, it said what it would do:

What the assistant printed (part)

I’ll add an optional `--category` argument that matches a report category exactly and combines with the status filter.

Read that sentence again. It contains three decisions, and nobody asked for them:

• One category. Grace wrote “just one category, or two when a lead has two teams”. The option takes only one.
• Exact match. --category Billing finds nothing, because the report says billing.
• A new key in every report. The change adds "category": null to every report, also when nobody uses the option. In Repository preparation, the baseline test caught exactly this key.

Each decision is reasonable, like the baker’s lemon cake. But nobody can say that the change is wrong, because the request never said what right is. The tests do not help: the assistant wrote them for its own decisions, so they pass.

The idea: a task brief with acceptance criteria

A task brief is a short written description of a task. It has five parts:

1. Observed behaviour: what happens now. Use a command and its real output.
2. Expected behaviour: what should happen after the change.
3. Inputs: the files, options and examples that the task uses.
4. Constraints: the limits. Which files may change, no new dependency, the style rules.
5. Evidence of done: the checks that show the work is finished.

The heart of the brief is a list of acceptance criteria. An acceptance criterion is one statement about behaviour. A person or a test can check it, and the answer is yes or no. For example, “the option is easy to use” is not a criterion. Two people can disagree about it. “You can give --category more than once” is a criterion. You run it twice and look.

The name comes from contracts. When a builder finishes a job, the client accepts it if the work meets the agreed criteria. You met a smaller version in Function contracts: what a function takes and what it promises to return.

A tiny example: one sentence becomes two criteria

Grace’s email has this sentence: “Can I ask the report for just one category, or two when a lead has two teams?” Turn it into checks with the Python course’s sample file, data/tickets.csv. It has 9 valid tickets, and 4 of them are open:

ID	Category	Priority
T-1001	billing	2
T-1003	login	1
T-1005	billing	3
T-1015	shipping	2

Now write each criterion with the result that you expect, from this table:

• One category. --category billing keeps T-1001 and T-1005. So selected is 2, by_category is {"billing": 2}, and average_priority is (2 + 3) / 2 = 2.5.
• Two categories. --category billing --category shipping keeps three tickets. So selected is 3, and by_category is {"billing": 2, "shipping": 1}.

You calculated the results by hand, before any code exists. That is the point. If the change prints selected 3 for --category billing, you know at once that it is wrong. You do not need to read the code to know it.

Larkfield: Amira writes the brief

Amira turns Grace’s email into docs/tasks/category-option.md. Some criteria come straight from the email. Others are decisions, and Amira checks them with Grace before she writes them down. Must Billing and billing give the same result? Grace says yes. Must the report stay the same without the option? Yes, because a spreadsheet reads it.

docs/tasks/category-option.md

# Task: add a --category option to the report

Request: Grace (support lead) wants the report for one category, or a few.

Observed now: `python -m ticket_cleaner data/tickets.csv` always counts every
category of the chosen status.

Expected: `python -m ticket_cleaner data/tickets.csv --category billing` counts
only the billing tickets of the chosen status.

## Acceptance criteria

1. With `--category NAME`, "selected", "by_category" and "average_priority"
   count only the tickets that have the chosen status and that category.
2. `--category` can be given more than once. A ticket in any of the given
   categories counts.
3. Capital letters and spaces around the name do not matter:
   `--category " Billing "` selects the billing tickets.
4. With `--category`, the report has one more key, "categories": the chosen
   names, in small letters, sorted. Without `--category`, the report is exactly
   the same as today.
5. "valid_records" and "rejected" do not change with `--category`: they
   describe the whole file.
6. A category that no ticket has is not an error: "selected" is 0, and the
   exit code is 0.
7. `--help` and the options table in README.md describe the option.

## Constraints

- Change only ticket_cleaner/cli.py, ticket_cleaner/report.py, README.md and
  the tests.
- No new dependency.
- Keep the style: `ruff check .` and `ruff format --check .` pass.

## Evidence of done

- `python -m pytest` passes, with new tests for criteria 1 to 6.
- `ruff check .` and `ruff format --check .` pass.
- The commands in criteria 1, 3 and 6 give the results above.

Look at what the brief does not say. It does not say how to add the option, or which function to change. It says what must be true afterwards. It leaves the how to the person, or the assistant, who does the work. The constraints are the only limits on the how.

Each criterion has a check. The diagram links each one to a command and its expected result:

The task brief links each requirement to the evidence that shows it is done. Every expected value comes from the sample file, before any code is written.

Text description of the diagram

The diagram has two columns. The left column is the brief. At the top, observed now: –category is an error, with exit code 2. Expected: –category billing counts only the billing tickets of the chosen status. Below are the seven criteria and the constraints. The right column has the check for each criterion. Each command starts with python -m ticket_cleaner data/tickets.csv.

1. One category: --category billing gives selected 2, by_category {"billing": 2} and average_priority 2.5.
2. More than once: --category billing --category shipping gives selected 3 and by_category {"billing": 2, "shipping": 1}.
3. Capital letters and spaces: --category " Billing " gives the same result as criterion 1.
4. The “categories” key only with the option: --category Shipping --category billing gives categories ["billing", "shipping"]. Without the option, tests/test_baseline.py still passes.
5. Whole-file counts: --category billing still gives valid_records 9 and 6 rejected rows, as without the option.
6. Unknown category: --category garden gives selected 0, an empty by_category and exit code 0.
7. Help and README: python -m ticket_cleaner --help lists –category, and the README’s options table has a row for it.

Constraints: git diff --stat lists only cli.py, report.py, README.md and files in tests/. The requirements files do not change. ruff check . and ruff format --check . pass.

Add the brief to your project

1. Check the start state. git status must say nothing to commit, working tree clean, and python -m pytest must print 25 passed.

2. Read Grace’s email: nextia-learning-labs/C15/emails/grace-feature-request.md. Find each criterion of the brief in it, or mark the criterion as Amira’s decision.

3. See the observed behaviour for yourself. Run the command of the brief’s “Expected” line:

   python -m ticket_cleaner data/tickets.csv --category billing

   Output

   usage: ticket_cleaner [-h] [--status {closed,open,pending}] [--output OUTPUT]
                         input
   ticket_cleaner: error: unrecognized arguments: --category billing

   Then show the exit code. On macOS and Linux, run echo $?; in PowerShell, run $LASTEXITCODE. It prints 2: the command is wrong, as the README’s table of exit codes says.

4. Make the folder docs/tasks, and copy the brief into it from the snapshot end-of-m02:

   macOS and Linux

   mkdir -p docs/tasks
   cp ../nextia-learning-labs/C15/snapshots/end-of-m02/docs/tasks/category-option.md docs/tasks/

   Windows

   New-Item -ItemType Directory -Force docs\tasks
   Copy-Item ..\nextia-learning-labs\C15\snapshots\end-of-m02\docs\tasks\category-option.md docs\tasks\

   You can also type it from the box above. Writing it yourself is the better exercise.

5. Run the checks. A brief changes no code, so the tests do not change. ruff also checks the format of Markdown files, so it counts one file more:

   python -m pytest
   ruff check .
   ruff format --check .

   Output (the end of each command)

   ============================== 25 passed in 0.03s ==============================
   All checks passed!
   18 files already formatted

6. Commit the brief:

   git add docs/tasks/category-option.md
   git commit -m "Write the task brief for the category option"

   Output

   [feature/category-option c1757ff] Write the task brief for the category option
    1 file changed, 39 insertions(+)
    create mode 100644 docs/tasks/category-option.md

   Your commit hash is different.

The brief is in the repository, next to the code. A reviewer can read it with the diff, and the pull request can link to it.

The same feature, with the brief

The course team gave this brief to the same assistant and model, with the five files that the task needs (transcript feature-bounded). The assistant began with a different plan:

What the assistant printed (part)

I’ll add a repeatable category option, normalize and apply the category filter after the status filter, and keep whole-file validation totals unchanged.

Each part of that sentence comes from a criterion. “Repeatable” is criterion 2, and “normalize” is 3. “After the status filter” and “whole-file … totals unchanged” are 1 and 5. After each session, a script ran eight hidden checks, one for each criterion (criterion 4 has two). Here are three recorded sessions:

Session	Prompt	Files changed	Lines	Tests	Hidden checks
feature-vague	one sentence	4	+77 −3	29 passed	4 of 8
feature-bounded	the brief	5	+145 −4	35 passed	8 of 8
feature-bounded-small	the brief, smaller model	5	+86 −3	30 passed	8 of 8

• One-sentence request
• The task brief

The smallest change was the wrong one. With the brief, both models passed all 8 checks; the one-sentence request passed 4.

Text description of the chart

A scatter chart. The horizontal axis is the number of lines changed, from 0 to 200. The vertical axis is the number of hidden checks passed, out of 8; the axis goes up to 10. The one-sentence request (“vague”) changed 80 lines and passed 4 checks. The brief with the larger model (“brief”) changed 149 lines and passed 8. The brief with the smaller model (“brief, small”) changed 89 lines and passed 8.

The vague change failed four checks: two categories, capitals and spaces, the categories key, and the unchanged report. Its own tests could not see this. Only the brief’s criteria could.

Notice two more things. First, the small diff was not the good one, so size is not the measure: the criteria are. Second, the brief did not make the work perfect. The larger model’s change failed ruff format --check (2 files would be reformatted), although the brief asked for it. And the same brief, sent a second time, gave a different change: +129 −5 instead of +145 −4. A brief tells you what to check. You still check it, in Module 3.

When to write a brief, and what it costs

Write a brief for any change that someone else makes, an assistant or a colleague, and that you will accept. It takes about fifteen minutes for a task like Grace’s. Most of that time goes to the questions to Grace. Somebody must answer those questions anyway: you now, or the assistant’s guesses later.

Keep it short. A brief of one screen is enough for a task of a few files. For a one-line change, such as “rename this variable”, one sentence is a brief: its criterion is “the tests still pass”.

A brief has costs. Expected values take thought: you calculate them by hand. A criterion can also be wrong, and then a correct change fails it. If a check fails, first ask whether the criterion is right. Then change the brief on purpose, in a commit, and tell Grace.

Do not let the brief turn into the code. “Use action="append" in make_parser()” is a design choice, not a criterion. Put it in the brief only when there is a reason, for example a team rule. Otherwise, write the behaviour, and let the person who does the work choose the how.

Common mistake

A criterion that nobody can check

Symptom: Tomás writes a brief with this criterion: “The category filter should work correctly and be easy to use.” The assistant’s change passes all the tests. Grace finds that --category Billing selects nothing, and Tomás says “but it works correctly”.

Cause: “correctly” and “easy to use” have no yes-or-no check. Each reader fills in their own meaning, so the criterion accepts any change. It is the vague request again, with more words.

Fix: for each criterion, write the command or the test that checks it, and the result that you expect. If you cannot write that check, the criterion is not finished. Here: --category " Billing " gives selected 2 on data/tickets.csv.

Your turn: make Tomás’s criteria checkable

Tomás wrote three more lines for a later version of the option. Rewrite each one as an acceptance criterion, with a command on data/tickets.csv and the result that you expect. Use the table of the open tickets above, and remember --status.

1. “The option should work with the other options.”
2. “Bad input should be handled.”
3. “The report should not get worse.”

Show an example answer

There is more than one good answer. Check that each of yours has a command and an exact result.

1. With --status, the categories apply to that status only. --status pending --category billing keeps only T-1004, the one pending billing ticket. Expected: selected 1, by_category {"billing": 1}, average_priority 2.0. In data/tickets.csv, the pending tickets are T-1004 (billing, priority 2) and T-1010 (LOGIN, priority 1).
2. “Bad input” must name the input. For example: “An empty name is an error: --category "" gives exit code 2 and the message --category needs a name.” This is a new decision. Grace must agree to it before it goes into the brief. Criterion 6 already covers a name that no ticket has: it is not an error.
3. “Not worse” means “the same”, and you can check “the same”. “Without --category, the report of data/tickets.csv is exactly the same as today: tests/test_baseline.py passes.” Criterion 4 already says this.

The last two show a common result: a vague line hides either a new decision or a criterion that you already have.

See it end to endUpgrade a dependency safelyCase study

The case study turns two rejected web addresses into tests and a written objective before the upgrade, and gives the assistant a bounded task.

Knowledge check

Check your understanding

Name the five parts of a task brief, with one line from Amira's brief for each part.

Show the answer

• Observed behaviour: “python -m ticket_cleaner data/tickets.csv always counts every category of the chosen status.”
• Expected behaviour: “--category billing counts only the billing tickets of the chosen status.”
• Inputs: data/tickets.csv, and the option --category NAME, which can be given more than once.
• Constraints: “Change only cli.py, report.py, README.md and the tests. No new dependency.”
• Evidence of done: “python -m pytest passes, with new tests for criteria 1 to 6”, and the commands of criteria 1, 3 and 6 give the expected results.

Check your understanding

Predict: on data/tickets.csv, what do --category billing --category shipping and --category garden give for 'selected'? Explain from the four open tickets.

Show the answer

The open tickets are T-1001 (billing), T-1003 (login), T-1005 (billing) and T-1015 (shipping). --category billing --category shipping keeps T-1001, T-1005 and T-1015: selected is 3, and by_category is {"billing": 2, "shipping": 1}. No ticket has the category garden, so selected is 0, by_category is {}, and the exit code is 0 (criterion 6). You can predict both before any code exists, and that is what makes them criteria.

Check your understanding

Tomás says: 'The vague request gave the smaller diff, and all 29 tests passed. Smaller is better, so it was the better result.' Answer him with the recorded sessions.

Show the answer

Size is not the measure. The one-sentence request changed 4 files (+77 −3) and its tests passed, but it passed only 4 of the 8 hidden checks. It accepted one category, matched the name exactly, and added "category": null to every report. Its tests passed because the assistant wrote them for its own decisions. The brief’s sessions changed more lines and passed all 8 checks. Without written criteria, nobody could call the small change wrong.

Recap

• A vague request leaves decisions open, and the assistant makes them. “Add a way to filter the report by category” gave one category, an exact match and a new key in every report. Its own tests passed.
• A task brief has five parts: observed behaviour, expected behaviour, inputs, constraints and evidence of done.
• An acceptance criterion is a statement about behaviour with a yes-or-no check. Write the command and the expected result, calculated by hand, before any code exists.
• With the brief, both recorded models passed all 8 checks. The smallest diff was the wrong one, and the brief did not remove the need to review.
• Your project has the brief in docs/tasks/category-option.md, committed on feature/category-option.

Next, you write the second task, Grace’s bug, and you choose what the assistant needs to see.

Next: Supply relevant context