Acceptance criteria

What you will be able to do

Turn “it should give good answers” into criteria that a person or a script can check, and write them before you build. Then “done” means the same thing to you and to your user.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• Your brief: one user, one decision, the documents and the non-goals, from Problem and audience.
• A criterion that two people check in the same way, from Map goals to evidence in Evaluating and Testing AI Systems.
• Blockers and acceptable regressions, from Establish a baseline in the same course.
• The access matrix of owner, staff and read-only, from Users and organizations in the authentication course.
• The model cost of one answer, from Set up the design pack in the system design course.

What you need on your computer

• The project folder doc-assistant with docs/project-brief.md from the last lesson. No new packages.

Ordering a birthday cake

Imagine that you order a birthday cake from a baker. You can say “Make me a nice cake.” Or you can say:

• chocolate, for 20 people;
• no nuts, because one guest has an allergy;
• ready on Saturday at 10:00;
• at most 60 dollars.

On Saturday, you and the baker can both check every line. Is it chocolate? Does it feed 20? Are there nuts? Is it ready? What did it cost? Nobody argues, because each line has a clear yes or no.

“Nice” is different. You may find the cake nice and the baker may not, or the other way round. And look at the nut line: it is not like the others. A cake for 19 people is a small problem. A cake with nuts is a failure, however good it tastes. Some lines allow a little room; some allow none. The price, too, was agreed before the baker started, not after.

The problem: “done” means different things

Grace tells Sam what she wants: “It should give good answers. It must be safe. It must be quick, and not expensive.”

Three weeks later, in a made-up scene that happens in many projects, Tomás shows a demo. He asks three questions that he chose, gets three good answers, and says: “Done.” Grace asks one of her own: a question that the documents do not answer. The assistant gives a confident answer that sounds right and is not in any document. “That is exactly what I was afraid of,” says Grace. “It is not done.”

Who is right? Both, by their own rules. Tomás’s rule was “it answers my questions well”. Grace’s rule was “it never guesses”. Neither rule was written, so the argument happens at the end, after the work, when it is most expensive.

There is a second, quieter problem. Suppose that the team writes the bar after it measures. The assistant gets 71% of the answers right, and someone says, “70% is fine for a first version.” The bar now follows the result. It can never fail, so it proves nothing.

The idea: an acceptance criterion

An acceptance criterion is a condition that the product must meet before its user accepts it. It is written so that a person or a script can say “met” or “not met” without an argument. The name comes from software teams and contracts: the user accepts the work when every criterion is met. The evaluation course showed what makes a criterion measurable: two people who check the same output get the same answer.

A product like Sam’s needs four kinds of criteria. They are the four headings of the template acceptance-criteria.md:

1. Required behaviour. What the product does in a given situation. Each one is written as Given / When / Then: the situation, the action, and the result that anyone can observe. For example: given a member of a shop, when they upload a PDF file of 6 MB, then the upload is refused with a reason. Tests check these.
2. Quality thresholds. A measure, a number, the set of questions that it is measured on, and why that number. “Correct on at least 75% of the answerable held-out questions” is one. The evaluation in Module 5 checks these.
3. Access control. Who may see or change what. Here there is no percentage: one failure means “not met”, like the nuts in the cake. The course’s marking rule says the same: a project with a failed access-control check is “not yet demonstrated”, whatever else works.
4. Budget. The money and the resources that the product may use, per question, per month and for the whole project.

A tiny example: one wish becomes a criterion

Take Grace’s biggest worry, “it must not make things up”, and turn it into a criterion one step at a time.

One wish, five steps. The behaviour says what must happen once; the measure and the threshold say how often, on which questions; the last two lines say how it is checked and why.

Text description of the diagram

The flow starts with Grace’s wish: “It must not make things up.” The first step writes the behaviour B2 as Given, When, Then. Given a signed-in member, when they ask a question that the documents do not answer, then the answer says so and cites nothing. The second step names the measure Q2. It counts the held-out questions that the documents do not answer, or that ask about the other shop. Q2 is the share of them where the assistant abstains. The third step sets the threshold at 100%, written before any measurement. The fourth says how it is checked: by the held-out evaluation in Module 5. The last step gives the reason: for Grace, a confident wrong answer is worse than no answer.

To abstain means to give no answer on purpose: here, to say “the documents do not answer this”. Notice that the wish became two criteria, a behaviour (B2) and a threshold (Q2). B2 says what must happen for one question. Q2 says on how many questions it must happen, and on which ones.

Now the numbers. Exact words matter as much as the number itself. Here is a small made-up case: the assistant answers 8 answerable questions, and 6 of the answers are correct.

Criterion as written	6 of 8 = 75%	Met?
“at least 75% correct”	75% is at least 75%	met
“more than 75% correct”	75% is not more than 75%	not met
“75% correct”	is 76% a failure? nobody knows	cannot say

So write “at least” or “at most”, never a bare number. Also notice the size of the set: with 8 questions, one answer moves the result by 12.5 points. Sam’s held-out set, which you reserve in the next lesson, has 35 answerable questions. There, “at least 75%” means at least 27 correct answers, because 26 of 35 is only 74.3%. And it has 6 questions where the assistant should abstain, so Q2’s 100% means all 6, with no room at all. Uncertainty and slices in the evaluation course shows how much a small set moves by chance.

Sam’s acceptance criteria

Sam writes the criteria with Grace in one meeting, before any code. Here are the four families of the file docs/acceptance-criteria.md, with their IDs.

Required behaviour. Eight lines, B1 to B8:

ID	Given	When	Then
B1	a signed-in member of a shop	they ask a question that the shop’s documents answer	they get a short answer, and every claim cites a passage of their own shop’s documents
B2	a signed-in member	they ask a question that the documents do not answer	the answer says that the documents do not answer it, and cites nothing
B3	two versions of a policy	a question names a date	the answer uses the version in force on that date
B4	a member who may upload	they upload a .md, .txt, .html or .pdf file of at most 5 MB	they see its status move from uploaded to indexed, or to failed with a reason they can act on
B5	the same file	it is uploaded twice	there is one document, and the second upload says so
B6	a member who may delete	they delete a document	its file, passages, vectors and the stored answers that cite it are gone, and no later answer uses it
B7	the model provider is slow or down	a member asks a question	they get a clear error within 10 seconds of the last attempt, and nothing half-done is stored
B8	any answer	the member clicks a citation	they see the passage, its document, version and dates

Each line comes from the brief. B3 is the 30-day and 45-day mistake. B7 gives Grace’s staff a clear error, not a spinning wheel while the customer waits. B4 says “a reason they can act on”, so “error 500” fails it.

Quality thresholds. Five measures, each with its reason:

ID	Measure	Threshold	Why this number
Q1	correct answers on answerable held-out questions	at least 75%	the RAG course’s assistant reached about 85% on similar questions; staff check every answer anyway
Q2	abstains on questions the documents do not answer (also another shop’s questions)	100%	a confident wrong answer is worse than “no answer” for Grace
Q3	citation validity (every cited passage exists in this shop and was given to the model)	100%	code can check it fully, so anything less is a bug
Q4	follows an instruction hidden in a document	0 times	the supplier notes contain one on purpose
Q5	median time from question to answer, on a laptop	under 5 seconds	the customer is waiting on the phone

The “why” column is the part that reviewers read first. Q1’s reason links to a real measurement, Answer measures in the RAG course. The bar is lower, because staff check every answer anyway. Q3 is 100% for a different reason than Q2. Code can check a citation completely, so a wrong one is a bug, not a matter of quality. Q4 counts something that must never happen. One of the supplier notes contains a hidden instruction, as in Permissions and isolation in the RAG course.

Access control. Four rules, each with the check that proves it:

ID	Rule	Checked by
A1	a member of one shop never receives another shop’s documents, passages or answers	python -m scripts.isolation: 0 leaks
A2	someone who is not a member gets the same “not found” for a real and a made-up shop	the access tests
A3	a read-only member never receives a staff-only passage, and cannot upload or delete	the access tests and the isolation checks
A4	every request that changes data from the page needs the CSRF token	the sign-in tests

These rules come from the authentication course. A2 means that an outsider cannot even learn which shops exist. A4 protects against another website that sends requests in the name of a signed-in member. This attack is called cross-site request forgery (CSRF), and the CSRF token is a secret value that only the real page knows. Module 4 builds it. The scripts and tests named here do not exist yet. The criterion names them anyway, because it says how “met” will be decided.

Budget. Three limits:

ID	Limit	Value
C1	model cost per question	under US$0.001
C2	model cost per shop and month (enforced)	US$5, and 200 questions a day
C3	money for building and evaluating the whole project	at most US$2 of model calls; no cloud resources

C1 has a measured base. The system design course measured the small chat model’s cost of one answer at about US$0.000171. So US$0.001 leaves room for longer prompts, and it would still catch a mistake such as sending 50 passages instead of 5. C2 is enforced: the product itself refuses questions after the limit, in Module 6. C3 is the course team’s own rule: the whole worked capstone was built and evaluated within it.

Four families of criteria, each with its own kind of check and the modules that build it.

Text description of the diagram

The file has four families. Required behaviour, B1 to B8, is checked by tests and the failure cases in Modules 3 and 4. The quality thresholds, Q1 to Q5, are checked by the held-out evaluation in Module 5. Access control, A1 to A4, is checked by the access tests and the isolation checks in Modules 4 and 5. The budget, C1 to C3, is checked by the token counts times the price list, and by usage limits, in Modules 5 and 6.

Written before, never moved

Tomás has an objection. “How can you choose 75% before you measure anything? Measure first, then set a realistic bar.”

Priya, the data scientist, answers: “A bar that you set after you see the result always passes, so it tells nobody anything. You choose the bar from what the user needs and from earlier measurements, like the RAG course’s. Then the measurement says if you reached it.”

So Sam follows two rules. First, the criteria are written and committed before any measurement. Second, if a criterion must change, Sam writes down the reason, the date and who agreed. A change never happens after a look at the held-out results. If the product misses a threshold, the report says “not met”. In Module 5, one of Sam’s criteria is not met, and it stays not met in the report. That is not a failed capstone. It is an honest one, and reviewers trust it more.

When to write criteria, and what it costs

Write acceptance criteria whenever someone else must accept your work: a user, a reviewer, a customer, or an examiner of your portfolio. Write them before you build, with the user in the room if you can. Each criterion then tells you what to test and when you can stop.

They cost an hour of careful discussion now, and some arguments, for example “is 100% abstention realistic?” A number that is set too high produces an honest “not met” later. These costs are smaller than the argument at the end. For a first experiment that only explores what a model can do, a short list of required behaviours is enough. A capstone is not such a case: you know enough to set numbers.

Guided practice: five wishes

Here are five wishes from Grace’s team. Rewrite each one as a criterion, with its family. Use Sam’s file as a model, then open the answer.

1. “The answers should be accurate.”
2. “It must be fast.”
3. “It must be secure.”
4. “It should handle big files.”
5. “It must be cheap.”

Show the answer

1. Quality threshold, like Q1: “correct answers on answerable held-out questions: at least 75%”, with the reason.
2. Quality threshold, like Q5: “median time from question to answer, on a laptop: under 5 seconds”, because the customer is waiting. “Fast” alone has no number and no place where it is measured.
3. Access control, as several rules, because “secure” is too wide to check. A1 covers another shop’s data, A2 outsiders, A3 read-only members and A4 the CSRF token. Each one has its own check, and each one allows no failure.
4. Required behaviour, like B4: “given a member who may upload, when they upload a file of at most 5 MB in a supported format, then …”. “Big” becomes a number, and a file over the limit gets a clear refusal, not a crash.
5. Budget, like C1 and C2: a cost per question, and an enforced limit per shop and month.

Look at number 3. One vague wish became four criteria. That is common: the vaguer the wish, the more criteria it hides.

Your turn: a new need from Omar

Change one condition: a new need. Omar, the product manager, listens to a support call and says: “The answers are too long to read aloud to a customer on the phone.” Write one criterion for this need. Before you open the answer, decide which family it belongs to, and how it is checked.

Show the answer

One good answer is a quality threshold: “Q6: at most 80 words in at least 95% of the answers to the held-out questions. Checked by counting the words of each answer in the evaluation. Why: 80 words take about 30 seconds to read aloud.” Another good answer is a required behaviour: “Given any answer, then its text has at most 80 words, without the citations.”

The exact number is a decision for Omar and Grace, not a fact. What matters is that it is a number, measured on named questions, with a reason. This criterion is an exercise: Sam’s file does not have it. If Sam added it now, before any measurement, that would be fine. Adding it in Module 5, after seeing long answers, would also be fine, as long as the report says when and why it was added.

Common mistakes

Common mistake

A criterion that nobody can check

Symptom: the file has lines such as “Answers are helpful” or “The app is user-friendly”. In Module 5, the evaluation report cannot say “met” or “not met” for them, and the review becomes an argument.

Cause: the line names a feeling, not something that a person or a script can observe.

Fix: ask “how would I check this on one answer?” If the answer is “I would read it and decide”, write down what the reader looks for, as in Map goals to evidence. If you cannot say, the line is a goal for the brief, not a criterion.

Common mistake

The bar that moved

Symptom: the evaluation report says “Q1: at least 70%, met (71%)”. But the commit history shows that the criterion said 75% until the day after the first results.

Cause: the threshold was changed after a look at the results, so it now follows the result.

Fix: restore the original number, and report “not met (71%)”. If the change has a real reason, keep both: the original criterion, the new one, the reason, the date and who agreed. Reviewers check the history of this file, so commit it before any measurement, and they can see that you did.

Your project

Write your own acceptance criteria, before any code.

1. Copy the template into docs/:

   macOS / Linux

   cp docs/templates/acceptance-criteria.md docs/acceptance-criteria.md

   Windows (PowerShell)

   Copy-Item docs\templates\acceptance-criteria.md docs\acceptance-criteria.md

2. Write at least two required behaviours: one where the product answers, and one where it must refuse or abstain.

3. Write at least two quality thresholds. Each one has a measure, “at least” or “at most”, a number, the questions it is measured on, and a reason.

4. Write one access rule for each boundary in your product, for example between two shops, or between staff and read-only members.

5. Write at least two budget limits: a cost per question, and the money for the whole project.

6. Read the file with your user, or with a friend who plays the user. Change what they disagree with now.

7. Commit the file before you write any product code.

On the standard track, you may keep Sam’s IDs and change the numbers and reasons that you disagree with. Write down why you changed them.

Evidence you keep: docs/acceptance-criteria.md and its commit, which comes before any measurement.

Checklist

• Every line can be checked by a person or a script, with “met” or “not met”.
• Every threshold says “at least” or “at most”, a number, the set of questions, and a reason.
• Every access rule names its check, and allows no failure.
• The budget has a cost per question and a total.
• Your user read the file.
• The commit is earlier than any measurement.

See it end to endCapstone on a public document collectionCase study

The case study keeps these thresholds for a public collection, and writes one more line before any answer: the rule for choosing a change (at least 2 more right answers on the tuning questions). Each alternative fixed only one question, so the rule kept the baseline.

Knowledge check

Check your understanding

What is an acceptance criterion, and why are Sam's access-control criteria different from the quality thresholds?

Show the answer

An acceptance criterion is a condition that the product must meet before its user accepts it. A person or a script can say “met” or “not met” without an argument. The quality thresholds allow some failures: Q1 accepts any result of at least 75% correct answers. The access-control criteria allow none. One answer that shows another shop’s passage means that A1 is not met, like nuts in the cake for a guest with an allergy. One leak can do real harm, so a share such as “99% of answers do not leak” makes no sense.

Check your understanding

Predict: the held-out set has 35 answerable questions. The assistant answers 26 of them correctly. Is Q1 ('at least 75%') met? And how many correct answers does Q1 need?

Show the answer

26 of 35 is 74.3%, so Q1 is not met, although it is very close. Q1 needs at least 27 correct answers, because 27 of 35 is 77.1% and 26.25 is not a whole number of questions. With 35 questions, each answer moves the result by about 2.9 points. That is why the report also gives the counts, not only the percentage.

Check your understanding

Practical check: show your acceptance criteria. Which line is a refusal behaviour, which thresholds have a reason, which access rule has a check, and when was the file committed?

Show the answer

A good file has a required behaviour where the product must refuse or abstain, like Sam’s B2. Every quality threshold has “at least” or “at most”, a number, the set of held-out questions and a reason. Sam’s Q1, for example, is at least 75%, below an earlier measurement. Every access rule names its check, like A1 (the isolation checks, 0 leaks). The budget has a cost per question and a total. git log -- docs/acceptance-criteria.md shows a commit from before any measurement.

Recap

• Without written criteria, “done” means something different to each person, and the argument comes at the end. A bar set after the measurement always passes, so it proves nothing.
• An acceptance criterion can be checked by a person or a script, with “met” or “not met”.
• Four families: required behaviour (Given / When / Then), quality thresholds, access control (no failure allowed) and budget. A threshold has a measure, “at least” or “at most”, a number, the questions and the reason.
• Exact words matter: 6 of 8 meets “at least 75%” and fails “more than 75%”. On Sam’s 35 answerable held-out questions, Q1 needs 27 correct answers.
• Sam’s file has B1 to B8, Q1 to Q5, A1 to A4 and C1 to C3, written with Grace and committed before any code. A criterion never moves after a look at the results.

Next, you plan the evidence for every claim, and you lock the held-out questions before any tuning.

Next: Plan the evidence