Rubrics and property checks
- Lesson 5
- 40 min
- Module 2: Plan and judge without one right answer
Where we are
In the last lesson, we decided where to spend our testing time, and we wrote acceptance criteria as Must and Must not rules. But a rule on paper does not tell you how to judge hundreds of replies that all use other words. In this lesson, we turn the rules into a rubric that two testers apply in the same way, and into property checks that the QA runner applies to every run.
Lesson 5 of 24 · Testing AI Applications: Manual and AI-Assisted QARubrics and property checks
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Decide whether an answer passes when there is no single right text. You judge answers with a short rubric that two testers apply in the same way. You also write property checks that the QA runner applies to every run.
Before you startWhat to know and to have ready
What you need to know
- Your risk table and Grace’s acceptance criteria, from Risk-based test planning. This lesson turns some of the criteria into checks.
- How to ask the app a question with a seed, and how a seed replays a run, from Meet the bug-hunt app.
What you need on your computer
- Your course folder
ai-qa, withpractice-app/and its.venv,templates/andworkspace/. - An internet connection once, to install PyYAML, the one package that the QA runner needs. The app itself needs nothing.
Tested with Python 3.14 on macOS. The Windows commands are written for PowerShell and were not tested on Windows.
A teacher marks two kinds of question
Section titled “A teacher marks two kinds of question”A teacher marks a mathematics test. The first question is “7 × 8 = ?”. The answer key says 56, and any other answer is wrong. Marking takes a second, and every teacher gives the same mark.
The second question is “Explain why plants need light.” No two pupils write the same sentence, so an answer key with one sentence would fail almost the whole class. The teacher uses a marking scheme instead. It gives one point for “light gives the plant energy”, one for “the plant uses it to make food” and one for a correct example. Two teachers with the same scheme give the same marks.
The teacher also has rules that every good answer meets, whatever its words: it is written in full sentences, in the language of the test. A pupil who answers in another language does not pass, however correct the content is.
The problem: an expected text fails good answers
Section titled “The problem: an expected text fails good answers”Testers write an expected result for each test case. For a calculator, the expected result is one number. In Testing a calculator, testing a new colleague, you saw that one expected text fails good answers. Now Esme measures how badly. She asks “How many days do I have to return an unused item?” with seed 1, and she saves the reply as the expected text:
Good question. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]Then she asks the same question with seeds 2 to 20. Of the 20 replies, 16 give the right window of 45 days. Those 16 replies use 14 different texts. Here are two of them, from seeds 3 and 9:
Here is what our policy says. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]You can return an unused item within 45 days of delivery for a full refund of the item price. [1] Is there anything else I can help with?All three replies are simulated (practice app), and all three are correct. Only one of the 16 correct replies is word for word the expected text: the reply of seed 1 itself. An exact-match test fails the other 15.
A real model varies in the same way. The course asked a real model the same question 10 times, with the same prompt, search and tools (recorded, real model). All 10 replies gave 45 days, in 8 different texts. Here are two of them:
You can return an unused item within 45 days of delivery. The 45-day period starts on the delivery date shown in your account. [1]You can return an unused item within 45 days of the delivery date shown in your account. [1]Tomás suggests the opposite: “Just check that the reply contains the word return.” That check passes all 20 simulated replies, including the 4 that do not give 45 days. Esme needs something between “the same text” and “any text with the word return”.
The idea: choose your test oracle
Section titled “The idea: choose your test oracle”You met the test oracle in Testing a calculator, testing a new colleague: whatever tells you whether a result is right. (In ancient Greece, an oracle was a place where people went to ask for the right answer.) For a calculator, the oracle is the expected number. For a support assistant, an expected text is a poor oracle, as you just saw. Testers of AI features use four better ones, from the strictest to the most flexible.
Text description of the diagram
The ladder has four rungs, from the bottom to the top.
Exact match: you need the expected text. It catches any change, but it fails good answers that use other words. It is cheap to run and almost impossible to write for answers that vary.
Rubric: you need a few short criteria and a reader, a person or a model. It catches wrong facts, missing parts and tone problems. It costs minutes of reading for each reply.
Property checks: you need rules that every good answer meets. They catch a wrong language, a missing or outdated source and invented amounts. A program checks them on every reply in seconds.
Metamorphic relations: you need a change to the input that must not change the answer. They catch answers that change with a name or a phrasing, and they need no expected answer. The next lesson covers them.
You climb the ladder when the rung below cannot judge your answers. Most test plans use several rungs together.
Rubrics: a marking scheme for answers
Section titled “Rubrics: a marking scheme for answers”A rubric is a short list of criteria that a reader uses to judge an answer, with a score for each level. It is the teacher’s marking scheme. A good rubric has three parts:
- Criteria with a yes/no test. “Every number, date and rule in the reply is in the cited policy” can be checked. “The reply is accurate” cannot, because two readers disagree on what accurate means.
- Score anchors that say what each score looks like, with an example.
- A pass rule, for example “score 3 or more”.
The templates folder has a rubric with five criteria and four scores. Here is its core:
1. Correct facts. Every number, date and rule in the reply is in the cited policy or the order. (Yes/No)2. Cites the source. The reply cites the policy that supports it, in its version in force. (Yes/No)3. Answers the question. The reply answers what the customer asked, not a nearby question. (Yes/No)4. Safe action. The reply asks before any action (cancel, refund) and acts only for this customer. (Yes/No)5. Language and tone. Same language as the customer; polite and calm. (Yes/No)
- 4: all five criteria are met.- 3: criteria 1, 3 and 4 are met; one small problem in 2 or 5.- 2: the reply is incomplete or partly wrong, but it does no harm.- 1: a wrong fact, a wrong action, a leak or a harmful reply.Pass: score 3 or more.A tiny example: three replies to a French question
Section titled “A tiny example: three replies to a French question”Esme scores three replies to the French version of the question: “Combien de jours ai-je pour retourner un article non utilisé ?” The returns policy in force on the app date is version 4: 45 days. The replies come from seeds 1, 9 and 2.
| Seed | Reply (start) | 1 Facts | 2 Source | 3 Answers | 4 Action | 5 Language, tone | Score |
|---|---|---|---|---|---|---|---|
| 1 | “Bonne question. Vous pouvez retourner un article non utilisé dans les 45 jours…” cites version 4 | Yes | Yes | Yes | Yes | Yes | 4: pass |
| 9 | “You can return an unused item within 45 days of delivery…” cites version 4 | Yes | Yes | Yes | Yes | No: English | 3: pass |
| 2 | “Thanks for asking. You can return an unused item within 30 days…” cites version 3 | No: 30 days is the old rule | No: old version | Yes | Yes | No: English | 1: fail |
The scores are the course author’s. The replies are simulated (practice app). The rubric does its job on seed 2: a wrong fact gives score 1, whatever else the reply does well. You met this outdated answer in Testing a calculator, testing a new colleague.
But look at seed 9. A French-speaking customer gets an English answer, and the rubric passes it with a 3. The rubric calls the wrong language a “small problem”. Grace’s criterion 4 says that the reply must be in the customer’s language. A score adds problems together, so a must-rule can disappear inside it. A rule that must never break needs its own check, on every run. That is the next idea.
Who reads with the rubric
Section titled “Who reads with the rubric”A rubric needs a reader. A person takes one or two minutes for each reply, so a person can read 30 replies, not 3,000. A model judge, a language model that scores replies with your rubric, can read thousands. But it makes mistakes of its own, and you must check it against your own scores first. Triage many outputs with a model judge shows how a tester uses one. Write a usable rubric in the evaluation course goes deeper into rubrics, and Calibrate reviewers shows how to measure whether two readers agree.
Property checks: rules every good answer meets
Section titled “Property checks: rules every good answer meets”The teacher’s rule “in the language of the test” needs no marking scheme. You can check it in a second, and it is true of every good answer, whatever its words. A property check is a rule of this kind for an AI feature. It names something that every good answer has, or never has, and a program can check it. The name comes from the everyday word: a property is a quality that a thing has.
The outline of this course names three properties of a good help desk answer:
- It answers in the customer’s language. A French question gets a French reply.
- It cites a policy. A policy answer has a source.
- It never quotes an amount that it was not given. A question about the return window gets no amount in dollars. A refund offer gives only the amount that the order gives.
Esme checks the reply of seed 9 against three properties by hand first.
Text description of the diagram
On the left is the customer’s question, in French: “Combien de jours ai-je pour retourner un article non utilisé ?” Below it is the reply of seed 9, in English: “You can return an unused item within 45 days of delivery for a full refund of the item price. [1] Is there anything else I can help with?” Its source is “Returns policy, version 4: Return window”.
On the right are three checks. Language: the question is French, the reply is English, so the check fails. Cites the returns policy: the reply cites the returns policy, version 4, so the check passes. No amount: the reply has no number like 12.95, so the check passes. The verdict at the bottom: one check fails, so the run fails.
Each check is simple, and none of them needs an expected text. The language check passes every French wording alike, and it fails an English reply even when its facts are right.
The QA runner applies properties to many runs
Section titled “The QA runner applies properties to many runs”Checking by hand works for three replies. A sometimes-failure needs many runs, so Esme lets a program do it. The QA runner in practice-app/qa/ reads test cases from YAML files. It runs each case several times, each time with a new seed, in a new conversation, on fresh test data. Then it applies the checks to every reply.
YAML is a plain-text format of key: value lines; you only edit it. Esme starts workspace/cases/properties.yaml with two cases. A line --- separates them:
# Property checks (Module 2): rules that every good answer follows, instead of one exact expected text.id: prop-refund-time-citestitle: An answer about refund times cites the refunds policymessage: How long does a refund take to reach my card?checks: - cites: refunds-and-payments - not_contains: "within 3 business days after we receive"runs: 10---id: prop-french-answer-in-frenchtitle: A French question gets a French answermessage: Combien de jours ai-je pour retourner un article non utilisé ?checks: - language: frruns: 10Each line under checks is one property. In the first case, cites: refunds-and-payments passes when the reply cites the refunds and payments policy. not_contains passes when the reply does not contain the text. “Within 3 business days after we receive” sounds right, but no policy says it: it mixes two real rules. A reply that says it has invented a time. In the second case, language: fr passes when the reply is in French. runs: 10 runs each case 10 times, with seeds 1 to 10.
Esme runs the file. The output:
QA run: 2 case(s) from ../workspace/cases/properties.yamlApp: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8PASS prop-refund-time-cites 10/10 passed (needs 10; seeds 1-10)FAIL prop-french-answer-in-french 7/10 passed (needs 10; seeds 1-10) first failing run: seed 2, trace tr-0001-1 - language: expected fr, the reply is in en
2 case(s): 1 passed, 1 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not runReport: reports/qa-<date>-<time>/report.mdRead it from the top. The runner used the app 2.3.0 with the simulated model. The refund-time case passed in all 10 runs. The French case passed in 7 of 10 runs, and it needs 10 of 10, so it fails. Its first failing run used seed 2, and its reply was in English. The report file lists every failing run (seeds 2, 8 and 9) with its reply. It also saves each failing conversation as JSON, ready to attach to a bug report.
This is a defect: the help desk answers some French questions in English. The case found it in 3 of 10 runs (simulated, practice app). Grace’s criterion 4 needs 10 of 10, and her decision makes it a high-severity bug. How sure is “3 of 10”? Repeat runs and reproduction rate shows how many runs you need.
Rubric or property check?
Section titled “Rubric or property check?”You usually need both. They catch different things and cost different amounts:
| Rubric | Property check | |
|---|---|---|
| Who applies it | A person, or a model judge that you checked | The QA runner, a program |
| Cost per reply | One or two minutes of reading | Less than a second |
| Catches | Wrong facts, missing parts, a nearby question answered, tone | Language, missing or outdated source, an amount that should not be there, a tool that should not be called |
| Misses | Problems the reader does not notice when tired; mistakes of an unchecked judge | Anything that no rule describes: a polite reply with the wrong advice |
So Esme puts every must-rule into a property check that runs on every reply. She reads a sample of replies with the rubric, to find the problems that no rule describes yet. Each new problem that she finds can become a new property.
Property checks have one more risk: a weak property passes bad answers. contains: return passes everything, as Tomás’s idea showed. A good property fails at least one bad answer that you can name. Before you trust a new property, try it on a reply that should fail.
Common mistake
A rubric that two testers apply differently
Symptom: Tomás writes a quick rubric: “Is the answer good? Score 1 to 5.” Esme and Tomás both score the same four replies to the returns question. These scores are constructed for this example:
| Reply | Esme | Tomás |
|---|---|---|
| Seed 1, 45 days, cites version 4 | 5 | 4 |
| Seed 4, 45 days, longer, with “The 45 days start on the delivery date” | 5 | 3 (“too long”) |
| Seed 9 to the French question, in English | 2 | 4 (“facts are right”) |
| Seed 2, 30 days, cites version 3 | 1 | 3 (“looks fine to me”) |
They agree on none of the four replies. On seed 2, Tomás passes a wrong answer, because he does not know that the return window changed to 45 days.
Cause: “good” is not a criterion. Each reader fills it with their own idea: Esme thinks of the policy, Tomás of length. And the rubric gives the reader no facts to check against.
Fix: write criteria with a yes/no test, and name the source of truth. The template does: “Every number, date and rule in the reply is in the cited policy, version in force.” Give score anchors with an example. Then both testers score the same 10 replies and compare. If they disagree on more than one, make the criterion that caused it more exact.
Consequence in AI work: a vague rubric gives numbers that look like measurements but are opinions. Two test rounds by two testers cannot be compared, and a wrong answer passes when the reader does not know the rule.
Guided practice: run your first property checks
Section titled “Guided practice: run your first property checks”-
Open a terminal in
ai-qa/practice-app/and activate the environment, as in Meet the bug-hunt app. The prompt starts with(.venv). -
Install PyYAML, the package that the QA runner needs:
Terminal window python -m pip install -r requirements.txtThe last line of the output is
Successfully installed PyYAML-6.0.3. -
In the folder
workspace/cases, which you have since Meet the bug-hunt app, create the fileproperties.yamlwith Esme’s two cases from this lesson. -
Check that your case files are valid. The runner reads them without running them:
Terminal window python -m qa check ../workspace/casesThe output names every case in the folder, your two new ones and the three from Module 1:
5 valid case(s): return-window-one-run, return-window-ten-runs, prop-refund-time-cites, prop-french-answer-in-french, smoke-first-answerIf you see
Case error, the message names the case to fix. YAML needs spaces, not tabs, before each- languageline. -
Run the Module 1 file first. It holds the return question twice, once with
runs: 1and once withruns: 10:Terminal window python -m qa run ../workspace/cases/one-run-is-not-enough.yamlPASS return-window-one-run 1/1 passed (needs 1; seed 1)FAIL return-window-ten-runs 8/10 passed (needs 10; seeds 1-10)first failing run: seed 2, trace tr-0001-1- contains: missing ['45 days']The same check passes with 1 run and fails with 10. This is the lesson of Module 1, now in a test case.
-
Run your new file:
Terminal window python -m qa run ../workspace/cases/properties.yamlYou see
PASS prop-refund-time-citesandFAIL prop-french-answer-in-french 7/10 passed, as above. The seeds are fixed, so your result is the same. -
Open the
report.mdfile that the last line names. Find the replies of seeds 2, 8 and 9. Score each one with the template rubric.
Show the scores for step 7
Seed 9 gets a 3 with the template rubric: right facts and source, but English. It passes the rubric and fails the property check, as in the lesson. Seeds 2 and 8 answer in English and give 30 days with the returns policy, version 3. That is a wrong fact (criterion 1), so each gets a 1. The property check found the language problem in all three runs. It did not see the 30 days, because no check in this case looks at the number of days. That is the job of criterion 1 of Grace’s list, and of the regression cases of later modules.
Your turn: a rubric and a property for refunds
Section titled “Your turn: a rubric and a property for refunds”Now the refund feature, the top of your risk table.
Part 1. Copy templates/rubric.md to workspace/plan/rubric-refund-reply.md. Adapt it to a reply about a return or a refund. Say which facts must be right (the amount, the number of days, the rule). Say which policy it must cite, and what a safe action is. Keep 4 to 6 yes/no criteria, score anchors and a pass rule.
Part 2. Add a third case to workspace/cases/properties.yaml, after a line ---. It checks the refund offer for the hose reel of order LK-200102 (test customer C-1001, the default). Think about Grace’s criteria 2 and 3: the amount from the order, and no refund before a yes. Use the checks of the runner’s README, in practice-app/README.md. Run the file again.
Part 3. If a run fails, open its reply in the report. Score it with your rubric.
Show a possible answer
Part 1. Esme’s rubric, the start of workspace/plan/rubric-refund-reply.md:
# Rubric: a reply about a return or a refund
## Criteria
1. **Correct facts.** Every amount, number of days and rule in the reply is in the cited policy or in the customer's order. (Yes/No)2. **Cites the source.** The reply cites the returns or refunds policy in its version in force (returns policy version 4). (Yes/No)3. **Answers the question.** The reply answers what the customer asked. (Yes/No)4. **Safe action.** The reply asks before it starts a refund, and acts only for this customer. (Yes/No)5. **Language and tone.** Same language as the customer; polite and calm. (Yes/No)
## Score
- **4:** all five criteria are met.- **3:** criteria 1, 3 and 4 are met; one small problem in 2 or 5.- **2:** incomplete or partly wrong, but no harm.- **1:** a wrong amount, a wrong action, a leak or a harmful reply.
Pass: 3 or more. A score of 1 is a bug.Part 2. The third case:
---id: prop-refund-amount-from-ordertitle: A refund offer gives the amount from the order (64.00 - 12.95 = 51.05)message: I want to return the hose reel from order LK-200102checks: - contains: "51.05" - tool_not_called: start_refundruns: 1051.05 is the price of the hose reel (64.00 dollars) minus 12.95 dollars return shipping. tool_not_called: start_refund checks Grace’s criterion 3: after the first message, no refund is started. The quotes around "51.05" keep it as text.
The run shows FAIL prop-refund-amount-from-order 9/10 passed, and the first failing run is seed 4: contains: missing ['51.05'].
Part 3. The reply of seed 4 (simulated, practice app):
I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 79.00 dollars (the item price, the shipping fee and a 15-dollar goodwill credit). Reply yes to confirm.Criterion 1: No. The order has no shipping fee to refund, and no policy gives a 15-dollar credit. Criterion 2: No, there is no source. Criteria 3, 4 and 5: Yes. Score 1: a wrong amount. Esme adds this worked example at the end of her rubric file. You write the bug report for it in A bug report template for AI behaviour. There are more defects in the refund feature than this case can see.
Knowledge check
Section titled “Knowledge check”Check your understanding
Esme saves the reply of seed 1 as the expected text and compares 20 runs with it. What goes wrong, and which oracle would you use instead for the return window?
Show the answer
An exact-match oracle fails good answers that use other words. Of 20 simulated runs, 16 replies gave the right 45 days, in 14 different texts. Only seed 1 matched the expected text, so 15 correct replies would fail. A real model varies too: 10 recorded runs gave 8 different texts. Use property checks instead: the reply contains 45, cites the returns policy, version 4, and is in the customer’s language. Read a sample with a rubric for what the rules cannot see.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With the template rubric, an English reply to a French question scores 3 and passes. Why, and what does Esme add?
Show the answer
The template counts the wrong language as a “small problem” in criterion 5, and score 3 allows one small problem. A score adds problems together, so a must-rule can disappear inside it. Esme checks the must-rule on its own, on every run, with the property language: fr. That case found English replies in 3 of 10 runs.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Tomás adds the property `contains: days` to a case for the return window. Which wrong reply does it still pass? Write a better property.
Show the answer
It passes the reply that gives 30 days with the returns policy, version 3, because “30 days” contains “days”. Every reply about a return window contains “days”, so the check fails nothing. Better: contains: 45 days together with cites: returns-policy@4, as in Grace’s criterion 1. Test a new property on a reply that should fail before you trust it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A test oracle decides whether a result passes. An exact expected text is a poor oracle for an AI feature. 15 of 16 correct simulated replies used other words, and a real model gave 8 texts in 10 runs.
- A rubric has yes/no criteria, score anchors and a pass rule. A score can hide a must-rule, such as the customer’s language.
- A property check is a rule that every good answer meets. Examples: the customer’s language, a cited policy in its version in force, no amount that it was not given. The QA runner applies properties to every run.
- On the French returns question, the runner found replies in English in 3 of 10 runs (simulated, practice app): a defect.
- Use both: properties for every must-rule on every run, and a rubric on a sample, to find what no rule describes yet.
Your workspace now: your notes from Module 1, workspace/plan/ with risk-table.md, acceptance-criteria.md, refund-risks.md and rubric-refund-reply.md, and workspace/cases/properties.yaml with three cases.
Next, you judge answers when you do not even know the right answer, by comparing two answers that must agree.
Next: Metamorphic testing