Metamorphic testing
- Lesson 6
- 40 min
- Module 2: Plan and judge without one right answer
Where we are
In the last lesson, we judged replies with a rubric and property checks. Both need you to know what a good answer looks like. But sometimes you do not know the right answer: should this customer get a late-delivery offer or not? In this lesson, we change the input in a way that must not matter, and we check that the two answers still agree.
Lesson 6 of 24 · Testing AI Applications: Manual and AI-Assisted QAMetamorphic testing
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Test an answer when you do not know the right answer. You change the input in a way that must not change the meaning of the answer, and you check that the two answers agree. This finds defects that no single expected answer can see.
Before you startWhat to know and to have ready
What you need to know
- Test oracles, rubrics and property checks, and how to run a case with the QA runner, from Rubrics and property checks.
- Your risk table, with the refund feature in detail, from Risk-based test planning.
What you need on your computer
- Your course folder
ai-qa, withpractice-app/,templates/andworkspace/, and PyYAML installed in the.venvofpractice-app/. In a new terminal, activate the environment again, as in Meet the bug-hunt app.
Tested with Python 3.14 on macOS. The Windows commands are written for PowerShell and were not tested on Windows.
A bathroom scale and your true weight
Section titled “A bathroom scale and your true weight”You do not know your exact weight. The bathroom scale is the only thing that tells you. So how can you test the scale? You cannot compare its number with the right answer, because you do not have the right answer.
But you can still catch a broken scale. Step on it twice: the two numbers must be the same. Step on it facing the wall: the number must not change. Step on it holding a 1-litre bottle of water: the number must go up by about 1 kg. If the scale shows a different weight when you face the wall, it is broken. You found the fault without knowing your true weight.
The problem: an answer that looks right on its own
Section titled “The problem: an answer that looks right on its own”Esme tests the late-delivery rule. Larkfield’s delivery policy says: if an order is more than 5 business days late, Larkfield refunds the shipping fee. Esme writes a customer message and signs it with a name, as customers often do:
My order is 8 business days late. What can you do? Thanks, OluwaseunHere is the reply with seed 1 (simulated, practice app):
Good question. Delivery times are counted in business days (Monday to Friday, except statutory holidays) from the day your order ships. [1]
Sources:[1] Delivery policy, version 4: Delivery times and fees (/policies/delivery-policy/v4#delivery-times-and-fees)Judge it with the tools of the last lesson. It is in English. It cites the delivery policy, version 4, the version in force. It quotes no amount. It is polite. All the property checks pass. A reader with the rubric may pass it too. Only a reader who knows the late-delivery rule sees that it does not answer “What can you do?”
Now Esme changes one thing: the name. Same message, same seed, signed Emily:
Good question. If your order arrives more than 5 business days late, we refund the shipping fee. [1]
Sources:[1] Delivery policy, version 4: Late and lost orders (/policies/delivery-policy/v4#late-and-lost-orders)Emily gets the refund of the shipping fee. Oluwaseun gets a sentence about business days. The only difference in the two messages is the name.
Text description of the diagram
The diagram has two columns. On the left is the message signed Emily, with seed 1. Its reply says: “If your order arrives more than 5 business days late, we refund the shipping fee.” It cites the delivery policy, version 4, section “Late and lost orders”. On the right is the same message signed Oluwaseun, with seed 1. Its reply says: “Delivery times are counted in business days.” It cites the delivery policy, version 4, section “Delivery times and fees”.
Below each reply are the same three property checks: in English, cites the delivery policy version 4, no amount. All three pass in both columns. At the bottom, a bar across both columns says: the offer differs, so the relation fails.
This is the problem. Each reply, alone, looks like a possible answer. To fail the reply to Oluwaseun with an expected text or a rubric, you must already know the right answer for this exact message. You cannot write the right answer for every message that every customer can write. But you know one thing without any policy at all: a customer’s name must not change what Larkfield offers.
The idea: compare two answers that must agree
Section titled “The idea: compare two answers that must agree”That is the bathroom-scale trick. You change the input in a way that must not change the answer, and you compare the two answers. This is a metamorphic relation: a change to the input, and the relation that the two outputs must keep. A test that uses one is a metamorphic test.
The name comes from metamorphosis, a change of form. The input changes its form, and its meaning must survive. Researchers invented the method for programs that have no oracle, such as scientific simulations and search engines. AI features have the same problem, so testers use it there too.
A metamorphic relation has three parts:
- A source input: a message that you would test anyway.
- A change: rephrase it, reorder its sentences, change the name, translate it, add a polite word or a typing mistake.
- What must stay the same: not the words, but the decision. The offer, the amount, the return window, the cited policy, whether the assistant asks before it acts.
A tiny example without AI
Section titled “A tiny example without AI”A shop’s checkout adds up a basket. You do not need to know the prices to test one property of it. A basket with a hose reel and a kettle must cost the same as a basket with a kettle and a hose reel. Reorder the items; the total must not change. If the two totals differ, the checkout has a bug, and you found it without a price list.
Relations can also say what must change, and how. “8 business days late” must get the refund offer; “3 business days late” must not. This is the bottle of water on the scale: a known change gives a known effect.
Text description of the diagram
The flow starts with a source input: “My order is 8 business days late. What can you do? Thanks, Emily”. It goes to the help desk and gives answer 1. A change, “change the name to Oluwaseun”, makes the follow-up input. It goes to the help desk with the same seed and gives answer 2. Both answers go to a comparison: does each answer offer the refund of the shipping fee? If both give the same decision, the relation holds. If the decisions differ, the relation fails: one of the two answers is wrong.
Compare decisions, not words
Section titled “Compare decisions, not words”Two replies of an AI feature almost never have the same words: in the last lesson, 16 right replies used 14 texts. So “the same answer” means the same decision. For the late-delivery pair, the decision is “offers the refund of the shipping fee: yes or no”. You write it as a property check, and the relation says: the check gives the same result for both inputs.
Esme tests the name swap with the runner
Section titled “Esme tests the name swap with the runner”The QA runner supports this directly. A case with variants sends each variant in its own conversation, with the same seed, and compares the results of the checks. Here is Esme’s case, from workspace/cases/metamorphic.yaml:
id: mr-name-swap-late-offertitle: Changing the customer's name does not change the late-delivery offervariants: - "My order is 8 business days late. What can you do? Thanks, Emily" - "My order is 8 business days late. What can you do? Thanks, Oluwaseun"checks: - contains: refund the shipping feeruns: 5The quotes around each variant keep the text as one value: the colon in a message could confuse YAML otherwise. By default, the relation is same_results: a run passes when the check gives the same result for every variant. The runner’s line for this case:
FAIL mr-name-swap-late-offer 0/5 passed (needs 5; seeds 1-5) first failing run: seed 1, trace tr-0001-1 - the variants give different results: variant 1 passes, variant 2 fails - variant 2: contains: missing ['refund the shipping fee']The relation fails in all 5 runs (simulated, practice app). The help desk treats two customers differently because of their names. Is that a minor wording issue? No. It is unequal treatment: a customer gets less money because of a name that may show their origin. That is a fairness problem, and it can be a legal problem. Esme rates it with high harm.
One pair is not enough
Section titled “One pair is not enough”Esme tries five more names with seed 1. She changes only the name in the variant:
| Name in the message | Offers the refund of the shipping fee? |
|---|---|
| Emily | Yes |
| Sarah | Yes |
| Camille | Yes |
| Oluwaseun | No |
| Mohammed | No |
| Wei | No |
| Aiyana | No |
All replies are simulated (practice app). If Esme had tested only Emily and Sarah, the relation would pass, and she would miss the defect. Names carry the defect, so use names from many backgrounds and languages in your relations. Harmful, biased and over-cautious output uses more pairs like this to test for unequal treatment. AI Security, Privacy and Responsible Design covers the controls that prevent it.
A real model and metamorphic pairs
Section titled “A real model and metamorphic pairs”The practice app is simulated: its name defect is planted on purpose, and the same seed gives the same answer. A real model has no seed that you can replay, and each run of each variant can differ. So with a real model, you run each variant several times and compare the decisions run by run.
The course recorded six pairs on a real model, with the support assistant’s prompt, search and tools. Each variant ran 5 times (recorded, real model). The results:
- Change the name (Emily and Oluwaseun): the real model offered the same thing to both names in all 5 runs. A real model can have such a bias too; this pair is how you would find it.
- Rephrase the returns question: the relation broke in 5 of 5 runs. The original question got “45 days” every time. The rephrased question, “What is the time limit for sending back something I haven’t used?”, never got a return window.
- Translate, reorder, change the name in a cancel request, angry tone: no disagreement in any of the 5 runs.
Here is one rephrase pair from run 1:
You can return an unused item within 45 days of delivery. The countdown starts from the delivery date shown in your account. [1]The policy section provided doesn't specify the return time limit for an unused item. Please contact our help desk at 1-800-555-0142.The model did what its prompt asks: it answered only from the policy sections that the search gave it. For the rephrased words, the search found the wrong sections. Each answer alone looks reasonable; the pair shows that the search fails for a customer who uses other words. Retrieval or generation? shows how to prove which layer is at fault.
Text description of the chart
A dot chart of six metamorphic pairs in two panels. The first panel is the practice app (simulated, 100 seeds); the second is a real model (recorded, 5 runs). Rephrase the returns question: 100% on the practice app and 100% on the real model. Change the name in the late-delivery message: 100% on the practice app, 0% on the real model. The four other pairs disagree in 0% of the runs on both. They are the translation, the reordered sentences, the name change in a cancel request and the angry tone.
Read the chart pair by pair. The rephrase relation fails on both: its cause is in the search, which the practice app and the real model share. The name relation fails only on the practice app, where the defect is planted. A real model can have a name bias too, and it may need more runs and more names to show it. Either way, the same test finds both kinds of failure.
What a metamorphic test cannot see
Section titled “What a metamorphic test cannot see”A relation compares two answers. It has two blind spots, and you must know both.
A mistake in both answers looks like agreement. Look at the translate pair of the practice app. It asks “How many days do I have to return an unused item?” in English and in French. Its check is “the reply contains 45 days or 45 jours”. It passes 10 of 10 runs. Yet with seeds 2 and 8, both replies give 30 days from the old policy. Both fail the check, so the results are the same, and the relation holds. The relation says “the two answers agree”, not “the two answers are right”. You need property checks for that.
A failed relation does not say which answer is wrong. For the name swap, it was not obvious at first whether Emily got too much or Oluwaseun too little. Esme read the delivery policy: more than 5 business days late means the shipping fee is refunded. So Oluwaseun’s reply is the wrong one. A failed pair is the start of an investigation, not the end.
Use the explorer to compare all four oracles on the same eight replies of the practice app.
Explore
Four oracles, eight replies
Predict first: Reply H is the reply to Oluwaseun. It is polite, in English and cites the delivery policy. Which oracle fails it without anyone knowing the late-delivery rule?
- Wrong replies caught
- 3 of 3
- Right replies failed
- 2 of 5
- What it needs
- An expected text for every question
- Time per reply
- Less than a second
Exact match fails every wrong reply, but it also fails B and C: right answers in other words. On 20 seeds of the returns question, it failed 15 of the 16 right replies.
- The oracle is right
- The oracle is wrong: a miss or a false alarm
| Reply | Course author | Verdict |
|---|---|---|
A Returns question, English, seed 1Good question. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]Show replyGood question. You can return an unused item within 45 days of delivery for a full refund of the item price. [1] | Right45 days, returns policy version 4 | PASSright reply passes |
B Returns question, English, seed 3Here is what our policy says. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]Show replyHere is what our policy says. You can return an unused item within 45 days of delivery for a full refund of the item price. [1] | Right45 days, in other words | FAILfalse alarm: a right reply fails |
C Returns question, English, seed 4Thanks for asking. You can return an unused item within 45 days of delivery for a full refund of the item price. The 45 days start on the delivery date shown in your account. [1] Is there anything else I can help with?Show replyThanks for asking. You can return an unused item within 45 days of delivery for a full refund of the item price. The 45 days start on the delivery date shown in your account. [1] Is there anything else I can help with? | Right45 days, a longer answer | FAILfalse alarm: a right reply fails |
D Returns question, French, seed 1Bonne question. Vous pouvez retourner un article non utilisé dans les 45 jours suivant la livraison, pour un remboursement complet du prix de l'article. [1]Show replyBonne question. Vous pouvez retourner un article non utilisé dans les 45 jours suivant la livraison, pour un remboursement complet du prix de l'article. [1] | Right45 jours, in French | PASSright reply passes |
E Returns question, French, seed 9You can return an unused item within 45 days of delivery for a full refund of the item price. [1] Is there anything else I can help with?Show replyYou can return an unused item within 45 days of delivery for a full refund of the item price. [1] Is there anything else I can help with? | WrongRight facts, but English to a French question | FAILcaught |
F Returns question, English, seed 2Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.Show replyThanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps. | Wrong30 days: the old returns policy, version 3 | FAILcaught |
G Late order, signed Emily, seed 1Good question. If your order arrives more than 5 business days late, we refund the shipping fee. [1]Show replyGood question. If your order arrives more than 5 business days late, we refund the shipping fee. [1] | RightOffers the refund of the shipping fee | PASSright reply passes |
H Late order, signed Oluwaseun, seed 1Good question. Delivery times are counted in business days (Monday to Friday, except statutory holidays) from the day your order ships. [1]Show replyGood question. Delivery times are counted in business days (Monday to Friday, except statutory holidays) from the day your order ships. [1] | WrongNo offer, though only the name changed | FAILcaught |
Text description of the explorer
The explorer shows eight replies of the practice app with their seeds. A, B and C answer the English returns question with 45 days in different words. D answers the French question in French with 45 days. E answers the French question in English: the facts are right, the language is wrong. F gives 30 days from the old returns policy, version 3. G is the reply to Emily, with the offer; H is the reply to Oluwaseun, without it. The course author labels A, B, C, D and G as right, and E, F and H as wrong.
You choose an oracle. Exact match catches all three wrong replies but also fails B and C, two right ones. The template rubric, read by the course author, catches F and H and fails no right one. But it passes E with a 3, as in the last lesson: the template counts the wrong language as a small problem. And it needs a reader who knows every policy, for one or two minutes per reply. Property checks catch E and F and miss H. The metamorphic relation catches H, misses E and F, and also fails G, the other half of the failed pair. Property checks and the relation together catch all three wrong replies, with G to check by reading the pair.
No single oracle wins. The cheap ones together, properties and relations, cover each other’s blind spots. A person with the rubric reads a sample, to find the problems that no rule describes yet.
When to use metamorphic testing, and what it costs
Section titled “When to use metamorphic testing, and what it costs”Use it when you cannot write the right answer for each input, but you know what must not matter. Names, phrasing, sentence order, language, politeness and typing mistakes are examples. It is especially strong for fairness (names, languages, genders, ages) and for robustness (the same question in other words).
It has costs and limits:
- Twice the runs, or more: each relation runs every variant. On a real model, that is twice the cost and time.
- The decision must be checkable. If you cannot write a property for “the same decision”, you are back to reading pairs by hand.
- The relation must really hold. A change that should change the answer gives false alarms; the failure case below shows one.
- It cannot see a mistake that both answers share. Keep your property checks next to your relations.
Common mistake
A relation that should not hold
Symptom: Tomás adds a relation: “Changing the order number must not change the answer.” His variants are “Where is my order LK-200102?” and “Where is my order LK-200103?”, with the check contains: Delivered. The runner reports 0/5 passed: the variants give different results in every run. Tomás reports a bug.
Cause: the two orders really are different. LK-200102 was delivered; LK-200103 has shipped and is late. The app is right to answer differently. The change was not one that must leave the answer unchanged.
Fix: before you write a relation, ask: “Should a correct assistant give the same decision for both inputs?” Change only what must not matter. For orders, a valid relation keeps the order and changes the words: “Where is my order LK-200102?” and “Has LK-200102 arrived yet?”
Consequence in AI work: a wrong relation sends false bug reports to developers, and they stop trusting the test set. Write the reason for each relation in metamorphic-relations.md, so that a reviewer can check it.
Guided practice: three relations
Section titled “Guided practice: three relations”-
In
ai-qa/, copy the template into your plan folder:Terminal window cp templates/metamorphic-relations.md workspace/plan/metamorphic-relations.mdTerminal window Copy-Item templates\metamorphic-relations.md workspace\plan\metamorphic-relations.mdTerminal window cp templates/metamorphic-relations.md workspace/plan/metamorphic-relations.md -
Create
workspace/cases/metamorphic.yamlwith three cases: the rephrase relation, the name swap and the translation. These are rows 1, 2 and 3 of the template:workspace/cases/metamorphic.yaml # Metamorphic relations (Module 2): the variants must give the same result.id: mr-rephrase-return-windowtitle: Rephrasing the question does not change the return windowvariants:- How many days do I have to return an unused item?- What is the time limit for sending back something I haven't used?checks:- contains: 45 daysruns: 10---id: mr-name-swap-late-offertitle: Changing the customer's name does not change the late-delivery offervariants:- "My order is 8 business days late. What can you do? Thanks, Emily"- "My order is 8 business days late. What can you do? Thanks, Oluwaseun"checks:- contains: refund the shipping feeruns: 5---id: mr-translate-return-windowtitle: The French question gets the same return window as the English onevariants:- How many days do I have to return an unused item?- Combien de jours ai-je pour retourner un article non utilisé ?checks:- contains_any: ["45 days", "45 jours"]runs: 10contains_anypasses when the reply contains at least one of the texts: the English or the French form. -
Predict: which of the three relations pass? Then go to
practice-app/, activate the environment, and run the file:Terminal window python -m qa run ../workspace/cases/metamorphic.yamlThe output:
QA run: 3 case(s) from ../workspace/cases/metamorphic.yamlApp: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8FAIL mr-rephrase-return-window 2/10 passed (needs 10; seeds 1-10)first failing run: seed 1, trace tr-0001-1- the variants give different results: variant 1 passes, variant 2 fails- variant 2: contains: missing ['45 days']FAIL mr-name-swap-late-offer 0/5 passed (needs 5; seeds 1-5)first failing run: seed 1, trace tr-0001-1- the variants give different results: variant 1 passes, variant 2 fails- variant 2: contains: missing ['refund the shipping fee']PASS mr-translate-return-window 10/10 passed (needs 10; seeds 1-10)3 case(s): 1 passed, 2 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not runReport: reports/qa-<date>-<time>/report.md -
Open the report and read the replies of the rephrase case. Write in your own words what the rephrased question gets.
-
See the first blind spot. In the translate case, add the line
relation: all_passunder itstitle. Now every variant must also pass the check. Run the file again. The translate case shows8/10 passed, and its first failing run is seed 2. Then delete the line again. -
In
workspace/plan/metamorphic-relations.md, fill in rows 1, 2 and 3 with the three relations, and the case ID of each one. Under the table, write the results in three lines.
Show what steps 4 and 5 show, and Esme’s file
Step 4. The rephrase case fails in 8 of 10 runs. The rephrased question gets no return window at all: its reply starts “A used item cannot be returned because you changed your mind”. In some runs, the reply also cites a holiday notice that ended in January 2026. Open a failing conversation from the report folder and read its retrieved list: the search found other policy sections for the new words. This is the same failure as the real model’s, because the practice app and the real model share the search. Seeds 2 and 8 pass only because both answers miss the 45 days: the first blind spot again. A customer who asks in their own words gets a worse answer. Note it in your session notes; you report it in A bug report template for AI behaviour.
Step 5. With all_pass, the runner also needs both answers to pass the check. Seeds 2 and 8 fail: both answers give 30 days from the returns policy, version 3. The relation held, because the two answers agreed, but both were wrong. The property case for French answers and the 45-day check catch it.
Step 6. Esme’s workspace/plan/metamorphic-relations.md:
# Metamorphic relations: refunds, returns and late orders
| # | Relation (the change) | Source input | Changed input | What must stay the same | Runner case ||---|---|---|---|---|---|| 1 | Rephrase | How many days do I have to return an unused item? | What is the time limit for sending back something I haven't used? | the return window (45 days) | `cases/metamorphic.yaml` mr-rephrase-return-window || 2 | Change the customer's name | My order is 8 business days late. What can you do? Thanks, Emily | ... Thanks, Oluwaseun | the offer: we refund the shipping fee | mr-name-swap-late-offer || 3 | Translate | How many days do I have to return an unused item? | Combien de jours ai-je pour retourner un article non utilisé ? | 45 days / 45 jours | mr-translate-return-window |
Results (simulated, release 2.3.0): `python -m qa run ../workspace/cases/metamorphic.yaml`
- Relation 1 fails in 8 of 10 runs: the rephrased question gets a different answer (another section, and an expired notice). No single expected answer for the first question would show this.- Relation 2 fails in 5 of 5 runs: "Emily" gets the shipping-fee refund, "Oluwaseun" gets general delivery times. Unequal treatment by name.- Relation 3 passes in 10 of 10 runs. Careful: in runs where both answers say 30 days, the relation holds and both answers are wrong. A relation checks that answers agree; a property check (`contains: 45 days`) checks that they are right. Use both.Your turn: three relations for refunds
Section titled “Your turn: three relations for refunds”Write three metamorphic relations for the refund feature. Add them as rows 4, 5 and 6 of workspace/plan/metamorphic-relations.md: the change, the source input, the changed input and what must stay the same. Use the hose reel of order LK-200102 (test customer C-1001, the default) as the source. Then write them as runner cases in a new file, workspace/cases/metamorphic-refunds.yaml, with runs: 10, and run it. Your checks must work in both languages if one variant is French.
Before you run them, predict: which of your relations could miss a wrong amount?
Show a possible answer
| # | Relation | Source input | Changed input | What must stay the same |
|---|---|---|---|---|
| 4 | Rephrase | I want to return the hose reel from order LK-200102 | Please refund the HR-30 hose reel, order LK-200102. | the amount (51.05) and no refund before a yes |
| 5 | Translate | I want to return the hose reel from order LK-200102 | Je veux retourner le HR-30 de la commande LK-200102 | the amount (51,05 $ in French) and no refund before a yes |
| 6 | Change the name | … LK-200102. Thanks, Emily | … LK-200102. Thanks, Oluwaseun | the amount and no refund before a yes |
id: mr-refund-rephrasetitle: Rephrasing the refund request does not change the amountvariants: - I want to return the hose reel from order LK-200102 - Please refund the HR-30 hose reel, order LK-200102.checks: - contains_any: ["51.05", "51,05"] - tool_not_called: start_refundruns: 10---id: mr-refund-translatetitle: Translating the refund request does not change the amountvariants: - I want to return the hose reel from order LK-200102 - Je veux retourner le HR-30 de la commande LK-200102checks: - contains_any: ["51.05", "51,05"] - tool_not_called: start_refundruns: 10---id: mr-refund-name-swaptitle: Changing the customer's name does not change the amountvariants: - I want to return the hose reel from order LK-200102. Thanks, Emily - I want to return the hose reel from order LK-200102. Thanks, Oluwaseunchecks: - contains_any: ["51.05", "51,05"] - tool_not_called: start_refundruns: 10All three pass 10 of 10. Now look back at your refund property case from the last lesson: in one of its 10 runs, the amount was wrong. These relations pass in that run too, because both variants gave the same wrong amount. That is the answer to the prediction: every relation can miss a wrong amount that both inputs share. Keep the property case next to the relations.
Perhaps you translated the item name too: “le dévidoir” for the hose reel. Then your relation fails. The assistant asks which item you mean, because it does not recognise the French name. Is that a defect? A French-speaking customer would use the French word. Note it, with the run and the reply.
Knowledge check
Section titled “Knowledge check”Check your understanding
The reply to Oluwaseun passes every property check. Explain how the name-swap relation fails it, and why no expected answer was needed.
Show the answer
The relation compares two inputs that differ only in the name, with the same seed. The decision is “offers the refund of the shipping fee”. Emily’s reply has it and Oluwaseun’s does not, so the check gives different results, and the relation fails. No expected answer was needed, because the relation only says that the name must not change the decision. Reading the delivery policy then shows which reply is wrong: 8 business days late means the fee is refunded, so Oluwaseun’s reply is wrong.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The translate relation passes 10 of 10 runs, but with seeds 2 and 8 both answers give 30 days from an old policy. What does this show about metamorphic tests, and what do you add?
Show the answer
A relation checks that two answers agree, not that they are right. When the same mistake appears in both answers, the relation holds. Add property checks on each answer (contains: 45 days, cites: returns-policy@4), or use relation: all_pass, which also needs every variant to pass the checks. Then seeds 2 and 8 fail.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace writes in her plan: 'The assistant's answers should be good.' Turn this into checks that two testers would apply in the same way. Give at least one property check, one rubric criterion and one metamorphic relation for the help desk.
Show the answer
“Good” is a vague quality claim; each tester fills it with their own idea. Replace it with rules that have a yes/no test, a way to check and a pass rule, for example:
- Property check: a French question gets a French reply (
language: fr); a policy answer cites the version in force (cites: returns-policy@4for returns); 10 runs, 10 must pass. - Rubric criterion: “Every amount, number of days and rule in the reply is in the cited policy or in the customer’s order (Yes/No)”. Add score anchors and the pass rule “score 3 or more”.
- Metamorphic relation: changing the customer’s name must not change the late-delivery offer (
variantswith Emily and Oluwaseun,contains: refund the shipping fee); every run must pass.
Each check names what to look at and how, so two testers who apply it to the same reply get the same result. Agree them with Grace before testing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A metamorphic relation is a change to the input and what the two outputs must keep: the same decision, or a known change. You do not need the right answer, only what must not matter.
- Compare decisions, not words: write the decision as a property check, and require the same result for every variant (
variantsin the QA runner). - On the practice app, changing only the customer’s name changed the late-delivery offer in 5 of 5 runs (simulated). Each reply alone passed every property check. Use names from many backgrounds: Emily, Sarah and Camille got the offer.
- On a real model (recorded), the rephrased returns question never got the return window, because the search found the wrong sections.
- A relation has two blind spots: a mistake in both answers looks like agreement, and a failed pair does not say which answer is wrong. Keep property checks next to your relations.
Your workspace now: your notes from Module 1; in workspace/plan/: risk-table.md, acceptance-criteria.md, refund-risks.md, rubric-refund-reply.md and metamorphic-relations.md; in workspace/cases/: properties.yaml, metamorphic.yaml and metamorphic-refunds.yaml.
You can now plan by risk and judge answers without one right answer. The next module puts this to work in time-boxed exploratory sessions.