Plan the evidence

What you will be able to do

Decide now what will prove each claim that you plan to make, and lock the final test questions before you tune anything. A claim without evidence is not made, and a test that you tuned on proves nothing.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• Your brief and your acceptance criteria, from Problem and audience and Acceptance criteria.
• A development set to tune on and a frozen holdout set to decide with, from Avoid contamination in Evaluating and Testing AI Systems.
• Grace’s test questions and their kinds, from Corpus and questions in the RAG course.

What you need on your computer

• The project folder doc-assistant with its brief and criteria committed. No new packages: the holdout script uses only the Python standard library.
• The labs repository from Set up the capstone repository, up to date (git -C nextia-learning-labs pull).

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) commands are written, not tested.

Buying a used car

A seller shows you a used car and says three things: “It was never in an accident. The brakes are new. It has only 40,000 km.”

A careful buyer asks for paper: an inspection report, the garage’s invoice for the brakes, and the service book. A good seller has this folder ready before the first buyer comes. And a good seller never says “it is a perfect car”, because no paper can prove that.

Your capstone is the car, and the reviewer of your portfolio is the buyer.

The problem: claims that nothing can prove

Sam writes a first draft of the README’s opening line. It is a made-up draft, but a very common one:

“An accurate, secure, production-ready AI assistant that saves shop staff hours every day.”

Amira, who reviews Sam’s code, asks for each word: “How will you show that?” Accurate: on which questions, how accurate? Secure: against what, tested how? Production-ready: it runs on one laptop, with made-up documents. Saves hours: nobody uses it for real work, so nobody can measure time saved. Two of these claims can be proven if Sam plans the evidence now. Two can never be proven in this project. Unsorted, the README makes all four, and a careful reviewer believes none.

The idea: an evidence plan

An evidence plan lists every claim that you plan to make in your README and your demo. For each claim, it names the evidence that a reviewer can open or run: a file, a test, a script’s run or a screenshot. It also names the module in which that evidence will exist. A last section lists what you will not claim, because your evidence cannot support it.

The rule is short: a claim without evidence is not made. You met the same idea in Map goals to evidence, for one evaluation. Here it covers the whole product.

A tiny example: one claim, three pieces of evidence

Take one claim from Sam’s brief: “A shop never sees another shop’s data.” What would convince a doubtful reviewer?

Evidence	What the reviewer does with it	Made in
tests/test_access.py	runs it: every wrong-shop request fails	Module 4
row-level security in migration 002	reads it: the database itself refuses other shops’ rows	Module 4
python -m scripts.isolation with “0 leaks”	runs it: hundreds of probes with the other shop’s IDs	Module 5

The three support each other. The tests show that the code checks. The migration shows that the database checks too, even if the code has a bug. The isolation run measures the result from outside. None of them exists today. The plan promises them and says when: Module 1 decides what to prove, and later modules make the proof.

Sam’s evidence plan

Sam’s plan has nine claims. The map below shows them against the modules that make their evidence. It also shows the scope from the brief at the top, because every claim must stay inside it.

Sam's scope and evidence map. Each claim has its evidence in the modules that make it. The held-out questions are locked now and opened only in Module 5.

Text description of the diagram

At the top is the scope from the brief, with an arrow down to a grid: every claim stays inside the scope. The grid’s rows are Sam’s nine claims; its columns are Modules 3 to 6. A green cell names the evidence that a module makes for a claim. Module 3 makes the first tests and a screenshot. Module 4 makes the access, failure and deletion tests and row-level security. Module 5 makes the most: the held-out results, the isolation checks and the cost per question. Module 6 makes the clean-computer run, CI, the smoke test and the Compose demo. Under the grid, a locked box holds the 41 held-out questions, locked in Module 1 and opened only in Module 5.

The plan ends with what Sam will not claim:

docs/evidence-plan.md (part)

## What I will not claim

- "Production-ready": it ran on one laptop, with synthetic documents and a practice identity provider.
- "Secure": it passed the checks listed here; nobody tested it as an attacker would.
- Anything about real shops, real users or time saved: nobody used it for real work.

These are Amira’s four words, sorted. “Accurate” became measured claims, and “secure” became narrow claims that tests can prove. “Production-ready” and “saves hours” are gone, with the reason written down. A reviewer who reads this section trusts the rest more.

Explore the plan, module by module

Move through the modules and watch which evidence exists at the end of each one. At Module 7, each claim also shows how Sam finally wrote it in the portfolio checklist.

Explore

Sam's evidence plan, module by module

Claims with all their evidence

0 of 9

Evidence that exists

0 of 16

Claims with no evidence yet

9 of 9

End of Module 1: 0 of 9 claims have all their evidence, 0 have part of it, and 9 have none. 0 of 16 pieces of evidence exist. The plan is a promise, not proof: a reviewer can open nothing yet. Module 1 makes no claim true; it says what will prove each one.

• Evidence exists
• Evidence planned

1. 1. A new user can upload a document and get a cited answer No evidence yet

   • the README's quick start, run on a clean computer planned for Module 6
   • a screenshot of the page planned for Module 3
   • tests/test_api.py planned for Module 3

2. 2. Answers cite their sources, and the citations are real No evidence yet

   • citation validity in the evaluation report planned for Module 5
   • tests/test_answers_unit.py planned for Module 3

3. 3. A shop never sees another shop's data No evidence yet

   • python -m scripts.isolation (0 leaks) planned for Module 5
   • tests/test_access.py planned for Module 4
   • row-level security in migration 002 planned for Module 4

4. 4. It says "no answer" instead of guessing No evidence yet

   • abstention on the held-out unanswerable and other-shop questions planned for Module 5

5. 5. Bad files, repeated uploads and a slow model are handled No evidence yet

   • python -m scripts.failure_cases planned for Module 4
   • tests/test_failures.py planned for Module 4

6. 6. Deleting a document removes everything No evidence yet

   • tests/test_deletion.py planned for Module 4

7. 7. One change improved it, with evidence No evidence yet

   • the evaluation report: baseline against candidate on the held-out questions planned for Module 5

8. 8. It runs from a clean environment No evidence yet

   • the CI workflow and the smoke test planned for Module 6
   • the Compose demo planned for Module 6

9. 9. It costs under US$0.001 per question No evidence yet

   • the cost per question in the evaluation report (recorded token counts × the price list) planned for Module 5

The nine claims of Sam's evidence plan, with each piece of evidence and the module that makes it. The slider shows the end of each module.

Text description of the explorer

The explorer lists Sam’s nine claims and their 16 pieces of evidence, each with the module that makes it. A slider chooses the end of a module, from 1 to 7. After Modules 1 and 2, no evidence exists. After Module 3, 3 pieces exist and no claim is complete. After Module 4, 8 pieces exist and 2 claims are complete; after Module 5, 13 and 7. After Module 6, all 16 pieces and all 9 claims are complete. Module 7 adds no evidence. It shows the claims as Sam finally wrote them, no stronger than the evidence. For example, “One change improved it” becomes “One change was tested with evidence”: Sam makes this change in Module 5, when the held-out results arrive.

The answer to the prediction is none: at the end of Module 1, a reviewer can open nothing. That is the purpose of a plan, not a weakness. Look at Module 7 too. The evidence does not only prove claims, it changes them: two merge, and several get more careful words. Sam plans the evidence now, and lets it write the final claims. When a claim changes, Sam changes it in docs/evidence-plan.md too, with the reason.

There is one claim that is not in the plan, and its evidence exists from this lesson: “The held-out questions were reserved before tuning.” The rest of this lesson makes that evidence.

Lock the final exam now

You know this idea from Avoid contamination in the evaluation course, and from the sealed exam paper in Train, validation and test sets. A held-out set is a set of test questions that you keep apart. Nobody reads its answers while the work goes on, and it decides the result once, at the end. The tuning set is open: you read its failures and change the system as often as you like.

Why lock it now, before any code? From Module 3, Sam tunes the prompt, the search mode and the number of passages. When a change helps on some questions and Sam keeps it, part of the gain can be luck on those exact questions. If the final questions are among them, the final score is too good, and nothing says so. The only safe moment to separate the sets is before the first change: today.

Sam’s question bank

The supplied question bank, evaluation/questions.jsonl, has 103 questions. Larkfield has 69: Grace’s 67 questions from the RAG course, and two that ask about the other shop. Bramble Books has 34, written for this course. No model wrote a question. Every question names the passages that answer it, and each passage was checked against the documents. Here is one question, a tuning question, with its fields:

evaluation/questions.jsonl (one line, formatted)

{"id": "B01", "tenant": "bramble", "question": "How long does delivery to Ontario take?",
 "language": "en", "kind": "answerable", "as_of": "2026-10-11", "abstain": false,
 "reference_answer": "4 to 8 days: delivery outside British Columbia takes 4 to 8 days (after shipping within 2 business days).",
 "keys": ["4 (to|-|–) ?8"], "must_not": [],
 "relevant": [{"doc_id": "delivery-policy", "version": "1", "section": "Shipping", "access": "public"}],
 "notes": "", "source": "written for this course"}

as_of is the date of the question, so the right version of a policy applies. abstain says whether the right answer is “the documents do not say”. A correct answer matches the keys patterns and contains none of the must_not patterns. relevant names the passages that answer it, and kind names the difficulty that it tests. There are nine kinds:

Kind	What the question tests	A tuning example
answerable	the documents answer it directly	“How long does delivery to Ontario take?”
version	two versions of a policy; the date decides	a novel bought on 10 October 2026: until when can it be returned?
table	the answer is only in a table	“Can a community group use the reading room for free?”
exact-code	a product or part code	“Which pump kit fits the PW-2200?”
contradiction	two documents disagree; a rule decides	the FAQ’s 5 business days against the policy’s 7 to 10
staff-only	the answer is in a staff-only document	the refund amount that staff may approve alone
injection-bait	a document hides an instruction for the model	“Can a customer cancel a pre-order for a full refund?”
unanswerable	the documents do not answer it	“Is there parking near the shop?”
other-shop	it asks about the other shop	the PW-2200’s warranty, asked at Bramble Books

This lesson shows only tuning questions as examples. The held-out questions stay unread until Module 5.

A tiny example: how the split works

The script scripts/holdout.py splits the bank in a fixed, fair way. It puts every question into a group by shop and kind. In each group, it shuffles the questions with a fixed seed (27), so the shuffle is the same on every computer. Then it keeps 40% of the group, rounded, as held out. If a group has two or more questions, both sets get at least one. Three groups, with every value:

Group	Questions	40%	Held out	Tuning
Bramble Books, version	3 (B13, B14, B15)	1.2	1 (B15)	2
Bramble Books, other-shop	2 (B33, B34)	0.8	1 (B34)	1
Larkfield, exact-code	11	4.4	4	7

Over all 17 groups, 41 questions are held out and 62 are for tuning. Because every group is split, the held-out set tests every kind of difficulty in both shops:

Kind	Held out	Tuning
answerable	12	19
contradiction	3	3
exact-code	4	7
injection-bait	2	4
other-shop	2	2
staff-only	5	6
table	5	7
unanswerable	4	7
version	4	7
all	41	62

A random split of all 103 at once could put both other-shop questions of a shop in one set by chance. Splitting by group prevents this.

The lock: a fingerprint of the file

After the split, the script writes heldout.lock. It holds the SHA-256 of heldout.jsonl: a fingerprint of 64 characters, computed from every byte of the file. If one character of the file changes, the fingerprint changes completely. So a later check can prove that the file is the same file, without anyone reading it.

Reserve once, commit at once. The held-out file and its fingerprint are committed before any tuning; the tuning set is open for everyday work.

Text description of the diagram

The question bank of 103 questions goes into the reserve command, which keeps 40% of each shop and kind with seed 27. It writes the held-out file (41 questions, locked), the lock with its SHA-256 fingerprint, and the tuning file (62 questions). The held-out file and the lock are committed before any tuning and used only in Module 5, once per compared version. The tuning file is used in Modules 3 to 5, as often as needed.

The commit matters as much as the lock. The lock proves that the file did not change; the commit’s place in the history proves when it was made, before any result existed.

Reserve and lock the questions

On the standard track, you copy the supplied question bank and the holdout script from the snapshot end-of-m01. On the own-idea track, see the note after the steps.

1. Copy the question bank, the holdout script and its tests into your project:

   macOS / Linux

   mkdir -p evaluation
   cp ../nextia-learning-labs/C27/snapshots/end-of-m01/evaluation/questions.jsonl evaluation/
   cp ../nextia-learning-labs/C27/snapshots/end-of-m01/scripts/holdout.py scripts/
   cp ../nextia-learning-labs/C27/snapshots/end-of-m01/tests/test_holdout.py tests/

   Windows (PowerShell)

   New-Item -ItemType Directory -Force evaluation
   Copy-Item ..\nextia-learning-labs\C27\snapshots\end-of-m01\evaluation\questions.jsonl evaluation\
   Copy-Item ..\nextia-learning-labs\C27\snapshots\end-of-m01\scripts\holdout.py scripts\
   Copy-Item ..\nextia-learning-labs\C27\snapshots\end-of-m01\tests\test_holdout.py tests\

   Copy only questions.jsonl from evaluation/. You make the other three files yourself in the next step.

2. Reserve the held-out questions:

   python -m scripts.holdout reserve

   Held out: 41 questions (bramble 14, larkfield 27). Kinds: answerable 12, contradiction 3, exact-code 4, injection-bait 2, other-shop 2, staff-only 5, table 5, unanswerable 4, version 4.
   Tuning set: 62 questions (bramble 20, larkfield 42).
   Wrote evaluation/heldout.jsonl, dev.jsonl and heldout.lock. Commit them now, before any tuning.

3. Look at the lock. Do not open heldout.jsonl.

   cat evaluation/heldout.lock

   {
    "sha256": "e8eda11340d636d700ee07c66a67fb407f94dd20f2367accfe7c83fec2c80757",
    "seed": 27,
    "share": 0.4,
    "questions": 41
   }

   On the standard track, your fingerprint is exactly this one. The same bank, the same seed and the same share give the same split on every computer. So your held-out set is Sam’s, byte for byte.

4. Commit at once, as the script says:

   git add evaluation scripts/holdout.py tests/test_holdout.py
   git commit -m "Reserve the held-out questions before any tuning"

5. Try to reserve again:

   python -m scripts.holdout reserve

   A held-out set is already reserved. Reserving again after you have seen results would let tuning leak into it. Use --force only if you have not tuned anything yet.

   The script refuses, and ends with exit code 1.

6. Check the lock. This is the command that every later module runs first:

   python -m scripts.holdout check

   Held-out set: 41 questions (bramble 14, larkfield 27). SHA-256 matches the lock (e8eda11340d6...).
   Tuning set (dev.jsonl): 62 questions (bramble 20, larkfield 42).

7. Run the tests:

   python -m pytest

   ..............                                                           [100%]
   14 passed in 0.12s

   The four new tests check four things. The held-out file matches its lock. No question is in both sets or lost. The split is repeatable. A changed file is refused.

On the own-idea track, write your own evaluation/questions.jsonl first, one question per line, with the same fields as B01. Write at least 30 questions, with at least two of each kind that you use, so that both sets get one. Use one tenant value if your product has no tenants. Keep the kinds that fit your documents, and always include unanswerable questions. Then run steps 2 to 7. The first test of tests/test_holdout.py expects Sam’s counts: change its expected line to the line that your holdout check prints.

Common mistakes

Common mistake

A key that was fixed after a look

Symptom: in Module 5, a held-out answer is marked wrong, and its key looks too strict. Someone changes the key in heldout.jsonl to .*, which accepts any answer. The next check refuses to run:

heldout.jsonl changed after it was reserved. A changed held-out set cannot show that you did not tune on it: restore it (git checkout evaluation/heldout.jsonl).

The tests fail too: 1 failed, 13 passed, in test_the_committed_heldout_set_matches_its_lock.

Cause: one changed character changed the fingerprint. The change itself may even be right. But it was made after a look at a held-out result, so it is tuning on the held-out set.

Fix: restore the file with git checkout evaluation/heldout.jsonl, and the check passes again. If the key is really wrong, name it in the evaluation report as a known problem. Report the result as it is. Sam’s report does exactly this for one strict key in Module 5.

Common mistake

Reserving again with --force

Symptom: after a few days of tuning, someone dislikes the held-out questions and runs python -m scripts.holdout reserve --force with another seed.

Cause: --force exists only for the first day, before any tuning, for example after you added questions to the bank. After tuning, a new split can move questions that you tuned on into the held-out set.

Fix: never use --force after the lock’s first commit; the Git history would show it. Add later questions to the tuning set only, and say so in the report.

When to hold out, and what it costs

Hold out questions whenever a final number must be believed by someone else: a release decision, an evaluation report, a portfolio. Reserve them before the first change to the system, and commit the lock at once.

It costs you 41 questions that you cannot learn from. And 41 is a small set: one answer moves the result by about 2.4 points. So a small difference between two versions can be chance, as Uncertainty and slices shows. A final score that nobody can trust costs more.

Your project

1. Copy the evidence-plan template into docs/: cp docs/templates/evidence-plan.md docs/evidence-plan.md (on Windows, Copy-Item docs\templates\evidence-plan.md docs\evidence-plan.md).
2. List every claim that you want to make in your README and your demo. Use your acceptance criteria: most criteria become a claim.
3. For each claim, name at least one piece of evidence that a reviewer can open or run, and the module that makes it.
4. Write “What I will not claim”, with a reason for each line.
5. Reserve your held-out questions with python -m scripts.holdout reserve, and commit at once.
6. Commit the evidence plan.

Evidence you keep: docs/evidence-plan.md, and the commit that adds heldout.lock, which comes before any tuning.

Checklist

• Every claim has at least one piece of evidence and a module.
• “What I will not claim” has at least two lines, each with a reason.
• python -m scripts.holdout check prints “SHA-256 matches the lock”.
• The lock’s commit is earlier than any tuning.
• You did not open heldout.jsonl.
• python -m pytest prints 14 passed (on the own-idea track, after you changed the expected line).

Module practice: your brief and milestone plan

You saw the worked example: Sam’s brief, criteria and evidence plan, with every intermediate file. Now do the same for your product.

Independent task. Hand in your project brief and its milestone plan. The brief names one user, one decision, the document collection with its permission, and at least three non-goals. The milestone plan has at least four milestones, each with an observable “done when” and a week. One complete journey comes before the hard cases. Attach your acceptance criteria and evidence plan, and the commit of your lock.

Show a worked solution (an own-idea example made up for this course)

Ana coaches at a football club (the example of Problem and audience).

• User and decision: the volunteer coaches of the junior teams decide whether a match goes ahead in bad weather, and what to tell the parents.
• Documents: 4 documents, public on the club’s website, used with the club’s written permission. They are the weather and pitch policy, the rules for messages to parents, and the league’s rules for postponed matches.
• Non-goals: no messages sent to parents by the assistant; no match scheduling; no medical advice; no cloud deployment in the core.
• Milestones:
  • Week 1, design pack: the club secretary reviewed it.
  • Week 2, one journey: a coach uploads the policy and gets a cited answer to “the pitch is frozen at 8:00; can the 10:00 match go ahead?”
  • Week 3, safe and reliable: the wrong-club, missing-source, invalid-file and timeout cases pass.
  • Week 4, evaluated: the held-out report is written.
  • Week 5, shipped: a clean computer runs the demo.
• Questions: 32 questions with one tenant, and at least two of each kind that she uses. She reserves them with the same script and commits the lock before any tuning.

Module check: every feature serves a stated need

The completion check of this module: every feature that you plan must support a need that your brief states. Make a table with one row per planned feature, and name the line of the brief that it serves. A feature with no line is removed, or the brief changes first, with a reason.

Show Sam’s table

Planned feature	The need in Sam’s brief
Upload a document and see its status	staff answer from “the shop’s own documents”, so they must add them and know when they are in use
A short answer with a citation for every claim	“a short answer with a citation for every claim”
“The documents do not answer this”	“or a clear ‘the documents do not answer this’”
The cited passage, its version and dates	staff check the answer before they speak; the brief’s two mistakes were an old version and a contradiction
Delete a document	an out-of-date policy caused the 30-day mistake
Sign-in, two shops, roles	“A shop never sees the other shop’s documents, passages or answers”
A clear error when the model is slow	“in a few seconds”, while the customer waits
Usage limits per shop	the budget (C2), which Grace and Ines agreed

Tomás’s agent, a chat history and sending answers to customers have no row: they are non-goals. A dark mode has no row either, so it waits.

See it end to endCapstone on a public document collectionCase study

The case study reserves 25 of its 60 questions about the GOV.UK guides with the same command, and commits them before any answer. The lock’s fingerprint is the same as the author’s, so anyone can repeat the split.

See it end to endReview a weak capstoneCase study

The case study reviews a submission whose held-out lock still matches, because it was written again after tuning. Three failing questions were removed, and 36 of 41 became “94.7%”. One git log on the held-out files shows the order of the commits.

Knowledge check

Check your understanding

Why must the held-out questions be reserved and committed in Module 1, before any code, and not in Module 5, just before the evaluation?

Show the answer

From Module 3, Sam tunes the prompt, the search and other settings on the questions that Sam reads. Every change that is kept because it helped on some questions picks up a little luck on those questions. If the final questions were among them, the final score would be too good, and nothing in the score would show it. Before any change is the only moment when no question has influenced the system yet. The commit records that moment: its place in the Git history proves that the split existed before any result.

Check your understanding

Predict: a learner on the standard track runs `python -m scripts.holdout reserve` on the supplied 103 questions. Will their held-out set be the same as Sam's? How can they check it without opening the file?

Show the answer

Yes. The split uses a fixed seed (27) and a fixed share (40%) on the same bank. So it gives the same 41 questions, in the same order, on every computer. To check it without opening the file, they compare the fingerprint in evaluation/heldout.lock with Sam’s: it starts with e8eda11340d6. python -m scripts.holdout check prints the first 12 characters. If one byte differed, the whole fingerprint would differ.

Check your understanding

Practical check: show your evidence plan and your lock. Name one claim with its evidence and module, one thing that you will not claim, and what `holdout check` and `git log` show.

Show the answer

A good answer names a claim with evidence that a reviewer can open or run. For example, “A shop never sees another shop’s data” has the access tests and row-level security in Module 4. It also has the isolation checks, with 0 leaks, in Module 5. It names a claim that the evidence cannot support, with the reason, for example “production-ready: one laptop, synthetic documents”. python -m scripts.holdout check prints “SHA-256 matches the lock”, and git log --oneline shows the commit that added heldout.lock before any later work.

Recap

• An evidence plan names, for every claim, the evidence that a reviewer can open or run, and the module that makes it. A claim without evidence is not made, and “What I will not claim” lists the rest with reasons.
• Sam’s plan has 9 claims and 16 pieces of evidence. At the end of Module 1, none exists; at Module 6, all do. The evidence also changes the claims: in Module 7, Sam writes them no stronger than the evidence.
• The held-out set is locked before any tuning: 41 of the 103 questions (Larkfield 27, Bramble Books 14). The split by shop and kind, with seed 27, tests every difficulty. The 62 tuning questions are open.
• heldout.lock keeps the SHA-256 fingerprint e8eda11340d6…; python -m scripts.holdout check refuses a changed file, and the commit proves when the split was made. The tests print 14 passed.

Module 1 recap. You set up a known-good starter and chose a track. You bounded the project: one user, one decision, a narrow collection and written non-goals. You wrote acceptance criteria that can be checked and will not move. You planned the evidence for every claim and locked the final questions. Your project now has the files of the snapshot end-of-m01, apart from its README, and no product code exists yet. That is on purpose.

Next, you prepare the documents themselves: a data pack with revisions, contradictions and questions that it cannot answer.

Next: Synthetic data pack