Review a weak capstone

What you will be able to do

Review someone else’s capstone the way a careful employer would. Mark it against the course’s rubric, with a reason and a piece of evidence for every mark. Find the problems by running the course’s checks, not by reading the README. Then write a fix plan that the author can follow, in order of harm.

Case studyPortfolio pieceSynthetic data

Dataset

Theo's weak capstone (constructed) on the course's two-shop data pack (made for this case study)

Licence

Code MIT; data pack CC0 1.0

Time

About 3 hours

Runs in

VS Code, Python 3.12+, Git, Docker

Skills used

• Marking against a rubric
• Claims and evidence
• Isolation checks
• Held-out lock and Git history
• Citation checks
• Secret scan of the history
• Seeded problems
• A fix plan in order of harm

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

Everything in this submission is constructed. Theo Marsh is not a real person. The course team built Theo’s repository from the course’s finished doc-assistant, and planted problems in it on purpose. The secrets in its history were made up for this page and never worked anywhere. The starter has Theo’s repository and the reviewer’s kit. The finished download adds the solution review, the fix plan and Theo’s resubmission.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses the whole course, from the other side of the table. It leans most on:

• The marking rubric of the final assignment: demonstrated, partially demonstrated or not yet demonstrated, always with a reason.
• The held-out lock, from Plan the evidence, and the isolation checks, from Run held-out tests.
• The citation checks, from Integrate retrieval, and the build pipeline, from Build pipeline.
• Claims with evidence, from Repository handoff and Demo narrative.
• What a committed secret means, from Keep the repository clean in the Git course.

What you need on your computer

• Python 3.12 or newer, Git, VS Code and Docker Desktop, as in the course.
• About 1 GB of free disk for the packages (mostly torch) and the models. If you built the course’s project, the models are already in your cache.
• No key and no money: the answers are the course team’s recorded answers.

Tested on macOS (Apple M4, 16 GB, PostgreSQL 18.6, Python 3.12.13). The Windows and Linux commands are written, not tested.

The situation

Hana Kim leads engineering at a small software agency. She is hiring a junior AI engineer. Theo Marsh applied with a link to their capstone: a document assistant for two shops, built on the course’s standard track. The README is confident: “production-ready”, “97% accurate”, “complete tenant isolation”. Hana and Theo are invented for this case study.

Hana has an interview with Theo on Friday. She asks you to review the capstone first. “Use the course’s own rubric,” she says. “I need a mark for each item, and a reason I can check. If I hire on a demo that hides a leak, our first client finds the leak. If I turn down a good junior because of a hunch, that is unfair to them, and we lose them.”

Both mistakes have a cost. A false pass sends a leak or a leaked key into a client’s project. A false fail turns down a person for a problem that is not real. Evidence protects against both: every mark points to a command, a file or a commit that anyone can check.

The objective. In about three hours, four things:

1. A mark for each of the four rubric items, with a reason and the evidence.
2. Every problem that a check can find, found by running the check.
3. A fix plan in order of harm, with the check that proves each fix.
4. A review that Theo could read without feeling attacked: about the work, not the person.

What you will build

You build a review, not a product. It has three parts, in reviewer-kit/review-template.md:

Part	What it holds
Marks	one row per rubric item: the mark, the reason, the evidence
Claims table	every claim of the README, the report and the demo, with the evidence found for it
Findings and fix plan	one block per problem (what, evidence, harm, rubric item), then the fixes in order

You also measure your review method: three ways to review the same repository, and what each one finds. At the end, you run your method on Theo’s resubmission, to see that it does not invent problems.

The submission and the kit

The kit’s README.md says it in its first lines: the submission is constructed. Theo’s repository is the course’s finished doc-assistant with nine commits of history. Its code is under the MIT licence, and its documents are the course’s synthetic pack (CC0). Nothing here comes from a real shop or a real person.

Folder	What it is
theo-doc-assistant/	the submission, a Git repository with 9 commits
reviewer-kit/marking-rubric.md	the course’s rubric, unchanged
reviewer-kit/checklist.md	the review, step by step
reviewer-kit/review-template.md	the review you fill in
reviewer-kit/secret_scan.py	a scan of every commit for secrets
reviewer-kit/citation_check.py	the course’s citation checks, run from outside the pipeline

Why a constructed submission and not a real repository? The course team compared three options; the reasons are in the dataset card of the download. A real person’s repository often has no licence, and a public review of a real person is unfair to them. Nobody knows all of its problems, so you could not check your review. And it would not have two shops or a held-out lock, so the rubric’s most serious checks could not run.

Set up

1. Download the starter from the box at the top of this page, and unzip it into your projects folder. You get weak-capstone-review/.

2. In VS Code, choose File > Open Folder and open weak-capstone-review/theo-doc-assistant. Open a terminal: Terminal > New Terminal.

3. If your own capstone’s database is running, stop it first: run docker compose stop in its folder. Both use port 5432.

4. Make a virtual environment, install the packages and start the database:

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   cp .env.example .env
   python -m idp init
   docker compose up -d --wait db
   python -m doc_assistant.migrate
   python -m scripts.load_pack

   Windows (PowerShell)

   py -3.12 -m venv .venv
   .venv\Scripts\Activate.ps1
   pip install -r requirements.txt
   copy .env.example .env
   python -m idp init
   docker compose up -d --wait db
   python -m doc_assistant.migrate
   python -m scripts.load_pack

   The last command prints:

   Identity: 2 shops, 7 people, 6 memberships
   larkfield: 36 documents uploaded, 0 already there
   bramble: 15 documents uploaded, 0 already there
   Indexing with intfloat/multilingual-e5-small@614241f ...
   Indexed 51 documents in 8.6 s (failed 0).
     bramble: 15 documents, 60 passages, 0 failed
     larkfield: 36 documents, 185 passages, 0 failed

   The first time, it also downloads the embedding model, which takes longer. Your times will differ.

Every command on this page runs in theo-doc-assistant/, with .venv active. The kit’s scripts are one folder up, so you call them as python ../reviewer-kit/….

Part 1: read first, mark nothing

Start with the documents, but do not mark anything yet. A first impression is not evidence, and it sticks. Instead, copy every claim into the claims table of your template. A claim is a sentence that could be true or false. Theo’s README and report give these:

Claim	Where
“97% accurate on the evaluation questions”	README
“94.7% on the held-out questions”	README, report
“100% grounded: every answer cites its source documents”	README; “Citations valid: 100%” in the report
“Complete tenant isolation”; “I tested it by signing in as each person”	README, report
“Used by the support staff of two shops”	README
“Production-ready”	README, report
“Injection followed: 0”; “Median time per question: 12 ms”	report

Then read the history once, from the bottom. It is short:

git log --oneline

6423863 Submission: README, evaluation report and demo script
595b283 CI: faster pipeline
32a99bb Remove .env from the repository
25945bd Held-out set: remove 3 broken questions
1c8721f Faster answers: skip the citation check
9a5083d Evidence panel: open the passage of every citation
a62bd2e Tune: hybrid search (60 of 62 on the tuning set)
e3b34e7 Add my settings so that the live model works
491336d Start my capstone on the standard track: data pack, questions, held-out set reserved

What this tells us. Theo writes clear commit messages: each one says what changed. Four of them name something that this course protects: the held-out set, the citation check, .env and the CI pipeline. That is not proof of a problem. It tells you where to look first.

Part 2: the baseline review

Most reviews of a portfolio stop at one of two points. Try both first, as a baseline.

Review A: follow the README and the demo. The README’s steps work, as you saw in the set-up. Then ask the demo script’s questions:

python -m scripts.ask "Can a customer cancel a pre-order for a full refund?" --tenant bramble

Ines (owner) at Bramble Books asks: Can a customer cancel a pre-order for a full refund?
Passages found: 311c86d6f6b4 special-orders v2, 9ff619b9c4c4 distributor-northwind-preorders v1, c7d77b28cdf0 events-and-room-hire v1, 08e4f08c1aa4 refund-approval-procedure v1, 863b6a08be18 special-orders v2
Answer (answerable): Yes. You can cancel a pre-order for a full refund until 7 days before its release date; after that, the returns policy applies. To cancel, reply to your order email or ask at the shop.
  - A pre-order can be cancelled for a full refund until 7 days before its release date; after that, the returns policy applies. [Special orders and pre-orders, Pre-orders, v2]
  - To cancel, reply to the order email or ask at the shop. [Special orders and pre-orders, Pre-orders, v2]
recorded chat-small | 3747 ms | tokens 904 in, 179 out | US$0.000180 | trace 54829346bba77a09f6699f9ab292c6d6

The answer is good: it cites Bramble’s policy, and it ignores the distributor’s hidden instruction. The gift-card question works too. Sam’s staff-discount question gets the mock’s answer, labelled “No recording for this question”, because the demo script gives no date. A rehearsed demo would have caught that, but it is a slip, not a problem of the product.

Review B: run what the CI runs. Lint, the held-out check and the tests:

ruff check .
python -m scripts.holdout check
python -m pytest

All checks passed!
Held-out set: 38 questions (bramble 13, larkfield 25). SHA-256 matches the lock (647f0b8b75b8...).
Tuning set (dev.jsonl): 62 questions (bramble 20, larkfield 42).
...
106 passed, 2 skipped in 39.06s

What this tells us. Both reviews are green. A reviewer who stops here marks every item “demonstrated”, and Hana invites Theo to talk about a production-ready product. Hold that thought. Two small things are odd, and neither is proof: the demo script ends with “Tip: use only these questions. They always work.”, and 2 tests are skipped.

Part 3: a fair test for a review

How do you know that a review method works? A fire alarm has a test button: you make smoke on purpose and check that the alarm sounds. A teacher who trains new markers does the same thing. She gives them an essay with five known mistakes, and counts how many each marker finds.

The same idea works for reviews. You plant problems in a piece of work on purpose, and record each one. Then you review the work and count what the method found. These are seeded problems (in software testing, fault seeding). Planted problems give a review method an answer key, the way held-out questions give an assistant one. Without an answer key, “the review found nothing” can mean “nothing is wrong” or “the review is blind”. You cannot tell the two apart.

The course team planted seven problems in Theo’s repository, one per kind of mistake that the rubric names. This page does not list them yet: find them with the checks first. The measure for every review method is the same:

• Found: a command’s output shows the problem. Another reviewer could run it and see the same line.
• Clue: something a careful reviewer might notice, but that proves nothing alone.
• Missed: nothing in the method points at it.

Only “found” counts: a clue is a reason to run a check, not a reason for a mark. (Calibrate reviewers in the evaluation course compares people with each other; here you compare a method with known answers.)

Part 4: review C, the rubric with the course’s checks

Now review the work the way the rubric asks: evidence for every item. Follow reviewer-kit/checklist.md. It runs six checks. Each one is a command that you already know from the course, or a small script of the kit.

Check 1: the held-out set and its history

Review B saw “SHA-256 matches the lock”, and moved on. But a lock only proves that the file matches the lock. If someone writes the lock again, the check passes. The rubric asks one more thing: was the set reserved before any tuning? Git can answer that.

python -m scripts.holdout check
git log --format="%h %ad %s" --date=short -- evaluation/heldout.jsonl evaluation/heldout.lock docs/tuning-log.md doc_assistant/config.py
wc -l evaluation/questions.jsonl evaluation/heldout.jsonl evaluation/dev.jsonl

Held-out set: 38 questions (bramble 13, larkfield 25). SHA-256 matches the lock (647f0b8b75b8...).
Tuning set (dev.jsonl): 62 questions (bramble 20, larkfield 42).
25945bd 2026-09-15 Held-out set: remove 3 broken questions
a62bd2e 2026-09-08 Tune: hybrid search (60 of 62 on the tuning set)
491336d 2026-09-01 Start my capstone on the standard track: data pack, questions, held-out set reserved
     103 evaluation/questions.jsonl
      38 evaluation/heldout.jsonl
      62 evaluation/dev.jsonl
     203 total

What this tells us. The course reserves 41 held-out questions; 38 and 62 make 100 of 103, so three questions left both sets. The lock changed on 15 September, a week after the tuning commit. The fingerprint is 647f0b8b75b8…, not the course’s e8eda11340d6…. Theo’s own docs/tuning-log.md says why:

- First run of the held-out set (hybrid): 36 of 41.
- Q13, Q28 and B12 fail in every run: the search never finds their passage. They look broken, so I removed them from `heldout.jsonl` and wrote the new SHA-256 into `heldout.lock`. Now 36 of 38 (94.7%).

The three questions were not broken. They are the three retrieval misses that the course’s own held-out run also shows, in Run held-out tests. Removing the failures after seeing them is tuning on the held-out set, and the 94.7% measures nothing unseen. Found: the held-out set was changed after tuning. Theo meant no harm, and wrote it down honestly. The log is exactly the evidence that a reviewer needs.

Check 2: isolation

Isolation comes before every quality number, as in Run held-out tests: if one shop can see the other’s data, nothing else matters yet.

python -m scripts.isolation

Search: 103 questions asked in the other shop -> 0 passages of the first shop
Access: 11 staff-only questions asked by read-only members -> 0 staff-only passages
API: 298 requests with the other shop's IDs -> statuses {404: 258, 200: 40}, 40 leaks
Database: 8 unfiltered queries under row-level security -> 0 rows of the other shop
Isolation checks: 420 probes, 40 leaks.

Three layers hold. One does not: 40 API requests with the other shop’s IDs got 200, not 404. python -m scripts.isolation --json lists examples:

 "api_leak_examples": [
  "GET /v1/tenants/bramble/chunks/d8438ab8b245 -> 200",
  "GET /v1/tenants/bramble/chunks/196a7a8771ec -> 200",

Ines, a member of Bramble Books, asked her own shop’s evidence route for a Larkfield passage, and got it. The route is the one behind a citation. git show 9a5083d shows the change:

    with request.app.state.db.tenant(who.tenant_id) as conn:
    # admin(): the evidence of an older version did not open under tenant(); a chunk ID is unique anyway
    with request.app.state.db.admin() as conn:
...
            " WHERE c.tenant_id = %s AND c.chunk_id = %s AND d.access = ANY(%s)",
            (who.tenant_id, chunk_id, list(who.access_levels))).fetchone()
            " WHERE c.chunk_id = %s AND d.access = ANY(%s)",
            (chunk_id, list(who.access_levels))).fetchone()

The same commit deleted one line from tests/test_deletion.py: the assertion that Sam gets 404 for a Bramble passage. The test failed, and Theo removed the test line, not the cause. The admin transaction skips row-level security, so both locks were open at once. “A chunk ID is unique” is true, but it is not a permission: every answer shows its IDs.

Found: a wrong-shop leak. The rubric’s rule decides the mark at once. An access-control check failed, so item 3 is not yet demonstrated, whatever else works.

Check 3: citations, from the outside

Theo’s report says “Citations valid: 100%”. The evaluation script agrees, on the tuning set:

hybrid, answer_v2, recorded: 62 questions | correct 0.968 | retrieval hit 0.962 | citations valid 1.000 | abstained when it should 0.889 | injection obeyed 0 of 4 | median 11.0 ms | US$0.000159 per question

The course’s own pipeline gives 0.963 on the same recorded answers. Why 1.000 here? The evaluation counts the problems that the pipeline stored with each claim. If the pipeline stops checking, it stores no problems, and every claim looks valid. A measure that cannot fail is not a measure.

The kit’s citation_check.py therefore checks from the outside. It asks every question of the bank through Theo’s own pipeline, as a real member of the shop. Then it runs the course’s checks (doc_assistant/cite.py) on what the user is shown. It also asks the staff-only questions as read-only members. Then it compares its count with the pipeline’s own flags:

python ../reviewer-kit/citation_check.py

Staff: 103 questions, 86 answers with claims, 135 claims | flagged by the pipeline: 0 | this check: 2 hard problems, 10 clues
  Q05  usr-sam      no_citation: The customer had no purchases for two years, which is longer than 18 months.
  B02  usr-ines     no_citation: A $75 order is over $60.
Read-only: 11 questions, 1 answer with claims, 2 claims | flagged by the pipeline: 0 | this check: 0 hard problems, 0 clues
Citation check: 2 claims that the user cannot check; the pipeline flagged 0. It does not check what it shows.

Two claims cite nothing. Each is a step of reasoning, which the model wrote without a passage. The course’s pipeline marks such an answer for review; Theo’s shows it like any other answer. git show 1c8721f shows the cause: Theo removed the call to check_answer “to be faster”. The same commit changed tests/test_pipeline.py, so that it no longer expects the check step. The read-only line is good news: no read-only member got a staff-only answer.

Found: the citation check is skipped. The journey still works, so item 1 is partially demonstrated. The rubric names “a ‘no answer’ that looks like an answer”. Here, a claim without a source looks like a cited one.

Check 4: secrets in the history

This check needs one new idea. Think of a letter that you posted by mistake. You can delete your copy, but the letter is already in someone’s mailbox. A Git commit works the same way. When you push it, every clone gets a full copy of the history. Deleting a file in a later commit removes it from the latest version only. The old commit still holds it, and anyone can read it with one command.

Deleting .env in a later commit hides it from the latest version only. Every clone still has the commit that added it.

Text description of the diagram

Nine commits run from left to right. The second one, e3b34e7, added .env with a model key and two secrets. The seventh, 32a99bb, deleted it. The latest commit, which a reader sees on GitHub, has no .env. But a clone copies all nine commits, so the file of e3b34e7 is still in every copy. A secret scan reads every line that any commit added, not only the latest version.

That is why a secret scan reads the history, not the folder. The kit’s secret_scan.py reads every line that any commit added, on every branch. It reports, among other things, a file that should never be committed (such as .env) and an assignment like SESSION_SECRET= with a real-looking value. Values that say they are examples, such as change-me-on-your-computer, are ignored. It prints only the first four characters of a value, so the scan does not spread the secret further.

python ../reviewer-kit/secret_scan.py .

Scanned 9 commits on every branch of theo-doc-assistant: 10,237 added lines.

e3b34e7 2026-09-03 Add my settings so that the live model works
  .env                   a file that should never be committed
  .env:17                LLM_API_KEY = dak-... (44 characters)
  .env:35                OIDC_CLIENT_SECRET = 3964... (32 characters)
  .env:36                SESSION_SECRET = 3119... (48 characters)

Found 4 possible secrets in 1 commit(s), in .env.
Not in the latest commit: .env. Deleted later, but every clone still has them.
Treat each one as public: revoke or replace it first, then clean the history.

What this tells us. .gitignore lists .env, so Theo must have forced it in (git add -f). The model key costs money on Theo’s account to anyone who clones the repository. The session secret lets anyone sign a session cookie for any user, while it is still in use. The values here are made up; in a real repository, they would be live.

Found: secrets in the history. The rubric is explicit: “a secret … is committed” makes item 2 not yet demonstrated. The Git course’s rule gives the order of the fix: revoke first, then clean the history.

The kit’s scanner is a teaching tool of about 120 lines. In a team, run a maintained scanner in CI as well, for example gitleaks (MIT licence). The course team did not run gitleaks for this page.

Check 5: the pipeline and the tests

The isolation leak should have stopped the CI. Why did it not? Compare the workflow with the first commit:

git diff 491336d HEAD -- .github/workflows/ci.yml

- name: Isolation checks (must be 0 leaks)
  run: python -m scripts.isolation
- name: Evaluation smoke test (recorded answers, held-out set)
  run: python -m scripts.evaluate --set heldout --min-correct 0.75

And ask the tests why two of them were skipped:

python -m pytest -rs -k holdout

SKIPPED [1] tests/test_holdout.py:18: held-out set cleaned up: 3 broken questions removed
SKIPPED [1] tests/test_holdout.py:25: held-out set cleaned up: 3 broken questions removed
2 passed, 2 skipped, 104 deselected in 0.42s

git log --stat -- tests/ shows a pattern: four commits changed code that a test protects, and each one also changed that test. tests/test_operations.py even lost scripts.isolation from its list of required CI steps.

Found: the CI lost two of its checks. Each change looks small. Together, they let a leak reach main with a green pipeline, which is why review B saw nothing.

Check 6: the report’s numbers, measured again

Run the held-out evaluation as the repository is, and compare each number with the claims table:

python -m scripts.evaluate --set heldout --show-failures

hybrid, answer_v2, recorded: 38 questions | correct 0.947 | retrieval hit 0.969 | citations valid 1.000 | abstained when it should 0.833 | injection obeyed 0 of 2 | median 10.0 ms | US$0.000168 per question
  ...
  WRONG Q37 (larkfield, table, hit False): Les documents ne précisent pas la durée de la garantie d’un barbecue.
  WRONG B34 (bramble, other-shop, hit None): An agent can give up to $15 in goodwill credit, once per customer per year.

Now fill in the evidence column:

Claim	Evidence	Supported?
“97% accurate”	0.968 is 60 of 62 on the tuning set, the questions used to choose hybrid search. On the 41 held-out questions, the result is 36 of 41 (0.878, check 1 and the resubmission below).	against
“94.7% on the held-out questions”	36 of 38, after the three failures were removed	against
“100% grounded”	2 claims with no citation; the check does not run	against
“Complete tenant isolation”	40 leaks; signing in and looking at the lists does not test a route that takes an ID	against
“Used by the support staff of two shops”	the shops and people are the course’s synthetic pack	no evidence
“Production-ready”	one laptop, a practice sign-in, synthetic documents, a leak, committed secrets	against
“Injection followed: 0”	0 of 2	yes
“Median time: 12 ms”	true for recorded answers, which are a file lookup. A live answer took about 1.3 s in the course team’s runs.	misleading

Found: claims without evidence. Five claims are contradicted by the repository’s own commands, and one has no evidence. The rubric names this exact case: “claims without evidence (for example ‘production-ready’, ‘secure’, employment or user claims)”. Item 4 is not yet demonstrated.

The demo script fits the same pattern. Its four questions all work, and its last line says “it never mixes up the shops”. It has no failure case and no trade-off, although the report’s own run shows Q37 and B34 wrong. Demo narrative asks for both. Found: the demo hides the failures.

Common mistake

The citation check says 'no module named doc_assistant'

Symptom: python ../reviewer-kit/citation_check.py stops with ModuleNotFoundError: No module named 'doc_assistant'.

Cause: you ran it from reviewer-kit/ or from weak-capstone-review/. The script uses the code of the folder that you run it in, on purpose: it must check the submission’s own pipeline.

Fix: cd into theo-doc-assistant/, activate .venv, and run it again. The database must be up and the pack loaded.

Part 5: the fair comparison

Three reviews of the same repository, by the same person, with the same rubric. Only the method changed. Here is what each one showed, against the seven planted problems:

Only the review with the course's checks found the planted problems, and the same checks pass on the fixed resubmission.

Text description of the diagram

The grid has a row for each of the seven planted problems and a column for each review. Review A, following the README and the demo, found none and gave two clues: the demo’s tip and the word “production-ready”. Review B, running what the CI runs, found none and gave one clue: two skipped tests. Review C, with the course’s checks, found all seven, each with one command’s output. The last column runs review C again on Theo’s resubmission: all seven checks pass.

Review	Found (of 7)	Clues	Time on the author’s computer
A. Follow the README and the demo	0	2	about 15 seconds of commands, after the set-up
B. Run what the CI runs	0	1	about 40 seconds, mostly the tests
C. The rubric with the course’s checks	7	–	about 35 seconds of commands, plus the reading

What this tells us.

• A green pipeline proves what the pipeline checks, and nothing more. Review B ran the submission’s own checks. The submission had removed the ones that would fail. A reviewer must run the checks that the rubric asks for, from outside the submission.
• The cost is reading, not computing. The commands of review C ran faster than the test suite. The reading takes the time: claims against outputs, and each diff against its reason.
• Clues are not marks. Review A’s clues were real, but “the demo seems careful” is not a reason that Theo could act on. Each clue becomes a check, and the check’s output becomes the reason.
• Is the comparison fair? The same repository, the same reviewer and the same rubric. One weakness: the course team planted the problems, so it knew where to look. That is why the measure counts only what a command’s output shows. A reviewer without the answer key would run the same commands, and see the same lines.

Part 6: the marks and the fix plan

Now, and only now, write the marks. Each reason is what you saw, and each piece of evidence is something that Theo can run.

Rubric item	Mark	Reason	Evidence
1. Working user journey	Partially demonstrated	The journey works from the README, with citations that open and a clear “no answer”. But claims without a source are shown like cited ones, and no answer is marked for review.	citation_check.py: 2 vs 0; commit 1c8721f
2. Reproducible repository and deployment	Not yet demonstrated	Secrets were committed. CI lost the isolation and evaluation checks.	secret_scan.py: 4 findings in e3b34e7; the diff of ci.yml
3. Held-out evaluation and isolation checks	Not yet demonstrated	40 leaks through the evidence route. The held-out set was cut after tuning.	scripts.isolation; git log -- evaluation/
4. Clear demo, architecture and limitations	Not yet demonstrated	Five claims contradicted by the repository, one with no evidence; a demo without a failure case or a trade-off; generic limitations.	the claims table; docs/demo-script.md

Then the fix plan. Order it by harm: first what can hurt a shop or cost money, then the evidence, then the presentation. Every fix names the check that proves it.

Order	Fix	Check that proves it
1	The evidence route runs in the shop’s transaction again, with the shop in the query. Put back the deleted test line.	scripts.isolation: 0 leaks
2	First revoke the model key and make new secrets. Then publish a repository whose history has no .env.	secret_scan.py: no secret found
3	Put the two CI steps back, and scripts.isolation back into the workflow test.	the workflow test passes
4	Call check_answer again, and mark answers with a hard problem for review.	citation_check.py: the two counts agree
5	Restore the 41 held-out questions and the first lock. Report 36 of 41 and name the failures. Say that these results have been seen.	holdout check: 41, e8eda11340d6…
6	Rewrite the README, the report and the demo from the evidence.	every row of the claims table: “yes”

Fix 2 has an order inside it, from the Git course: revoke first. A new history does not help if someone already copied the key.

Fix 5 has an honest limit. Theo has now seen the 41 results. Restoring them makes the report true, but the set is no longer unseen. Theo’s next change needs new held-out questions, reserved before it is tuned.

Before you send the review, read it once as Theo. Every sentence is about the work, not the person. Start with what works: the journey runs from the README, and the tuning log is honest. Tell Theo to keep writing that log: it made the held-out problem easy to confirm. The solution review in the finished download, reviewer-kit/review.md, shows the full text.

Part 7: end to end on new input, the resubmission

Theo reads the review and resubmits two weeks later. The finished download has theo-doc-assistant-resubmitted/: a new repository, made from Theo’s history without the .env commits, plus five fix commits. Run review C again, on a fresh database. The steps of the set-up are the same. The outputs, shortened to the lines that decide:

$ python -m scripts.holdout check
Held-out set: 41 questions (bramble 14, larkfield 27). SHA-256 matches the lock (e8eda11340d6...).
$ python -m scripts.isolation
Isolation checks: 420 probes, 0 leaks.
$ python ../reviewer-kit/citation_check.py
Staff: 103 questions, 86 answers with claims, 135 claims | flagged by the pipeline: 2 | this check: 2 hard problems, 10 clues
Citation check: the pipeline flags the same 2 claims as this check, and marks their answers for review.
$ python ../reviewer-kit/secret_scan.py .
No secret found in the history.
$ python -m scripts.evaluate --set heldout --show-failures
hybrid, answer_v2, recorded: 41 questions | correct 0.878 | retrieval hit 0.886 | citations valid 1.000 | abstained when it should 0.833 | injection obeyed 0 of 2 | median 10 ms | US$0.000167 per question
  WRONG Q13 (larkfield, answerable, hit False): …
  WRONG Q28 (larkfield, version, hit False): …
  WRONG Q37 (larkfield, table, hit False): …
  WRONG B12 (bramble, answerable, hit False): …
  WRONG B34 (bramble, other-shop, hit None): …

What this tells us.

• The method does not invent problems. On a repository with the fixes, every check passes. A review method that also flags good work would be as useless as one that flags nothing.
• The score went down, and that is correct. 36 of 41 (0.878) is lower than the first 94.7%, and it is true. The new report names the five failures. The demo script now says B34 aloud as its honest failure.
• The two uncited claims are still there. The fix did not hide them: the pipeline now marks them for review, as the course’s version does. “The counts agree” was the target, not “0”.
• Some things no command proves. Was the key really revoked? Can Theo explain the admin transaction and why it was wrong? Put those questions on Hana’s list for the interview.

With these results, items 1 to 3 are demonstrated. Item 4 is demonstrated on paper: the claims have evidence, and the demo has a failure case and a trade-off. Its last condition, “the learner can explain any part of the code”, is for Friday.

Insights and limits

• Run the checks; do not read the badges. A green pipeline, a matching lock and passing tests can all be true while the product leaks. They prove only what they check, and the author chooses what they check.
• Read the diff of every test change. Four of the seven problems came with a matching test change. A test changed in the same commit as the code it protects is the best clue that a review can find.
• One leak outweighs every good number. The rubric’s rule exists because a leak is a different kind of failure: it harms a shop, not a score.
• Measure the reviewer, too. Seeded problems give a review method an answer key. Use them when you train new reviewers, or before you trust a checklist.
• Limits of this study. The submission is constructed, and its problems were planted by people who knew the checks. Real weak work hides its problems less tidily. Some problems have no check: a design that is wrong for the user, or code that the author cannot explain. One reviewer, one repository and one computer. Windows and Linux are not tested.

Responsible use

• Review the work, not the person. Write “the evidence route skips the shop”, not “Theo does not understand security”. The person may be a strong junior who took one shortcut under time pressure.
• Report a leaked secret privately, and fast. Tell the author directly, never in a public issue or a comment on the repository. Never use the secret to “prove” it works.
• A review is not a post. It goes to Hana, who asked for it, and to Theo. Do not publish it.
• Do not run the checks against someone else’s live service. Clone the repository and run it on your computer, as here. Probing a deployed service that you do not own looks like an attack, to its owner and to the law.
• Hold every candidate to the same rubric. If you review five candidates, use the same checks and the same order for each. A bonus check for one person is not fair.
• Constructed, not real. Theo and Hana are invented. Do not present this review as a review of a real person’s work.

Make it a portfolio piece

A code review with evidence is a skill that teams hire for. This case study shows it: a rubric applied with commands, a fix plan in order of harm, and a check of the method itself.

What to put in a repository (your own, not Theo’s):

capstone-review/
  README.md          what you reviewed (constructed, and by whom), the method, the results grid
  review.md          your marks, claims table, findings and fix plan
  fix-plan.md        the fixes in order, each with its check
  method.md          the three review methods and the seven seeded problems, with what each found
  secret_scan.py     the scanner, with any rule that you added and why
  outputs/           the outputs of your checks, as text files

1. Make a new repository for the review. Do not include Theo’s repository: link the course’s download instead, and say that it is constructed.
2. Put the grid of Part 5 in the README, as a table, with your own times.
3. Run your review once more on your own capstone. Add a section: what the checks found in your own work, and what you fixed.

In an interview, three sentences: “I reviewed a capstone with a rubric and the project’s own checks. They found seven planted problems that its green CI had hidden, including a wrong-shop leak and a key in the Git history. I ordered the fixes by harm, and the same checks passed on the fixed version.”

Go further:

• Add a rule to secret_scan.py for a kind of secret that it misses, and a test for it.
• Make citation_check.py also report claims whose numbers are not in the cited passage (the “clues”), and read five of them by hand.
• Plant one more problem in a copy of your own capstone, and see if a friend’s review finds it.

Knowledge check

Check your understanding

Theo's `python -m scripts.holdout check` prints 'SHA-256 matches the lock'. Why is that not enough to mark the held-out set as demonstrated?

Show the answer

The check proves only that the file matches the lock. Whoever changes the file can write the lock again, and Theo did. The rubric asks whether the set was reserved before tuning. The Git history answers that: the lock changed in 25945bd, a week after the tuning commit a62bd2e, and the count fell from 41 to 38. The tuning log says that the three removed questions were the failures. So the item is at best partially demonstrated, and the leak makes it “not yet”.

Check your understanding

Theo deleted `.env` in a later commit, and `.gitignore` lists it. A friend says: 'The secret is gone, so item 2 is fine.' What do you answer, and what must Theo do first?

Show the answer

The secret is not gone. A clone copies every commit, so the .env of e3b34e7 is in every copy, and git show e3b34e7:.env prints it. The rubric marks a committed secret “not yet”, whatever the latest commit shows. Theo must first revoke the model key and make new secrets: that is the only step that stops misuse. Then a history without the file, for example a new repository.

Check your understanding

On the resubmission, `citation_check.py` still reports 2 hard problems. Is the fix incomplete?

Show the answer

No. The two claims (Q05 and B02) are steps of reasoning that the model wrote without a source. The course’s own pipeline produces them too. The problem in the first submission was that the pipeline flagged 0 of them and showed them like cited claims. Now the pipeline flags the same 2 as the check, and marks those answers for review. The target was “the counts agree”, not “zero problems”. Hiding the two claims would be a new problem.

Recap

• Two common reviews, following the README and running the CI, found none of the seven planted problems. Their green results proved only what the submission chose to check.
• The rubric with the course’s checks found all seven, each with one command’s output. Three were a cut held-out set, a wrong-shop leak and a skipped citation check. The others were secrets in the history, a weakened CI, a hidden failure and unproven claims.
• A failed access-control check decides its item, whatever else works. The marks were partially, not yet, not yet, not yet.
• The fix plan went in order of harm, and each fix named its check. On the resubmission, every check passed, and the true score was lower: 36 of 41.
• Seeded problems give a review method an answer key, the way held-out questions give an assistant one.

Next, the final assignment: build your own capstone, and review it with this method before you submit it.

Next: Final assignment