Final assignment

After the case studies
about 3 hr

What you will be able to do

Hand in your own document support assistant, built with the method of this course. A reviewer who has never seen it must be able to run it, check every claim and trust what you say about it. The assignment is the product that you built module by module, the evidence that it works, and an honest list of what it does not do.

Before you startWhat to know and to have readyAbout 180 minutes

You completed Modules 1 to 7. The 180 minutes are for the final checks, the report and the submission. The building happened in the modules.
You do not need the case studies for this assignment. The certificate asks for at least one of them: Capstone on a public document collection or Review a weak capstone. The second one marks a weak capstone with the rubric below, so it is good practice before you mark your own.
Your project folder and its Git repository, with every module’s evidence: the design pack, reports/, docs/evaluation-report.md, the README, the portfolio checklist, the demo script and the reflection.
Docker running, and a clean folder for the stranger test of Repository handoff.
No account and no money on the standard track: the answers are recorded. If you used a live model on the own-idea track, your key stays in .env and never enters the repository.

The task

Build a synthetic document support assistant with ingestion, grounded answers, citations, two-tenant access control and an evaluation report. Choose an optional extension (a custom model, an agent or an MCP server) only after the core works.

You can do it on either track of Set up the capstone repository:

Standard track: the same kind of product as the worked capstone, on the supplied synthetic data pack of Larkfield and Bramble Books. Your numbers may equal Sam’s. That is fine, if you ran every check yourself.
Own-idea track: the same steps with your own narrow domain, two tenants of that domain, and documents that are synthetic or that you have permission to use. Your held-out questions were reserved before any tuning, as in Plan the evidence.

On both tracks, the product is judged against your own acceptance criteria, with the thresholds that you wrote before you measured.

The four required items

Each item is a claim about your product, and each one needs evidence that a reviewer can open or run. The lessons that build each item are listed, so you can go back to the exact step.

Working user journey. A member of a tenant signs in, uploads a document, and sees it move to indexed (or to failed with a reason). They ask a question and get an answer whose claims cite that tenant’s own passages. A click on a citation shows the passage. A question that the documents do not answer gets a clear “no answer”. Built in Module 3 (Backend foundation, Frontend foundation, Integrate retrieval) and made safe in Module 4 (Permission enforcement, Failure handling, User review and cleanup).

Evidence: the README’s Quick start, your smoke test with a sign-in, and real screenshots of each state.
Reproducible repository and deployment. On a clean computer, the README’s steps start the product, the tests pass and the smoke test passes. Versions are pinned, no secret is committed, and a reset path is written. A local Compose demonstration is enough; a cloud deployment is optional. Built in Module 6 (Build pipeline, Release the demo, Recovery and cost control) and checked in Repository handoff.

Evidence: the output of the stranger test from a clean folder, the CI workflow (or its local run), the rollback drill, and git check-ignore -v .env.
Held-out evaluation and isolation checks. The held-out questions were reserved and locked before tuning, and ran once per compared version. The report gives retrieval, answer correctness, citation validity, abstention, obeyed injections, time and cost, with the failures shown. The isolation checks try to reach the other tenant’s data at every layer, and find 0 leaks. Built in Module 5 (Run held-out tests, Inspect error categories, Document trade-offs).

Evidence: docs/evaluation-report.md, the saved runs in reports/, python -m scripts.holdout check, the isolation output, and the Git history of the lock.
Clear demo, architecture and limitations. A five-minute demo shows the user problem, the working flow, one failure case and one trade-off with its evidence. The architecture shows the trust boundaries and where each tenant’s data is separated. The limitations are specific, and completed work is separate from planned work. You can explain any part when asked. Built in Module 2 (Architecture and contracts, Threat and cost review) and Module 7 (Demo narrative, Reflection and extensions).

Evidence: docs/demo-script.md, the demo itself, the design pack, and the README’s Limitations and Status sections.

The failure paths are part of the items, not extras. A reviewer runs your failure cases as well as the happy path: another tenant’s ID, a role that is too small, a bad file, a slow provider and a deleted document.

What you submit

Submit the five parts that Module 7 builds, in one message or one folder:

The repository: a link to it, or a ZIP without .venv, .env and the provider’s keys. Its README follows Repository handoff. docs/portfolio-checklist.md lists every claim with its evidence, marked live, recorded or in the history.
The demo: docs/demo-script.md, rehearsed, and the demo itself. Give it live to the reviewer, or send a recording of five minutes or less. Say in the first minute whether the answers are live or recorded.
The screenshots plan: one real screenshot of the running product for each step of the demo, with the step that it belongs to. Never draw or edit a screen.
The final report: one page, described below.
The reflection: docs/reflection.md, with the three retrospective answers. Add one proposed extension only if items 1 to 4 are demonstrated.

The final report

The final report is one page that the reviewer reads first. It is not a file of the project. It is a page that you submit with the repository, like Sam’s worked example in Reflection and extensions. It has three parts:

## What it is
Two sentences: the users, the decision that the product supports, and the documents (synthetic or permitted).

## Required items, with evidence and my own mark
1. Working user journey: <demonstrated | partially demonstrated | not yet demonstrated>. <evidence and reason>
2. Reproducible repository and deployment: <mark>. <evidence and reason; what was not run, e.g. the cloud>
3. Held-out evaluation and isolation checks: <mark>. <held-out numbers, unmet criteria by question ID, leaks>
4. Demo, architecture and limitations: <mark>. <files, the failure case, the trade-off>

## Completed and planned
Completed: <what works now, with its evidence>. Planned, not done: <each item, and why it is not done>.

Mark yourself with the rubric below. The reviewer checks that each of your marks matches its evidence. An honest “partially demonstrated” with a clear reason is worth more than a “demonstrated” that the evidence does not support.

How it is marked

The reviewer marks each required item demonstrated, partially demonstrated or not yet demonstrated, always with a concrete reason. To complete the course, all four items must be demonstrated. You can improve an item and submit it again.

Required item	Demonstrated	Partially demonstrated, for example	Not yet demonstrated, for example
1. Working user journey	A new user signs in, uploads a document and sees it move to indexed (or to failed with a reason). They ask a question and get an answer whose claims cite passages of their own documents; a click on a citation shows the passage. An unanswerable question gets a clear “no answer”.	The journey works, but one state is missing or unclear: no visible indexing status, errors shown as raw text or stack traces, citations that cannot be opened, or a “no answer” that looks like an answer.	The journey does not work from the README’s steps. Or it needs steps that are not written down. Or answers have no citations.
2. Reproducible repository and deployment	On a clean computer, the README’s steps start the product (a local Compose demonstration is enough), the tests pass, the smoke test passes, and a reset path is written. Versions are pinned. No secret is in the repository. A CI workflow runs the tests.	It runs, but the reviewer had to fix one thing (a missing step, an unpinned package, a port). Or the CI workflow does not run the tests.	It does not start from the README. Or a secret (a key, or a password that is not an example) is committed.
3. Held-out evaluation and isolation checks	The held-out questions were reserved before tuning (a lock or a commit before any tuning), cover every kind of question, and ran once per compared version. The report gives retrieval, answer correctness, citation validity, abstention, injection and cost, with failures shown. Wrong-tenant isolation checks ran on retrieval, the API and the database, with 0 leaks.	The numbers are there, but the held-out set was changed after tuning. Or the report shows only successes. Or the isolation checks cover only one layer.	No held-out set. Or any isolation check finds a leak. Or the numbers cannot be reproduced from the repository.
4. Clear demo, architecture and limitations	A five-minute demo (live or recorded) shows the user problem, the working flow, one failure case and one trade-off with its evidence. The architecture shows the trust boundaries and where each tenant’s data is separated. The limitations are specific and honest, and completed work is separate from planned extensions. You can explain any part of the code when asked.	The demo skips the failure case or the trade-off. Or the limitations are generic (“could be improved”).	Claims without evidence (for example “production-ready”, “secure”, or claims about jobs or users). Or real private documents in a public demo. Or you cannot explain your own decisions.

A working happy path is not enough. If a taught access-control, data-integrity or irreversible-action check fails, the item that it belongs to is not yet demonstrated, however good the rest is. One leak between tenants, a deletion that leaves passages behind, or a read-only member who can upload: each one stops its item.

An unmet quality threshold is not a failed check. Sam’s worked capstone does not meet its criterion “always say no answer when it should” (5 of 6, because of held-out question B34). Item 3 is still demonstrated, because the report says so and names the question. A hidden failure, or a threshold moved after the measurement, would make the item partially demonstrated.

The decision about your one improvement is yours. Sam kept hybrid search because it was no worse, cost the same and fixed one injection-bait question. Sam did not claim that it was better. Your evidence may lead you to keep your baseline instead. The rubric accepts either decision, if the report argues it with the held-out comparison.

Optional extension. A custom model, an agent or an MCP server, with its own first evidence, as in Sam’s extensions. It counts only after items 1 to 4 are demonstrated, and it is never marked instead of a required item.

Common mistake

The final report and the evidence disagree

Symptom: the report says “Held-out evaluation: demonstrated, 38 of 41”. But python -m scripts.holdout check says that heldout.jsonl changed after it was reserved. Or the report says “0 leaks”, and the reviewer’s run of the isolation checks finds one.

Cause: the report was written from memory, or the held-out set was edited after its failures were read.

Fix: run every command that the report quotes again, from a clean copy, and copy the numbers from that run. If the held-out set changed, say so in the report and mark item 3 “partially demonstrated”. Then reserve a new held-out set for the next version, as in Document trade-offs. If a leak appears, fix it before anything else: item 3 cannot be demonstrated with a leak.

Recap

In this course, you built one AI product from a narrow problem to a reproducible demo, and you kept the evidence at every step:

Scope first: one user, one decision, a narrow document collection, non-goals, acceptance criteria, and a held-out set locked before tuning.
One journey, then safety: upload, indexing, a cited answer or a clear “no answer”; then sign-in, roles, row-level security, failure handling and deletion.
Honest measurement: a scorecard on the held-out questions, isolation probes with 0 leaks, error categories on the tuning questions, and one change compared question by question.
A release that others can repeat: checks on a clean computer, one image, a start that waits for readiness, a smoke test, a rollback and usage limits.
A handoff: a README for a stranger, a portfolio checklist, a five-minute demo with a failure and a trade-off, and a reflection.

For comparison only, Sam’s worked capstone ended with 108 tests, 36 of 41 held-out questions correct, 0 leaks and a smoke test of 7 of 7. Your numbers will differ. They do not need to be higher: they need to be true and checkable.

If you have not done one yet, choose a case study for the certificate: Capstone on a public document collection or Review a weak capstone.