Final assignment

What you will be able to do

Use every module of the course on one change that real knowledge bases meet every month: new documents, a new version, a notice that expires and a document that only staff may see. You prove with measurements that the assistant still finds the right evidence, cites it, says “the documents do not say” when they do not, and forgets what it must forget.

Before you startWhat to know and to have readyAbout 300 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• Start from your own policy-assistant at the end of Module 6, or copy the course’s snapshot end-of-m06 with the steps of Use a snapshot. Run python -m pytest first: it must show 90 passed, 1 deselected before you change anything.
• Some tests pin the course’s own collection and question set, for example 36 documents and 67 questions. They fail after Parts 1 and 2, as expected. Update their numbers, and list in your report each test that you changed and why.
• Part 5 needs live model calls, because recorded answers cannot answer new questions. Use a free local model with Ollama, or your own key, as the project README describes. No payment is required.
• Tested with Python 3.12 and the versions in requirements.txt, on macOS.

The change: three new documents

Mei brings three documents from Grace’s team. You write them yourself, from the facts below, in the same Markdown format and front matter as the other documents in corpus/documents/:

1. Holiday returns notice (public, version 1, effective from 2026-11-20 to 2027-01-31, the last day for returns, as in the earlier holiday notice). Items bought between 20 November 2026 and 15 January 2027 can be returned until 31 January 2027, even when the normal return window has ended. It does not apply to gift cards or to custom-cut products.
2. Returns policy, version 5 (public, effective from 2027-02-01). The same as version 4, except one new rule: opened large appliances (pressure washers, lawn mowers, generators) have a restocking fee of 10% of the price, unless the item is faulty.
3. Suspected fraud procedure (staff, version 1, effective from 2026-11-01). What an agent does when a return looks fraudulent: do not accuse the customer, do not refund, open a case for the payment team, and reply with the neutral text in the document.

Grace asks one question that decides whether the update ships: “A customer bought a lawn mower that was delivered on 25 November 2026, and asks on 10 January 2027 whether they can still return it. Version 4’s 45 days ended on 9 January; only the holiday notice allows it. Will the assistant quote the holiday notice, and will it never show the fraud procedure to a customer?”

The documents and questions are synthetic. No real customers.

Part 1: write and ingest the documents

Uses Preserve metadata and Updates and deletions.

1. Write the three documents with complete front matter (doc_id, title, version, effective_from, effective_to where it applies, owner, access, language and doc_type). Version 5 keeps the doc_id of the returns policy, in a new file such as returns-policy.v5.md.
2. Retire version 4 on the day before version 5 starts, as in the module practice of Operational handoff. Version 4 is the HTML file returns-policy.v4.html, so add the line <meta name="effective_to" content="2027-01-31"> under its effective_from line. Without it, both versions are in force from 1 February 2027. Add a row for each new document to corpus/inventory.csv, and set the effective_to of version 4 there too.
3. Run python -m policy_assistant ingest. Show that only the new and changed documents were processed, and show info before and after.
4. Show with chunks that every new chunk has a stable ID, its document, version, section and access label, and trace three of them back to a line in the source file.

Part 2: extend and freeze the question set

Uses Corpus and questions and Retrieval measures.

1. Write 12 new questions, Q68 to Q79, as new lines of questions/questions.jsonl. Give each line every field that the other lines have: id, question, language, kind, as_of, public_ok, abstain, reference_answer, keys, must_not, relevant, relevant_chunks, wrong_version, tags and notes. The project refuses a line with a missing field. Copy a similar question as a model, for example Q22 for a date question or Q51 for a staff-only question. Write at least 3 with an “as of” date that the holiday notice or version 5 decides (one on 10 January 2027, one on 15 February 2027), 2 that only the restocking-fee rule answers, 2 that the documents cannot answer, 2 staff-only questions about the fraud procedure, 1 in French, and 2 ordinary questions. Give each its reference answer and relevant passages, and whether a public agent may see the answer.
2. Freeze the set: commit it (or save a copy with the time) before you run any measurement on it. Write one sentence on why, from Answer measures.

Part 3: retrieval, measured fairly

Uses Lexical, dense and hybrid search, Reranking and query changes and Find where it fails.

1. Run eval on all 79 questions for the course’s default (rerank, top 5) and one other setting that you choose and justify, for example --k 10 or --method dense. Change one thing only. eval --k 1 gives hit@1.
2. Report hit@5 (or recall@k), MRR and hit@1, overall and for your 12 new questions, and list every new question that each setting misses, with the rank of its first relevant passage. eval reports only the whole set and each kind. For your 12 questions, change the question IDs in the Python loop of Find where it fails: it prints the rank of the first relevant passage and recall@5.
3. For the date questions, show that the date filter returns the version in force on the question’s date, and what happens without the filter.

Part 4: access, before the model sees anything

Uses Permissions and isolation.

1. For a public audience, show that no fraud-procedure chunk reaches the context of any question, including your staff-only questions, and that those questions are answered with “the documents do not say” or a referral, not with staff text.
2. Show the same questions for a staff audience, and that the answers then cite the procedure. The answers in this part need your live model, as in Part 5.
3. Explain in two or three sentences why filtering after retrieval would not be enough, with your own numbers.

Part 5: grounded answers and abstention

Uses Assemble context, Answer and cite and Handle difficult evidence.

1. Run ask on your 12 questions with your live model, for the default setting.
2. For each answer, report what the citation checks found, and score it by hand for correctness, support and citation accuracy, as in Answer measures.
3. Show at least one abstention that is right, and explain at least one answer that is wrong, with the stage where it failed.

Part 6: the notice expires

Uses Updates and deletions and Operational handoff.

1. Remove the holiday notice from the collection, as Mei would after it ends, and ingest again.
2. Prove with search, chunks and ask that no index entry or new answer still returns or cites its passages, and that your 10 January 2027 question now abstains or answers from the version in force, as your documents say.
3. Run audit. It checks only the course’s recorded answers, not your live ones. So keep the output of your Part 5 ask runs in a file, and list the answers that cite a passage of the notice. In a real assistant, these are the cached answers to drop.
4. Update the handoff record (info and the README): document fingerprint, models with revisions, prompt version, costs and the rebuild command.

What to submit

Your policy-assistant folder (without .env, the index and the virtual environment) with the new documents and questions and REPORT.md:

# Policy assistant: holiday notice, returns v5, fraud procedure
## Documents and ingestion        (Part 1: front matter, incremental ingest, traced chunks)
## Question set                   (Part 2: the 12 questions, frozen before measuring, and why)
## Retrieval                      (Part 3: two settings, one change, the measures and the misses)
## Access                         (Part 4: public vs staff, and why filtering first matters)
## Answers and abstention         (Part 5: citation checks and hand scores, one right abstention, one failure by stage)
## Deletion                       (Part 6: proof that the notice is gone everywhere)
## Handoff and limits             (versions, costs, rebuild command; synthetic data, one live model, what you did not test)

Check your work

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a reason. To complete the course, all four required items must be demonstrated.

• Traceable chunks and citations (required): every new chunk traces to its source line; every citation in the answers is checked, and wrong ones are reported.
• A retrieval evaluation set (required): 12 new questions with relevant passages, frozen before measurement; two settings that differ in one thing, measured on all 79 questions.
• Evidence-based abstention (required): unanswerable and staff-only questions are answered with “the documents do not say” for the public audience, and at least one right abstention is shown.
• Update and deletion behaviour (required): incremental ingestion, the version in force by date, and proof that the expired notice no longer appears anywhere.

There are no key numbers to compare for your new questions: they and your live model are yours. The course’s own results for the 67 original questions are in Retrieval measures and Answer measures.

Recap

You maintained a knowledge base the way a team must: new documents with clear versions and access labels, a frozen question set, one change at a time, access enforced before the model sees anything, answers that cite or abstain, and deletions that really delete. If you have not done one yet, choose a case study for the certificate: Answer questions from public guidance, Search technical documentation or Answer from versioned regulations.