Reflection and extensions

What you will be able to do

Look back at the project to learn from it, and plan what could come next without breaking what works. Write what went well, what did not, and what you would do differently, each with evidence. Then propose one optional extension, with the course to take first and the evidence that it would need.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• The README, the portfolio checklist and the demo script, from Repository handoff and Demo narrative.
• The error categories and why the held-out set is used only once per compared version, from Inspect error categories and Run held-out tests.
• The marking rubric of the final assignment: docs/marking-rubric.md in end-of-m06.

What you need on your computer

• The copy of end-of-m06 from the last lesson, with the database running and the pack loaded. Only one command in this lesson needs it.

Tested on macOS (Apple silicon).

The match video

A football team plays on Saturday. On Monday, the players and the coach watch the video of the match together. They do not argue about who was to blame. They stop the video at three kinds of moment. Here, the passing worked: keep it. Here, we lost the ball four times in the same way: why? Here, we would do it differently next time: what exactly will we train this week?

Each moment has evidence: the video, the clock, the numbers of passes. “We played badly” is not a moment; “we lost the ball four times on the left side, after a long pass” is. Only after this do they talk about new players for next season. A new striker does not help a team that loses the ball in defence.

The problem: two bad reflections and one shiny extension

At the end of the project, Sam writes a first draft of the reflection in five minutes:

“The project went well overall. I learned a lot about RAG, auth and Docker. Some things were hard because of time. Next, I will add an agent, because agents are the future.”

Kwame reads it and asks three questions. What went well, exactly, and how do you know? What was hard, and what would you change? And the agent: what would it do, what must you learn first, and how would you know that it works?

The last sentence is the dangerous one. Tomás has asked for an agent since Module 1. Suppose the agent drafts and sends replies to customers by itself. Look at what the assistant does today with held-out question B34, asked at Bramble Books:

Ines (owner) at Bramble Books asks: How much goodwill credit can an agent give at Larkfield?
Answer (answerable): An agent can give up to $15 in goodwill credit, once per customer per year.

The right answer is “Bramble’s documents do not describe Larkfield”. Today, a member of staff reads this answer and decides. An agent that acts alone would send it. A known weakness of the core becomes a wrong action in front of a customer. The extension adds risk before the core is ready to carry it.

A second trap hides in the word “next”. Sam now knows the five held-out questions that failed. Suppose Sam changes the prompt until B34 passes, then runs the held-out set again and reports 37 of 41. That number is no longer honest: the held-out set helped to choose the change, so it has become a tuning set.

The idea: a retrospective with evidence, then a gated extension

A retrospective is a planned look back at finished work, to improve the next piece of work. The name comes from the Latin for “looking back”. It asks three questions, and every answer needs evidence:

1. What went well, and why? Keep it in the next project.
2. What did not go well, and what caused it? Find the cause, not a person to blame.
3. What would you do differently? One concrete change, small enough to try next time.

A retrospective is not a list of excuses, and it is not a score. Its value is in the third answer: a change that you can act on.

An extension is optional work that adds a new ability after the core works. In this course there are three kinds: a custom model, an agent and an MCP server. Each has its own course with skills that this course does not teach. So every extension proposal has a gate before it, and three parts after it:

The extension gate. An extension starts only when the four required items are demonstrated, and each one names its prerequisite course and the first evidence it must produce.

Text description of the diagram

The diagram starts with a question: are rubric items 1 to 4 all demonstrated? If not, the arrow leads to “fix the core first”: an extension never replaces a required item. If yes, the arrow leads to “choose one extension”, which has three branches. A custom model needs the model-serving course first, and its first evidence is the same held-out report, the cost and the p95 time on the same laptop. An agent that drafts replies needs the agents course, and its first evidence is a test that it never acts without approval. An MCP server for the search needs the MCP course, and its first evidence is the isolation checks run through the tool, with 0 leaks.

The prerequisites are the skills that you must have first, and the course that teaches them. The first evidence is the one check that would show the extension works and keeps the core safe. Writing it before you build anything is the same habit as the evidence plan of Module 1.

A tiny example: one lesson, three ways

Take one thing that went wrong for Sam: the first evaluation scores looked worse than the answers were. Write it as a retrospective line three ways:

Version	Text	Can Sam act on it?
An excuse	“The evaluation was unfair because the keys were bad.”	No. It blames the tool and hides the cause.
Vague	“Evaluation is hard; I need to be more careful.”	No. “More careful” is not a change.
Useful	“Some keys were stricter than the reference answers: Q46 wanted ‘5 to 10’ and the answer said ‘5–10’. Next time, I will read every failure before I trust a score, and test each key against its reference answer.”	Yes: a cause, evidence, and one change to try.

The useful line names the evidence: Q46 and the two spellings. It names the cause: a key stricter than its reference answer. And it names one change: test each key against its reference. The football team would stop the video at that moment.

The worked capstone: Sam’s reflection

Sam rewrites the draft as docs/reflection.md, in end-of-m06. Each line now has its evidence.

What went well. Writing the acceptance criteria and reserving the held-out questions in the first week. Every later argument (“is it good enough?”) had a number to point at. Tomás’s “skip the held-out set, we are short of time” also had an answer: reserving it took ten minutes, in week one. Second, one complete journey before any extra feature: the mock adapter made the page, the upload and the job states work before any model was involved. Third, isolation checked in four layers: search, access level, API and database. Row-level security found nothing new, which is the point of a second lock.

What did not go well. The strict keys, as in the tiny example. And the local models need about 1 GB of downloads and most of the image’s size. So the demo is slow to start on a new computer.

What I would do differently. Write the four failure cases as tests on the first day of Module 4, before the code that handles them. And keep a short decision log from the first week: one line per decision, with its date and its evidence. At the end, Sam had to find several decisions again in the commit history.

Where the remaining failures are

A retrospective also looks at what is still wrong, to choose the next work. Sam runs the evaluation once more, with the failures shown. The answers are recorded, so this repeats the report’s first pass exactly:

python -m scripts.evaluate --set heldout --show-failures

hybrid, answer_v2, recorded: 41 questions | correct 0.878 | retrieval hit 0.886 | citations valid 1.000 | abstained when it should 0.833 | injection obeyed 0 of 2 | median 9 ms | US$0.000167 per question
  answerable       12  correct 0.833  hit 0.833
  contradiction     3  correct 1.0  hit 1.0
  exact-code        4  correct 1.0  hit 1.0
  injection-bait    2  correct 1.0  hit 1.0
  other-shop        2  correct 0.5  hit None
  staff-only        5  correct 1.0  hit 1.0
  table             5  correct 0.8  hit 0.8
  unanswerable      4  correct 1.0  hit None
  version           4  correct 0.75  hit 0.75
  WRONG Q13 (larkfield, answerable, hit False): The documents do not say whether Larkfield sells extended warranties.
  WRONG Q28 (larkfield, version, hit False): The documents do not say whether a Rewards member pays for a return label in this case.
  WRONG Q37 (larkfield, table, hit False): Les documents ne précisent pas la durée de la garantie d’un barbecue.
  WRONG B12 (bramble, answerable, hit False): The documents do not say whether we can ship a book to a customer in Seattle.
  WRONG B34 (bramble, other-shop, hit None): An agent can give up to $15 in goodwill credit, once per customer per year.

• Retrieval missed the passage
• Answered another shop's question
• No failure

The kept version on the 41 held-out questions (first pass), by kind: answer = answerable, shop = another shop's question, others = the five kinds with no failure. Four of the five failures are retrieval misses; the fifth is B34.

Text description of the chart

A column chart of the share of held-out questions answered correctly, by kind. Answerable questions: 83% (10 of 12; Q13 and B12 failed). Version questions: 75% (3 of 4; Q28 failed). Table questions: 80% (4 of 5; Q37 failed). These three columns are the same colour: in each failure, retrieval did not find the right passage. Questions about another shop: 50% (1 of 2; B34 failed), in a second colour. The five other kinds together, exact codes, contradictions, staff-only, injection bait and unanswerable questions: 100% (18 of 18), in a third colour.

Four of the five failures have hit False: the right passage never reached the model. The answers to them are honest refusals, not wrong facts. The fifth, B34, is the opposite: a confident answer that should be a refusal. Sam writes both into the reflection’s next steps: French tuning questions (Q37), and a prompt rule for questions that name another shop (B34).

Sam also writes the rule that protects the next round. These five questions are now known, so they cannot measure a fix for themselves. Before any change, Sam will write new tuning questions like them, and tune on those. A new held-out set measures the next version. Avoid contamination in the evaluation course explains why.

Sam’s extensions

Sam proposes the three extensions in a table, and builds none of them in this course:

Extension	What it adds	Course to take first	First evidence it would need
Custom model	a small model served locally for the answer step, to cut cost and keep documents on the computer	Deploying AI Models for Real Users	the same held-out report, the cost per question, and the p95 time on the same laptop
Agent	drafts a reply to the customer and asks staff to approve it; it never sends alone	AI Agents and Workflow Orchestration, especially Approval boundaries	a test that it never acts without approval, and the held-out report of the drafts
MCP server	the assistant’s search as a tool for other AI clients, with the same per-shop access checks	MCP: Connect AI Applications to Tools and Data, especially Identity and permission	the isolation checks run through the MCP tool: 0 leaks

Look at the agent’s row. Tomás’s idea comes back, but in a safe form: the agent drafts, and a person approves. Its first evidence is a test of exactly the risk that B34 showed. Kwame accepts the table. He also asks Sam to rank them: Sam chooses the MCP server first, because its first evidence reuses the isolation checks that already exist.

Guided practice: the module task, step by step

The module ends with one deliverable: your product, ready for someone else. First see it for the worked capstone, with every part visible.

1. Open the four files of the handoff in end-of-m06: README.md, docs/portfolio-checklist.md, docs/demo-script.md and docs/reflection.md.

2. Open docs/marking-rubric.md. It has four required items: a working user journey; a reproducible repository and deployment; held-out evaluation and isolation checks; and a clear demo, architecture and limitations.

3. For each item, find the evidence in the four files. Item 1: the README’s Quick start and scripts/smoke.py (7 of 7). Item 2: the stranger test and the CI workflow. Item 3: the evaluation report and scripts.isolation (0 leaks). Item 4: the demo script, the design pack and the Limitations section.

4. Read Sam’s final report in the box below. It is one page that a reviewer reads first.

Sam’s final report (worked example)

Final report: the document assistant (one page)

## What it is
A document assistant for two shops (Larkfield, Bramble Books): answers from each shop's own
documents with a citation for every claim, or a clear "no answer". Synthetic documents (CC0).

## Required items, with evidence and my own mark
1. Working user journey: demonstrated. README Quick start; scripts/smoke.py: 7 of 7 checks passed;
   screenshots of upload, cited answer, no answer, read-only refusal.
2. Reproducible repository and deployment: demonstrated (local Compose; cloud steps written, not run).
   Stranger test on a fresh copy: 108 tests passed; CI workflow; rollback drill in docs/runbook.md.
3. Held-out evaluation and isolation: demonstrated, with one criterion not met. 36 of 41 correct,
   citations 100%, 0 of 2 injections followed; isolation 0 leaks. Abstention 5 of 6: B34 fails
   (Q2 not met), named in the report.
4. Demo, architecture, limitations: demonstrated. docs/demo-script.md (failure case and trade-off),
   docs/design-pack/, README Limitations.

## Completed and planned
Completed: the four items above. Planned, not done: cloud deployment, French tuning questions,
a fix for B34 with a new held-out set, the MCP extension (docs/reflection.md).

Notice item 3. Sam marks it “demonstrated” because the rubric asks for an honest held-out evaluation and isolation checks with 0 leaks, and both are there. The unmet threshold is reported, not hidden. A reviewer may still ask about B34, and Sam has the answer ready.

Independent task: hand over your product

Deliver your own product, ready for a reviewer who has never seen it. Submit four things:

1. The repository, with the handoff README of Repository handoff, the portfolio checklist, and the stranger test passed from a clean folder.
2. The demo script, docs/demo-script.md, rehearsed, with a failure case and a trade-off, as in Demo narrative.
3. The screenshots plan: one real screenshot of the running product for each step of the demo, with the step it belongs to.
4. The final report: one page, like Sam’s. For each of the four rubric items, the evidence and your own mark (demonstrated, partially demonstrated or not yet demonstrated) with a reason. Then the completed and planned work.

Also write docs/reflection.md: three retrospective answers with evidence, and one proposed extension with its prerequisite course and first evidence.

How a reviewer checks your submission (worked solution)

• The repository: the reviewer clones it into a new folder and follows the README. Every check prints what the README says. git check-ignore -v .env lists .env. A failure here makes item 2 “not yet demonstrated”, whatever else works.
• The demo script: five parts, five minutes or less. The failure case and the trade-off are present. Every question was rehearsed (the rehearsal outputs show recorded or live, never mock). Without the failure case or the trade-off, item 4 is “partially demonstrated”.
• The screenshots plan: each step has a real screenshot of the running product. A drawn or edited screen is a claim without evidence.
• The final report: each mark matches the evidence. “Demonstrated” with a leak in the isolation checks is wrong: a failed access-control check makes item 3 “not yet demonstrated”. An unmet quality threshold, reported honestly, does not.
• The reflection: each line names evidence and one change. The extension names a course to take first and a first check, and nothing in it is described as done.

Completion check: explain it yourself

The outline’s completion check for this module: explain the implementation independently, and separate completed work from planned extensions. Answer these without your notes, then open the worked answers.

Worked answers for the worked capstone

• What happens when Ines asks a question? The API checks her session, reads her membership in Bramble Books from its own table, and finds her role. The pipeline searches only Bramble’s passages that her role may read and that are in force on the date. It uses vectors and keywords, fused. It sends the top 5 passages and the prompt to the provider (recorded by default), checks that every citation is one of those passages, stores the answer, and returns it.
• Why can Ines never see Larkfield’s passages? Every query has Bramble’s tenant_id; row-level security in PostgreSQL checks it again; the search filters by shop and access level; and a citation lookup uses the same filters. scripts.isolation tests search, the access level, the API and the database: 0 leaks.
• What is completed, and what is planned? Completed: one journey for two shops, sign-in and roles, row-level security, failure handling, deletion, the held-out evaluation, the local Compose demo with CI, limits and a tested rollback. Planned, not done: the cloud deployment (written, not run), French tuning questions, a fix for B34, and the extensions.

Then answer the same three questions for your own product, aloud, to a friend or into a recording. If you cannot explain a part, read that code again until you can. The rubric marks “the learner cannot explain their own decisions” as “not yet demonstrated”.

Your project

1. Write the retrospective, docs/reflection.md: what went well, what did not, and what you would do differently. Give each line its evidence, and give the last answer one concrete change.
2. Look at your remaining failures with your evaluation’s failure list. Write the next steps, and the rule for the next round: new tuning questions, and a new held-out set before you measure a fix.
3. Propose one extension only if your four required items are demonstrated. Name what it adds, the course to take first, and its first evidence. Do not build it in this course.
4. Write the final report, and explain your implementation aloud, as in the completion check.

Checklist

• Each retrospective line names evidence (a question ID, a number, a commit), not a feeling.
• “What I would do differently” has at least one change small enough to try in the next project.
• The next steps do not reuse the current held-out set to measure a fix.
• The extension names its prerequisite course and its first evidence, and is listed as planned, not done.
• The final report gives each rubric item a mark and a reason, and its marks match the evidence.
• You can explain how one question flows through your product, and why one shop never sees the other’s data.

Common mistakes

Common mistake

The extension before the core

Symptom: the project has an agent or an MCP server, but the stranger test fails, or the isolation checks were never run.

Cause: the extension looked more impressive than the core. Tomás asked for “an agent first” in Module 1.

Fix: stop the extension and finish the core until the four rubric items are demonstrated. An extension is never marked instead of a required item.

Consequence in AI work: an agent multiplies the weaknesses of what it calls. B34’s wrong answer, sent by an agent without approval, becomes a wrong promise to a customer.

Common mistake

Fixing the held-out failures and reporting the new score

Symptom: the next version reports 37 or 38 of 41 on the same held-out questions, after a change that targets exactly the questions that failed.

Cause: the failures were read, and the change was tuned until they passed. The held-out set took part in the choice.

Fix: write new questions of the same kinds, tune on those, and reserve a new held-out set before you measure. Report the old held-out score as the score of the old version.

Common mistake

A reflection that is a list of excuses

Symptom: every line explains why something was not your fault: the time, the tools, the data.

Cause: the reflection is written for a grade, not to improve the next project.

Fix: for each problem, name the cause that you control and one change that you will try. A good reflection makes the reader trust you more, because it shows what you learned.

Knowledge check

Check your understanding

What are the three questions of a retrospective, and what turns an answer into a useful one?

Show the answer

What went well, and why? What did not go well, and what caused it? What would you do differently? An answer is useful when it names evidence (a question ID, a number, a commit) and a cause, and when the last answer gives one concrete change that you can try. “I need to be more careful” is not a change; “test each answer key against its reference answer” is.

Check your understanding

Tomás proposes an agent that answers customers' emails by itself, using the assistant. Using B34, explain the risk, and rewrite the proposal so that it passes the extension gate.

Show the answer

B34 shows that the assistant answers some questions about another shop with its own shop’s numbers. Today, staff read every answer before they act. An agent that sends alone would send the wrong $15 to a customer. Rewritten: only after the four required items are demonstrated, an agent drafts a reply and a member of staff approves it; it never sends alone. Prerequisite: the agents course (approval boundaries). First evidence: a test that it never acts without approval, and the held-out report of the drafts.

Check your understanding

Predict: Sam changes the prompt until B34 is answered correctly, runs the held-out set again and gets 37 of 41. What is wrong with reporting '37 of 41 on held-out questions'?

Show the answer

The held-out set took part in the choice: Sam read its failures and changed the prompt until one passed. Its score is now optimistic, like a tuning score, and it no longer estimates new questions. Sam should write new questions like B34, tune on them, and reserve a new held-out set to measure the change. The honest held-out score of the current version stays 36 of 41.

Check your understanding

Module check (completion): for the worked capstone, explain in three sentences how a question becomes an answer, and list two completed and two planned items.

Show the answer

The API checks the session and reads the member’s role from its own membership table, never from the token. The pipeline searches only the shop’s passages that the role may read and that are in force on the date, with vectors and keywords, fused. It sends the top 5 with the prompt to the provider, checks every citation against those passages, and stores the answer. Row-level security checks the shop again in the database. Completed: for example, row-level security with 0 leaks in the isolation checks, and the held-out evaluation (36 of 41). Planned, not done: for example, the cloud deployment (steps written, not run) and a fix for B34 with a new held-out set.

Recap

• A retrospective asks what went well, what did not, and what you would do differently, each with evidence and one change to try.
• The remaining failures guide the next steps, but the held-out set cannot measure a fix for its own failures. Write new questions, and reserve a new held-out set.
• An extension is optional and comes only after the four required items are demonstrated. Each proposal names what it adds, the course to take first, and its first evidence.
• The module’s deliverable is the repository, the demo script, the screenshots plan and a one-page final report, and you can explain all of it yourself.

Module recap. You made the project work without you. A README answers five questions, and a portfolio checklist links every claim to its evidence. A rehearsed five-minute demo shows a failure and a trade-off. An honest reflection ends with a gated extension.

Course recap. You scoped one product for one user, reserved held-out questions before any tuning, built one complete journey, and made it safe for two shops. You measured it honestly, including a change that did not prove better and a criterion that is not met. Then you shipped it as a reproducible demo and handed it over with its evidence. Next, a case study applies the same method to a public document collection.

Next: Capstone on a public document collection