Repository handoff

What you will be able to do

Make your repository usable by a person who has never seen it. A stranger must be able to run it and understand how it works. They must see where the data comes from, how good it is, and what it does not do. Every claim in it points to evidence.

Before you startWhat to know and to have readyAbout 80 minutes

What you need to know

• The evidence plan and its list “What I will not claim”, from Plan the evidence.
• The evaluation report, from Document trade-offs.
• The Compose demo, the smoke test and the runbook, from Release the demo and Recovery and cost control.
• How .gitignore keeps secrets out of Git, from Keep the repository clean in the Git course.

What you need on your computer

• Your project at the end of Module 6. For the worked capstone, use the snapshot end-of-m06, the clean copy that Module 6 worked on: python -m pytest prints 108 passed.
• Docker running, about 5 GB of free disk, and room for a second copy of the project for a clean test.

The snapshot end-of-m06 already contains Sam’s documents of this module: README.md, docs/portfolio-checklist.md, docs/demo-script.md and docs/reflection.md. Module 7 adds no code. You read Sam’s versions and write your own. Tested on macOS (Apple silicon); the Windows (PowerShell) commands are written but not tested.

A recipe for a friend

Your friend loved your soup and asks for the recipe. You know it by heart, so you write three lines: “Onions, carrots, stock. Cook until done. Add salt.” Your friend tries it, and it fails. Which stock? How many carrots? How long is “until done”?

A good recipe card answers the cook’s questions before they ask. It lists what you need, with amounts. It gives the steps in order, and says why one step matters: “fry the onions first, or the soup tastes raw”. It says where a special ingredient comes from. It says how the soup tastes when it is right. And it says what does not work: “do not freeze it; the potatoes go soft”.

The person who wrote the recipe cannot see these gaps. They know the answers, so the short card looks complete to them. Only a person who cooks from the card finds what is missing.

The problem: the project works for Sam

Sam’s project passes every test. The Compose demo starts, the smoke test passes, and the evaluation report is written. But much of it lives in Sam’s head and on Sam’s laptop. Omar wants to show the project to a hiring manager next week. Amira will run it on her own computer first. Neither of them was there when Sam made each choice.

Sam asks Amira to read the README before she runs anything. She soon finds a real problem. The settings table said this:

| `RETRIEVAL_MODE` | `hybrid_rerank` | `dense` (vectors only) or `hybrid_rerank` (vectors + keywords, fused, reread by a cross-encoder) |

The course team found the same gap while writing this lesson, with one grep across the README and the code:

grep -n "RETRIEVAL_MODE" README.md .env.example compose.yaml doc_assistant/config.py

README.md:65:| `RETRIEVAL_MODE` | `hybrid_rerank` | `dense` (vectors only) or `hybrid_rerank` (vectors + keywords, fused, reread by a cross-encoder) |
.env.example:22:RETRIEVAL_MODE=hybrid
compose.yaml:64:      RETRIEVAL_MODE: ${RETRIEVAL_MODE:-hybrid}
doc_assistant/config.py:92:        retrieval_mode=get("RETRIEVAL_MODE", s.retrieval_mode),

The default in config.py is hybrid too. The README described an older plan. In Module 5, Sam measured the reranker and did not keep it, because it was slower and not better. Nobody updated the table. A reviewer who read it would believe that a cross-encoder rereads every search. They would expect about 0.4 s more per question, and then find numbers in the report that do not match.

No test failed, and no error appeared. A document can be wrong while the code is right, and only a reader finds it. We say that the documents drift away from the code.

The idea: write the handoff for a stranger

A repository handoff is the moment when the repository must work without its author. Its main part is the README, the first file that every reader opens. A handoff README answers the five questions of any new reader, in this order:

1. Can I run it? Setup: what to install, with versions; the commands, in order; how to check that it works; how to reset it.
2. How does it work? Architecture: one picture of the parts, and the settings that change its behaviour.
3. Where does the data come from? Data provenance: who made the documents, under which licence, which models it uses, and what was recorded.
4. How good is it? Evaluation: the headline numbers, and a link to the full report.
5. What does it not do? Limitations: what was not tested, the known failures, and what is planned but not done.

These are the recipe card’s five kinds of line. Each answer points to evidence that the reader can open or run.

A handoff README answers five reader questions. Each section ends in evidence that the reader can open or run (green).

Text description of the diagram

The diagram has three columns. On the left are five questions of a new reader. Each question has an arrow to the README section that answers it, and then to the evidence. “Can I run it?” leads to the Quick start, and then to two commands: pytest, which prints 108 passed, and scripts.isolation, which must print 0 leaks. “How does it work?” leads to the Architecture section and the design pack. “Where does the data come from?” leads to Data provenance: the data card, the checksums and the lock file of the held-out questions. “How good is it?” leads to Evaluation and the evaluation report. “What does it not do?” leads to Limitations and status, and to the portfolio checklist.

Why does this matter for a portfolio? A hiring manager gives your repository a few minutes. They will not run every test. They read the README, perhaps run one command, and look for one honest limitation. A README that answers the five questions, with evidence, shows the skill that the job needs.

A tiny example: one claim, three ways

The limitations and the evaluation are the hardest sections to write, because each line is a claim. Take one claim about the worked capstone, and write it three ways:

Version	Text	What a reviewer can do with it
Unsupported	“Secure multi-tenant assistant, production-ready.”	Nothing. “Secure” and “production-ready” have no test. Sam’s evidence plan forbids both words.
Vague	“Tenants are isolated.”	Ask “how do you know?” The answer is not in the repository.
With evidence	“A shop never sees the other shop’s data: python -m scripts.isolation checks search, the API and the database, and finds 0 leaks (430 probes in the report’s run).”	Run the command, and see 0 leaks in a few seconds.

The third version is longer, but every part of it can be checked. It names the claim, the command, the result and the size of the test. It also says “shop”, not the team’s word “tenant”, because the reader is not on the team.

The same rule applies to a failure. “Some edge cases could be improved” is vague. “Asked at Bramble Books about Larkfield’s goodwill credit, it answers with Bramble’s own limit (held-out question B34)” is specific. The reader learns exactly what to watch.

The worked capstone: Sam’s README, section by section

Open README.md in your copy of end-of-m06. It has about 1,300 words. Read it with the five questions in mind.

What it does, then Quick start. The first paragraph says what the product does, for whom, in two sentences. The second says what it costs to run: nothing, because the answers are recorded and the search runs on the CPU. “What it does” lists the user journey in four lines. The Quick start begins with what you need: Python 3.12 or newer, Git, Docker, and about 5 GB of disk. Then come the commands, each with a short comment that says why it is there:

README.md (Quick start, part)

cp .env.example .env                    # then change POSTGRES_PASSWORD (and the two URLs) in .env
python -m idp init                      # the practice identity provider's keys; writes two secrets to .env
docker compose up -d --wait db          # PostgreSQL with pgvector; --wait: until it is ready
python -m scripts.load_pack             # the two shops, their people and their 51 documents (downloads the models once)

The section also gives the Windows differences, the five people to sign in as, three checks from the terminal, and two ways to reset. The reset matters: a reviewer who breaks the data must be able to start again with one command.

Architecture. One paragraph, one text picture, and a table of the settings that change behaviour. It does not repeat the design pack. It links to docs/design-pack/architecture.md and docs/design-pack/threat-and-cost.md for the details. The drift was in this section. The fixed row now reads:

| `RETRIEVAL_MODE` | `hybrid` | `dense` (vectors only), `hybrid` (vectors + keywords, fused), `dense_rerank` or `hybrid_rerank` (then reread by a cross-encoder: measured, not better here, about 0.4 s slower) |

The text picture changed too: it says “retrieve (vectors + keywords, fused)”, not “reranked”.

Data provenance. Every document, shop and person is made up for the course, under CC0, and data/dataset.md is the data card. The section names the two local models with their licences, and says that the answers in recordings/ came from chat-small. It says that people wrote the 103 questions, and that 41 were reserved before any tuning. A reviewer who asks “did the model write its own test?” finds the answer here.

Evaluation. One paragraph of headline numbers, then a link to the full report. Sam added the paragraph during the handoff, so that a reader knows how well it works without opening a second file:

README.md (Evaluation, first paragraph)

On the 41 held-out questions (real answers of chat-small, recorded): 36 correct, every citation valid,
5 of the 6 questions that need a "no answer" refused, 0 of 2 hidden instructions followed, about 1.3 s and
US$0.00017 per question. The isolation checks found 0 leaks (430 probes in the report's run). The one
improvement, keyword search next to the vectors, gained on the tuning questions but not on the held-out
ones: it is kept because it is no worse at the same cost, not because it is better.

Look at the last sentence. It repeats the honest result of Document trade-offs, and does not round it up to “an improvement”.

Limitations and status. The section has six specific limits, one per line. They are: one tested platform, synthetic documents, recorded answers only for the course’s questions, the practice provider without passwords, the cloud steps not run, and strict answer keys. Sam added a seventh line with the known failures, B34 and Q37, by their question IDs. Then a short Status section separates completed work from planned work, and links the portfolio checklist and the demo script.

Check the README against the code

Drift comes back each time the code changes. So after each change, check the settings and the commands again. For the settings, grep the README together with .env.example, compose.yaml and config.py, as above. After the fix:

grep -n "RETRIEVAL_MODE" README.md .env.example compose.yaml

README.md:65:| `RETRIEVAL_MODE` | `hybrid` | `dense` (vectors only), `hybrid` (vectors + keywords, fused), `dense_rerank` or `hybrid_rerank` (then reread by a cross-encoder: measured, not better here, about 0.4 s slower) |
.env.example:22:RETRIEVAL_MODE=hybrid
compose.yaml:64:      RETRIEVAL_MODE: ${RETRIEVAL_MODE:-hybrid}

For the commands, there is only one real check: run them, in order, in a clean place. That is the guided practice below.

The portfolio checklist: every claim and its evidence

The README makes claims. The demo of the next lesson makes more, and so does your CV. The portfolio checklist, docs/portfolio-checklist.md, collects all of them in one table. Each row has the claim, the evidence and the kind of evidence. Sam uses three kinds:

• Live: the reviewer runs it on their own computer and sees the result. The smoke test, the isolation checks and the failure cases are live.
• Recorded: a real run, kept in the repository, that the reviewer can read but not repeat exactly. The held-out answers of chat-small are recorded. scripts.evaluate replays them, but no model answers again.
• In the history: a fact that Git proves. The held-out set was locked in a commit before any answer existed.

Sam's portfolio checklist as a picture: each claim points to evidence that is live (green), recorded (blue) or in the Git history (orange).

Text description of the diagram

On the left are seven of Sam’s claims. Three of them point to live evidence in green, which the reviewer runs: the Quick start with the smoke test, which prints 7 of 7 checks passed; the isolation checks, with 0 leaks; and the failure cases, with the statuses 404, 415 and 413, a failed upload and a 504. Three claims point to recorded evidence in blue: the evaluation report with 5 of 6 refusals and B34 named; the baseline and the candidate, two passes each; and the cost, from the token counts times the price list. The last claim, that the held-out questions were reserved before tuning, points to the Git history in orange: the lock file, committed before any answer.

The kind matters to the reviewer. A live claim is the strongest, because they see it happen. A recorded claim is honest only with its label. Suppose your demo replays recorded answers, and you let the audience think that a model answered just now. Then you claim something false. Sam’s table has a column for the kind, and the template adds one more: the date when you last checked the claim yourself.

Sam’s checklist ends with two lists: completed and planned (not done). A cloud deployment is in the second list. Its steps are written in docs/deployment.md, but nobody ran them, so it cannot move to the first list.

Guided practice: the stranger test

The only way to find a missing step is to follow the README as a stranger would. Use a new folder, a new environment, and nothing from your shell history. Do it for the worked capstone first. The outputs below are from the course team’s run on a fresh copy of end-of-m06.

1. Copy the snapshot to a new folder, as if you cloned it for the first time:

   macOS / Linux

   cp -R ../nextia-learning-labs/C27/snapshots/end-of-m06 ~/handoff-check
   cd ~/handoff-check

   Windows (PowerShell)

   Copy-Item -Recurse ..\nextia-learning-labs\C27\snapshots\end-of-m06 $HOME\handoff-check
   Set-Location $HOME\handoff-check

2. Open README.md. Follow its Quick start one line at a time. Use no command that the README does not give.

3. Write down every place where you hesitate. Each one is a missing line in the README.

4. If another PostgreSQL already uses port 5432, add DB_PORT=5433 to .env. Then use 5433 in the two URLs. The course team used another port for this reason.

5. After python -m idp init, read the output. It ends with these two lines:

   Wrote a new SESSION_SECRET (the API signs its session cookies with it).
   The provider keeps only the client secret's SHA-256. Never commit .env or .idp/.

6. After python -m scripts.load_pack, read the last lines:

   Indexed 51 documents in 8.4 s (failed 0).
     bramble: 15 documents, 60 passages, 0 failed
     larkfield: 36 documents, 185 passages, 0 failed

   The first run on a new computer also downloads the two models, about 1 GB, and takes longer.

7. Run the tests:

   python -m pytest

   ........................................................................ [ 66%]
   ....................................                                     [100%]
   108 passed in 38.29s

8. Ask the README’s question:

   python -m scripts.ask "How long is a gift card valid?" --tenant bramble

   The answer and its last line:

   Answer (answerable): A gift card is valid for 3 years from the date of purchase.
     - A gift card is valid for 3 years from the date of purchase. [Gift cards, How long they last, v1]
   ...
   recorded chat-small | 4117 ms | tokens 768 in, 63 out | US$0.000108 | trace b399659f3e72321038cfa8eaa85e9189

   The first question takes about 4 s, because it loads the embedding model.

9. Run the isolation checks:

   python -m scripts.isolation

   Search: 103 questions asked in the other shop -> 0 passages of the first shop
   Access: 11 staff-only questions asked by read-only members -> 0 staff-only passages
   API: 294 requests with the other shop's IDs -> statuses {404: 294}, 0 leaks
   Database: 8 unfiltered queries under row-level security -> 0 rows of the other shop
   Isolation checks: 416 probes, 0 leaks.

   The number of API requests grows with the answers stored in the database, so your count may differ. The number of leaks must be 0.

10. Make the folder a Git repository, and ask Git which files it ignores:

    git init -q && git add -A
    git check-ignore -v .env .idp

    .gitignore:2:.env  .env
    .gitignore:3:.idp/  .idp

    If Git prints nothing for .env, stop. Your secrets would go into the next commit.

11. Look for secret settings in the files that Git will commit:

    git grep -nE "^(LLM_API_KEY|SESSION_SECRET|OIDC_CLIENT_SECRET|POSTGRES_PASSWORD)="

    .env.example:5:POSTGRES_PASSWORD=change-me-on-your-computer
    .env.example:17:LLM_API_KEY=

    Only the example file matches, with a placeholder and an empty key.

12. Stop the database and delete its data with docker compose down --volumes. Then delete the test folder.

The course team’s run passed every step. On the own-idea track, the commands change, but the test is the same.

Your project

Now write the handoff of your own product. On the own-idea track, the five questions are the same; only the answers change.

1. Write the README with the five sections: Quick start (with versions, a reset path and three checks), Architecture (one picture and the settings), Data provenance, Evaluation (headline numbers and a link), and Limitations with a Status section. Start from Sam’s README, and replace every fact with yours.
2. Check it against the code. grep every setting in the README, .env.example and your settings file. If they differ, fix the README, or fix the code if the README says what you meant.
3. Fill the portfolio checklist. Copy docs/templates/portfolio-checklist.md to docs/portfolio-checklist.md. Write one row for each claim in your README, your demo and your CV. Mark each one live, recorded or in the history, with the date when you last checked it.
4. Run the stranger test. Clone your repository from your Git host into a new folder, and follow the README exactly. Better still, ask a friend to do it while you watch and say nothing.
5. Keep the evidence: the stranger’s notes (where they hesitated, and what you changed), the output of the three checks, and the commit that fixed the README.

Checklist

• A stranger ran the Quick start from a clean folder, and every check printed what the README says.
• The README names versions, a reset path, and the operating systems that you tested.
• Every setting in the README equals the default in the code.
• Data provenance says who made the documents, under which licence, and which models and recordings you use.
• Every claim in the README has a row in the portfolio checklist, marked live, recorded or in the history.
• The limitations name at least one known failure by its question or case, and separate completed from planned work.
• git check-ignore lists .env and every other file with a secret, and no private document is in the repository.

Common mistakes

Common mistake

The README describes last month's project

Symptom: a setting, a command or a number in the README is not what the code does. Sam’s README said hybrid_rerank; the code’s default was hybrid.

Cause: the README was written once and not updated when a decision changed. Tests check the code, not the text.

Fix: after each decision, grep the setting in the README, .env.example, compose.yaml and the settings file. Quote numbers from the report, not from memory. Run the stranger test again before you share the link.

Consequence in AI work: a reviewer who finds one wrong line stops trusting the others. In a team, the next person changes the wrong setting and measures the wrong system.

Common mistake

A step that lives only in your shell

Symptom: the stranger test stops at a step that works on your laptop. For example:

error while interpolating services.db.environment.POSTGRES_PASSWORD: required variable POSTGRES_PASSWORD is missing a value: Set POSTGRES_PASSWORD in .env

Cause: you did something once, by hand, weeks ago: you created .env, exported a variable, downloaded a model, or started a database. The README never mentions it, because you forgot that it was there. Here, the stranger skipped cp .env.example .env, and Compose refused to start a database without a password.

Fix: write the missing step into the README, at the place where the stranger needs it, and run the test again from a new folder. Always run it from the start, never from the middle.

Common mistake

Private documents in a public repository

Symptom: on the own-idea track, the repository contains real documents of your employer or of another real company, or recordings that quote them.

Cause: those documents were the easiest data to find, and the repository became public for the portfolio.

Fix: remove them from the repository and from its history, and use synthetic or openly licensed documents with a data card, as in Synthetic data pack. Recordings contain the passages that the model saw, so record them again. Ask the owner before you publish anything that is not yours.

See it end to endCapstone on a public document collectionCase study

The case study ends with a portfolio README: the held-out result and the recommendation in the first lines, then the three wrong answers by question ID, and limitations that are specific to the collection.

See it end to endReview a weak capstoneCase study

The case study checks a README’s claims against the repository’s own commands. “97% accurate” was the tuning set, “complete tenant isolation” met 40 leaks, and a deleted .env was still in the history.

Knowledge check

Check your understanding

Name the five questions that a handoff README answers. For the worked capstone, give one piece of evidence for each.

Show the answer

Can I run it: the Quick start, checked by python -m pytest (108 passed) and python -m scripts.isolation (0 leaks). How does it work: the Architecture section and the design pack in docs/design-pack/. Where does the data come from: data/dataset.md (synthetic, CC0), SHA256SUMS and evaluation/heldout.lock. How good is it: the evaluation report and python -m scripts.evaluate --set heldout (36 of 41 correct, recorded answers). What does it not do: the Limitations and Status sections and docs/portfolio-checklist.md, with B34 named and the cloud deployment planned, not done.

Check your understanding

Predict: Sam's README said that the default retrieval mode was hybrid_rerank, and every test passed. Why did no test catch it, and what would a reviewer probably conclude?

Show the answer

The tests check what the code does, and the code was right: its default was hybrid. Nothing checks the words of the README. A reviewer would believe that a cross-encoder rereads every search, and would expect about 0.4 s more per question. Then they would find numbers in the report that do not match, and stop trusting the README. The checks are a grep of the setting across the README, .env.example, compose.yaml and config.py, and the stranger test.

Check your understanding

Sort these three claims into live, recorded or in the history, and say why: (a) 36 of 41 held-out questions correct; (b) 0 leaks across the two shops; (c) the held-out set was reserved before any tuning.

Show the answer

(a) Recorded: the answers are real answers of chat-small, kept in recordings/. scripts.evaluate replays them, but no model answers again. (b) Live: the reviewer runs python -m scripts.isolation on their own computer and sees 0 leaks. (c) In the history: evaluation/heldout.lock was committed before any answer was generated, and Git shows the order of the commits.

Recap

• A repository handoff makes the project work without its author. The README answers five questions in order: can I run it, how does it work, where does the data come from, how good is it, and what does it not do.
• Documents drift from the code, and no test fails. Sam’s README said hybrid_rerank; the code said hybrid. Check settings with grep, and commands with the stranger test from a clean folder.
• Write each claim with its evidence: the command, the result and the size of the test. Never “secure” or “production-ready”.
• The portfolio checklist lists every claim with its evidence, marked live, recorded or in the history, and separates completed from planned work.

Next, you turn the same evidence into a five-minute demo, with a failure case and a trade-off.

Next: Demo narrative