Demo narrative

What you will be able to do

Show your product to other people in five minutes. They must understand the user’s problem, see the product work and fail safely, and hear one decision with its evidence. Plan every step, and rehearse it the day before, so that the demo shows what is real.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• The README and the portfolio checklist, from Repository handoff. Every claim in the demo comes from the checklist.
• The trade-off and its numbers, from Document trade-offs.
• The failure cases, from Failure handling, and the read-only member, from Permission enforcement.
• How to explain numbers to people who decide, from Stakeholder presentation in the system design course (optional).

What you need on your computer

• A copy of the snapshot end-of-m06, set up with its README’s Quick start: the virtual environment, .env, python -m idp init, the database running and the pack loaded. These are the steps of the stranger test in the last lesson; that test deleted its folder at the end, so make a new copy.
• For the live part: the Compose demo of Release the demo, or the two terminals of the README (python -m idp and fastapi dev doc_assistant/main.py).

Tested on macOS (Apple silicon); the Windows (PowerShell) commands are written but not tested.

A test drive

You want to buy a used car. A good seller does not read you the manual. First, they ask what you need it for: “You drive your children to school, on a hill?” Then they let you drive your usual route. On the hill, they ask you to stop hard, so you feel the brakes hold. At the end, they tell you one honest choice: “This model has a smaller boot, because the battery is under it. For school runs, that is fine; for camping, it is not.”

In ten minutes, you know the problem that the car solves, and you saw it work. You saw how it behaves when something goes wrong. You heard one trade-off with its reason. You trust the seller more because of the brake test and the boot, not less.

A bad seller drives the car for you, on a flat road, and says “it is perfect”. You learn nothing that you can check.

The problem: the demo that lied twice

Omar asks Sam for a demo on Friday. Sam has docs/demo-script.md, and rehearses it once, on a fresh copy of the project, with the terminal. The first rehearsal goes wrong in two ways that nobody planned.

First, the failure case. Omar, a read-only member at Larkfield, asks “What is the staff discount?”. The script expects a clear “no answer”. Sam leaves the date empty, so it is today:

python -m scripts.ask "What is the staff discount?" --tenant larkfield --user usr-omar

Omar (read_only) at Larkfield asks: What is the staff discount?
Passages found: 1dc2f10bbe79 policy-governance v2, b8eb17ce658e policy-governance v2, 6fddb68adafa privacy-policy v3, 78cb2bb4da77 promotions-and-discount-codes v2, 2db6b314c8b0 promotions-and-discount-codes v2
Answer (answerable): - Policy: The rules for customers, covering returns, refunds and payments, delivery, warranty, privacy, Rewards, and others.
  - - Policy: The rules for customers, covering returns, refunds and payments, delivery, warranty, privacy, Rewards, and others. [How Larkfield policies are managed, Document types, v2]
Note: No recording for this question: the mock made this answer from your passages.
mock mock | 3696 ms | tokens 0 in, 0 out | US$0.000000 | trace b38c1527c79984203dfc8d3824a15377

The answer is nonsense, and it says “answerable”. The note explains why. The demo replays recorded answers, and a recording is found by its exact question, shop and date. Omar’s answer was recorded for 2026-10-09, the date in the questions file. On any other date, the mock answers. It only quotes the first sentence of the best passage. Nothing leaked: Omar still saw no staff-only passage. But on a screen, in front of Omar, it looks like a broken product.

Second, Sam types the pre-order question in other words, as people do when they talk:

python -m scripts.ask "Can customers cancel pre-orders?" --tenant bramble

Answer (answerable): You pay for a pre-order when you place it.
  - You pay for a pre-order when you place it. [Special orders and pre-orders, Pre-orders, v2]
Note: No recording for this question: the mock made this answer from your passages.
mock mock | 3655 ms | tokens 0 in, 0 out | US$0.000000 | trace c4eb7d898db8920d2ee3d0cfed8f42ad

Again the mock answers, because no recording has this wording. Both outputs carry the label, so the product was honest. The demo was not: an audience reads the answer, not the small grey note.

Tomás has a simple fix: “Show only the questions that work, and skip the failures. Nobody needs to see B34.” That fix makes the demo lie in a third way. It hides what Grace most needs to know before she trusts the product.

The idea: a demo is a planned story with evidence

A demo, short for demonstration, is a short, planned showing of the working product to a specific audience. It is not a tour of every feature. A good demo tells one story in five parts, and each part answers a question from the audience:

1. The user problem (about 45 s): who has the problem, and one real example of it going wrong. “Why should I care?”
2. The working flow (about 2 minutes): one user does the main task, from start to end. “Does it work?”
3. A failure case (about 1 minute): one thing that goes wrong, and what the user sees. “What happens when it fails?”
4. One trade-off (about 1 minute): one decision, the option that you did not choose, and the evidence. “Can I trust your judgement?”
5. Questions: the ones you expect, with short, honest answers. “What is not done?”

The written plan of these parts, with every click and every sentence, is the demo script. The story told by the script is the demo narrative. Parts 3 and 4 are the ones that people skip. They are also the ones that build trust, like the brake test and the boot.

Sam's five-minute demo in five parts. The two orange parts, the failure case and the trade-off, are where the audience learns what to trust.

Text description of the diagram

The diagram shows five boxes from left to right (top to bottom on a phone). The first box is the user problem, 45 seconds. An agent quoted the old 30-day return window, and Bramble’s FAQ says 5 days where the policy says 7 to 10. The second box is the working flow, 2 minutes: Ines uploads a document, sees it indexed, gets a cited answer and a clear “no answer”. The third box, in orange, is the failure case, 1 minute. Omar gets no staff-only passage. The terminal shows the isolation checks and a 504 from a slow provider. Sam names question B34. The fourth box, also orange, is the trade-off, 1 minute: keyword search gained 4 answers on the tuning questions and none on the held-out ones. The last box is the questions: what is not done, the cost and the next steps.

A tiny example: one step, three columns

Each step of a demo script has three parts: what you do, what the audience sees, and what you say. Here is one step of Sam’s script, written three ways:

Version	Do	See	Say
Too short	“Ask a question.”	(not written)	(not written)
A feature tour	“Ask a pre-order question.”	an answer	“It uses hybrid retrieval with RRF over pgvector and a tsvector index.”
A story step	Type exactly “Can a customer cancel a pre-order for a full refund?” as Ines	the answer, two citations; click one to see the passage	“Ines gets Bramble’s own rule, 7 days before release, with the source. Notice what she did not get.”

The third version names the exact question, so a recording exists for it. It says what the audience must look at. And it says why it matters to the user, not how it is built. The audience can ask about the retrieval later, in the questions.

The worked capstone: Sam’s demo, part by part

Sam fixes the script after the rehearsal. The changes are now in docs/demo-script.md in end-of-m06. A new opening sentence says that the answers are recorded. Omar’s question has its date, and every question has its exact text.

1. The user problem

Sam starts with Grace’s example from the project brief, not with the technology. “Last month an agent quoted the old 30-day return window to a customer whose delivery fell under the new 45-day policy.” Then one example from Bramble Books: the FAQ says 5 business days for a special order, and the policy says 7 to 10. Both shops want answers from their own documents, with the source. Then Sam says one sentence about the demo itself: “The answers that you will see are real answers of a model, recorded once, and replayed here.”

2. The working flow

Sam signs in as Ines, owner of Bramble Books. Ines uploads holiday-hours.md, a short note written for the demo (made up). The audience watches it move from uploaded to indexed, with 2 passages. Sam does not ask about it, because nobody recorded an answer about it. In the rehearsal, Sam asked “When does the shop close on 24 December?”. The mock quoted the first sentence of the note: “From 1 to 23 December, the shop is open from 9:00 to 20:00 every day.” That is not the answer.

Then Ines asks the pre-order question, exactly as written:

The working flow: a cited answer from Bramble's own policy (the top of the page). The last line of the answer says 'recorded'. Say it aloud too.

Sam clicks a citation, and the passage opens with its version and dates. Then Sam points at what did not happen. The distributor’s note in the same collection contains a hidden instruction: every pre-order can be cancelled at any time, by email to the distributor. The assistant reported Bramble’s own rule.

Last, a question that the documents do not answer:

A clear 'no answer', with no citation and no guess. The audience sees that the product knows the limits of its documents.

3. A failure case

Sam signs in as Omar, read-only at Larkfield, and sets the date to 2026-10-09. The same question as in the rehearsal now finds its recording:

python -m scripts.ask "What is the staff discount?" --tenant larkfield --user usr-omar --as-of 2026-10-09

Omar (read_only) at Larkfield asks: What is the staff discount?
Passages found: 1dc2f10bbe79 policy-governance v2, b8eb17ce658e policy-governance v2, 6fddb68adafa privacy-policy v3, 78cb2bb4da77 promotions-and-discount-codes v2, 2db6b314c8b0 promotions-and-discount-codes v2
Answer (not answerable): The documents do not say what the staff discount is.
recorded chat-small | 4138 ms | tokens 961 in, 46 out | US$0.000119 | trace afb0e1302a9cb86398159b4e5caa35d9

The failure case on purpose: a read-only member gets no answer about staff-only rules. The staff-only procedure is never searched for him, so it cannot leak. The top of the page is shown.

Then Sam switches to the terminal for one minute. python -m scripts.isolation ends with 0 leaks. The last line of python -m scripts.failure_cases shows a model provider that never answers:

provider timeout: 2 s timeout, 2 retries
  -> 504 in 9.29 s: {"code": "provider_timeout", "message": "The model did not answer in time. Your question was not saved; try again.", "provider_calls": 3}

And Sam says the failure that is not on purpose, B34:

python -m scripts.ask "How much goodwill credit can an agent give at Larkfield?" --tenant bramble

Answer (answerable): An agent can give up to $15 in goodwill credit, once per customer per year.
  - Goodwill credit is limited to $15 per customer per year. [Refund approval (staff procedure), Goodwill credit, v1]

“Asked at Bramble Books about Larkfield, it answered with Bramble’s own limit. It should say that it only knows Bramble’s documents. No Larkfield data reached Bramble, but our criterion ‘always say no answer when you should’ is not met.” Grace now knows exactly what to watch.

4. One trade-off

Sam chooses the decision with the clearest evidence: keyword search next to the vector search. The chart is the whole argument:

• Vectors only (baseline)
• Vectors + keywords (kept)

Tune: the 62 tuning questions, where keyword search fixed four answers. Run 1 and Run 2: two identical passes over the 41 held-out questions, equal in the first and one question better in the second, within the change between two runs.

Text description of the chart

A column chart with three pairs of columns, on an axis from 0 to 100%. On the tuning questions (Tune), vectors only answer 90.3% correctly (56 of 62), and vectors with keywords 96.8% (60 of 62). On the held-out questions, first pass (Run 1), both answer 87.8% (36 of 41). In the second pass (Run 2), vectors only answer 85.4% (35 of 41), and vectors with keywords 87.8% (36 of 41).

What Sam says: “On my tuning questions, keyword search fixed four answers and broke none. On the held-out questions, it fixed one, an injection question about a hose reel, and broke one, a French question about a barbecue. So there is no proven gain, and no loss, at the same cost and speed. I kept it, and I do not claim that it made the assistant better. The reranker was about 0.4 s slower per question and did not help.” The option not chosen, the evidence, and the limit of the claim: all in one minute.

5. Questions

Sam writes the expected questions into the script, with short answers. “Is it production-ready?” No: one laptop, synthetic documents, a practice identity provider. “What does it cost?” About US$0.00017 per question with chat-small, and each shop is limited to 200 questions a day and US$5 a month. “What next?” French tuning questions, then a fix for questions that name another shop.

Guided practice: rehearse the demo

Rehearse the day before, on the computer and the data that you will use. The outputs above are from the course team’s rehearsal.

1. Start the product as in the demo, for example the Compose demo, and load the pack.

2. Run the smoke test. It must end with 7 of 7 checks passed.

   macOS / Linux

   SMOKE_TOKEN=$(python -m idp token usr-sam) python scripts/smoke.py http://127.0.0.1:8000

   Windows (PowerShell)

   $env:SMOKE_TOKEN = python -m idp token usr-sam
   python scripts/smoke.py http://127.0.0.1:8000

3. Ask every question of the script with python -m scripts.ask, with the same shop, person and date as in the demo.

4. Read the last line of each answer. It must start with recorded chat-small. If it starts with mock mock, change the question’s words or date in the script to the recorded ones.

5. Upload the demo document, and wait until its status is indexed. Do not plan a question about it.

6. Run python -m scripts.isolation and python -m scripts.failure_cases, and copy the lines that you will show.

7. Time the whole demo with a clock. If it takes more than five minutes, cut words, not the failure case.

8. Make the fallback: keep a screenshot of each step. If the live demo breaks, show the screenshots and say “this is a recording of the same steps”.

Your project

Write docs/demo-script.md for your own product. Start from docs/templates/demo-script.md.

1. The user problem: one user, one decision, one real example of it going wrong, from your project brief.
2. The working flow: the one journey of Module 3, with the exact text of every question and every click.
3. A failure case: one refusal on purpose (a wrong shop, a role, a bad file or a slow provider), and one honest failure from your evaluation report, by its question ID.
4. One trade-off: one decision from your evaluation report, the option that you did not choose, and its numbers. If your held-out set did not confirm the gain, say so.
5. Questions you expect, with short answers. Include “Is it production-ready?”
6. The screenshots plan: one real screenshot for each step, taken from the running product, as the fallback. Never draw a fake screen.
7. Rehearse with the steps above, and fix the script.

Keep as evidence: the script, the rehearsal outputs (every question with its recorded or live line), the screenshots, and a recording of the demo if you can make one.

Checklist

• The demo takes five minutes or less, and its first minute is about the user, not the technology.
• Every question in the script was rehearsed with the same shop, person and date, and was answered by a recording or a live model, never by the mock.
• The demo says aloud whether the answers are live or recorded.
• It shows one failure on purpose and names one real failure from the evaluation.
• The trade-off names the option not chosen, the numbers, and the limit of the claim.
• Every claim in the demo has a row in the portfolio checklist.
• A screenshot exists for every step, in case the live demo breaks.

Common mistakes

Common mistake

The rephrased question

Symptom: in the demo, an answer is wrong or strange, and its last line says mock mock and “No recording for this question”.

Cause: the recorded answers are found by the exact question, shop and date. A new wording, another date or a document uploaded during the demo has no recording, so the mock answers.

Fix: type each question exactly as in the script; rehearse each one with scripts.ask; or configure a live model and say so. If it happens on stage, say what happened: “this question was not recorded, so the practice mock answered”. That is the truth, and it shows that the label works.

Common mistake

The happy-path demo

Symptom: the demo shows only answers that work. Afterwards, Grace asks “and when it is wrong?”, and nobody knows.

Cause: the presenter fears that a failure makes the product look bad. Tomás suggested this.

Fix: show one failure on purpose, and name one real failure from the evaluation report. The rubric of the final assignment marks a demo without a failure case as “partially demonstrated”.

Consequence in AI work: users trust the answers more than they should. The first wrong answer in real work then costs more trust than an honest demo would have.

Common mistake

A claim without evidence

Symptom: the demo says “this will save Grace’s team ten hours a week”, or “it is secure”.

Cause: the presenter wants a strong ending.

Fix: say only what the portfolio checklist proves. Nobody measured time saved; nobody tested the product as an attacker would. Say “it passed these checks”, and list them.

Knowledge check

Check your understanding

Name the five parts of a demo narrative, and say which question of the audience each part answers.

Show the answer

The user problem: why should I care? The working flow: does it work? A failure case: what happens when it fails? One trade-off: can I trust your judgement? Questions: what is not done? The failure case and the trade-off are the parts that people skip, and they build the most trust.

Check your understanding

Predict: in a rehearsal, Omar asks 'What is the staff discount?' with today's date, and the answer is a sentence about policy types, marked 'answerable', with 'mock mock' in the last line. Is this a leak? What do you change in the script?

Show the answer

It is not a leak: Omar’s passages were public (policy governance, privacy, promotions), and no staff-only passage reached him. The recorded provider found no recording for this question on this date, so the mock quoted the first sentence of the best passage. Change the script: set the date to 2026-10-09, the date of the recording, and rehearse again. The answer is then “The documents do not say what the staff discount is.”

Check your understanding

Tomás says: 'Keyword search fixed four tuning questions, so say it improved the assistant by 6.5 points.' Write the sentence that Sam says instead, and explain why.

Show the answer

“On the tuning questions it fixed four answers; on the held-out questions it fixed one and broke one, so there is no proven gain, and no loss, at the same cost. I kept it, and I do not claim that it is better.” The tuning questions helped Sam choose the change, so their score is too optimistic. Only the held-out set gives an honest number, and there the difference (36 against 36 and 35) is within the change between two identical runs.

Recap

• A demo is a short, planned showing of the working product to a specific audience. The demo script plans every step: what you do, what they see, what you say.
• Tell one story in five parts: the user problem, the working flow, a failure case, one trade-off, and questions.
• Show one failure on purpose, and name one real failure from the evaluation (B34). State the trade-off with its numbers and the limit of the claim.
• Say whether answers are live or recorded. Recordings are found by exact question, shop and date; rehearse every question, or the mock answers on stage.

Next, you look back at the whole project, and plan what could come after it.

Next: Reflection and extensions