Problem and audience
- Lesson 2
- 60 min
- Module 1: Select and bound the project
Lesson 2 of 22 · Portfolio Capstone: Build, Evaluate and Ship an AI ProductProblem and audience
1. Select and bound the project
2. Prepare data and architecture
3. Build one complete user journey
4. Add reliability and access control
5. Evaluate and improve
6. Deploy and operate
7. Prepare the portfolio presentation
What you will be able to do
Decide who your product is for, which decision it helps with, which documents it reads and what it will not do. A project with one user and one decision is small enough to finish and specific enough to test.
Before you startWhat to know and to have ready
What you need to know
- The starter, its design-pack templates and your track, from Set up the capstone repository.
- A document collection with an inventory, and the questions that people ask of it: Corpus and questions in RAG: Building AI That Uses Your Data.
- Tenants: Larkfield and Bramble Books as two organizations whose data never mixes, from Users and organizations in Authentication, Permissions and Multi-Tenant Applications.
What you need on your computer
- The project folder
doc-assistantfrom the last lesson, with its first commit. If it is missing, copy the snapshotstartagain, as in Set up the starter. - No new packages. This lesson is about decisions; you write one file.
A food truck with a short menu
Section titled “A food truck with a short menu”Two friends open a food business in the same month.
The first friend opens a restaurant “for everyone”: 80 dishes from 12 countries, three meals a day, delivery and catering. After a month, the kitchen is tired, many dishes are poor, and she cannot tell which ones lose money. When a customer complains, she does not know where to start.
The second friend parks a food truck next to a group of offices. She serves one kind of customer: office workers with a 30-minute lunch break. She does one job: a hot meal in under five minutes. Her menu has five dishes. On the side of the truck, a sign says: “No table service. No delivery. Cards only.” After a month, she knows exactly how she is doing. She counts the meals that she served in under five minutes, and she knows which dish sells.
The sign is as important as the menu. Every “no” on it is a decision that she made once, so she does not have to argue about it every day.
The problem: an idea you cannot test
Section titled “The problem: an idea you cannot test”In the first meeting, Tomás, an AI engineer, describes the product that he wants Sam to build:
“An AI assistant for shops. It chats with customers, answers any question from any document, sends refund emails by itself, speaks every language, and runs in the cloud. Let’s add an agent first: that is what people want to see.”
It sounds impressive. Sam tries a simple test: write the first test question and its right answer. Sam writes “Can I return this?”, and then gets stuck at once:
- Who asks? A customer needs a short, polite reply. A member of staff needs the rule and its exceptions, for example the approval limit for a refund.
- Which shop? Larkfield sells garden tools; Bramble Books sells books. Their return rules are different.
- On which date? Larkfield changed its return window from 30 to 45 days. The right answer depends on the date of the delivery.
- From which document? “Any document” names no collection, so nobody can check that the answer comes from the right one.
Sam cannot write even one test question that has one right answer. So nobody could ever say if the product works. This is the real cost of a vague idea: it is not only big, it is untestable. And Tomás’s list has five large features. A five-week capstone with five half-built features has nothing to show at the end, because no single journey works from start to finish.
The idea: one user, one decision, a narrow collection, and non-goals
Section titled “The idea: one user, one decision, a narrow collection, and non-goals”The fix is to answer four questions before you build anything. Together, the answers are the project’s scope: the boundary of what the project does.
- One user. One kind of person with a role, for example “the help-desk staff of a small shop”. Not “everyone”.
- One decision. The thing that this person must decide or do, which the product makes better or faster. For example: “what to tell a customer who asks about a policy”. We say decision because the product’s value lies in a choice that a person makes with its help.
- A narrow document collection. Exactly which documents the product reads: what they are, how many, in which formats, who owns them, and why you may use them.
- Non-goals. The things that the product will not do, written down on purpose. A non-goal is not a weakness. Like the sign on the food truck, it is a decision that protects your time, and it shows a reviewer that you chose.
Here is a quick test for “narrow enough”. Can you write ten real questions that this user asks, each with its right answer and the document that contains it, in about 30 minutes? If you can, the scope is narrow enough to test. If you cannot, the user, the decision or the collection is still too vague.
A tiny example: from an idea to five lines
Section titled “A tiny example: from an idea to five lines”Here is a small example that the course made up. Ana wants “an AI for my football club”. She answers the four questions:
| Question | Ana’s first answer | Ana’s answer after the test |
|---|---|---|
| Who uses it? | “The club” | The volunteer coaches of the junior teams |
| Which decision? | “Anything about the club” | Whether a match goes ahead in bad weather, and what to tell the parents |
| Which documents? | “All our files” | The club’s weather and pitch policy, its safeguarding rules for messages to parents, and the league’s rules for postponed matches: 4 documents, all public on the club’s website |
| What will it not do? | (nothing) | No messages sent to parents by itself, no match scheduling, no medical advice |
| One test question | “Is the match on?” | “The under-10 pitch is frozen at 8:00 on Saturday. Can the 10:00 match go ahead?” Right answer: no, a frozen pitch means postponement; the source is section 2 of the weather policy |
Look at the last row. The first test question has no right answer, because “the match” and “on” depend on facts that the question does not give. The second question has one right answer and one source. Ana’s narrower scope made the product testable. It also made it smaller: she can finish it.
Sam’s brief
Section titled “Sam’s brief”Sam does not choose the user alone. Sam meets Grace, Larkfield’s support lead, and Ines, the owner of Bramble Books. Their shops are the two organizations of the authentication course. Sam asks how their staff work today and goes through the four questions with them. The diagram shows the result.
Text description of the diagram
The flow starts with “an AI assistant for shops” and asks four questions in turn. Who uses it? The help-desk staff of two shops. Which decision? Answering a customer’s policy question correctly, while the customer waits. From which documents? Each shop’s own policies: 36 for Larkfield, 15 for Bramble Books. What will it not do? Five non-goals: no answers to customers, no chat history, no agent, no other languages, no cloud in the core. The result is a project that Sam can finish, test and demonstrate.
Sam writes the answers into docs/project-brief.md. Here is its first part:
## User and decision
- Who uses the product: the help-desk staff of two small shops, Larkfield (home and garden) and Bramble Books (an independent bookshop).- The decision it helps with: answer a customer's policy question correctly, from the shop's own documents, while the customer waits.- How they do it today, and what goes wrong: staff search a shared folder and read several documents. Last month a Larkfield agent quoted the old 30-day return window to a customer whose delivery fell under the new 45-day policy, and a Bramble Books volunteer quoted the FAQ's "5 business days" for a special order that took 9.The last line is the most useful one. It names two real mistakes that the product must prevent: an old version of a policy, and two documents that contradict each other. Both become test questions in this module and difficulties in the data pack of Module 2. “Staff need answers faster” would give you nothing to test.
The next part says which documents, and why Sam may use them:
## Documents
- Which documents the product reads: each shop's own policies, staff procedures, help articles and supplier notes. Larkfield: 36 documents (Markdown, HTML and PDF). Bramble Books: 15 documents.- Who owns them, and why we may use them: every document is synthetic, written for this course; no real company wrote them. Licence CC0.Larkfield’s 36 documents are the collection of the RAG course, unchanged. Bramble Books’ 15 documents were written for this course, with the same kinds of difficulty. Both are synthetic: made up, so they contain no real customer and no real company’s rules. CC0 means that anyone may use them for anything.
Then come success and the non-goals:
## What success looks like
- A member of a shop asks a question in plain words and gets, in a few seconds, a short answer with a citation for every claim, or a clear "the documents do not answer this".- A shop never sees the other shop's documents, passages or answers.
## Non-goals
- No answers sent to customers directly: staff read the answer and decide.- No chat history and no follow-up questions: one question, one answer.- No agent that takes actions (refunds, emails). Tomás asked for one; it is a possible extension after the core works.- No other languages than the documents' (English, and French for Larkfield's two Quebec documents).- No cloud deployment in the core: a reproducible local demonstration.Each non-goal has a short reason. “Staff read the answer and decide” is also a safety decision: a person checks every answer before a customer hears it. The diagram puts the scope and the non-goals side by side.
Text description of the diagram
The left group, in scope, holds six things. One question gets one answer, and every claim cites the shop’s own documents. There is a clear “no answer”, upload and delete, two shops that never mix, and a local demonstration. The right group, with dashed borders, holds the five non-goals of the brief. A dotted arrow leads from the non-goals to Module 7’s extensions: the agent is possible after the core works.
Tomás and the agent
Section titled “Tomás and the agent”Tomás does not give up easily. “Everyone wants agents. Without one, the demo looks small.”
Sam answers with the brief: “Grace’s staff need a correct answer while the customer waits. An agent that sends refund emails solves a different problem and needs its own safety checks. If the core works, it is the first extension that I propose in Module 7.” What an agent needs, its tools, limits and approval steps, fills a course of its own: AI Agents and Workflow Orchestration.
This is the most common mistake in capstones: too many features before one flow works. A small product that works from start to finish, with evidence, impresses a reviewer more than five impressive pieces that do not connect.
Milestones
Section titled “Milestones”The last part of the brief is the plan. A milestone is a point in the project with an observable result: a “done when” that anyone can check, not only a date. “Work on the backend” is not a milestone. “Grace uploads a document, sees it indexed, asks a question and gets a cited answer” is one. On the day, it is true or it is not.
| Milestone | Done when | Week |
|---|---|---|
| Design pack | the architecture and the threat and cost review are written and reviewed by Kwame | 1 |
| One journey | Grace uploads a document, sees it indexed, asks a question and gets a cited answer | 2 |
| Safe and reliable | the wrong-shop, missing-source, invalid-file and provider-timeout cases pass | 3 |
| Evaluated | the held-out report is written and Priya reviewed it | 4 |
| Shipped | a clean computer runs the demo from the README; Omar has seen the demo | 5 |
The order matters. “One journey” comes before “Safe and reliable”: first one complete flow, then the hard cases. A milestone that names a person, such as Kwame or Priya, also says who must agree that it is done.
When to narrow, and what it costs
Section titled “When to narrow, and what it costs”Narrow the scope whenever you must finish and prove something in fixed time: a capstone, a pilot or a first version for a customer. One user and one decision give you test questions with one right answer. Those questions are the base of everything in Modules 3 to 5.
Narrowing has a cost. Some people get nothing, for example the shops’ customers. The demo looks smaller than Tomás’s idea. And a narrow brief is harder to write than a wide one, because every line is a choice that somebody can question. A wide scope can suit a short experiment that only explores ideas. It does not suit a portfolio project, where a reviewer asks above all: “Does it work, and how do you know?”
Guided practice: features and needs
Section titled “Guided practice: features and needs”Every feature must support a need that the brief states. Below are seven features that people asked for. For each one, find the line of Sam’s brief that it supports, or say that it is a non-goal. Then open the answer.
- Every claim in an answer cites a passage of the shop’s documents.
- A chat history, so staff can ask follow-up questions.
- Staff upload a PDF policy and see when it is ready.
- The assistant emails the answer to the customer.
- A dark mode for the page.
- Answers in French to French questions about Larkfield’s two Quebec documents.
- Staff delete a document that is out of date.
Show the answer
- Supported: “a short answer with a citation for every claim”. It also helps the decision: staff can check the answer before they speak.
- Non-goal: “No chat history and no follow-up questions: one question, one answer.”
- Supported: “each shop’s own policies” can only be read if staff can add them. The status matters because staff must know when a new policy is in use.
- Non-goal: “No answers sent to customers directly: staff read the answer and decide.”
- No stated need. Nothing in the brief asks for it. It is not forbidden, but it waits until every milestone is done. If Grace’s staff work in a dark room and ask for it, the brief changes first.
- Supported, within limits: the non-goal says “No other languages than the documents’”. French is a language of two of Larkfield’s documents, so a French question about them is in scope.
- Supported: a policy that is out of date causes the very mistake in the brief (the old 30-day return window). Deleting it, so that no later answer uses it, serves the decision.
A feature with no stated need, such as number 5, is not “wrong”. It is not part of this project until the brief gives it a reason.
Your turn: another user, the same documents
Section titled “Your turn: another user, the same documents”Change one condition: the user. Imagine that Bramble Books wants the assistant for its customers on its website, not for its staff. The documents stay the same. Before you open the answer, predict: which lines of the brief change, and which new non-goals or rules appear?
Show the answer
Almost every line changes, although the documents are the same:
- User and decision: a customer who wants to know, for example, whether they can return a gift. The decision is the customer’s own, and no member of staff checks the answer first.
- Documents: the three staff-only documents of Bramble Books (wholesale terms, refund approval and the staff discount) must be excluded. A customer must never see them.
- Non-goals and rules: the answer goes directly to a customer, so the non-goal “staff read the answer and decide” is gone. The product now needs stricter rules: polite wording, no internal procedures, and a clear way to reach a person.
- Success: “staff get an answer while the customer waits” becomes “the customer gets a correct answer without calling the shop”.
This is why the user comes first. The same documents serve a very different product when the person who reads the answer changes. Sam keeps the staff version: it is safer, because a person checks every answer, and it matches what Grace and Ines asked for.
Common mistakes
Section titled “Common mistakes”Common mistake
Too many features before one flow works
Symptom: after three weeks, the project has a chat window, half of an agent, a login page and a script that reads PDF files. None of them connects to the others. Nobody can upload a document and get a cited answer from start to finish.
Cause: the brief had no non-goals, or nobody read them. Every new idea seemed small on its own day.
Fix: go back to the milestones. Finish “One journey” before you start anything else, and move every other feature to the non-goals or to Module 7’s extensions. Module 3 builds exactly one journey for this reason.
Common mistake
Private documents in a public demo
Symptom: on the own-idea track, a learner uses their employer’s internal documents “because they are realistic”. The demo is on a public portfolio page.
Cause: realistic data seemed more impressive than permission. But the documents belong to someone else, and they can contain names, prices and procedures that must not be public.
Fix: use only public documents with an open licence, documents with written permission from their owner, or synthetic documents. If you used private ones, take the demo down, remove the documents from the repository and its history, and tell the owner. Synthetic documents with the right difficulties make an equally strong portfolio piece; Module 2 shows how to write them.
Your project
Section titled “Your project”Write your own brief.
-
Copy the template into
docs/:Terminal window cp docs/templates/project-brief.md docs/project-brief.mdTerminal window Copy-Item docs\templates\project-brief.md docs\project-brief.md -
Open
docs/project-brief.mdand replace every line in angle brackets. -
Under “How they do it today”, name at least one real mistake that your product must prevent.
-
Write at least three non-goals, each with a short reason.
-
Do the ten-question test. If you cannot write ten questions with their answers and sources in about 30 minutes, narrow the user, the decision or the documents.
-
Commit the brief.
On the standard track, your user, decision and documents are close to Sam’s. Write the brief in your own words anyway, and choose your own non-goals and milestones. On the own-idea track, name each source of your documents and its licence or permission.
Evidence you keep: docs/project-brief.md and its commit, and your ten test questions (you need them in Plan the evidence).
Checklist
- One kind of user, named by role.
- One decision, in one sentence.
- A named document collection: what, how many, which formats, who owns it, why you may use it.
- At least one real mistake that the product must prevent.
- At least three non-goals, each with a reason.
- Milestones with an observable “done when”; “one journey” comes before the hard cases.
- Ten test questions with their answers and sources.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name the four parts of a project's scope in this lesson, and say in one sentence what each one protects you from.
Show the answer
- One user: answers that try to suit everyone and suit nobody. The user decides the wording, the documents and the risks.
- One decision: a product with no clear job, so no test question has one right answer.
- A narrow document collection: answers that nobody can check against a named source, and documents that you may not use.
- Non-goals: new features that arrive every week until no single journey works. Each non-goal is a decision made once.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: Tomás says, 'Write the brief later. Build first, and see what users like.' What goes wrong in this capstone if Sam agrees?
Show the answer
Sam cannot write test questions with one right answer, because nothing says who asks, which shop, which date or which document. So in Module 5 there is nothing fixed to measure against, and every result can be explained away. New features arrive without a filter, and after three weeks no single journey works. The brief costs about an hour now. Without it, “see what users like” becomes “build many half-features and prove none of them”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: show your brief. Which user, which decision, which documents, which non-goals, and which test question shows that the scope is narrow enough?
Show the answer
A good answer names one kind of user by role and one decision in one sentence. It names a document collection with its size, formats and permission, and at least three non-goals with reasons. On the standard track, for example:
- User: the help-desk staff of Larkfield and Bramble Books.
- Decision: answer a customer’s policy question correctly while the customer waits.
- Documents: the two shops’ synthetic policies, 36 and 15 documents, CC0.
- Non-goals: no answers to customers, no chat history, no agent.
A test question has one right answer and one source. For example, “How long does delivery to Ontario take?” at Bramble Books: 4 to 8 days, from the delivery policy.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A vague idea such as “an AI assistant for shops” is not only big: it is untestable, because no test question has one right answer.
- A scope answers four questions: one user, one decision, a narrow document collection, and non-goals, the things that the product will not do.
- The ten-question test: if you can write ten real questions with their answers and sources in about 30 minutes, the scope is narrow enough.
- Sam’s brief: help-desk staff of Larkfield and Bramble Books answer customers’ policy questions from 36 and 15 synthetic documents, with five non-goals. The agent waits until the core works.
- A milestone has an observable “done when”. One complete journey comes before the hard cases.
- Every feature must support a need that the brief states.
Next, you turn “it should give good answers” into criteria that a person or a script can check.
Next: Acceptance criteria