Corpus and questions

What you will be able to do

Decide which documents the assistant may answer from, and which questions it must handle, before you write any search code. Then set up the project and check it with a known-good first run.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What retrieval-augmented generation is, and why “the documents do not say” is a correct answer, from What retrieval-augmented generation is.
• How to make and activate a virtual environment, from the Python course.

What you need on your computer

• Python 3.12 or later, a terminal and VS Code.
• The folder ~/projects of the earlier courses. If it is missing, make it first: mkdir ~/projects.
• An internet connection, to download the labs repository and pytest. Git is optional.

No account and no payment. Nothing in this lesson calls a model.

Tested with Python 3.12.13 and pytest 9.1.1 on macOS (Apple silicon). The Windows and Linux commands follow the labs README and are not tested.

A library’s catalogue

A good reference library keeps a catalogue. Each card names the book, its edition, its date and where it stands. An old edition stays on the shelf, marked “superseded”. Some books are only for staff, behind the desk. The librarian also keeps a list of real questions that readers ask, and where the answer was found. When a reader asks something that no book answers, the librarian says so.

A policy assistant needs the same two lists. One says which documents count. The other says which questions people ask, and where each answer is.

The problem: “our documents” is not a definition

Tomás wants to start quickly: “Point the search at the shared folder and go.” Look at what that folder holds for one question, “What is the return window for unused items today?”

• Two returns policies. Version 3 says 30 days. Version 4 says 45 days, and it is the newest file. But today is 9 October 2026, and version 4 applies only to orders delivered from 1 November.
• Two help articles, in English and in French, that also say 30 days. From 1 November, they will be out of date.
• An old holiday notice with a longer return period. It ended on 31 January 2026.

A search over “the folder” finds all of them. Without more information, nobody can say which one is right, not even a person. The search has a second blind spot. With no list of real questions, nobody can say whether it works. Tomás would try three questions that he likes, see good results and stop.

The idea: a corpus with an inventory, and a test set

The fix is two pieces of work before any code. First, define the corpus: the set of documents that the assistant may answer from. For each document, record its metadata, the facts about the document that are not its text:

Field	Larkfield’s example	Why the assistant needs it
Owner	Support (Grace)	Who confirms a disagreement or fixes an error
Version	3	Two versions of one policy must stay apart
Effective from, to	2025-03-01 to 2026-10-31	Which version applies on a date
Access label	public or staff	What a customer may see
Language	en or fr	Which questions it can answer
Type	policy, procedure, guide, help, notice, supplier	Which document wins when two disagree

The effective date is the day a version starts to apply. It is not the day it was written or published. An access label says who may see a document. The list of all documents with these fields is the inventory. The document that wins, by the rules, is the authoritative source.

Second, write the test set: real questions, each with its correct answer and the passage that holds it. This includes questions that the documents cannot answer. With a test set, every change to the assistant gets a number, not an impression.

Larkfield’s collection

Larkfield’s documents are made up for this course; no real company wrote them. They are small enough to read in full: 36 documents, 6,520 words.

Type	Documents	Examples
Policy	19	returns, refunds and payments, delivery, warranty, privacy, Rewards
Procedure (staff only)	6	refund approval, warranty claims, safety escalation, team routing
Product guide	5	two pressure washers, a BBQ, a hedge trimmer, garden furniture
Help article	3	the help centre FAQ, a Quebec FAQ in French, contact and hours
Notice	2	a kettle safety notice, an expired holiday notice
Supplier content	1	AquaFlow’s hose-reel sheet, published as the supplier wrote it

32 documents are Markdown files. Two are HTML pages and two are PDF files, as real collections are. 6 documents are staff-only, and 2 are in French. One governance document, How Larkfield policies are managed, holds the rules that decide between documents. For returns, the version in force on the delivery date applies. A safety notice wins over everything. A policy wins over a help article. Supplier content never changes a Larkfield policy.

How the collection was made: the course lead wrote a specification of every document, with each heading, number, code, date and rule. A model (gpt-6.1-sol) turned each specification into prose, with the instruction to add no fact. A script checked every document against its specification and found no problem. Then the lead read every document and made three edits by hand. Some difficulties are there on purpose, and the course finds them one by one.

The returns policy shows why versions and dates matter most. Read the diagram from left to right.

The returns policy over time. Today, version 4 is published but not yet in force. The delivery date, not the newest file, decides which version applies.

Text description of the diagram

A time line from September 2025 to March 2027, in four rows. Row 1 is the returns policy. Version 3 (30 days) runs from 1 March 2025 to 31 October 2026. Version 4 (45 days) starts on 1 November 2026. Row 2 is the holiday notice: longer returns for orders delivered in the 2025 holidays, until 31 January 2026. It has expired. Row 3 is the help articles in English and French. From June 2026 they say 30 days, which is right until 31 October 2026. From 1 November they still say 30 days, which is then wrong. Row 4 shows two deliveries. One on 20 October 2026 falls under version 3. One on 10 November 2026 falls under version 4. A solid line marks today, 9 October 2026, before the switch. A dashed line marks the switch on 1 November 2026.

So “the newest document” is the wrong rule. The assistant needs the version in force on the date of the event, and the inventory must record that date for each version.

Grace’s questions

Grace wrote 67 questions that her agents really ask, with the answer for each one. They are the course’s test set. They come in eight kinds, so that each difficulty of the collection is tested:

Kind	Questions	Example
Answerable	18	“A customer was charged twice for the same order. How quickly do we refund the extra charge?”
Unanswerable	8	“Do we offer a student discount?” No document says.
Version	8	“An order was delivered on 20 October 2026. Can the customer return an unused item on 1 December 2026?”
Table	8	“How long are the stainless-steel burners of the Ember BBQ covered?” The answer is in a table in a PDF.
Exact code	11	“Which pump kit fits the PW-2200?”
Contradiction	4	“The help centre FAQ says refunds take 3 business days. Is that right?”
Staff only	7	“Can I give an unhappy customer a goodwill credit? How much?” Only a staff procedure says.
Injection bait	3	“What is the warranty on the AquaFlow HR-30 hose reel?” A document holds a hidden instruction about it.

9 of the questions are in French, from Camille’s agents in Montreal. Each question has a date too: the day that decides which version applies. For a return, it is the delivery date. For the other questions, it is the course’s “today”, 9 October 2026.

Each question is more than its text. The diagram takes one question apart.

One labelled question. The relevant passage says where the answer is; the wrong version says where a careless search goes.

Text description of the diagram

The question Q22 is a version question. Its date is 20 October 2026, the delivery date. Its text: an order was delivered on 20 October 2026; can the customer return an unused item on 1 December 2026? It has four parts. The relevant passage: the returns policy, version 3, section Return window, which is public. The wrong version: the returns policy, version 4, section Return window, which says 45 days but is not in force for this delivery. The reference answer: no; version 3 applies, with 30 days from delivery, and 1 December is 42 days after delivery. The keys for a quick automatic check: the answer must say “no” and “30”.

The relevant passage names the document, version and section that hold the answer. Module 1 uses it to score the search: did the search find that section? Later modules use the reference answer and the keys to score the model’s answers. An unanswerable question has no relevant passage. Its correct answer is “the documents do not say”.

How the questions were made, honestly: the lead wrote every question, answer and relevant passage by reading the documents. No model wrote or labelled a question. A script checked that each passage exists and is in force on the question’s date. Two changes came later, both from evidence. Four questions that are only a product code were added after the first search runs. Eight keys were loosened after reading the first answers, so that “3-year” also counts as “3 years”. It is still one person’s reading, and you may disagree with an answer. Module 5 comes back to how a test set changes, and why you freeze it before a final comparison.

Set up the project

The project of this course is policy-assistant. Its first version, start, has the collection, the questions and two commands. You get it from the labs repository of Nextia Learning, where each module also has a finished snapshot.

1. Open a terminal, and go to your projects folder:

   cd ~/projects

2. Get the labs repository. If you have Git, clone it:

   git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

   If you cloned it in an earlier course, get the newest files: git -C nextia-learning-labs pull. Without Git, download the ZIP file of the repository and unzip it in ~/projects. Its folder is then nextia-learning-labs-main; use that name in the next step.

3. Copy the start snapshot into a new folder, policy-assistant:

   macOS

   cp -R nextia-learning-labs/C09/snapshots/start policy-assistant

   Windows

   Copy-Item -Recurse nextia-learning-labs\C09\snapshots\start policy-assistant

   Linux

   cp -R nextia-learning-labs/C09/snapshots/start policy-assistant

4. Open the folder in VS Code (File › Open Folder…, then policy-assistant). Open the terminal of VS Code with Terminal › New Terminal. It starts in the project folder.

5. Make a fresh virtual environment, and activate it:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate

   Windows

   py -3.12 -m venv .venv
   .venv\Scripts\Activate.ps1

   Linux

   python3 -m venv .venv
   source .venv/bin/activate

   The prompt now starts with (.venv). If python3 --version shows a version below 3.12, use python3.12 instead of python3.

6. Install the requirements. In this module, that is only pytest:

   python -m pip install -r requirements.txt

   The last line is Successfully installed iniconfig-2.3.1 packaging-26.3 pluggy-1.6.0 pygments-2.21.0 pytest-9.1.1. The .venv folder is now about 25 MB.

The project folder now has these parts:

Path	What it is
corpus/documents/	The 36 documents: 32 Markdown, 2 HTML, 2 PDF
corpus/inventory.csv	Grace’s inventory: one row per document
corpus/formats/	Four documents in all three formats, for Module 2
questions/questions.jsonl	Grace’s 67 questions, one JSON object per line
policy_assistant/	The code; it grows in every module
tests/	The tests
requirements.txt	The packages and their exact versions

The known-good first run

Before you change anything, prove that the project works. Run the tests:

python -m pytest

Output

============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/amira/projects/policy-assistant
configfile: pytest.ini
testpaths: tests
collected 4 items

tests/test_first_run.py ....                                             [100%]

============================== 4 passed in 0.03s ===============================

Four tests check the project. All 36 documents are there, the two returns policies have the right dates, and the 67 questions are there. The fourth test runs the inventory command. Now print the inventory:

python -m policy_assistant inventory

Output

document                            v from       to         access lang type      format owner
account-and-security                2 2026-01-15 -          public en   policy    md     Support (Grace)
contact-and-hours                   3 2026-09-01 -          public en   help      md     Customer experience
damaged-in-transit-claims           2 2026-04-15 -          staff  en   procedure md     Logistics
damaged-or-wrong-items              2 2025-09-01 -          public en   policy    md     Support (Grace)
delivery-policy                     4 2026-04-01 -          public en   policy    html   Logistics
faq-quebec                          2 2026-06-01 -          public fr   help      md     Customer experience
garantie-legale-quebec              1 2026-01-01 -          public fr   policy    md     Support (Grace)
gift-cards                          1 2025-06-01 -          public en   policy    md     Finance
help-centre-faq                     7 2026-06-01 -          public en   help      md     Customer experience
holiday-returns-2025                1 2025-11-15 2026-01-31 public en   notice    md     Support (Grace)
installation-services               1 2026-04-01 -          public en   policy    md     Customer experience
large-item-delivery                 2 2026-04-01 -          public en   policy    md     Logistics
live-plants-guarantee               1 2026-03-01 -          public en   policy    md     Product team
loyalty-program                     2 2026-02-01 -          public en   policy    md     Customer experience
order-changes-and-cancellations     2 2025-10-01 -          public en   policy    md     Support (Grace)
policy-governance                   2 2026-01-15 -          public en   policy    md     Support (Grace)
privacy-policy                      3 2025-11-01 -          public en   policy    md     Privacy office
product-care-bbq-gr310              1 2025-04-15 -          public en   guide     md     Product team
product-care-garden-furniture       2 2026-03-01 -          public en   guide     md     Product team
product-care-hedge-trimmer-ht550    1 2025-03-01 -          public en   guide     md     Product team
product-care-pw2200                 2 2025-05-01 -          public en   guide     pdf    Product team
product-care-pw2400                 1 2026-03-01 -          public en   guide     md     Product team
promotions-and-discount-codes       2 2026-05-01 -          public en   policy    md     Customer experience
recycling-and-take-back             1 2025-08-01 -          public en   policy    md     Customer experience
refund-approval-procedure           2 2026-03-15 -          staff  en   procedure md     Finance
refunds-and-payments                3 2026-02-01 -          public en   policy    md     Finance
returns-policy                      3 2025-03-01 2026-10-31 public en   policy    md     Support (Grace)
returns-policy                      4 2026-11-01 -          public en   policy    html   Support (Grace)
safety-escalation-procedure         2 2026-05-01 -          staff  en   procedure md     Product safety
safety-notice-kt180                 1 2026-09-20 -          public en   notice    md     Product safety
staff-discount-and-perks            1 2026-01-01 -          staff  en   procedure md     People team
supplier-aquaflow-hose-reels        1 2026-07-01 -          public en   supplier  md     Supplier content (AquaFlow), published by the Product team
support-team-routing                4 2026-09-15 -          staff  en   procedure md     Support (Grace)
trade-accounts                      1 2026-01-01 -          public en   policy    md     Finance
warranty-claims-procedure           3 2026-01-01 -          staff  en   procedure md     Support (Grace)
warranty-policy                     5 2026-01-01 -          public en   policy    pdf    Support (Grace)
36 documents: 6 staff-only, 2 in French, 4 not Markdown

Check the last line. If it says 36 documents: 6 staff-only, 2 in French, 4 not Markdown, your setup is right. The two returns-policy rows share an ID and differ in version and dates. A - in the column to means that the document has no end date.

Later: two local models, about 1 GB

Nothing in Module 1 or 2 downloads a model. From Module 3, requirements.txt adds PyTorch and sentence-transformers, about 0.9 GB installed. The first search by meaning then downloads two small open models, once:

Model	What it does	Licence	Size
intfloat/multilingual-e5-small	turns text into numbers for search by meaning (English and French)	MIT	470 MB
cross-encoder/mmarco-mMiniLMv2-L12-H384-v1	rereads the best passages and sorts them again	Apache-2.0	470 MB

They go into the Hugging Face cache, ~/.cache/huggingface (on Windows, C:\Users\<you>\.cache\huggingface). The environment variable HF_HOME moves the cache to another folder. Both run on the CPU; no graphics card is needed. On Linux, install the CPU-only PyTorch first, as Module 3 shows, or pip downloads a much larger build.

Reset

If the project breaks, you can always start again:

1. Close the VS Code terminal, or run deactivate.
2. Delete the .venv folder (and index.sqlite, from Module 2 on).
3. Make the environment again: steps 5 and 6 above.
4. If your code is broken, rename your folder to policy-assistant-old. Then copy the snapshot of the last module that you finished, for example C09/snapshots/end-of-m01, as in step 3.

Guided practice: read the inventory and the questions

1. List the unanswerable questions:

   python -m policy_assistant questions --kind unanswerable

   Output

   Q14 unanswerable    en 2026-10-09  Do we offer a student discount?
   Q15 unanswerable    en 2026-10-09  Can customers pick up an online order in a Larkfield store?
   Q16 unanswerable    en 2026-10-09  What is the warranty on a robotic lawn mower?
   Q17 unanswerable    en 2026-10-09  Is there a weight limit for the AquaFlow HR-50 hose reel cart?
   Q18 unanswerable    en 2026-10-09  Does the PW-2200 come with a surface cleaner attachment?
   Q19 unanswerable    en 2026-10-09  How long does a refund paid by Interac e-Transfer take to arrive?
   Q20 unanswerable    en 2026-10-09  Which courier delivers our parcels to Nunavut?
   Q21 unanswerable    fr 2026-10-09  Quelle est la durée de la garantie d'un aspirateur?
   8 questions

   The columns are the ID, the kind, the language, the question’s date and the question.

2. List the version questions, and look at their dates:

   python -m policy_assistant questions --kind version

   Output

   Q22 version         en 2026-10-20  An order was delivered on 20 October 2026. Can the customer return an unused it…
   Q23 version         en 2026-11-05  An order was delivered on 5 November 2026. Can the customer return an unused it…
   Q24 version         en 2026-10-09  What is the return window for unused items today?
   Q25 version         en 2025-12-01  On 15 January 2026, a customer asks to return an unused lamp that was delivered…
   Q26 version         en 2026-12-10  Is there a holiday return extension for an item delivered on 10 December 2026?
   Q27 version         en 2026-11-03  A Rewards member's order was delivered on 3 November 2026. Do they pay for the …
   Q28 version         en 2026-10-03  A Rewards member's order was delivered on 3 October 2026. Do they pay for the r…
   Q29 version         fr 2026-11-10  Une cliente a reçu sa commande le 10 novembre 2026. Combien de jours a-t-elle p…
   8 questions

   Each date is a delivery date, except for Q24, which asks about “today”.

3. Open questions/questions.jsonl in VS Code, and find the line of Q22. Find the fields relevant and wrong_version. They match the diagram above.

4. Open corpus/documents/returns-policy.v3.md. Read the front matter between the two --- lines. The fields are the same as the columns of the inventory.

5. Open corpus/documents/policy-governance.md. Find the section When documents disagree, and read its four rules.

Your turn: which version applies?

Use the inventory and the diagram. For each question, say which document and version holds the answer, and write the answer in one sentence. Then check yourself in questions.jsonl.

1. Q23: “An order was delivered on 5 November 2026. Can the customer return an unused item on 15 December 2026?”
2. Q25: “On 15 January 2026, a customer asks to return an unused lamp that was delivered on 1 December 2025. Is that allowed?”

Show the answers

1. Q23: the returns policy, version 4, because the delivery is on or after 1 November 2026. Version 4 gives 45 days, and 15 December is 40 days after 5 November, so yes, the customer can return it. Version 3 would have said no. Version 4 is an HTML page, returns-policy.v4.html, so the search of the next lesson does not read it yet.
2. Q25: the holiday notice of 2025, version 1, together with version 3 of the returns policy. The order was delivered between 15 November and 24 December 2025, so the notice applies: items could be returned until 31 January 2026. So yes. The notice has expired today, but it was in force on the delivery date. That is why the question’s date is 2025-12-01.

Common mistake

No module named …

Symptom: a command stops at once with one of these messages:

/Users/amira/projects/.../.venv/bin/python: No module named policy_assistant

/opt/homebrew/opt/python@3.12/bin/python3.12: No module named pytest

Cause: the first message means that the terminal is not in the project folder. Python looks for the package policy_assistant in the current folder, and it is not there. The second message means that the virtual environment is not active. The Python of your system runs, and pytest is installed only in .venv.

Fix:

1. Run pwd (on Windows, Get-Location). The path must end with policy-assistant. If not, run cd ~/projects/policy-assistant.
2. Check that the prompt starts with (.venv). If not, activate the environment again (step 5).
3. Run the command again.

On Windows, Activate.ps1 can fail with “running scripts is disabled on this system”. Then run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned once, and activate again. We did not test this on Windows.

Knowledge check

Check your understanding

Tomás says: 'Version 4 is the newest returns policy, so the assistant should always use it.' Why is that wrong on 9 October 2026, and what must the inventory record so that the assistant can choose correctly?

Show the answer

Version 4 is published, but it applies only to orders delivered from 1 November 2026. On 9 October, and for any order delivered before 1 November, version 3 (30 days) applies. The governance document says so: for returns, the version in force on the delivery date applies. So the inventory must record each version’s effective-from and effective-to dates, and each question needs the date of its event. “Newest” is not a rule.

Check your understanding

Why does Grace's test set include questions that the documents cannot answer, such as 'Do we offer a student discount?'

Show the answer

Because “the documents do not say” is a correct answer, and the assistant must be tested on it. A search always returns some passages, and a model can write a fluent answer from passages that do not answer the question. Without unanswerable questions in the test set, an assistant that always answers would look perfect. With them, you can measure how often it guesses.

Check your understanding

Practical check: run python -m policy_assistant inventory. How many documents are staff-only, and which document holds the rules for disagreements between documents?

Show the answer

Six documents are staff-only: their access is staff. They are all procedures. Their IDs are damaged-in-transit-claims, refund-approval-procedure, safety-escalation-procedure, staff-discount-and-perks, support-team-routing and warranty-claims-procedure. The rules for disagreements are in policy-governance (How Larkfield policies are managed), section When documents disagree.

Recap

• The corpus is the set of documents that the assistant may answer from. The inventory records each document’s metadata: owner, version, effective dates, access label, language and type.
• Larkfield has 36 documents: 32 Markdown, 2 HTML, 2 PDF; 6 staff-only; 2 in French. They are made up for this course.
• The newest version is not always the one in force. For returns, the delivery date decides.
• Grace’s test set has 67 questions of eight kinds, including 8 that the documents cannot answer. Each answerable question names its relevant passage.
• The known-good first run: python -m pytest gives 4 passed, and the inventory ends with 36 documents: 6 staff-only, 2 in French, 4 not Markdown. To reset, delete .venv, or copy a snapshot.

Next, you build the first search over the documents, and you measure how often it finds the relevant passage.

Next: A keyword-search baseline