What retrieval-augmented generation is

What you will be able to do

Understand what retrieval-augmented generation adds to a language model, and what it cannot fix, before you build one. You see real answers of a model with and without Larkfield’s documents, and you learn when another approach fits better.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• How an application sends a prompt to a model and reads a structured answer, from Building Reliable Applications with LLM APIs. This course reuses its adapter, its mock provider and its recorded answers.
• What a context window is, from Context and memory in LLMs: How They Work and How to Choose One. It helps, but it is not required.

What you need

• A browser. This lesson has nothing to install. You set up the project in the next lesson.

No account and no payment. Search runs on your own computer. The answers in this course are real answers of real models, recorded once and replayed by the project. A live model is optional, never required.

Models change fast, so the numbers in this course are a snapshot. The method lasts longer than the numbers.

An open-book exam

Think of two exams on the same law course. In the closed-book exam, you write from memory. If you do not remember a rule, you guess, and the examiner cannot see where your answer came from. In the open-book exam, you may bring the textbook. A good student does not read the whole book during the exam. She finds the two or three pages that matter, reads them, writes the answer and notes the page: “see section 4.2”.

The open-book answer is better for three reasons. It comes from the book, not from memory. The examiner can check it, because the page is named. And if the book does not cover the question, an honest student writes “the book does not say”, instead of a guess.

The problem: the model alone

Larkfield, the home-and-garden shop of the earlier courses, now wants a policy assistant for its help desk. An agent asks a question in plain words, for example “Can a customer return a used pressure washer after 45 days?” The assistant must answer from Larkfield’s own policy documents. Grace, the support lead, owns those documents. Amira, a developer, builds the assistant.

Amira starts with the simplest test: she asks a model with no documents. This is the closed-book exam. She sent all 67 of Grace’s test questions to two models on Azure: gpt-6-luna (Larkfield calls it chat-small) and gpt-6.1-sol (chat-strong). The prompt said only that the model works for Larkfield’s help desk. These are six of the real answers.

1. What is the return window for unused items today?
   The model alone (chat-small): I don’t have Larkfield’s current return policy details here, so I can’t confirm the return window for unused items. Please check the Returns page or your order confirmation.
   Larkfield's documents: 30 days from delivery (returns policy version 3, in force until 31 October 2026). Version 4, with 45 days, applies to orders delivered from 1 November 2026.
2. Which pump kit fits the PW-2200?
   The model alone (chat-strong): I can’t confirm the compatible pump kit from “PW-2200” alone. Could you share the unit’s brand and serial number, or a link to its product page, so I can help identify the correct kit?
   Larkfield's documents: PK-2200-07 (it fits the PW-2200 only; the PW-2400 uses PK-2400-03).
3. A customer has not bought anything from us for two years. What happened to their Rewards points?
   The model alone (chat-small): Their Rewards points expired after 24 months without a purchase.
   Larkfield's documents: They have expired: points expire after 18 months without any purchase.
4. A Rewards member's order was delivered on 3 October 2026. Do they pay for the return label?
   The model alone (chat-small): No. As a Rewards member, they receive a free return label; the order delivered on 3 October is also within the return window.
   Larkfield's documents: Yes. Version 3 of the returns policy applies (delivered before 1 November 2026): 12.95 dollars of return shipping is taken from the refund. The free label for Rewards members starts with version 4.
5. The help centre FAQ says refunds take 3 business days. Is that right?
   The model alone (chat-small): Yes, refunds typically take 3 business days to process. Your bank or payment provider may take a little longer to show the funds in your account.
   Larkfield's documents: No. The refunds and payments policy says the money usually appears within 5 to 10 business days after the refund is sent (within 2 business days of approval). The policy wins over the FAQ (rule 2); report the disagreement to Grace.
6. Une cliente a reçu sa commande le 10 novembre 2026. Combien de jours a-t-elle pour retourner un article non utilisé?
   The model alone (chat-small): Elle dispose de 30 jours à compter de la réception de la commande pour retourner un article non utilisé.
   Larkfield's documents: 45 jours après la livraison (politique de retour, version 4, pour les commandes livrées à partir du 1er novembre 2026). La FAQ du Québec, qui dit 30 jours, est un article d'aide plus ancien : la politique l'emporte.

Read the first two answers. Most answers were like them: the model said that it does not have Larkfield’s policy, and asked for it. That is honest, but it does not help the agent. Then read the next four. Here chat-small answered with confidence, and it was wrong. Larkfield’s Rewards points expire after 18 months, not 24. The free return label starts only for orders delivered from 1 November 2026. The help centre’s “3 business days” is the one sentence that the refund policy contradicts. And in French, it gave 30 days for a delivery that falls under the new 45-day policy.

By the automatic check of this course, chat-small answered 10 of the 59 answerable questions correctly, and chat-strong 7. Some of those were luck: “2,400 PSI” for the PW-2400 pressure washer is in the product’s name. No answer named a document, and no answer could know which version of a policy applies today. So the problem is bigger than invented facts. The agent needs evidence: the passage that the answer comes from, in the version that applies.

Why not give the model every document?

Tomás, an AI engineer at Larkfield, has a shortcut: “The collection is small. Put all of it in the prompt.” It is a fair idea, and Amira tested it. All 36 documents, cut into 185 passages, take about 17,848 tokens. A token is a piece of a word; models count and charge their input in tokens.

Input tokens for one question to chat-small: the whole collection in the prompt (20 questions), and only the 5 best passages that search found (67 questions).

Text description of the chart

A column chart with two columns. With every document in the prompt, the median question takes about 17,850 input tokens. With only the 5 best passages, it takes 905. The first is about 20 times the second.

On 20 test questions, the whole collection did as well as the five best passages, even a little better. With 36 short documents, it fits easily. But each question sends 20 times more text, and the cost grows with it. The prompt also carries the six staff-only documents to every user, and it grows with every new document. A shortcut that works today can fail next year. Assemble context in Module 4 measures this in full: cost, staff text and citations.

The idea: find the pages, then answer from them

The open-book exam gives the idea. Before the model answers, the application finds the few passages that matter. A passage is a short piece of a document, such as one section under a heading. The application puts those passages into the prompt with the question. Then the model writes the answer from them and names the passage behind each claim.

This is retrieval-augmented generation, or RAG. The name lists the three steps. Retrieval: search the documents and return the best passages. Augmentation: add those passages to the prompt, so the prompt is bigger than the question alone. Generation: the model writes the answer. A citation is the reference to the passage that supports a claim, like the page number in the exam.

Retrieval-augmented generation as an open-book exam. The no branch matters as much as the yes branch.

Text description of the diagram

The diagram has two parts. The first part, open the book, is the search. An agent’s question and Larkfield’s documents (36 files, with versions and access labels) go into step 1, retrieve. Retrieval returns the best passages, and each passage keeps its ID, its version and its dates. The passages go into the prompt. The second part, write the answer, has step 2, augment: one prompt with the rules, the question and the passages. Step 3, generate: the model writes from the passages only. Then a decision: do the passages answer the question? If yes, the answer has a citation for every claim. If no, the assistant says that the documents do not say, and the agent asks Grace. The no branch is drawn dashed: it is a correct result, not an error.

Look at the decision at the end. A good assistant has two correct results, not one. It answers with citations, or it says “the documents do not say”. An assistant that always answers will sometimes answer from nothing, like the closed-book model.

One question through the three steps

Follow the Rewards question through the pipeline, with the real recorded outputs. The question: “A customer has not bought anything from us for two years. What happened to their Rewards points?”

Without documents, chat-small answered: “Their Rewards points expired after 24 months without a purchase.” It sounds sure, and it is wrong.

Step 1, retrieve. The search finds five passages. The first one is the section Expiry of Larkfield’s Rewards policy, with its ID, version and dates:

The first passage, as the model sees it

[df98766ea4cf] Larkfield Rewards | policy, version 2, in force from 2026-02-01, no end date | section: Expiry
Larkfield Rewards
Expiry

Points expire after 18 months without any purchase.

Step 2, augment. The project puts the rules, the question and the five passages into one prompt: about 950 tokens.

Step 3, generate. The same model, chat-small, now answers from the passages. This is the project at the end of the course, replaying the recorded answer:

python -m policy_assistant ask Q05 (course end)

Question (date 2026-10-09): A customer has not bought anything from us for two years. What happened to their Rewards points?
Passages: df98766ea4cf, 726059051b70, 39da3ee0855c, d0b1b8403935, 1b1c047750bf (555 tokens, estimated)
Answer (answerable): Their Rewards points expired after 18 months without a purchase, so after two years without buying, the points have expired.
  - Rewards points expire after 18 months without a purchase. [df98766ea4cf] ok
  - After two years without a purchase, the customer’s points have expired. [df98766ea4cf] CHECK: unsupported:0.43
gpt-6-luna-2026-09-22 | 948 tokens in, 154 out | 1.548 s

The answer is now right, and each claim names its passage, so an agent can check it in seconds. Look also at the last claim. The code in the project marks it CHECK: the passage does not say “two years”; the model worked that out. Here the reasoning is right, but code cannot prove it. Module 4 teaches what such a check can and cannot do.

What the documents change for Larkfield

The same two models answered all 67 questions twice: once alone, once with the five best passages. This is the end result of the course, so you can see where the work goes.

Correct answers to the 59 answerable questions, by the course's automatic check: each model alone, and with the five best passages from the course's final search.

Text description of the chart

Four rows on a scale from 0 to 59. The model chat-small alone has 10 correct answers, and chat-strong alone has 7. With the five best passages, chat-small has 51 and chat-strong has 50.

With the passages, both models answer most questions correctly. The stronger and more expensive model is not better here: the evidence matters more than the model. The 8 questions that the documents cannot answer are not in this chart. With passages, both models said “the documents do not say” on all 8. Here is one of them:

python -m policy_assistant ask Q14 (course end)

Question (date 2026-10-09): Do we offer a student discount?
Passages: 6ed852264d3b, c13d27947ed1, f02ee0910bbf, b16f983ce250, ea0bff365b2c (410 tokens, estimated)
Answer (the documents do not answer): The documents do not say whether we offer a student discount.
gpt-6-luna-2026-09-22 | 802 tokens in, 67 out | 1.288 s

The search still returned five passages, about a staff discount and discount codes. The model read them and saw that none of them answers the question. That is the “no” branch of the diagram.

Where it fails

RAG moves the problem: the answer is now only as good as the passages. Each step can fail, and the course gives each failure its own lesson.

• The right document is missing, or not readable. Most warranty periods are only in a table inside a PDF file. A search that cannot read the PDF never finds them. Module 2 reads PDF and HTML files.
• The search misses the right passage. The question says “garden sprinkler”, and the table says “Watering”. Module 3 compares search methods on exactly such questions.
• The wrong version. Version 4 of the returns policy (45 days) is already published, but version 3 (30 days) applies until 31 October 2026. Module 3 adds a date filter.
• The model does not settle a conflict. Asked whether the help centre’s “3 business days” for refunds is right, chat-small put the help article and the returns policy side by side. It did not apply Larkfield’s rule that a policy wins over a help article. Search had also missed the refund policy’s passage with the real time. Module 4 handles conflicts and abstention.
• The citation is real, and the claim is still wrong. One supplier document says “lifetime warranty”, and the warranty policy says 1 year. Citing the supplier passage does not make the claim true. Modules 4 and 6 return to it.

Module 5 measures each step on its own, so that you can say which step failed.

When to use something else

RAG is not always the right tool. Three other approaches fit some jobs better.

A longer prompt. If the documents are few, short, public and stable, put them all in the prompt. You saw that it works on Larkfield’s 36 documents. It costs more tokens per question, and it stops working when the collection grows or has staff-only text. Context and memory in the language models course explains why a large context window does not mean that the model uses everything in it.

Search alone. Sometimes the agent only needs the right page, not a written answer. An agent who types a part number such as PK-2200-07 wants the parts table. Search with no model is cheaper and faster, and it cannot invent anything. You build that search in the third lesson of this module.

Fine-tuning. Fine-tuning trains a model further on your own examples. It changes how the model behaves, for example how it sorts messages. The case study Fine-tune a small transformer in Deep Learning and Transformers Explained shows it. It is a poor way to teach facts that change. Larkfield’s return window changes on 1 November, and a fine-tuned model cannot cite the page that it learned from.

What RAG costs: an index that you must keep up to date, a search on every question, and a longer prompt. In this course, search runs on your computer for free. One answer from chat-small with five passages costs about US$0.00017.

What you will build

In this course you build policy-assistant, a Python project, one module at a time. It runs in a terminal, and you write it in VS Code. At the end of the course, an agent’s question gives this:

python -m policy_assistant ask Q22 (course end)

Question (date 2026-10-20): An order was delivered on 20 October 2026. Can the customer return an unused item on 1 December 2026?
Passages: 7839ded2d52b, c236034e24d2, beb2688295be, 3b5ac04156e7, 1b1c047750bf (664 tokens, estimated)
Answer (answerable): No. The return window is 30 days from delivery, so 1 December is outside the window for an item delivered on 20 October.
  - Unused items can be returned within 30 days of delivery. [7839ded2d52b, c236034e24d2] ok
  - 1 December is outside the 30-day return window for an item delivered on 20 October. [7839ded2d52b] ok
gpt-6-luna-2026-09-22 | 1047 tokens in, 237 out | 2.348 s

The search found version 3 of the returns policy, because the order was delivered before 1 November. The answer has two claims, each with its passage IDs, and the code checked both. Here is the path:

1. This module: the documents, Grace’s 67 test questions, and a keyword search with its first score. Every later step must beat that score.
2. Module 2: read Markdown, HTML and PDF files, cut them into passages, and give each passage an ID and its source.
3. Module 3: search by meaning with a small open model, combine search methods, and filter by date and access.
4. Module 4: build the prompt, ask for cited claims, check every citation in code, and abstain.
5. Module 5: measure search and answers, and find which step failed.
6. Module 6: update and delete documents, protect staff-only text and hand the system over.

Everything runs on your computer. The answer step replays the recorded answers of the real models, so you need no key. If you want, you can connect a free local model or your own key later.

You work with Larkfield’s team. Amira, a developer, builds the assistant. Grace, the support lead, owns the policies and the questions. Mei, a data engineer, prepared the data in the SQL course. Omar, the product manager, asks what agents see and what it costs. Tomás, an AI engineer, suggests shortcuts. Priya, a data scientist, owns the measures. Camille, an agent in Montreal, asks questions in French.

Common mistake

The answer sounds sure, so the model must know

Symptom: Tomás asks the model alone about the Rewards points. It answers in one clear sentence: “Their Rewards points expired after 24 months without a purchase.” No hedge, no question back. He passes it to an agent, who tells a customer.

Cause: a model writes the most likely text, not a checked fact. Larkfield’s policy was never part of its training, so it filled the gap with a common rule from other shops. A confident tone is part of the likely text; it is not evidence.

Fix: never accept a policy answer without its passage. Ask the model to answer only from retrieved passages, with a citation for each claim, and to say “the documents do not say” otherwise. With the passage Expiry in the prompt, the same model answered 18 months and cited it.

What it means for a real AI service: judge an answer by its evidence, not by its tone. An assistant that cannot point to a source should not answer a policy question.

Knowledge check

Check your understanding

In your own words: what do the three steps of retrieval-augmented generation do, and what does the assistant do when the documents do not answer the question?

Show the answer

Retrieval searches the documents and returns the few passages that best match the question, each with its ID and version. Augmentation puts those passages into the prompt with the question and the rules. Generation is the model writing the answer from those passages, with a citation for each claim. When the passages do not answer the question, the assistant says “the documents do not say” instead of guessing. Then the agent asks the owner of the policy.

Check your understanding

Omar says: 'The model alone declined most questions. So it is safe: it never says anything wrong.' What do the recordings show?

Show the answer

The model alone declined most questions, but not all. chat-small answered some with confidence and was wrong:

• 24 months for Rewards points; the policy says 18.
• A free return label for an October delivery; it starts in November.
• “Yes” to 3 business days for refunds; the policy says 5 to 10 business days after Larkfield sends the refund.
• 30 days, in French, for a November delivery; the new policy gives 45.

It also never named a source or a version. So an agent cannot tell a right answer from a wrong one.

Check your understanding

Larkfield wants the assistant to always write replies in Grace's friendly style. Is retrieval or fine-tuning the better fit for that, and why?

Show the answer

Fine-tuning, or clear instructions in the prompt, fits a style better. A style is a behaviour that stays the same, and fine-tuning changes behaviour. Retrieval fits facts that change and must be cited, such as the return window. The two can be combined: the style from the model or the prompt, the facts from the retrieved passages.

This course ends with three case studies. Each one builds a cited assistant on real open documents, and each one is a portfolio piece.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Answer questions from public guidanceBuild a cited question-answering assistant over open government guidance pages, with abstention for questions the guidance does not cover and a measured evaluation.
• Search technical documentationCompare keyword, vector and hybrid retrieval with reranking on openly licensed technical documentation, where exact names and paraphrases both matter.
• Answer from versioned regulationsAnswer questions about rules that change over time, retrieve the version in force on a given date, and cite it.

Recap

• Retrieval-augmented generation (RAG) is an open-book exam for a model. Retrieve the best passages, augment the prompt with them, then generate an answer with a citation for every claim.
• Alone, Larkfield’s models mostly declined. Sometimes they answered with confidence and were wrong (24 months instead of 18). They never named a source or a version. With five passages, chat-small answered 51 of 59 answerable questions correctly.
• A correct assistant has two results: a cited answer, or “the documents do not say”.
• Every step can fail. A document can be missing or unreadable, or the search misses a passage. The version can be wrong. The model can ignore the evidence, or cite a real passage for a wrong claim.
• A longer prompt fits a few short, public, stable documents. Search alone fits when the agent needs the page. Fine-tuning changes behaviour, not facts that change.

Next, you meet Larkfield’s documents and Grace’s questions, and you set up the project on your computer.

Next: Corpus and questions