Handle difficult evidence

What you will be able to do

Make the assistant honest when the evidence is hard. It says “the documents do not say” when they do not, and shows both sides of a conflict with the rule that decides. It uses the version in force, and never stretches a passage that is only close to the question.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Claims, citations and cite.py, from Answer and cite.
• Versions and effective dates, and the date filter, from Preserve metadata and Lexical, dense and hybrid search.
• Precision, recall and their trade-off, from The precision–recall trade-off in Machine Learning: From Problem to Reliable Model.

What you need on your computer

• ~/projects/policy-assistant after Answer and cite, with .venv active: python -m pytest shows 74 passed, 1 deselected.

This lesson adds no project file. You read ask output, and in the module practice you write a short script. No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are not tested.

A colleague with the policy binder

You ask an experienced colleague a question, and she opens the policy binder. If the binder says nothing about it, she tells you so. If a page covers something close but not your case, she does not stretch it. If two pages disagree, she shows you both and turns to the page with the rules. If there is an old page and a new page, she checks the dates.

A poor colleague answers every time, from memory or from the nearest page. The answer sounds the same either way, so you cannot tell which kind of colleague you have.

The problem: a plausible answer from close evidence

Q16 asks: “What is the warranty on a robotic lawn mower?” Larkfield’s warranty table lists corded lawn mowers (3 years) and the Volt battery range, but no robotic mowers. The documents do not answer.

The local model gemma3:4b answered anyway (Ollama 0.40.1). Here is its answer, in the format of ask, with the project’s checks. The mock has no gemma3:4b recordings, so you cannot replay it:

Answer (answerable): Robotic lawn mowers have a warranty period of 3 years. This warranty covers the robotic lawn mower itself, but not the blades. The warranty policy states that ‘Corded power tools | hedge trimmers such as the HT-550, corded lawn mowers | 3 years | blades: not covered’.
  - Corded power tools | hedge trimmers such as the HT-550, corded lawn mowers | 3 years | blades: not covered [d9a5bd8ea531] ok

The one claim copies a real row of the table, so every check says ok. The wrong part is “Robotic lawn mowers have a warranty period of 3 years”. It is in the answer but in no claim, so no check ever saw it. An agent would read a confident, cited answer.

Both hosted models gave the honest answer. Run it:

python -m policy_assistant ask Q16

Question (date 2026-10-09): What is the warranty on a robotic lawn mower?
Passages: d9a5bd8ea531, e50c70a93a32, 82cf269d9624, a4dc9ec49818, fc2580a01927 (732 tokens, estimated)
Answer (the documents do not answer): The documents do not say what warranty applies to a robotic lawn mower.
gpt-6-luna-2026-09-22 | 1164 tokens in, 78 out | 1.236 s

Same passages, same prompt, two behaviours. This lesson is about getting the second one, and about checking for the first.

The idea: four kinds of difficult evidence

The passages for a question either answer it, or they fall into one of four difficult kinds. Each kind has an honest response.

Four kinds of difficult evidence, each with a real question from Grace's set, and the honest response.

Text description of the diagram

One question starts the diagram: what do the passages say? Four branches follow. First, nothing about it, for example Q14 about a student discount: abstain, the documents do not say. Second, something close but not the answer, for example Q16 about a robotic mower, where the table lists corded mowers: abstain, or answer only what the passages say. Third, two documents disagree, for example Q45, where the help centre FAQ says 3 business days and the refunds policy 5 to 10: show both and apply the rule, a policy wins over a help article. Fourth, two versions, for example Q22 with version 3 of the returns policy, 30 days, and Q23 with version 4, 45 days: use the version in force on the date that decides.

To abstain means to give no answer and say why: “The documents do not say.” It is not a failure of the assistant. For an agent, it is useful information: search somewhere else, or ask Grace.

Absent evidence: abstain

The prompt’s rule 3 asks for this: “If the passages do not answer the question, set answerable to false, write in answer that the documents do not say, and give no claims. Do not guess.” Grace’s set has 8 questions that the documents do not answer. Examples are a student discount, store pickup and the warranty on a vacuum cleaner.

python -m policy_assistant ask Q14

Question (date 2026-10-09): Do we offer a student discount?
Passages: 6ed852264d3b, c13d27947ed1, f02ee0910bbf, b16f983ce250, ea0bff365b2c (410 tokens, estimated)
Answer (the documents do not answer): The documents do not say whether we offer a student discount.
gpt-6-luna-2026-09-22 | 802 tokens in, 67 out | 1.288 s

chat-small and chat-strong abstained on all 8 of these questions. gemma3:4b abstained on 2 of them, answered 5, and failed on 1 with an error from Ollama (HTTP 500). Its 5 answers show how a small model fills the gap:

Question	gemma3:4b’s answer	What the documents say
Q14 student discount	“Larkfield offers a 10% discount to customers who sign up for their newsletter. […] However, there is no specific student discount offered.”	Nothing about students.
Q16 robotic mower	“Robotic lawn mowers have a warranty period of 3 years.”	Corded mowers only.
Q19 refund by Interac e-Transfer	Card refund times: sent within 2 business days, then 5 to 10.	Interac is used when a card is closed; no time is given.
Q20 courier to Nunavut	“Larkfield uses Purolator for express parcels to Nunavut, the Northwest Territories and Yukon.”	A courier code list (PUR: Purolator, for express parcels) and a FAQ about the territories. Neither says this.
Q21 vacuum (in French)	“La garantie d’un aspirateur est de 2 ans pour les modèles PW-2200 et PW-2400.”	The table has no vacuums; the PW models are pressure washers.

Each answer starts from something real in the passages: a newsletter discount, a table row, a code list. That is why they sound right. Q20 joins two true passages into a false fact.

Evidence that is only close: answer only what it says

Q16 and Q20 are the second kind: the passages are about something near the question. The honest answer says what the passages cover and stops there. “The warranty table lists corded lawn mowers (3 years). It does not mention robotic mowers.” The agent then knows what is known and what is not.

Code cannot see this case, as the Q16 example showed: the claim was a true quote, and the false sentence was not a claim. Read the answer text, not only the claims.

Q36 shows the other side: “Can a customer in Halifax choose express delivery?” Both hosted models abstained, because search missed the delivery table. But even that table never names Halifax; it says that express delivery is not available in Nova Scotia. Rule 1 of the prompt forbids outside knowledge, such as where Halifax is. Whether the assistant may add such common facts is Grace’s decision, not the model’s.

Conflicting evidence: show both, then apply the rule

Larkfield has one real contradiction. The help centre FAQ says: “Refunds reach your card within 3 business days after we receive your return.” The refunds and payments policy says two things: Larkfield sends the refund within 2 business days of approval, and the money usually appears within 5 to 10 business days. Larkfield’s governance document settles it with rule 2: a policy wins over a help article.

python -m policy_assistant ask Q45 --model chat-strong

Question (date 2026-10-09): How long does a refund take to reach the customer's card?
Passages: cc7dbb7f12be, 2e762c0284bb, 9c59e2caec7f, 7839ded2d52b, 2c590823561f (451 tokens, estimated)
Answer (answerable): The documents disagree: the FAQ says refunds reach the card within 3 business days after the return is received, while the refunds and payments policy says they usually appear within 5–10 business days after being sent. Both are in force on 2026-10-09, but the Returns policy directs you to the refunds and payments policy for arrival times, so use that timing. Under that policy, the refund is sent within 2 business days of approval.
  - The FAQ says refunds reach the card within 3 business days after the return is received. [cc7dbb7f12be] ok
  - The refunds and payments policy says the money usually appears within 5–10 business days after being sent, depending on the bank. [2e762c0284bb] ok
  - Both documents are in force on 2026-10-09 and give conflicting refund timelines. [cc7dbb7f12be, 2e762c0284bb] CHECK: unsupported:0.29
  - Use the refunds and payments policy's arrival timing because the Returns policy explicitly refers to it. [2c590823561f, 2e762c0284bb] ok
  - The refund is sent to the original payment method within 2 business days of approval. [2e762c0284bb] ok
gpt-6.1-sol-2026-09-29 | 854 tokens in, 424 out | 6.891 s

This is the shape of a good answer to a conflict: both sides, each with its own citation, and a decision. chat-small’s answer has the same shape. The flagged claim is a true summary.

But look at the reason. The model chose the policy because the returns policy refers to it, not because of rule 2. The passage with the rules, When documents disagree in the governance document, was not in the top 5 for any of the four contradiction questions. The model cannot apply a rule that it never sees.

When search also misses one side, the answer is weaker. For Q46, “The help centre FAQ says refunds take 3 business days. Is that right?”, the policy’s 5-to-10-day passage was not retrieved. Both models reported the FAQ’s 3 days and said that the passages do not confirm it, but neither could give the right number. Run python -m policy_assistant ask Q46 to see it.

So a conflict needs two things in the context: both sides, and the rule. Two fixes are possible: always add the governance rules to the context, or write the precedence rules into the prompt. Each one changes the requests, so each must be measured on the whole question set before you keep it. Find where it fails in Module 5 shows how to test one change at a time.

The safety notice for the KT-180 kettle (Q48) tests rule 1, a safety notice wins over every other document. Both models answered it correctly. The notice itself says: “This offer applies even if the return window or the warranty has ended.” So no rule was needed.

Outdated evidence: the version in force

Larkfield’s returns policy has two versions. Version 3 gives 30 days and applies to orders delivered until 31 October 2026. Version 4 gives 45 days and applies from 1 November 2026. Version 4 is already published on the course’s date, 9 October 2026, but it is not yet in force. The governance document says that for returns, the delivery date decides.

Which document applies depends on the delivery date. Five of Grace's version questions on the timeline of the returns policy, the 2025 holiday notice and the help centre FAQ.

Text description of the diagram

A timeline from September 2025 to January 2027. Row 1, the returns policy: version 3, 30 days, in force from 1 March 2025 to 31 October 2026, then version 4, 45 days, from 1 November 2026, with no end date. Row 2, the holiday returns notice 2025, in force to 31 January 2026. Row 3, the help centre FAQ, from 1 June 2026, which says 30 days, also after 1 November: it goes out of date on that day. A dashed vertical line marks 1 November 2026: version 4 starts. Row 4, the delivery dates of five questions: Q25 on 1 December 2025, during the holiday notice; Q28 on 3 October 2026 and Q22 on 20 October 2026, both under version 3; Q23 on 5 November 2026 and Q26 on 10 December 2026, both under version 4. A short mark shows the course date, 9 October 2026.

The question date of Q22 is its delivery date, 20 October, so the date filter lets only version 3 into the context. The answer says 30 days. Q23’s delivery date is 5 November, so only version 4 comes in:

python -m policy_assistant ask Q23

Question (date 2026-11-05): An order was delivered on 5 November 2026. Can the customer return an unused item on 15 December 2026?
Passages: 62603868db65, 89af4f96760d, 963dae4229eb, c236034e24d2, 1afe5bb16a9e (463 tokens, estimated)
Answer (answerable): Yes. The Returns policy version in force on the delivery date allows 45 days for unused items, so 15 December is within the return window. The help-centre FAQ says 30 days, but the policy specifies that the version in force on the delivery date applies.
  - Version 4 of the Returns policy was in force when the order was delivered, and it provides a 45-day return window. [62603868db65, 89af4f96760d, 963dae4229eb] ok
  - An unused item delivered on 5 November can be returned on 15 December within that 45-day window. [963dae4229eb] ok
  - The FAQ says 30 days, but the policy says the version in force on delivery applies. [c236034e24d2, 62603868db65] ok
gpt-6-luna-2026-09-22 | 874 tokens in, 499 out | 3.722 s

The FAQ, still saying 30 days, was in the context too. The answer names it and explains why the policy wins. That is outdated evidence handled well.

What happens without the date filter? Amira recorded the version questions once more with the filter off (these recordings come with Module 6). Both versions reached the context, and the models still chose correctly on most of them. chat-strong on Q22: “Version 4 allows 45 days, but it takes effect on 1 November 2026 and does not apply to this delivery.” The dates in the header did the work. The filter still keeps the wrong version out of the budget and out of the citations.

Q26 shows the cost of the filter. “Is there a holiday return extension for an item delivered on 10 December 2026?” With the filter, the expired 2025 notice is out of the context, and both models answer “The documents do not say whether there is a holiday return extension.” That is safe, but not helpful. Without the filter, chat-small answered: “No holiday return extension is stated for an item delivered on 10 December 2026; the 2025 notice had ended, so only the returns policy applies. The applicable return window is 45 days.” The passage that proves that something ended is itself out of date. A filter cannot know that.

The trade-off: abstain or answer

Abstaining has a cost too. Treat “abstain” as an alert that the assistant raises, as in The precision–recall trade-off. Here are chat-small’s 67 recorded answers with reranked search:

Real condition	Abstained	Answered	Total
Documents do not answer	8right to abstain	0answered without support	8total
Documents answer	5abstained, answer existed	54answered	59total
Total	13total	54total	67total

• Share of real cases that are flagged (recall): 8 ÷ 8 = 100%
• Share of flags that are real (precision): 8 ÷ 13 = 61.5%

chat-small (gpt-6-luna on Azure), reranked search: it abstained on all 8 questions that the documents do not answer, and on 5 of the 59 that they answer.

Text description of the table

A table with two rows and two columns. Rows: the documents do not answer (8 questions) and the documents answer (59 questions). Columns: abstained and answered. Documents do not answer: 8 abstained, right to abstain; 0 answered without support. Documents answer: 5 abstained although an answer existed (Q13, Q26, Q28, Q36, Q60); 54 answered. Totals: 13 abstentions and 54 answers. Share of the unanswerable questions where it abstained (recall): 8 ÷ 8 = 100%. Share of the abstentions that were right (precision): 8 ÷ 13 = 61.5%.

And gemma3:4b’s 65 completed answers with the same search (2 requests failed with HTTP 500):

Real condition	Abstained	Answered	Total
Documents do not answer	2right to abstain	5answered without support	7total
Documents answer	0abstained, answer existed	58answered	58total
Total	2total	63total	65total

• Share of real cases that are flagged (recall): 2 ÷ 7 = 28.6%
• Share of flags that are real (precision): 2 ÷ 2 = 100%

gemma3:4b (Ollama 0.40.1), reranked search: it abstained on only 2 of the 7 unanswerable questions that it answered, and never on an answerable one.

Text description of the table

The same table for gemma3:4b, 65 completed answers; Q15 and Q31 failed with an HTTP 500 error from Ollama. Documents do not answer: 2 abstained, right to abstain; 5 answered without support (Q14, Q16, Q19, Q20, Q21). Documents answer: 0 abstained; 58 answered. Totals: 2 abstentions and 63 answers. Recall of abstention: 2 ÷ 7 = 28.6%. Precision: 2 ÷ 2 = 100%.

The two models sit at opposite ends. chat-small catches every unanswerable question and pays with 5 needless abstentions. gemma3:4b never abstains needlessly and pays with 5 unsupported answers.

Look closer at chat-small’s 5. For Q13, Q28, Q36 and Q60, the passage with the answer was not in the top 5. From what the model saw, abstaining was right. For Q26, the date filter removed the expired notice that the answer needs. So these are problems of search and context, not of the model. Better search and better context fix them, not a bolder model. Find where it fails sorts them stage by stage.

Which end is better for Larkfield? An agent who reads “the documents do not say” loses a minute: they search, or ask Grace. An agent who reads a confident wrong answer may tell a customer the wrong warranty. For a help desk, a wrong answer costs more than an abstention. So Grace prefers high recall of abstention, and the team fixes the false abstentions in search.

When to abstain, and what it costs

Abstain when no passage answers the question, or when the passages are only close to it. Do not abstain when two documents disagree: show both, and name the rule if it is in the context. Every needless abstention is a question that the agent must answer alone. So measure both errors on your question set, and fix needless abstentions where they start, usually in search.

Common mistake

Every claim is cited, so the answer is supported

Symptom: Omar asks for a quick quality check before the pilot: “Count the answers where every claim says ok.” Tomás reports that 4 of gemma3:4b’s 5 answers to unanswerable questions pass every check. Q16 is one of them: “Robotic lawn mowers have a warranty period of 3 years.”

Cause: the checks look only at claims. gemma3:4b’s claim for Q16 is a true quote of the warranty table. The false statement is in the answer text, and no claim carries it. An answer that should be an abstention can still pass every claim check.

Fix: check the answer, not only its claims. For an unanswerable question, the right result is answerable: false with no claims, and Grace’s question set says which questions those are. Module 5 turns this into a measure: abstention on the unanswerable questions, and support for the claims, scored separately. Until then, read the answer text of every answer whose question is unanswerable.

What it means for a real AI service: a support check is only as good as what it reads. A model can put its guess in a sentence that is not checked.

Module practice: supported and unanswerable questions

Ask the assistant 4 questions that the documents answer and 4 that they do not, and look at the result of each. Start python in the project folder:

Python, in ~/projects/policy-assistant

from policy_assistant.store import Store
from policy_assistant.embed import LocalEmbedder
from policy_assistant.rerank import Reranker
from policy_assistant.search import Retriever
from policy_assistant.providers import MockProvider
from policy_assistant.evaluate import load_questions
from policy_assistant.assistant import ask

retriever = Retriever(Store("index.sqlite"), LocalEmbedder(), Reranker())
questions = load_questions()
for qid in ["Q24", "Q30", "Q38", "Q48", "Q14", "Q15", "Q16", "Q20"]:
    q = questions[qid]
    r = ask(q.question, q.as_of, retriever, MockProvider(), "chat-small", "rerank", language=q.language)
    flagged = sum(not c.ok for c in r.checks)
    print(f"{qid} {q.kind:12} {'answered ' if r.answer.answerable else 'abstained'} claims {len(r.answer.claims)} flagged {flagged}")

Press Enter on the empty line after the loop. The output:

Q24 version      answered  claims 2 flagged 0
Q30 table        answered  claims 1 flagged 0
Q38 exact-code   answered  claims 1 flagged 0
Q48 contradiction answered  claims 3 flagged 0
Q14 unanswerable abstained claims 0 flagged 0
Q15 unanswerable abstained claims 0 flagged 0
Q16 unanswerable abstained claims 0 flagged 0
Q20 unanswerable abstained claims 0 flagged 0

Now your task:

1. Look at the claims of the four supported questions. In the same Python session, print them with their cited passages:

   Python (continued)

   r = ask(questions["Q48"].question, questions["Q48"].as_of, retriever, MockProvider(), "chat-small", "rerank")
   for claim in r.answer.claims:
       print(claim.text)
       for cid in claim.chunk_ids:
           print("   ", cid, retriever.store.chunk(cid).text.replace("\n", " "))

   Do the same for Q24, Q30 and Q38. For each claim, decide: does the passage say it? Write “accept” or “reject” and one reason.

2. Change the list to ["Q05", "Q22", "Q32", "Q19", "Q21"], and change "chat-small" to "chat-strong". Predict, for each question, whether it answers or abstains and whether a claim is flagged. Then run the loop.

3. For every flagged claim, run ask with --model chat-strong and decide: a true conclusion, or an unsupported claim?

4. Compare with gemma3:4b’s answers to Q19 and Q21 in the table of Absent evidence. Write one sentence for Grace about each model’s behaviour on questions that the documents do not answer.

Show the answers for steps 2 and 3

Q05 answerable   answered  claims 2 flagged 0
Q22 version      answered  claims 2 flagged 1
Q32 table        answered  claims 2 flagged 1
Q19 unanswerable abstained claims 0 flagged 0
Q21 unanswerable abstained claims 0 flagged 0

Two claims are flagged, and both are true conclusions. Q22: 1 December is 42 days after delivery, and the window is 30. Q32: a glass panel that cracks after 18 months is outside its 1-year warranty. Accept both, after you check the arithmetic.

For Grace: “chat-strong abstained on all 8 questions that the documents do not answer. gemma3:4b answered 5 of the 7 that it completed, each from a nearby passage, so a person must read every answer.”

Module check

The completion check for this module: reject fabricated citations, and find unsupported claims even in plausible answers. Six claims follow, real and constructed. For each one, read the question, the answer, the cited passage and what cite.py reports. Then decide.

Explore

Accept or reject the claim

Answered

0 of 6

Right

0 of 0

Claim 1 of 6: Read the passage and the check. Do you accept the claim?

Claim 1 of 6: A number from the passage, in the wrong place

gemma3:4b (local, Ollama 0.40.1)

Question Q50, date 2026-10-09

Can I give an unhappy customer a goodwill credit? How much?

Answer

You can give an unhappy customer a goodwill store credit of up to 25 dollars without approval, once per customer in 12 months [9c59e2caec7f]. However, for amounts over 25 dollars, a team lead or support lead approval is required [2f517f5673f1]. Refunds over 300 dollars require approval from the support lead (Grace) [e50c70a93a32].

Claim

Refunds over 300 dollars require approval from the support lead (Grace) [e50c70a93a32].

cites e50c70a93a32

→

Cited passage

[e50c70a93a32] Warranty claims procedure | procedure, version 3, in force from 2026-01-01, no end date | section: Approval limits

Approval limits

- An agent may approve a repair, replacement or refund worth up to 300 dollars.
- From 300 dollars up to 1,000 dollars, a team lead must approve.
- Over 1,000 dollars, the support lead (Grace) must approve.

What cite.py reports

- Refunds over 300 dollars require approval from the support lead (Grace) [e50c70a93a32]. [e50c70a93a32] ok

The claim is false. Reject: not supported. The passage says that a team lead approves from 300 to 1,000 dollars, and the support lead over 1,000. The claim reuses 300 for another rule. The numbers and words are all in the passage, so every check passes. Only reading catches it.

Source: recordings/local_gemma3.jsonl, Q50, gemma3:4b, claim 3 (a course recording, not in the project)

Six claims with their cited passages and the real result of cite.py. Real claims show their model and recording date; one claim is constructed and labelled.

Text description of the explorer

Six claims, one at a time. Each card shows the question, the answer, the claim with its cited IDs, the cited passages, and the line that cite.py prints. One control: accept the claim; reject it because the citation is not real; or reject it because the passage does not support it.

1. Q50, gemma3:4b, recorded: “Refunds over 300 dollars require approval from the support lead (Grace)”, citing the warranty claims procedure’s approval limits. The check says ok. Reject: not supported. The passage says that a team lead approves from 300 to 1,000 dollars, and the support lead over 1,000.
2. Q50, gemma3:4b, recorded: “For amounts over 25 dollars, a team lead or support lead approval is required”, citing the approval limits for manual refunds. The check says number_not_found:25. Reject: not supported. The passage gives 100 and 500 dollars.
3. Q42, gemma3:4b, recorded: “The GR-310 needs three BR-310-2s”, citing the BBQ parts passage. The check says number_not_found:2,BR-310. Accept: the passage says three are needed; the plural “s” confused the number check.
4. Q23, chat-strong, recorded: “15 December is within the 45-day return window for an order delivered on 5 November 2026”, citing the version 4 Return window passage. The check says unsupported:0.43. Accept: 15 December is 40 days after 5 November.
5. Constructed: “Unused items can be returned within 30 days of delivery”, citing 7839ded2d52c. The check says missing_id. Reject: the citation is not real. The real passage is 7839ded2d52b.
6. Q16, gemma3:4b, recorded: a claim that copies the corded power tools row of the warranty table. The check says ok. Accept the claim: the table says it. But the answer around it, “Robotic lawn mowers have a warranty period of 3 years”, is not supported.

You have finished the module when:

• python -m pytest shows 74 passed, 1 deselected, and python -m policy_assistant ask Q22 gives the answer of Answer and cite.
• You can reject a made-up citation, such as c23617e24d2, and explain which check catches it.
• You can find the unsupported claim in a plausible answer, such as gemma3:4b’s “Refunds over 300 dollars require approval from the support lead”, even when every check says ok.
• You can explain why chat-small abstained on Q13, Q28, Q36 and Q60, and why that is a search problem.

Reset. If your project is broken, rename it and copy the snapshot end-of-m04, as in Use a snapshot:

macOS / Linux

cd ~/projects
mv policy-assistant policy-assistant-old
cp -R nextia-learning-labs/C09/snapshots/end-of-m04 policy-assistant

Windows (PowerShell)

cd ~\projects
Rename-Item policy-assistant policy-assistant-old
Copy-Item -Recurse nextia-learning-labs\C09\snapshots\end-of-m04 policy-assistant

Make the virtual environment again, install the requirements and run python -m policy_assistant ingest, as the snapshots README shows. python -m pytest then prints 74 passed, 1 deselected. The snapshot has the guided steps of this module, not the module practice. These steps were tested on macOS; the Windows commands are not tested.

See it end to endAnswer questions from public guidanceCase study

The public guidance case study answers questions from government pages with citations, and measures abstention on questions that the guidance does not cover.

Knowledge check

Check your understanding

chat-small abstained on all 8 unanswerable questions, and on 5 answerable ones. Use precision and recall to describe this, and say which error matters more for Larkfield's agents.

Show the answer

Treat abstention as the alert. Recall: 8 of the 8 unanswerable questions, 100%. Precision: 8 of its 13 abstentions were right, 61.5%. For a help desk, a missed abstention (a confident answer without support) costs more than a needless one (the agent searches or asks Grace). In 4 of the 5 needless abstentions, search missed the passage, so the fix is in search.

Check your understanding

For Q26, the date filter removed the expired 2025 holiday notice, and both models said that the documents do not say. Without the filter, chat-small answered correctly. Should Larkfield turn the filter off?

Show the answer

No. Without the filter, wrong versions reach the context and the citations, and they use the budget. The cost of the filter is that the model cannot say that the notice ended. One possible improvement: keep recently expired notices searchable, with their end date in the header, and measure the change on the whole question set.

Check your understanding

In Q45, both models showed both sides of the refund contradiction. Why could they not cite rule 2, and what could make them apply it?

Show the answer

The passage with the rules, When documents disagree, was not in the top 5. So the model never saw rule 2. chat-strong chose the policy for another reason: the returns policy refers to it. To apply the rule, the rule must reach the model: always add the governance rules to the context, or write the precedence into the prompt. Either change must be measured on the whole question set before it is kept.

Recap

• Evidence can be absent, only close, conflicting or outdated. The honest responses: abstain; answer only what it says; show both sides and the rule; use the version in force.
• To abstain is to say “the documents do not say”. The hosted models abstained on all 8 unanswerable questions; gemma3:4b on 2.
• A conflict needs both sides and the rule in the context. The date filter keeps the wrong version out, and can also remove the passage that says something ended.
• Abstention is a precision and recall trade-off. For a help desk, a missed abstention costs more; most needless abstentions are search problems.
• Claim checks do not read the answer text.

Next, Module 5 measures all of this on the whole question set: retrieval first, then the answers.

Next: Retrieval measures