Choose a model for bilingual government answers

What you will be able to do

Run the whole method of this course on real public data, in two languages. You test four models on official Government of Canada answers, score them blind with a rubric, and measure their speed and cost. Then you write the decision memo, with a fallback.

Case studyPortfolio piece

Dataset

Two Health Canada pages, English and French (Open Government Portal)

Licence

Open Government Licence – Canada (pages); MIT (code)

Time

About 3 hours

Runs in

A spreadsheet; Python 3.12+ optional

Skills used

• Check the licence of a source
• Build a bilingual test set
• Blind rubric scoring
• Latency and cost per question
• Must-haves before scores
• Decision memo with a fallback

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter has the official pages, the 24 questions and the recorded answers of four models. It also has the rubric, a blind scoring sheet, the scripts and a memo template. The finished project adds our scores, results and memo, so that you can compare your work with ours. You need no API key and no programming: a spreadsheet is enough.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses the whole course. The lessons it leans on most:

• The kinds of test question, from Build a representative test set.
• The rubric and blind scoring, from Score outputs consistently.
• Latency, tokens and cost per request, from Measure practical performance.
• Must-haves before weights, from Weighted selection; the fallback, from Fallback and change planning; the memo, from Communicate the recommendation.
• Hosted and local models, from Hosted and self-hosted choices, and data terms, from Licensing and data terms.

What you need

• A spreadsheet program (LibreOffice Calc, Excel, Numbers or Google Sheets), and a program that opens .zip files.
• Some French helps, but you do not need it: the key answers are in English. If you read no French, score the English answers and use our French scores.
• Optional: Python 3.12 or newer, to run the checks and the summary yourself.
• Optional: Ollama, to run a local model. Or your own Anthropic or Google API key, to run a hosted model: 24 questions cost less than 0.40 US dollars.

Hosted models change often: your own results will differ. The method is what lasts.

The situation

Hélène Tremblay runs a community centre in Ottawa that helps newcomers. Its information desk answers questions in English and in French: about health insurance, about the water at home, about government services. Every answer must come from an official source, and the desk links to that source.

The desk is busy. Hélène wants an assistant that drafts an answer from the official page, in the language of the question. A person reads each draft, corrects it and sends it, so the assistant never answers the public alone. But on a busy day, a draft with a wrong fact can slip through: wrong facts are the main risk.

Samir Haddad, a bilingual adviser at the desk, judges the French answers. Ana Lima, a volunteer data analyst, runs the comparison. In this case study, you are Ana. Hélène needs a recommendation that she can defend to the centre’s board: which model, at what cost, and what happens when it fails.

The centre and the people are invented. The pages and every model answer on this page are real.

What you will make

A project folder that another person can check, and a one-page decision memo.

The course's method in four stages. Each stage leaves a file in the project, so another person can check the decision.

Text description of the diagram

The method has four stages. First, the test: official pages in English and French, and 24 questions of four kinds. Second, the runs: one prompt, with the page and the question, goes to four models, and every answer is logged. Third, the evidence: automatic checks, a blind rubric score for each answer, and measurements of time, tokens and cost. Fourth, the decision: must-haves remove options first; the memo states the choice, the fallback, and when to test again.

Set up the project

1. Download the starter with the button in the box at the top of this page. Unzip it. You get a folder government-answers.

2. Open README.md in a text editor, and scoring/blind_answers.csv in your spreadsheet program. The sheet has 96 answers: 24 questions, four answers each.

3. Optional, with Python: open a terminal in the folder, and make a virtual environment.

   macOS

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt

4. Optional: run python scripts/check.py. Part 4 shows its output.

Part 1: the sources, and their licence

The problem: “public” is not “open”

A government page is free to read. That does not make it free to copy into a project and publish. Ana wants to put the page text in the project folder, give it to models, and share the folder. Each of those is a copy.

The general Canada.ca terms say, in short: you may reproduce the pages for non-commercial purposes, “unless otherwise specified”; commercial redistribution needs written permission. A shared project, or a paid service built on it, can be commercial. So the general terms are not enough.

The idea: find the licence of the exact page

Some Canada.ca pages are “otherwise specified”. The department lists them on the Open Government Portal (open.canada.ca) as a dataset, with a licence. Ana searched the portal for frequently asked questions in both languages. She found two pages with a record each:

Page	Topic	Record on the portal	Licence	Date modified
How publicly funded health care coverage works / Comment fonctionne la couverture de soins de santé financés par des fonds publics	Health insurance: moving, travelling, what is covered	Canada Health Act - Frequently Asked Questions	Open Government Licence – Canada	2025-07-16
Questions And Answers On Drinking Water Treatment Devices / Questions et réponses sur les dispositifs de traitement de l’eau de consommation	Water filters, wells, certification	Questions And Answers On Drinking Water Treatment Devices	Open Government Licence – Canada	2023-05-30

Both records list the English and the French page as their resources. The Open Government Licence – Canada lets anyone copy, adapt and distribute the information, also commercially, with an attribution. Its standard attribution is one sentence: Contains information licensed under the Open Government Licence – Canada.

What this tells us. The licence belongs to the page, not to the website: two pages on the same site can have different terms. In dataset.md, Ana records for each page the two URLs, the portal record, the licence, the “Date modified” and the day of the download (2026-10-08). It is a record of what she checked, not legal advice.

The licence also says what it does not give. It does not allow the Government of Canada’s names, logos or symbols. It does not allow a use that suggests official status or endorsement. This matters for the assistant’s design: see Responsible use.

The project keeps the text of each page’s main part, unchanged, in data/pages/. The text keeps the originals’ small typing errors, such as “Ca nada” in the French water page. The models must work with real pages.

Part 2: build the test set

The problem: a few easy questions prove nothing

A model that answers “Is cosmetic surgery covered?” well can still invent a phone number when the page has none. Ten routine questions would hide that. The test set must contain the questions where models fail.

The idea: four kinds, in pairs

Ana writes 12 questions, each in English and in French. A pair asks the same thing in both languages. So a difference between the two answers comes from the language, not from the question. The course’s multilingual kind is built in: every question has its twin. The four other kinds come from Build a representative test set:

Kind	Pairs	What it tests	Example (English)
Routine	6	The page answers it directly	“The water from our cottage well smells strongly of sulphur. Is it dangerous to drink?”
Difficult	2	Two parts, or a trap in the wording	“Our filter is certified to NSF standard 53. Is the water safe after filtering?”
Ambiguous	2	The answer depends on something the question does not say	“Is my tap water safe to drink without a filter?”
Unanswerable	2	The page does not say, but a model may “know” it from elsewhere	“What is the phone number of Quebec’s health insurance board, the RAMQ?”

The routine questions follow the pages’ own questions, in a person’s words. Ana wrote the others for this case study. The trap in the NSF question is real: the page says these filters remove parasites, but “will not remove viruses or bacteria”. The RAMQ question is a trap too. The page has one phone number, and it belongs to a federal office, not to the RAMQ.

Each question has a key answer from the page, in data/questions.csv. Ana writes the keys before she sees any model answer, as in the answer key comes first.

What this tells us. Half of the questions are not routine, far more than the desk’s real mix. That is on purpose: the rare questions carry the risk. And 24 questions can show only a large difference between models: a start, not proof.

Part 3: one prompt, four models

The prompt

Every model gets the same instructions, the page in the language of the question, and the question. The full prompt is data/prompt.md. Its rules:

Answer with one JSON object and nothing else:
{"language": "en" or "fr", "answerable": true or false,
 "answer": "your answer to the person", "quote": "one sentence copied from the page"}

- Use only the page. Do not add facts, numbers, names, addresses or phone numbers that are not on the page.
- Answer in the language of the question (English or French), in 2 to 5 sentences, in plain words.
- "quote": copy one sentence from the page, word for word, that supports your answer.
- If the page does not answer the question, set "answerable" to false, say so, and say where to ask.
- If the answer depends on something the question does not say, say what it depends on.

The quote field is a cheap check against invented facts: a script can see whether the sentence is really on the page. The answerable flag lets the desk send “the page does not say” answers to a person first.

The four models

Model	Where it runs	Why it is in the test
Claude Sonnet 5.5 (claude-sonnet-5-5)	Hosted, Anthropic	Near the top of this course’s earlier runs, at half the time of the largest Claude model
Gemini 3.5 Flash (gemini-3.5-flash)	Hosted, Google	A lower price per token; it reasons before it answers
Gemma 3 4B (gemma3:4b)	Local, Ollama, Mac mini M4 16 GB	No price per question; the questions never leave the centre
Mistral 7B (mistral:7b)	Local, Ollama, same computer	A larger local model

Ana ran all four with scripts/run_models.py, at temperature 0 where the model allows it. Current Claude models reject the temperature setting, so Sonnet ran with its default. She ran the hosted models twice, to see how much the answers change between runs. Every answer is in the project’s runs/ folder.

The same page, a different number of tokens

The first measurement comes before any score. The prompt and the page are the same for all models, but each model’s tokenizer cuts them differently:

Prompt with the page, mean tokens	English	French	French ÷ English
Claude Sonnet 5.5	3,073	4,639	1.51
Gemini 3.5 Flash	1,991	2,734	1.37
Gemma 3 4B	2,001	2,744	1.37

What this tells us. A French question costs Sonnet 51% more input tokens than the same question in English. Part of that is the French page, which has 20% more words. The rest is the tokenizer, which cuts French words into more pieces (why the count matters). So Ana compares the cost per question, in each language, not the price per token.

Part 4: the automatic checks

The problem: 96 answers are a lot to read

Some failures need no judgement: invalid JSON, an English answer to a French question, a quote that is not on the page. A script finds these in a second, so the people who score can spend their time on the meaning.

Run the checks

scripts/check.py checks every answer. Is it valid JSON with the four fields, and in the language of the question? Is the answerable flag right, and is the quote on the page? It also counts the sentences, and looks for phrases that must not appear, such as a lead limit that is not on the page.

python scripts/check.py

anthropic_claude-sonnet-5-5                24 answers | JSON 22 | language 22 | answerable 20 | quote on page 22 | forbidden phrase 0
anthropic_claude-sonnet-5-5_run2           24 answers | JSON 23 | language 23 | answerable 21 | quote on page 23 | forbidden phrase 0
gemini_gemini-3.5-flash                    24 answers | JSON 24 | language 24 | answerable 24 | quote on page 24 | forbidden phrase 0
gemini_gemini-3.5-flash_run2               24 answers | JSON 24 | language 24 | answerable 24 | quote on page 24 | forbidden phrase 0
ollama_gemma3-4b                           24 answers | JSON 23 | language 14 | answerable 20 | quote on page 17 | forbidden phrase 0
ollama_mistral-7b                          24 answers | JSON 24 | language 19 | answerable 19 | quote on page 15 | forbidden phrase 2
Wrote results/checks.csv

What this tells us. Two problems are visible before anyone reads an answer. First, the local models often answer French questions in English: Gemma 3 4B did it for 9 of the 12 French questions, Mistral 7B for 5. Second, Mistral 7B used a forbidden phrase twice: a lead limit that the page does not give. Sonnet’s two failed JSON checks of the first run are worth a look.

Common mistake

The model corrects itself, and breaks the format

Symptom: Claude Sonnet 5.5’s answer to “Does the public health insurance plan pay for cosmetic surgery?” is not valid JSON. The answer itself is correct.

What the model wrote (shortened):

{"language": "en", "answerable": true, "answer": "No. ...", "quote": "Examples of uninsured health services include:"}

Wait, the quote must be a full sentence from the page and support the answer.

{"language": "en", "answerable": true, "answer": "No. ...", "quote": "Services provided in a hospital or by a doctor that are not considered medically necessary are not covered by a public health insurance plan."}

Cause: the model noticed that its first quote broke a rule of the prompt, and wrote a corrected object after it. A person reads this as fine. A program that expects one JSON object fails. The same thing happened on one more question (D1-en), and on another question in the second run.

Fix: the desk’s tool must check the JSON and ask again once when it is not valid. Taking the last object is tempting, but a rule that guesses can hide other errors. Count the retries: they cost time and money.

Consequence in AI work: format compliance is part of quality, not a detail. A model that is excellent on the content can still fail in production on 1 answer in 12. Only a test that uses the real format finds it.

Part 5: score blind with the rubric

A scorer who sees the model’s name reads a famous model more kindly, so Ana scores blind, as in Score blind.

The rubric

This is Module 4’s rubric: the same four criteria, 0 to 2 points each, and the same meaning for each score. It is an adapted rubric in one way: format also asks for a quote copied from the page. Otherwise, only the examples change. The official page takes the place of Larkfield’s policy, and “holding back” means saying that the page does not answer, not sending the question to a person. The full anchors are in the project’s RUBRIC.md:

Criterion	2	1	0
Correctness	Everything agrees with the page; it adds nothing from elsewhere	One small error that would not change what the person does (a softer or stronger wording than the page)	A false or invented fact that could change what the person does: a critical error
Completeness	All the points the person needs	One needed point is missing	The main answer is missing, or there is no answer
Format	Valid JSON with the four fields, the language of the question, 2–5 sentences, a quote copied from the page	Usable after a small fix: a reworded quote, one sentence or more than 5, a code fence, a few words in the other language	The tool cannot use it: not valid JSON, a missing field, or an answer in the wrong language
Abstention	Holds back exactly when it should: says when the page does not answer, or what the answer depends on. The answerable flag agrees with the text	Too cautious: holds back on a question, or a part of one, that the page answers. Or the flag is false on a good answer, so a person checks it first	Not cautious enough: answers as if the page said something that it does not say. Or answers for one case when the answer depends on something unsaid. Or gives no answer

A critical error fails the answer for this use, whatever its total. That rule comes from the situation: wrong facts are Hélène’s main risk.

The blind sheet

scripts/make_blind.py makes scoring/blind_answers.csv. For each question, the four answers get the labels A, B, C and D in a random order, which changes from question to question. The model names are in scoring/blind_key.csv: open it only at the end. The sheet also shows what the automatic checks found, so you do not check those by hand again.

Try it: one question, four answers

Here is the French version of the NSF question: « Nous allons camper et boire l’eau d’un lac. Notre filtre est certifié selon la norme NSF 53. L’eau est-elle sûre après le filtrage? » The key: not on its own; NSF 53 cyst filters remove parasites such as Giardia, not viruses or bacteria; other options exist; ask a specialist.

Before you read our scores, score the four answers yourself, and decide which one you would send.

Label	Answer (as written)
A	“According to the information provided, if your water is contaminated with protozoa like Cryptosporidium or Giardia, a device certified to NSF/ANSI Standard 53 will effectively remove them. However, it doesn’t eliminate viruses or bacteria. To ensure the water is safe, you should consult with a water treatment specialist…”
B	« Pas forcément. Un filtre certifié NSF 53 pour la réduction des kystes parasitiques peut servir à traiter l’eau contaminée par Cryptosporidium, Giardia ou d’autres protozoaires. Par contre, il n’enlève ni les virus ni les bactéries. Pour une eau de lac, la page indique d’autres options : faire bouillir l’eau, la désinfecter chimiquement… »
C	« Un filtre certifié selon la norme ANSI/NSF 53 peut être utilisé en toute confiance pour éliminer les kystes parasitiques comme Giardia ou Cryptosporidium. Cependant, ce type de dispositif ne retirera pas les virus ni les bactéries… L’eau n’est donc pas totalement sûre… »
D	« Oui, l’eau est sûre après le filtrage si votre filtre est certifié selon la norme NSF 53. »

Our scores (correctness, completeness, format, abstention):

Label	Scores	Total	Note	Model
A	2, 1, 0, 2	5	Correct, but in English, and no other options	Gemma 3 4B
B	2, 2, 2, 2	8	Complete, in French	Claude Sonnet 5.5
C	2, 1, 2, 2	7	Correct; no other options	Gemini 3.5 Flash
D	0, 0, 1, 2	3	“Yes, the water is safe”: false. Critical error	Mistral 7B

What this tells us. Answer D is short, confident, in good French, and dangerous: a busy person might send it. Answer A is right but in the wrong language, so the desk would have to translate it. A total of 3 against 8 does not show how different D is: that is why critical errors are counted on their own.

Score all 96 answers, or at least the 48 of one language. It takes about an hour. Then open blind_key.csv, and compare your scores with ours in the finished project. Where you disagree, read the rubric’s anchor again, and write down why. In Module 4’s agreement check, two readers agreed on 24 of 28 replies on all four criteria, and on 28 of 28 for “ready or not”. Expect a few small differences; a difference on “ready” means an anchor needs work.

Part 6: the results

With scoring/scores.csv filled, scripts/summarise.py joins the scores, the runs and the prices:

python scripts/summarise.py

Model                Score/8    EN    FR   Full marks (95%) Ready Critical Median s  p95 s US$/1,000
Claude Sonnet 5.5       7.75  7.58  7.92    20/24 (64%-93%)    22        0      2.5    3.9     10.17
Gemini 3.5 Flash        7.88  7.92  7.83    21/24 (69%-96%)    24        0      4.9    8.4     16.27
Gemma 3 4B (local)      5.88   6.5  5.25      4/24 (7%-36%)     8        0      9.4   19.3       0.0
Mistral 7B (local)      4.71  5.17  4.25      3/24 (4%-31%)     9        7      8.2   21.8       0.0
Wrote results/summary.csv and results/by_kind.csv

Ready is Module 4’s bar: correctness 2 and no criterion at 0, so a person can send the answer after a light check.

• Hosted
• Local

Mean rubric score on the 24 questions. Hosted models in the first colour, local models in the second.

Text description of the chart

A column chart of the mean score out of 8 on the 24 questions. Claude Sonnet 5.5: 7.75. Gemini 3.5 Flash: 7.88. Gemma 3 4B, local: 5.88. Mistral 7B, local: 4.71. The two hosted models are almost equal; the local models are 2 to 3 points lower.

• Hosted
• Local

Critical errors: answers with a false or invented fact that could change what a person does.

Text description of the chart

A column chart of critical errors out of 24 answers. Claude Sonnet 5.5: 0. Gemini 3.5 Flash: 0. Gemma 3 4B: 0. Mistral 7B: 7. Only Mistral 7B made critical errors.

What this tells us. The two charts tell different stories about the local models. Gemma 3 4B makes no critical error, but it loses points everywhere else: the wrong language, a weak format, missing parts. Mistral 7B makes 7 critical errors. In both languages, it gave a RAMQ phone number and a lead limit that are not on the page. In both, it said that the lake water is safe. It also said that a visit for pneumonia is not covered. A mean score alone would hide this.

Quality, time and cost together

• Hosted
• Local

Quality against time. Up and to the left is better.

Text description of the chart

A scatter chart with the median seconds per question on the horizontal axis (0 to 14) and the mean score on the vertical axis (4 to 9). Claude Sonnet 5.5: 2.5 s, 7.75. Gemini 3.5 Flash: 4.9 s, 7.88. Mistral 7B: 8.2 s, 4.71. Gemma 3 4B: 9.4 s, 5.88. The hosted models are faster and better; Sonnet is about twice as fast as Gemini at the same quality.

Per question, mean	Claude Sonnet 5.5	Gemini 3.5 Flash
Input tokens	3,856	2,363
Visible output tokens	246	123
Hidden reasoning tokens (billed as output)	counted in the output	1,291
Price, input / output, US$ per million tokens	2.00 / 10.00	1.50 / 9.00
US$ per 1,000 questions: English / French	8.24 / 12.09	14.15 / 18.39

Prices from the Anthropic and Google pricing pages. Check them again before you use them.

What this tells us. Gemini has the lower price per token and the higher cost per question. Before each answer, it “thinks” for about 1,300 hidden tokens that you pay for (cost per ticket). French costs about 45% more than English with Sonnet, and 30% more with Gemini. For the desk’s 1,000 questions a month, either model costs less than 20 US dollars.

The local models cost nothing per question, but they need a computer and someone to keep them working. They were also slower here. But other programs used the same computer during the runs, so their times are less reliable.

Is the difference between the hosted models real?

Sonnet has 20 answers with full marks out of 24, Gemini 21. The 95% intervals, 64–93% and 69–96%, overlap almost completely: one answer in 24 is noise (how sure 24 tickets can be). On quality, this test cannot separate them. It can separate them on time and cost.

The second run adds a warning. Gemini, at temperature 0, gave a different text for 9 of the 24 questions. Sonnet, which does not accept a temperature setting, gave a different text for 23 of 24. The meaning stayed the same in the answers we read, and the answerable flag changed once. A score from one run is a sample, not a fixed property of the model.

Part 7: where the models fail

By kind of question

Mean score (of 8)	Routine	Difficult	Ambiguous	Unanswerable
Claude Sonnet 5.5	7.83	7.50	7.50	8.00
Gemini 3.5 Flash	8.00	7.50	7.75	8.00
Gemma 3 4B	6.33	6.25	4.25	5.75
Mistral 7B	6.58	3.75	3.75	1.00

What this tells us. On routine questions, all four models come close to usable. The rare kinds are where they separate. Mistral 7B scores 1.00 of 8 on unanswerable questions: it answered all four as if the page said it. With only routine questions, Ana would have ranked Mistral above Gemma, and the desk would have sent invented phone numbers.

By language

Mean score (of 8)	English	French	Difference
Claude Sonnet 5.5	7.58	7.92	+0.33
Gemini 3.5 Flash	7.92	7.83	−0.08
Gemma 3 4B	6.50	5.25	−1.25
Mistral 7B	5.17	4.25	−0.92

What this tells us. The local models lose about 1 point in French, mostly because they answer in English. A French speaker who gets an answer in English is not served, even if every fact is right.

Common mistake

The model knows an answer that the page does not give

Symptom: asked for the RAMQ’s phone number, Mistral 7B answers: “The phone number for Quebec’s health insurance board, the RAMQ, is 1-800-665-5300.” It sets answerable to true. The page has no RAMQ number.

Cause: the model learned phone numbers, limits and rules from its training text. When the page is silent, it fills the gap from that memory, against the prompt’s rule. We did not check whether the number is real. It does not matter: the desk’s rule is “from the official page”, and this number is not on it. Mistral did the same for the lead limit (“0,010 mg/L”) in French.

Fix: test unanswerable questions, in each language, before you choose. In the tool, check every number in an answer against the page. The hosted models passed all four unanswerable questions here, but 4 questions are not proof.

Consequence in AI work: a model’s memory is not a source. For this desk, an answer that the page does not support is a hallucination, even when it happens to be true. In a public service, a wrong phone number sends people to the wrong place.

Two more patterns appear in the notes of scores.csv:

• A rule from the wrong section. Asked about a holiday in the United States, Gemma 3 4B used the page’s rule for travel inside Canada (“covered for emergency hospital and physician services”). It did not say that outside Canada only part of the cost is covered. The answer reads well and misleads.
• A flag that disagrees with the text. Sonnet’s answer to “How long can I stay outside my province?” is one of the best answers in the test. But it set answerable to false, because the page gives no number of days. In the desk’s tool, that flag sends the draft to a person first. Here that is harmless, so the rubric gives abstention 1, not 0.

Part 8: the decision

Must-haves first

A high mean score cannot make up for a failed must-have. Ana writes three from the situation, before she looks at the scores:

1. No critical error. Mistral 7B fails: 7 critical errors.
2. French as good as English, and every French answer in French. Gemma 3 4B fails: 9 of 12 French answers in English, and 1.25 points lower in French.
3. Data terms that fit personal questions. No provider tier that uses the content to improve its products. Google’s pricing page says that its free tier does, and its paid tier does not. So Gemini is acceptable only on the paid tier. Read each provider’s terms as in Licensing and data terms.

Both local models are out. Their main advantage, that questions never leave the centre, does not help if the answers are wrong or in the wrong language. If the centre had a strict rule that no question may leave its building, the decision would change. A local model would then need a larger test, a larger model, or no assistant at all.

Then the order of what matters

The two hosted models pass. Hélène’s order is: quality first, then cost, then time. On quality they are equal on this test. On cost, Sonnet is about 40% cheaper per question. On time, Sonnet is about twice as fast. So Sonnet goes first. Gemini is the fallback: tested on the same 24 questions, with the same prompt, at the same quality.

Sonnet’s one weak point is the format: 1 or 2 answers in 24 that are not valid JSON. The tool can catch that, so it is a design requirement, not a reason to reject the model.

The memo

The finished project’s MEMO.md is one page, with the parts of Communicate the recommendation. Its recommendation:

Use Claude Sonnet 5.5 (hosted, Anthropic) to draft answers, with a person on the desk
checking every draft before it is sent. Use Gemini 3.5 Flash (hosted, Google, paid tier)
as the fallback. Do not use the two local models for this task.

The fallback section says what happens in each case:

Event	What the desk does
Sonnet does not answer, or takes over 10 s, for 5 minutes	Switch to Gemini 3.5 Flash, paid tier, same prompt
Both hosted models are down	Answer by hand from the pages, as today
An answer is not valid JSON	Ask the same model once more; then a person writes it
answerable is false	The draft goes to a person first, flagged

And when to test again: a new model version or a retirement notice, a price change, or a new topic. Also after any critical error in a real draft, and every six months in any case. The project makes that cheap: the same scripts and questions, plus new ones.

Write your own memo in the starter’s MEMO.md before you open ours. Your must-haves may differ. For example, a centre with no budget at all would need a different decision, and a different memo.

Limits

• Small and narrow. 24 questions, 2 pages, 1 department. “No critical error in 24 answers” is compatible with a true rate of up to about 14% (a 95% interval). The person who checks each draft stays necessary.
• One scorer. The author scored all 96 answers with the names hidden. But the author also ran the models, and some styles are easy to recognise. A second scorer should score a sample.
• One day. The hosted models and prices are those of the day of the runs. Answers changed between two runs on the same day.
• Local times. Measured while other programs used the same computer. On a quiet computer they are faster, but the quality does not change: a second Gemma run gave the same 24 texts.
• The page is the reference here. The water page (2023) names “Industry Canada”, today Innovation, Science and Economic Development Canada. The models were scored against the page. A real desk must also keep its sources up to date.

Responsible use

Not an official voice. The licence forbids a use that suggests official status. The assistant drafts for the centre’s staff. Every answer that goes out says that it comes from the centre, links to the official page, and gives the page’s date. It never uses the Government of Canada’s name or symbols as its own.

A person checks every draft. The test found no critical error for the chosen model, but on 24 questions only. The desk’s adviser reads each draft against the linked page before sending it. Answers flagged answerable: false go to a person first.

Personal information. People tell an information desk about their health, their status and their family. Send the model only the question and the page, not names, file numbers or addresses. Use a provider tier that does not use the content to train or improve its products, and check the retention period in its terms.

Equal service in both languages. A model that is good in English and weaker in French gives French speakers a worse service. The paired test set measured this. Keep the pairs in every new test. Add the languages that the desk’s clients really use, as Module 1 did with the tokens per language.

Make it a portfolio piece

This case study shows an employer that you can choose a model with evidence. It has a licence check, a representative test and a blind rubric. It has the cost and latency per question, and a decision with a fallback. Many model choices are made on a demo. Yours can be checked and repeated.

What to put in the repository:

government-answers/
  README.md          what it is, the result in three lines, how to rerun it
  MEMO.md            your decision memo
  dataset.md         sources, licence check, attribution, known gaps
  RUBRIC.md          the four criteria and their anchors
  LICENSE            MIT, for the code
  data/              pages, questions, prompt, prices
  runs/              the recorded answers
  scoring/           the blind sheet, the key, your scores
  results/           checks.csv, summary.csv, by_kind.csv
  scripts/           run_models.py, check.py, make_blind.py, summarise.py

1. Start from your project folder, with your own scores and memo. Make sure that no file contains an API key: keys belong in environment variables only.
2. Make the repository: git init, then git add . and git commit -m "Bilingual model comparison".
3. Push it to GitHub as a public repository. Keep the attribution sentence of the OGL – Canada in the README.
4. Add one result of your own: a model that we did not test, a third page, or your agreement with our scores.

How to describe it in an interview, in two sentences: “I compared two hosted and two local language models for a bilingual information desk, on 24 paired English and French questions from openly licensed government pages, scored blind with a four-criterion rubric, with cost and latency per question. I recommended a model with a tested fallback; the key finding was that the local models failed on unanswerable questions and on French, which a routine-only test would have hidden.”

Go further:

1. A second scorer. Ask a person who reads French to score 20 answers blind. Compute how often you agree, and fix the anchors where you do not.
2. A third page. Find another Canada.ca page with an OGL record on the Open Government Portal. Add four question pairs, and rerun the four models.
3. A cheaper model. Claude Haiku 5.5 cost US$0.10 / US$0.50 per million tokens, about 20 times less than Sonnet. Does it pass the must-haves? Run it, log the cost, and update the memo.
4. A larger local model. If your computer has the memory, test a larger open model. Check whether it answers French in French, and whether it holds back on the unanswerable questions.

Knowledge check

Check your understanding

Gemini 3.5 Flash has a lower price per token than Claude Sonnet 5.5, but costs more per question. Why, and what does that mean for a comparison?

Show the answer

Gemini reasons before it answers: about 1,300 hidden tokens per question, billed at the output price. Sonnet used more input tokens, because its tokenizer cuts the same page into more pieces, but it thought little. So the price per token says little on its own. Compare the cost per question, measured on your own questions and in each language.

Check your understanding

Mistral 7B scored 6.58 of 8 on routine questions, higher than Gemma 3 4B. Why is it still out, and what would a test with only routine questions have shown?

Show the answer

It made 7 critical errors. It invented a phone number and a lead limit, and it said “the water is safe” for a filter that does not remove bacteria. A critical error fails a must-have, whatever the mean score. A routine-only test would have ranked Mistral above Gemma and hidden all of it, because the errors are in the difficult and unanswerable questions.

Check your understanding

Sonnet has 20 of 24 answers with full marks, Gemini 21. A board member says: Gemini is better. What do you answer?

Show the answer

One answer in 24 is noise: the 95% intervals (64–93% and 69–96%) overlap almost completely, and a second run changed many texts. On quality, this test cannot separate the two. The decision rests on what the test can separate: cost per question and time. To separate them on quality, you would need many more questions.

Recap

• Check the licence of the exact page. Canada.ca’s general terms are non-commercial; pages with a record on the Open Government Portal are under the OGL – Canada, with an attribution.
• Build the test in pairs across languages, and include difficult, ambiguous and unanswerable questions: that is where models separate.
• Let a script check the format, the language and the quote. Then score the meaning blind, with anchors, and count critical errors on their own.
• Compare the cost per question, not per token: tokenizers and hidden reasoning change it, and French costs 30–50% more than English here.
• Must-haves remove options before any score. Here they removed both local models.
• Write the memo with the limits, a tested fallback, and the events that start a new test.

Next, the second case study: a privacy rule keeps the data on your own computer, so only local models can be chosen.

Next: Choose a model that runs on your own hardware