Final assignment

What you will be able to do

Recommend a model for Larkfield’s bilingual support assistant in a new situation, and write the recommendation so that someone else can check it. You use the fixed 24 tickets and the recorded runs, and set your own must-haves and weights. Then you build a weighted decision table and deliver a one-page memo with its evidence.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

• You completed Modules 1 to 5. Each part below names the lessons that it uses.
• You do not need the case studies of Module 6 for this assignment. (The certificate asks for at least one of them.) To see the whole method on real data first, open Choose a model for bilingual government answers or Choose a model that runs on your own hardware.

What you need

• A browser, and a spreadsheet (any spreadsheet program works) or paper and a calculator.
• A document for the memo.
• No model access and no account. You use the recorded runs. If you want to run a model yourself, label those results live and the course’s results recorded.

Hosted models and prices change often: use the numbers of this page, with their date.

The situation

Larkfield adds a chat window to its website. Customers in Quebec and the rest of Canada now write in English and French in the chat, as well as by email. Grace still wants drafts that an agent checks before sending. Three things change:

1. Volume: about 25,000 tickets a month, instead of about 1,350.
2. Budget: Omar raises the budget for model calls from US$40 to US$150 a month.
3. Speed: a chat customer waits for the answer. Grace sets a new rule: 95% of drafts must be ready within 8 seconds (the 95th percentile, p95, at most 8 seconds).

Everything else stays as in Module 5: the same 24 tickets, the same policy, the same prompt and the same recorded runs. These numbers are fictional; the runs and the prices are real.

The evidence

The table has every option. Quality is Module 4’s rubric: the score of 10, and the replies ready to send, with the 95% interval. Valid JSON and the reply language are simple checks of the raw output. Prices are in US$ per million tokens, from the providers’ pages. The cost per ticket uses the mean tokens of the 24 tickets: input tokens × input price + (output + reasoning tokens) × output price.

Option	Valid JSON (of 24)	Right language, French and mixed (of 12)	Rubric score (of 10)	Ready (of 24, 95% interval)	Median s	p95 s	Price in / out	US$ per ticket	Stable model
Claude Haiku 5.5	24	12	9.84	24 (86% to 100%)	2.2	3.2	0.1 / 0.5	0.00027	yes
Claude Sonnet 5.5	24	12	9.90	22 (74% to 98%)	1.8	3.8	2 / 10	0.00365	yes
Claude Opus 5.5	24	12	10.00	24 (86% to 100%)	3.9	5.7	4 / 20	0.00935	yes
Gemini 3.5 Flash	24	12	9.95	24 (86% to 100%)	4.4	5.9	1.5 / 9	0.01090	yes
Gemini 3.1 Pro (preview)	24	12	9.90	24 (86% to 100%)	9.8	21.5	2 / 12	0.01483	no (preview)
Gemma 3 4B (local)	24	11	5.78	6 (12% to 45%)	4.8	5.8	local	0.00000	yes
Mistral 7B (local)	23	12	7.45	12 (31% to 69%)	4.2	8	local	0.00000	yes
Qwen3 4B (local, 4,000 tokens)	0	12	6.51	14 (39% to 76%)	64.3	140.3	local	0.00000	yes

The local models have no price per token. Their cost is the computer, power and staff time, which this table does not include.

Part 1: representative test cases

Uses Build a representative test set.

1. Describe the 24 tickets in two or three sentences: how many of each kind, and how many in French or mixed languages.
2. Choose five tickets for the memo’s appendix, one of each kind. For each one, write one sentence: what it tests, and what a wrong reply would cost Larkfield.
3. The chat is a new channel. Name two kinds of chat messages that the 24 tickets do not cover. For each one, write one example message, in English or French, and say how you would add such tickets before launch.

Evidence for part 1: the description, the five tickets with your sentences, and the two missing kinds with examples.

What a complete answer has

The counts: 12 routine (6 English, 6 French), 3 difficult, 3 ambiguous, 3 unanswerable, 3 mixed-language; 12 tickets in French or mixed. The five chosen tickets each test something different, for example S07 (French, 8 days late: free shipping, and the language bias seen in Module 2), D03 (French, 17 days late: the “lost” rule), A02 (“Je veux annuler.”: the reply must ask, not guess), U02 (an injury: must go to a person) and M03 (French greeting, asks for English). Missing kinds for chat, for example: very short messages without context (“still nothing??”), several messages that belong to one question, a customer who switches language in the middle of the chat, or messages with typing errors. New tickets are written with an answer key before any model sees them, and get their own kind.

Part 2: transparent criteria

Uses Weighted selection and Licensing and data terms.

Do this part before you compute any total.

1. Write the must-haves. Keep the four of Module 5, with the new budget, and add Grace’s speed rule. For each must-have, write who needs it and why, in one sentence.
2. Choose the weights of quality, cost, latency and privacy. Write one sentence of reason for each. Think about what the chat changes.
3. Write the rule that turns each measurement into a score from 0 to 10. You can keep the rules of Module 5; the cost rule now uses the new budget.

Evidence for part 2: the five must-haves, the four weights with reasons, and the four scoring rules, with the date on which you set them.

Part 3: cost and latency assumptions

Uses Measure practical performance and Fallback and change planning.

1. Compute the cost per month at 25,000 tickets for every hosted option. Show one computation in full, from tokens and prices to US$ per month.
2. Apply the budget and the p95 rule. Write which options fail, and by how much.
3. List your assumptions: every number that you chose or copied, with its source and date.

Evidence for part 3: the monthly costs, one full computation, the options that fail the budget or the speed rule, and the list of assumptions.

Check your numbers

At 25,000 tickets a month: Haiku 5.5 about US$6.75, Sonnet 5.5 about US$91.25, Opus 5.5 about US$233.75, Gemini 3.5 Flash about US$272.50, Gemini 3.1 Pro about US$370.75. Opus, Gemini Flash and Gemini Pro are over US$150. p95 above 8 seconds: Gemini 3.1 Pro (21.5) and Qwen3 4B (140.3). Mistral 7B is at exactly 8.0 and passes, if your rule says “at most 8”. Small differences in the cents are fine if you rounded the tokens.

Part 4: the weighted decision table

Uses Weighted selection and Communicate the recommendation.

1. Remove the options that fail a must-have. Write the failure next to each one.
2. Compute the score of each criterion and the weighted total for the options that are left.
3. Rank them. Say whether the quality scores of the first two are a tie (less than about 0.2 apart on 24 tickets), and which criterion makes the gap.
4. Find the weight at which the leader changes: change one weight in steps of 5 and keep the others fixed.
5. Write one paragraph about what the table does not show, and whether it changes your choice.

Evidence for part 4: the table with the failures, the scores and the totals; the ranking; whether quality is a tie and where the gap comes from; the weight at which the leader changes; your paragraph.

Part 5: limitations and fallback

Uses Uncertainty and errors and Fallback and change planning.

1. Write at least four limitations of your evidence. Turn each one into a check before launch or a trigger.
2. Design the fallback path. For each step, write which failure it covers. Name a low-cost fallback or alternative.
3. Compute what the fallback model costs during a 4-hour outage, and during a whole month if it had to replace the primary model.
4. Write at least four triggers to run the comparison again.

Evidence for part 5: the limitations with their checks, the fallback path, the two costs and the triggers.

Part 6: the memo

Write the memo in the seven parts of Communicate the recommendation: recommendation, requirements, evidence and results, assumptions, limitations, fallback and alternatives, triggers and checks before launch. At most one page, plus an appendix with the tables of Parts 1 to 5. Put the version and the date at the top.

Check your work

There is no reviewer for this course yet. Review your memo with this table, or ask a colleague to use it. Mark each item as demonstrated, partially demonstrated or not yet demonstrated, with a reason. The course is complete when every item is demonstrated. You can improve an item and review it again as often as you need.

Required evidence	Demonstrated when
Representative test cases	The memo describes the 24 tickets by kind and language; five tickets, one of each kind, say what each tests and what a wrong reply costs; two kinds of chat messages that are missing are named, with an example each and a plan to add them before launch.
Transparent comparison criteria	The must-haves are pass-or-fail and each has an owner and a reason; the weights were set before the totals, with a reason each; each score has a one-sentence rule; the table shows the failures, the scores and the totals; the memo says whether the quality scores tie and which criterion makes the lead, and names the weight at which the leader changes.
Cost and latency assumptions	The monthly cost of every hosted option is correct for 25,000 tickets, with one full computation; reasoning tokens are counted as output; the p95 rule is applied with the recorded p95; every chosen or copied number has a source and a date; the memo says that the volume and budget are estimates.
Documented limitations and fallback choice	At least four limitations, each turned into a check before launch or a trigger; the fallback path names the failure that each step covers, and the outage fallback does not share the primary model’s provider; the low-cost fallback or alternative is named with its cost; the fallback’s own settings are stated (for Gemini 3.5 Flash, a large output limit); at least four triggers.

A clear recommendation is not enough on its own. Maybe the memo picks a model that fails a must-have, chooses the weights after the totals, ignores the hidden reasoning tokens in the cost, or names a fallback from the same provider for an outage. Then that item is not yet demonstrated, even if everything else is good.

Common mistake

The table chooses, and nobody reads past it

Symptom: the memo recommends the option with the highest total, and says nothing about what the total leaves out.

Cause: the cost criterion counts only model calls. A model that sends more tickets to a person costs agent time, which is not in the table. At 25,000 tickets, a difference of 2 hand-offs in 24 tickets is about 2,000 extra hand-offs a month.

Fix: after the ranking, read the differences between the first two options ticket by ticket. Turn what matters into a check before launch, for example: “Measure the hand-off rate of the chosen model on 200 new chat tickets; if it is more than X%, use the second option.” A real AI service is judged by its whole cost, not only by its bill for model calls.

Recap

• A selection is a chain of evidence: representative cases, criteria set in advance, measured cost and speed with their assumptions, and an honest account of the limits.
• Must-haves remove options; weights rank the rest; on 24 tickets, quality scores less than about 0.2 apart are a tie, so say which criterion makes the lead.
• A fallback must not share the failure that it covers, and it must be tested with its own settings.
• The memo puts the recommendation first, gives every number a source and a date, and says when to decide again.

Next: if you have not done one yet, complete a case study of Module 6. The certificate asks for at least one: Choose a model for bilingual government answers or Choose a model that runs on your own hardware.