Communicate the recommendation

What you will be able to do

A good choice that nobody can read is lost the first time someone asks “why this model?”. Write the recommendation as a one-page memo with fixed parts. Then Grace can act on it today, and someone else can check it, or redo it, next year.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Larkfield’s must-haves and weights, why Haiku 5.5 ranked first although the quality scores tie, and the weight at which the leader changes, from Weighted selection.
• The fallback path and the triggers to compare again, from Fallback and change planning.
• A 95% interval for a share, from Estimate uncertainty in Essential Mathematics and Statistics for AI.

What you need

• A browser, and a text editor or a document for the memo.

Hosted models change often: your own results will differ.

The mechanic’s report

Your car goes in for a yearly check. A good mechanic does not only say “it’s fine”. The report says what was checked (brakes, tyres, lights) and what was found (front pads at 30%). It says what they recommend (new pads within six months) and what they could not check (the gearbox). And it says when to come back: in 10,000 km, or earlier if the brakes make noise.

You can act on that report at once. A second mechanic can read it next year and continue the work. That is what a recommendation memo must do for a model choice.

The problem: “Haiku won, 9.30”

Amira finishes the table and sends Grace one line: “Haiku 5.5 won with 9.30. Let’s use it.”

See what each reader does with that line:

• Omar reads “won” and tells the managers that Larkfield has the best model. But the four options tied on quality. Haiku won because it is the cheapest.
• Tomás asks why not Opus 5.5, which is 2nd on the leaderboard and had full marks on all 24 tickets. The line does not say that 24 tickets cannot separate Opus from Haiku, or that Opus costs 35 times as much per ticket.
• Camille does not know which French replies were read, or what was found. In one French reply (S11), Haiku named the “Forgot password” link only in English. And it sent the French warranty ticket S12 to a person, although the policy answers it.
• Next year, a retirement notice arrives. Nobody remembers the tickets, the weights or the settings. The comparison starts again from nothing.

How a result was measured matters too. A quick phrase check marked one Haiku reply as wrong because it contains “send us your password”. The full sentence is “Please don’t send us your password or full card number by message.” A phrase check cannot read “don’t”. The rubric of Module 4, scored by reading, gave that reply full marks. So give every result one sentence about how it was measured, and what that measure can and cannot see.

The idea: a decision record

A decision record is a short document that connects three things. The requirements say what the business needs. The evidence is what you measured. The chosen option is what you recommend, with its fallback. It also says what limits the evidence, and what would make you decide again. Teams that build software use decision records for every choice that is hard to undo. A model choice is one of them.

A decision record connects requirements, evidence and the chosen option, and keeps the decision true over time. When a trigger fires, the same tickets run again and the record gets a new version.

Text description of the diagram

Four groups from left to right, joined by arrows. Requirements: the must-haves (valid JSON, French, data terms, budget) and the weights (quality 60, cost 15, latency 15, privacy 10). Evidence: the 24 fixed tickets of five kinds; the recorded runs with scores, seconds and tokens; the prices and terms, with dates. Decision: the chosen model, Claude Haiku 5.5, and the fallback, Gemini 3.5 Flash and then no draft. Keep it true: the limits (24 tickets, one run, a snapshot) and the triggers (a notice, a price change, failures, new kinds of tickets). A dashed arrow goes back from “keep it true” to “evidence”, labelled “a trigger fires: run the 24 again”.

For a model choice, the record has seven parts. Put them in this order, because a busy reader stops after the first one:

1. Recommendation: the model, the fallback, and one sentence of reason.
2. Requirements: the must-haves and the weights, who set them, and why.
3. Evidence and results: the test, the results, and how certain they are.
4. Assumptions: the numbers that you chose, not measured: volume, budget, prices with their date.
5. Limitations: what the evidence cannot show.
6. Fallback and alternatives: what happens when the chosen model fails, and the next choice for other reasons.
7. Triggers and before launch: when to decide again, and what to check before the first customer sees a draft.

A tiny worked example: weak and strong sentences

The difference between a weak memo and a strong one is often one sentence at a time. A strong sentence says what was measured, on what, when, and how sure you are.

Weak	Strong
Haiku 5.5 is the best model.	On 24 test tickets, four options passed the must-haves and could not be separated on quality (rubric scores 9.84 to 10.00 of 10). Haiku 5.5 ranked first (9.30) because it is the cheapest.
It is cheap.	At about 1,350 tickets a month (our estimate) and the current prices, Haiku 5.5 costs about US$0.36 a month.
It is fast.	The median time was 2.2 seconds per ticket, and 95% of tickets took 3.2 seconds or less.
French works.	All 12 French and mixed replies were ready to send by the rubric. One (S11) named a link only in English.

Each strong sentence is longer, but it can be checked. A reader can disagree with “about 1,350 tickets a month”, and see at once what changes.

The parts, on Larkfield’s choice

Results, with their uncertainty

Show the results of the options that passed the must-haves, with an interval. With 24 tickets, a share has a wide interval, and the picture shows it better than any sentence.

Module 4's rubric on the 24 tickets, for the four options that passed the must-haves. Top: ready replies with 95% Wilson intervals. Bottom: each option's rubric score minus Haiku's, with the paired 95% interval. Every difference crosses zero: on 24 tickets, these four options cannot be told apart on quality.

Text description of the chart

Two panels. Upper panel, ready replies, axis from 60% to 100%: Haiku 5.5, Opus 5.5 and Gemini 3.5 Flash 100% (24 of 24, interval 86% to 100%); Sonnet 5.5 92% (22 of 24, interval 74% to 98%). All intervals overlap. Lower panel, the rubric score of each option minus Haiku’s, on the same tickets, axis from −0.4 to +0.4 with a line at zero: Sonnet 5.5 +0.05 (−0.13 to +0.23); Opus 5.5 +0.16 (−0.01 to +0.33); Gemini 3.5 Flash +0.10 (−0.04 to +0.25). Every interval crosses zero.

The intervals tell the reader what “tied” means. Write it in words too: “On 24 tickets, the four options cannot be separated on quality. The choice between them rests on cost and latency, and on the details of the rubric results, as explained below.”

Assumptions

An assumption is a number or a fact that you chose or copied, not measured. Each one gets its value, its source and its date, so that a reader can replace it:

• Volume: about 1,350 tickets a month (about 45 a day).
• Budget: US$40 a month for model calls, about one hour of an agent’s time (Omar).
• Prices: Haiku 5.5 US$0.10 and US$0.50 (for prompts up to 100,000 tokens), Gemini 3.5 Flash US$1.50 and US$9.00 per million input and output tokens. Gemini’s output price includes its thinking tokens.
• Tokens per ticket: the means of the recorded runs. Haiku 5.5 used 796 input and 378 output tokens per ticket.
• The data: the 24 tickets were written for this course. They are not real customer tickets.

Models and prices change fast. These numbers are a snapshot. The method lasts: the tickets, the must-haves, the weights and the scoring rules.

Limitations

A limitation is something the evidence cannot show. Writing it down is not weakness. It tells the reader where the decision could be wrong, and what to check next. Larkfield’s are:

• 24 tickets are a start, not proof. One ticket moves a share by about 4 points. The quality scores of the four options cannot be separated.
• The rubric is a judgement. A second reader scored 28 of the replies. The two readers agreed on all four criteria for 24 of the 28, and on ready or not ready for all 28.
• One run per model. The same ticket can get a different reply on another run. Current Claude models do not accept a temperature setting, so Larkfield cannot ask Haiku to choose the most likely token each time.
• The answer key was corrected once. For D02 (a late refund to an expired card), the key said the policy answers it. The policy does not, so Module 4 changed the key: a person is needed. Every hosted model sent D02 to a person.
• Haiku is cautious on warranty tickets. It sent S06 and S12 to a person, although the policy answers them.

Fallback and alternatives

From the last lesson: if Haiku 5.5 fails (an error, a timeout or invalid JSON), the app calls Gemini 3.5 Flash with an 8,000-token limit. If that fails too, the ticket goes to the agents’ queue with no draft. The fallback is low-cost: it costs nothing while it waits, and about US$0.011 per ticket when it is used. The quality alternative is Opus 5.5, with full marks on all 24 tickets, for about US$12.63 a month. It is the next choice if agents find that Haiku’s drafts need too many fixes. It does not cover an outage, because it comes from the same provider.

Triggers, and what to check before launch

A trigger is an event that makes you run the comparison again. Write each trigger as something a person can see: a retirement notice for Haiku 5.5 or Gemini 3.5 Flash; a price change; a fallback rate above 2% of tickets in a week; agents who send back more than one Haiku draft in ten; agents reporting a kind of ticket that the 24 do not cover; and a fixed date three months after launch.

Before launch, some things must be rechecked, because the evidence is from a test and not from production:

1. Run the 24 tickets with the final prompt and the exact production settings.
2. Score the new replies with the rubric. Camille scores every French reply; Grace and Camille score a shared sample and compare. (On the test runs, two readers agreed on ready or not ready for 28 of 28 replies.)
3. Test the fallback path: switch Haiku off and check that Gemini drafts arrive, marked “fallback”.
4. Check that the data terms of both providers are accepted in writing.
5. Check the prices again on the providers’ pages.

One memo, several readers

Omar wants the cost and what customers will notice. Grace wants the quality bar. Amira wants the settings. Do not write three memos. Put the recommendation and its one-sentence reason first, in plain words, and keep each part short. Put the settings and the full results table in an appendix.

When to use it, and what it costs

Write a decision record for every model choice that reaches customers, and for every choice that someone will have to repeat. Keep it to one page plus an appendix. Store it next to the test tickets and the runs, and give it a version and a date. When a trigger fires, add a new version; do not overwrite the old one.

A one-page memo takes an hour or two to write. Its value comes later: when Omar asks why, when a model is retired, and when a new person joins the team.

See it end to endChoose a model for bilingual government answersCase study

The case study ends with a memo in this format, for real English and French questions.

Common mistake

The memo hides the tie and the limits

Symptom: the memo says “Haiku 5.5 won” and has no limitations section. Three weeks after launch, agents complain that many warranty tickets come to them with no answer. Omar asks why nobody said that Haiku is cautious on warranty tickets, and that it won on cost, not on quality.

Cause: the writer wanted a clear answer, and treated uncertainty as something to hide. The readers trusted the memo more than the evidence allowed.

Fix: state the tie and the reason for the final choice. Give every result with its number of tickets and an interval. Write the limitations, and turn each one into a check before launch or a trigger. A clear recommendation and honest limits do not conflict. For a real AI service, the limits are where the first problems in production come from, so they must be visible before launch.

Check your understanding

Check your understanding

Name the seven parts of Larkfield's decision record, in order, and say why the recommendation comes first.

Show the answer

Recommendation; requirements; evidence and results; assumptions; limitations; fallback and alternatives; triggers and checks before launch. The recommendation comes first because many readers stop after the first part. It must say the model, the fallback and one sentence of reason, so that a reader who reads only that can act correctly.

Module practice

Guided practice → independent task. You saw each part of the memo above, with Larkfield’s numbers. Now deliver the complete selection memo with a low-cost fallback, on your own.

1. Open the explorer in Weighted selection. Keep Grace’s weights and the must-haves on.
2. Write the recommendation in at most three sentences: the model, the fallback, and the reason, including the tie on quality.
3. Write the requirements: the four must-haves and the four weights, with who set each and why.
4. Write the results for the four options that passed. Include the weighted totals, the cost per month, the median and 95th-percentile seconds, and one sentence about uncertainty.
5. Write the assumptions, with values, sources and dates.
6. Write at least four limitations.
7. Write the fallback path. Name a low-cost fallback or alternative, and say which failure it covers and which it does not.
8. Write at least four triggers and at least three checks before launch.
9. Read the memo as Omar. If a sentence uses a word he does not know, change it.

The memo should fit on one page, about 400 to 600 words, plus a small table.

Show a model answer

Model selection for the bilingual support assistant. Version 1, 2026-10-08. Author: Amira, for Grace.

1. Recommendation. Use Claude Haiku 5.5 to draft replies. If it fails, use Gemini 3.5 Flash (8,000-token limit), and then the agents’ queue with no draft. On our 24 tickets, the four options that passed our must-haves tied on quality; Haiku ranked first on our weighted table because it is by far the cheapest and among the fastest.

2. Requirements. Must-haves: valid JSON on all 24 tickets (Grace: the help desk reads it); the right language on all 12 French and mixed tickets (Camille); a paid service that does not train on our tickets, and a stable model (Grace); under US$40 a month (Omar). Weights, set before the results by Grace: quality 60, cost 15, latency 15, privacy 10.

3. Results (24 tickets; quality = the rubric of Module 4, scored by reading). Four of eight options passed the must-haves:

Option	Rubric score (of 10)	Ready (of 24)	Weighted total	US$ a month	Median / p95 seconds
Claude Haiku 5.5	9.84	24	9.30	0.36	2.2 / 3.2
Claude Sonnet 5.5	9.90	22	9.20	4.92	1.8 / 3.8
Claude Opus 5.5	10.00	24	8.81	12.63	3.9 / 5.7
Gemini 3.5 Flash	9.95	24	8.67	14.72	4.4 / 5.9

Out: Mistral 7B and Qwen3 4B (invalid JSON), Gemma 3 4B (wrong language on one mixed ticket), Gemini 3.1 Pro (preview model). Uncertainty: the four rubric scores are within 0.2 of each other, and every paired 95% interval against Haiku includes zero, so we cannot separate them on quality. Haiku’s lead comes from cost: 13 times cheaper per ticket than Sonnet. If cost gets a weight below about 6, Sonnet ranks first, by less than the noise.

4. Assumptions. About 1,350 tickets a month (45 a day). Budget US$40 a month. Prices per million input / output tokens: Haiku 5.5 US$0.10 / US$0.50, Gemini 3.5 Flash US$1.50 / US$9.00 (output includes thinking). Mean tokens per ticket from the runs. The 24 tickets were written for this course. Models and prices change: these numbers are a snapshot.

5. Limitations. 24 tickets, one run per model. The rubric is a judgement: two readers agreed on ready or not ready for 28 of 28 sampled replies, and on every criterion for 24 of 28. The answer key for D02 was corrected (a person is needed). Haiku sent two answerable warranty tickets (S06, S12) to a person, and named a link only in English in one French reply (S11). Current Claude models do not accept a temperature setting, so replies can differ from run to run.

6. Fallback and alternatives. Fallback for errors, timeouts and invalid JSON: Gemini 3.5 Flash at another provider (rubric 9.95, 24 of 24 ready). It costs nothing while unused, about US$0.011 per ticket when used. Last step: the agents’ queue, with no draft. Quality alternative: Opus 5.5 (full marks, about US$12.63 a month). It covers drafts that need too many fixes, not an outage of Anthropic’s service.

7. Triggers and before launch. Run the 24 tickets again on: a retirement notice for Haiku 5.5 or Gemini 3.5 Flash; a price change; a fallback rate above 2% in a week; agents sending back more than one draft in ten; a new kind of ticket from the agents; and three months after launch. Before launch: run the 24 tickets with the final prompt and settings; Camille scores the French replies with the rubric; test the fallback path; accept both providers’ data terms in writing; check the prices again.

A reviewer marks the practice demonstrated when the memo has all seven parts, every number has its source or date, the tie or the uncertainty is stated, the fallback names the failure that it covers, and at least three checks before launch are listed.

Module check

Check your understanding

Retrieval: what is the difference between a must-have and a weighted preference? Give one Larkfield example of each.

Show the answer

A must-have is a pass-or-fail rule: an option that fails it is out, whatever its other scores. Example: valid JSON on all 24 tickets. A weighted preference is something you want more of but can trade: it gets a score from 0 to 10 and a weight. Example: cost, with a weight of 15, scored as 10 × (1 − monthly cost ÷ budget).

Check your understanding

Predict: Anthropic halves the price of Claude Sonnet 5.5, and nothing else changes. What happens to the cost score, the ranking and the memo?

Show the answer

Sonnet’s monthly cost goes from about US$4.92 to US$2.46, so its cost score goes from 8.77 to about 9.39. With Grace’s weights, its total rises by about 0.15 × 0.62 = 0.09, to about 9.29. Haiku has 9.30, so the two are now level: Haiku still leads by about 0.01. The quality scores tie. So the choice now rests on the rubric details: Haiku’s cautious hand-offs against Sonnet’s two French replies that were not ready. The price change is a trigger. Recompute, and add a new version of the memo with the new price, its date and the new reasoning. Keep the old version.

Check your understanding

Completion check: justify Larkfield's choice with task evidence, and identify what should be rechecked before launch.

Show the answer

Justification: on Larkfield’s 24 tickets of five kinds, Haiku 5.5 passed all four must-haves: valid JSON 24 of 24, the right language on 12 of 12 French and mixed tickets, a stable model with checked terms, and US$0.36 a month. Its rubric score was 9.84 of 10, with 24 of 24 replies ready. That cannot be separated from the other three options that passed: the paired intervals include zero. With the weights set in advance (60, 15, 15, 10), it ranked first (9.30), because it is by far the cheapest and among the fastest. The fallback is Gemini 3.5 Flash at another provider, then the agents’ queue. Recheck before launch: the 24 tickets with the final prompt and production settings; Camille’s rubric scores of the French replies; Haiku’s hand-offs on warranty tickets; the fallback path, switched on for real; both providers’ data terms, accepted in writing; the current prices. A choice justified only by a leaderboard or by “9.30 wins” is not demonstrated.

Recap

In this module you:

• Turned business needs into must-haves (pass or fail) and weights (how much each preference counts), and ranked only the options that passed. A public leaderboard chose which models to test, not which to use.
• Read a tie on quality as a result, split a lead into its parts, and found the weight at which the leader changes.
• Planned for outages, retirements and prompts that break on a new model, with a fallback that does not share the failure, and triggers to compare again.
• Wrote a decision record: recommendation, requirements, evidence with uncertainty, assumptions, limitations, fallback and triggers, and the checks before launch.

Reset. Nothing in this module changes files on your computer. To start the explorer again, reload the page of Weighted selection or click Reset to Grace’s weights.

Next: the final assignment asks for your own memo, under different business numbers. The case studies of Module 6 apply the whole method to new data: Choose a model for bilingual government answers and Choose a model that runs on your own hardware.

Next: Final assignment