Weighted selection

What you will be able to do

Turn Larkfield’s business needs into a decision you can check. First, a few must-haves remove the options that cannot work. Then weights say how much each remaining quality matters, and the options are ranked on your own measured evidence.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The 24 test tickets (written for this course, not real customer tickets) and their five kinds, from Build a representative test set.
• How the replies were scored, from Score outputs consistently.
• The quality–latency–cost table, and why one or two tickets of difference is noise, from Measure practical performance.
• Hosted and local models, and what data terms say, from Hosted and self-hosted choices and Licensing and data terms.
• A mean, and an interval on a test score, from Describe a dataset and Estimate uncertainty: Error bars on a leaderboard, in Essential Mathematics and Statistics for AI.

What you need

• A browser. There is no code in this lesson.

Hosted models change often: your own results will differ.

Choosing a flat to rent

Think of how you choose a flat to rent. Some things are not negotiable. You have a dog, so the flat must allow pets. The rent must fit your budget. A flat that fails one of these is out, even if it has a beautiful balcony. You do not trade “no dogs” against “lots of light”.

Among the flats that are left, you have preferences: light, a short trip to work, a quiet street. These are not equal. The trip matters every day, the balcony only in summer. So, without saying it, you give each preference a weight. A flat that is a little darker but 20 minutes closer can win.

Choosing a language model for Larkfield works the same way, in two steps and in this order: first remove what cannot work, then weigh what is left. This lesson makes both steps explicit. Then Grace, Omar and anyone else can see how the choice was made.

Two ways to choose badly

Tomás’s shortcut: the public leaderboard

Tomás has an answer before the meeting starts. “Take the model at the top of the leaderboard. Millions of people voted. Why test it ourselves?”

A public leaderboard ranks models by one score from a general test. On the Text Arena leaderboard, people compared answers to their own prompts and voted for the better one. Claude Opus 5.5 (with high effort) was 2nd of 414 models, Gemini 3.1 Pro (preview) 17th, Gemini 3.5 Flash 28th and Claude Sonnet 5.5 32nd. Claude Haiku 5.5 was not on the list.

Now put Larkfield’s own evidence next to it. The quality is Module 4’s rubric score on the 24 tickets, recorded on the same day:

Model	Leaderboard rank	Rubric score (of 10)	Ready replies (of 24)	Median seconds per ticket	US$ per 1,000 tickets
Claude Opus 5.5	2	10.00	24	3.9	9.35
Gemini 3.1 Pro (preview)	17	9.90	24	9.8	14.83
Gemini 3.5 Flash	28	9.95	24	4.4	10.90
Claude Sonnet 5.5	32	9.90	22	1.8	3.65
Claude Haiku 5.5	not listed	9.84	24	2.2	0.27

On quality, Tomás’s pick does well: Opus 5.5 scored 10.00. But so did nearly every hosted model. In Measure practical performance, 24 tickets could not tell them apart. The clear differences are in the columns that the leaderboard does not show. Opus costs 35 times as much per ticket as Haiku. Gemini Pro takes more than twice as long as Gemini Flash. And Gemini Pro is a preview model, which the provider can shut down at short notice.

The leaderboard is not wrong. It answers a different question: “Which model do many people prefer for their own prompts?” Larkfield’s question is narrower. Which model follows this policy, in English and French, returns valid JSON, sends the right tickets to a person, and fits this budget? No public leaderboard measures that. Even its French ranking is uncertain. Opus 5.5 was 11th there, from only 201 votes. The page gives its possible rank as anywhere from 1st to 110th.

Use a leaderboard to choose which models to test. Do not use it to choose which model to use.

Omar’s shortcut: add everything up

Omar likes numbers. “Then give each model one score for quality, cost, latency and privacy, take the average, and pick the highest.”

Amira tries it with equal weights. The winner is Mistral 7B, a local model. It costs nothing per ticket and keeps the data in the building, so it scores 10 on cost and on privacy. But on the recorded run, it returned invalid JSON on 1 of the 24 tickets. The help desk reads the JSON to fill in the draft. One ticket in 24 with no draft is about 55 tickets a month. A high score on cost cannot repair a broken format. The average hid a failure that makes the option unusable.

Both shortcuts fail for the same reason: one number replaces the questions that the business really asks.

The idea: must-haves first, then weights

The method has two steps, as with the flat.

1. A must-have requirement is a pass-or-fail rule. If an option fails it, the option is out, whatever its other scores. Write each must-have so that the evidence answers it with yes or no: “valid JSON on all 24 tickets”, not “good format”.
2. A weighted preference is a quality that you want more of, but can trade. Each one gets a score from 0 to 10 and a weight: how much it counts. The weighted total of an option is the sum of each score times its share of the weight.

These weights are not the model’s weights of Module 1, the numbers it learned in training. Here, a weight only says how much a preference counts.

A table that holds the options, their scores, the weights and the totals is a weighted decision table. It is also called a decision matrix.

The order matters. Must-haves come first, so that no weight can buy back a failure. Then the weights rank only the options that can really work.

A tiny worked example

Take three options and two preferences: quality, with weight 70, and cost, with weight 30. The weights add up to 100, so the shares are 70% and 30%. The must-have is “valid JSON on all 24 tickets”.

Option	Valid JSON	Quality score	Cost score	Weighted total
Sonnet 5.5	24 of 24	9.90	8.77	0.7 × 9.90 + 0.3 × 8.77 = 9.56
Haiku 5.5	24 of 24	9.84	9.91	0.7 × 9.84 + 0.3 × 9.91 = 9.86
Mistral 7B	23 of 24: out	7.45	10.00	not computed

Mistral is out before any multiplication. Between the two that are left, Haiku wins with these weights. Change the weights to 100 and 0, so that only quality counts: Sonnet gets 9.90 and Haiku 9.84. The weights are a decision, and the result depends on them. That is why you write them down.

Where the scores come from

A score from 0 to 10 is not measured. You make it from a measurement, with a rule. The rule must be short enough to say in one sentence, so that someone else can check it. These are the rules in this lesson:

Criterion	What is measured	The rule for the score
Quality	Module 4’s rubric: 4 criteria, 0 to 2 points each, on each of the 24 tickets	mean points per reply ÷ 8 × 10
Cost	Cost per month at 1,350 tickets	10 × (1 − monthly cost ÷ the budget of US$40)
Latency	Median seconds per ticket	10 at 1 second, 0 at 21 seconds, a straight line between
Privacy	Where the ticket text goes	10 on Larkfield’s computer, 5 hosted with checked terms

For example, Sonnet 5.5 costs US$0.00365 per ticket, so US$4.92 a month at 1,350 tickets. Its cost score is 10 × (1 − 4.92 ÷ 40) = 8.77. The volume and the budget are fictional Larkfield numbers. Larkfield gets about 45 tickets a day, so about 1,350 a month. Omar’s budget, US$40 a month, is about one hour of an agent’s time. The prices are real, from the providers’ pages.

Larkfield’s must-haves

Grace, Omar and Camille agree on four must-haves. Each one comes from a person and a reason.

• Valid JSON on all 24 tickets (Grace). The help desk reads the JSON. A reply in any other form is a ticket with no draft.
• The right language on every French and mixed-language ticket (Camille). Twelve of the 24 tickets are in French or mix French and English. A customer who gets a reply in the wrong language has a bad experience, whatever the content.
• Data terms and a stable model (Grace, from Licensing and data terms). A paid service that does not train on Larkfield’s tickets, and a model with a published retirement policy. A preview model can be shut down at short notice: Google shut down Gemini 3 Pro preview 16 weeks after its release.
• Under US$40 a month at 1,350 tickets (Omar).

The must-haves remove four of the eight recorded options before any weight is applied. The number after each name is its result on that check.

Text description of the diagram

Eight options recorded on the 24 tickets enter on the left. The first check asks “valid JSON on all 24?”: Mistral 7B (23 of 24) and Qwen3 4B (0 of 24; it writes its reasoning before the JSON) are out. The second check asks “right language on all 12 French and mixed tickets?”: Gemma 3 4B (11 of 12) is out. The third check asks “data terms and a stable model?”: Gemini 3.1 Pro is out, because it is a preview model. The fourth check asks “under US$40 a month?”: all four remaining options pass. Claude Haiku 5.5, Claude Sonnet 5.5, Claude Opus 5.5 and Gemini 3.5 Flash go on to the weights and the ranking.

Every local model is out. That is not a rule against local models. Each one failed a check that the business needs, on this test. The case study Choose a model that runs on your own hardware starts from a privacy must-have that only local models can pass.

Larkfield’s weights

Grace sets the weights before she looks at the totals: quality 60, cost 15, latency 15, privacy 10. Her reasons, in one sentence each:

• Quality 60. A person checks every draft, but a wrong draft costs that person time and can reach a customer.
• Cost 15. All four options are under the budget, so cost should decide only between options that are close.
• Latency 15. An agent opens a ticket and waits for the draft. A few seconds matter; a fraction of a second does not.
• Privacy 10. All hosted options passed the data-terms must-have. The weight still rewards an option that keeps the text in the building.

Now use the explorer. Make your prediction first.

Explore

Must-haves first, then weights

First place

Haiku 5.5 (9.30)

Lead over second

0.10 over Sonnet 5.5

Options in the ranking

4 of 8

Weights: quality 60%, cost 15%, latency 15%, privacy 10%. Haiku 5.5 ranks first with 9.30 of 10. Sonnet 5.5 is second, 0.10 behind. Their quality scores (9.84 and 9.90) are too close to separate on 24 tickets, so the lead comes from the other criteria. 4 options are out before the weights apply: Mistral 7B, Gemini Pro, Gemma 3 4B, Qwen3 4B.

• Quality
• Cost
• Latency
• Privacy
• Out: fails a must-have

Scores from 0 to 10, the weighted total, and the must-haves

Rank	Option	Quality × 60%	Cost × 15%	Latency × 15%	Privacy × 10%	Total	Must-haves
1	Haiku 5.5	9.84	9.9	9.4	5	9.30	passes all four
2	Sonnet 5.5	9.90	8.8	9.6	5	9.20	passes all four
3	Opus 5.5	10.00	6.8	8.6	5	8.81	passes all four
4	Gemini Flash	9.95	6.3	8.3	5	8.67	passes all four
out	Mistral 7B	7.45	10	8.4	10	–	Valid JSON on all 24 tickets: 23 of 24
out	Gemini Pro	9.90	5	5.6	5	–	Data terms and a stable model: preview model: can be shut down at short notice
out	Gemma 3 4B	5.78	10	8.1	10	–	Right language on every French and mixed ticket: 11 of 12
out	Qwen3 4B	6.51	10	0	10	–	Valid JSON on all 24 tickets: 0 of 24

The eight options recorded on Larkfield's 24 tickets, ranked by their weighted total. The bars show how much each criterion gives to each total. Options that fail a must-have are out (dashed) unless you switch the must-haves off.

Text description of the explorer

The explorer has four sliders, one weight each for quality, cost, latency and privacy, from 0 to 100. Each weight’s share of the total is shown next to it. A checkbox applies the must-haves; it starts checked. The readout shows the option in first place, its lead over the second, and how many options are in the ranking. Below it are stacked bars, one per option, and a table with each score, the weighted total and the must-haves.

The scores from 0 to 10 for quality, cost, latency and privacy are: Haiku 5.5: 9.84, 9.9, 9.4, 5. Sonnet 5.5: 9.90, 8.8, 9.6, 5. Opus 5.5: 10.00, 6.8, 8.6, 5. Gemini 3.5 Flash: 9.95, 6.3, 8.3, 5. Gemini 3.1 Pro (preview): 9.90, 5.0, 5.6, 5. Gemma 3 4B: 5.78, 10, 8.1, 10. Mistral 7B: 7.45, 10, 8.4, 10. Qwen3 4B: 6.51, 10, 0, 10.

With Grace’s weights (60, 15, 15, 10) and the must-haves on, Haiku 5.5 ranks first with 9.30, Sonnet 5.5 is second with 9.20, then Opus 5.5 with 8.81 and Gemini 3.5 Flash with 8.67. Gemini 3.1 Pro, Gemma 3 4B, Mistral 7B and Qwen3 4B are out. With equal weights, Haiku 5.5 ranks first with 8.54 and Sonnet 5.5 is second with 8.32. With equal weights and the must-haves off, Mistral 7B ranks first with 8.96, Haiku 5.5 is second with 8.54 and Gemma 3 4B third with 8.46, although Mistral fails the JSON check and Gemma the language check.

With equal weights, Haiku 5.5 still wins, with 8.54: it is the cheapest option that passes. Switch off the must-haves, and Mistral 7B takes first place with 8.96: the failure that Omar’s average hid. Turn the must-haves on again, and no weight can bring Mistral back. That is the point of the first step.

Reading the result: where does the lead come from?

With Grace’s weights, Haiku 5.5 has 9.30 and Sonnet 5.5 has 9.20. Haiku leads by 0.10. Is that a real lead, or noise? Split each total into its parts:

Option	Quality part	Cost part	Latency part	Privacy part	Total
Haiku 5.5	0.60 × 9.84 = 5.90	0.15 × 9.91 = 1.49	0.15 × 9.39 = 1.41	0.50	9.30
Sonnet 5.5	0.60 × 9.90 = 5.94	0.15 × 8.77 = 1.32	0.15 × 9.61 = 1.44	0.50	9.20
Difference	−0.04	+0.17	−0.03	0	+0.10

The quality part favours Sonnet by 0.04. That is one rubric point on one of the 24 tickets. In Measure practical performance, you saw that 24 tickets cannot separate the hosted models. In score points, the paired 95% interval for Haiku − Sonnet runs from −0.23 to +0.13: it includes zero. A useful rule here: on 24 tickets, two rubric scores that are less than about 0.2 apart are a tie. All four hosted options that passed, from 9.84 to 10.00, are within it.

The cost part is different. Haiku costs US$0.00027 per ticket and Sonnet US$0.00365, 13 times more. That comes from token counts and prices, and another run of the 24 tickets would not change it much. So the honest reading is: on quality, the options tie; Haiku 5.5 leads because it is much cheaper and almost as fast. The explorer says the same in its status line.

A tie on quality is useful. It tells you that the choice depends on cost and on things the table does not show. The rubric results of Module 4 show three of them:

• Hand-offs. Haiku sent 2 warranty tickets to a person, S06 and S12, although the policy answers them. That is cautious, not harmful, but each one is work for an agent. Sonnet sent none.
• French replies. Sonnet had 2 French replies that were not ready: S09 thanks the customer for a photo that was never sent, and S12 contradicts itself. Haiku’s French replies were all ready.
• The money is small. At 1,350 tickets a month, Haiku costs about US$0.36 and Sonnet about US$4.92. Even Opus 5.5, with full marks on all 24 tickets, costs only about US$12.63.

Grace recommends Haiku 5.5. The test cannot separate the options on quality, and Haiku is the cheapest and among the fastest. She also names Opus 5.5 as the next choice if agents find that Haiku’s drafts need too many fixes. She writes the tie and her reasons in the memo. You write that memo in Communicate the recommendation.

How sensitive is the choice?

A decision that flips with a small change of weight is fragile, so find the point where it flips. Keep quality at 60, latency at 15 and privacy at 10, and change the cost weight:

Cost weight	First	Second	Gap
0	Sonnet 5.5	Haiku 5.5	0.08
5	Sonnet 5.5	Haiku 5.5	0.01
10	Haiku 5.5	Sonnet 5.5	0.05
15 (Grace)	Haiku 5.5	Sonnet 5.5	0.10
30	Haiku 5.5	Sonnet 5.5	0.24

Below a cost weight of about 6, Sonnet ranks first. But its lead then comes only from quality, which is a tie, and from a median that is 0.4 seconds shorter. So the table never chooses Sonnet with confidence. The memo must say: “If cost does not count, Sonnet 5.5 and Haiku 5.5 tie. If cost counts at all, Haiku 5.5 is first.” A reader who disagrees with a weight can see what their own weight would choose.

When to use it, and what it costs

Use a weighted decision table whenever more than one person has a say, or the choice must be explained later. It makes each judgment visible: who set each must-have, each weight and each scoring rule.

Rules that keep it honest:

• Set the weights before you see the totals. Otherwise you move them until your favourite wins (see the common mistake below).
• Keep must-haves few and real. If you would give up a must-have for a good price, it is a preference. Give it a weight instead.
• Use few criteria. Every new criterion takes weight from the others. Four is enough here.
• Look for criteria that cannot change the ranking. If every remaining option has the same score on a criterion, its weight does nothing. Here, privacy is 5 for every hosted option that passed. It counts again when the options change.
• The table ranks evidence. It does not make evidence. With 24 tickets, quality scores less than about 0.2 apart are a tie, whatever weight quality has.

The cost is time: an hour to agree on must-haves and weights, and the discipline to write them down. That is much less than the cost of a choice that nobody can explain when a model is retired, as the next lesson shows.

Guided practice: equal weights and the must-haves

Use the explorer above.

1. Click I have made my prediction. Set all four weights to 25.
2. Write down the option in first place and its total.
3. Clear Apply the must-haves first. Write down the new first place, and which must-have it fails. The last column of the table shows the failure.
4. Select the checkbox again. Click Reset to Grace’s weights.
5. Write one sentence that explains why the two rankings differ.

Expected results

Step 2: Haiku 5.5, 8.54. Step 3: Mistral 7B, 8.96. It fails “Valid JSON on all 24 tickets: 23 of 24”. Haiku 5.5 is second with 8.54, and Gemma 3 4B third with 8.46; Gemma fails the language check. Step 5, for example: “Without the must-haves, the local models win on cost and privacy, and the weights cannot see that one of them breaks the help desk’s format.”

Your turn: privacy becomes a must-have

Change one condition. Larkfield’s lawyer says that ticket text must not leave Larkfield’s own computers. Privacy is now a must-have, not a weight.

Which options pass all five must-haves? What do you recommend to Grace?

Example answer

No option passes. The new must-have removes every hosted option: the three Claude models and both Gemini models. The three local models had already failed: Mistral 7B and Qwen3 4B on valid JSON, Gemma 3 4B on the language of one mixed ticket.

The right recommendation is “none of these options, yet”. Do not quietly loosen a must-have to get a winner. Tell Grace what would have to change. One way is to test larger local models. Another is to change how the app calls a local model, for example by checking the JSON and asking again when it is invalid. Then measure the 24 tickets again. The case study Choose a model that runs on your own hardware does this work.

Common mistake

The weights were chosen after the results

Symptom: the team’s favourite model always ranks first, and the weights have odd values such as “quality 47, latency 23”. Nobody can say why latency counts 23.

Cause: someone looked at the totals and moved the weights until the favourite won. The table now records a preference, not a reason.

Fix: write the must-haves, the weights and the scoring rules before you compute any total. Add who set each weight, the reason and the date. If you change a weight later, keep the old one and write why, then show both rankings. A real AI service is compared again many times: when a model is retired, a price changes or new kinds of tickets appear. Weights with a written reason make each new comparison quick and fair.

Check your understanding

Check your understanding

Tomás says that the model at the top of the leaderboard will be best for Larkfield. Give two reasons from this lesson why that does not follow.

Show the answer

The leaderboard measures a different task: people’s preferences on their own prompts. It does not test Larkfield’s policy, French replies, JSON format or hand-offs. And it ignores Larkfield’s constraints: cost, latency and data terms. On the 24 tickets, the hosted models could not be separated on quality (9.84 to 10.00 of 10). But Opus 5.5 (2nd) cost 35 times as much per ticket as Haiku 5.5 (not listed). Gemini 3.1 Pro (17th) was a preview model, and it took more than twice as long as Gemini 3.5 Flash (28th). A leaderboard is a good list of models to test, not a choice.

Check your understanding

Why must the must-haves be applied before the weights, and not as one more weighted criterion?

Show the answer

A weight lets one quality make up for another. A must-have is something that nothing can make up for: an option with invalid JSON breaks the help desk, whatever it costs. As a weighted criterion, a large score on cost and privacy could make up for it. That is what happened with equal weights and no must-haves: Mistral 7B ranked first although it fails the JSON check.

Check your understanding

With Grace's weights, Haiku 5.5 has 9.30 and Sonnet 5.5 has 9.20. Omar says: Haiku is the better model. What do you answer?

Show the answer

Not better: cheaper. Split the gap of 0.10 into parts. The quality part favours Sonnet by 0.04, one rubric point. On 24 tickets, the quality scores are a tie: less than 0.2 apart, and the paired interval includes zero. The lead comes from cost: Haiku costs 13 times less per ticket, and another run would not change that. The memo must say three things. The options tie on quality. Haiku wins on cost. With a cost weight below about 6, Sonnet ranks first, by less than the noise.

Recap

• A public leaderboard answers a general question. Use it to choose which models to test, not which model to use.
• First, must-haves: pass-or-fail rules that remove options which cannot work. Then weights: how much each preference counts among the options that are left.
• A score from 0 to 10 comes from a measurement and a one-sentence rule. Write the rule down.
• On Larkfield’s 24 tickets, four of eight options pass the must-haves. They tie on quality. With Grace’s weights, Haiku 5.5 ranks first because it is much cheaper, and the reason is written down.
• Split a lead into its parts, and find the weight at which the leader changes. Say both in the memo.

Next: Fallback and change planning