Measure practical performance

What you will be able to do

Find out what each model costs Larkfield in time and money, next to its quality, and how sure the comparison can be with 24 tickets. Quality alone does not choose a model: a correct reply that takes 20 seconds, or costs 50 times more, may be the wrong choice.

NotebookOptional: build the table in a notebookColab · Kaggle · GitHub

The notebook is optional: this lesson works in the browser with a calculator. It loads the recorded runs and the rubric scores of all 192 replies, and builds the quality–latency–cost table, the cost per ticket and the intervals of this page with pandas. It needs no API key.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The rubric and the “ready to send” rule, from Score outputs consistently.
• Tokens, and why the same text has a different number of tokens for each model, from From text to tokens.
• Hidden reasoning tokens, from Reasoning and multimodality, and latency, from Resource trade-offs.
• The median and percentiles, from Describe a dataset: Percentiles in Essential Mathematics and Statistics for AI. A 95% confidence interval, from Estimate uncertainty in the same course.

What you need

• A browser and a calculator. A phone calculator is enough. The notebook is optional.

Hosted models were called from Toronto; local models ran on a Mac mini M4 with 16 GB. Hosted models and prices change often: your own results will differ.

Price per litre, cost per trip

Two cars stand at a dealer. Fuel for the first costs less per litre. That does not tell you which car is cheaper to drive. The first car may burn twice as much fuel per kilometre. What you pay depends on the price and on how much each trip uses.

Providers price language models the same way: a price per million tokens. And, like cars, models use different amounts for the same trip.

The problem: three numbers that do not compare

Omar asks a simple question: “Which model is cheapest for one ticket?” Amira opens the price pages. Claude Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens. Gemini 3.5 Flash costs $1.50 and $9. Flash looks cheaper on both.

Then she looks at ticket S01 in the recorded runs:

                    input tokens   visible output tokens   hidden reasoning tokens   seconds
Claude Sonnet 5.5        795              141                     (inside output)        1.9
Gemini 3.5 Flash         566               99                          933               3.9

The same prompt is 795 tokens for Claude and 566 for Gemini, because their tokenizers split text differently. Both replies are 60 to 70 words long. But Gemini 3.5 Flash also produced 933 reasoning tokens: text that the model writes for itself before the answer. The reply does not show them, but the provider bills them as output. Claude also thinks before it answers; it counts its thinking inside its output tokens, so 141 is the full bill.

So a price per token cannot answer Omar. He needs the cost per ticket.

Cost per ticket

The idea: multiply each kind of token by its price, and add.

cost per ticket = input tokens × input price + (output tokens + reasoning tokens) × output price

Prices are per million tokens, so divide by 1,000,000. For S01:

Claude Sonnet 5.5:  795 × $2 ÷ 1,000,000  +  141 × $10 ÷ 1,000,000
                  = $0.00159              +  $0.00141                  = $0.0030

Gemini 3.5 Flash:   566 × $1.50 ÷ 1,000,000  +  (99 + 933) × $9 ÷ 1,000,000
                  = $0.00085                 +  $0.00929                   = $0.0101

The model with the lower prices costs more than three times as much for this ticket, because of the reasoning tokens that nobody sees. You can only find this by measuring tokens on your own task.

For 1,000 tickets, multiply by 1,000: about $3 for Sonnet and $10 for Flash on tickets like S01. The table further down uses the mean over all 24 tickets.

Local models have no price per token. Gemma 3 4B and Mistral 7B ran on a Mac mini that Larkfield would own. Their cost is the computer, the electricity and the people who keep it running, which Hosted and self-hosted choices compared. This module did not measure electricity, so the table says “no per-token price”, not “free”.

End-to-end latency: the median and the slow tail

Latency is the time that one request takes. Here it is end-to-end: from the moment the program sends the ticket to the moment the whole reply has arrived, including the network. That is the time a support agent waits for a draft.

One number is not enough, because times vary from ticket to ticket. Each dot below is one ticket.

• Hosted API
• Local (Mac mini M4)

End-to-end seconds for each of the 24 tickets (one dot per ticket), with the median. Blue: hosted APIs, called from Toronto. Orange: local models on a Mac mini M4. Qwen3 4B is not shown: its median is 64 s, off this scale.

Text description of the chart

Seven rows of 24 dots, from 0 to 25 seconds. Claude Haiku 5.5: all tickets between about 1.5 and 3.5 s, median 2.2 s. Claude Sonnet 5.5: about 1 to 4.5 s, median 1.8 s. Claude Opus 5.5: about 2.5 to 8 s, median 3.9 s. Gemini 3.5 Flash: about 3 to 9 s, median 4.4 s. Gemini 3.1 Pro: spread from about 5 to 24.5 s, median 9.8 s; its three slowest tickets are S06, S12 and D02. Gemma 3 4B: most tickets near 4 to 6 s, median 4.8 s, and one ticket at 15.7 s, the first one, which includes loading the model. Mistral 7B: median 4.2 s, with the first ticket at 16.7 s for the same reason.

The median (p50) is the typical wait: half of the tickets are faster. The 95th percentile (p95) is the wait that 95% of tickets stay under: it describes the slow tail. Grace’s agents feel the tail. A draft that usually takes 10 seconds and sometimes 24 seconds breaks their rhythm more than a steady 4 seconds.

Read two things in the chart.

• Gemini 3.1 Pro is slow and uneven. Median 9.8 s, p95 21.5 s. Its slowest tickets are the judgment calls: on S06 (a warranty question) it produced 2,447 reasoning tokens and took 22.3 s.
• The first local ticket is slow. Gemma 3 4B took 15.7 s on S01 and about 5 s afterwards. The first request loads the model into memory. Other work shared the Mac mini during this run, so the load was slower than usual: on a quiet computer, Module 3 measured about 6.5 s for the load and about 4.0 s for a ticket after it. A real service loads the model once, at start-up.

With 24 tickets, the p95 is almost the slowest ticket: there are only one or two tickets above it. Treat it as a rough sign of the tail, not a promise. A real latency target needs hundreds of requests, under real load.

The quality–latency–cost table

Now put the three measurements side by side, for all eight runs. Quality is the number of ready replies of 24, from the rubric of Lesson 2. The 95% interval next to it is explained in the next section.

Model	Qualityready replies of 24 (95% interval)	Latencyseconds per ticket: median · p95	CostUS$ per 1,000 tickets
Claude Haiku 5.5hosted API	2486–100%	2.2p95 3.2	$0.27796 in · 400 out
Claude Sonnet 5.5hosted API	2274–98%	1.8p95 3.8	$3.65796 in · 146 out
Claude Opus 5.5hosted API	2486–100%	3.9p95 5.7	$9.35796 in · 294 out
Gemini 3.5 Flashhosted API	2486–100%	4.4p95 5.9	$10.90560 in · 1054 out (974 reasoning)
Gemini 3.1 Pro (preview)hosted API	2486–100%	9.8p95 21.5	$14.83560 in · 1052 out (950 reasoning)
Gemma 3 4Blocal, Mac mini M4	612–45%	4.8p95 5.8	no per-token price (your hardware and power)
Mistral 7Blocal, Mac mini M4	1231–69%	4.2p95 8	no per-token price (your hardware and power)
Qwen3 4B (4,000-token limit)local, Mac mini M4	1439–76%	64.3p95 140.3›	no per-token price (your hardware and power)

Quality (ready replies of 24, with a 95% interval), latency (median and p95 seconds per ticket) and cost (US dollars per 1,000 tickets, from the mean tokens per ticket and the standard prices). Bars share one scale per column. Hosted models were called from Toronto; local models ran on a Mac mini M4 (16 GB).

Text description of the table

Eight rows. Claude Haiku 5.5: 24 ready (interval 86 to 100%); 2.2 s median, 3.2 s p95; $0.27 per 1,000 tickets. Claude Sonnet 5.5: 22 ready (74 to 98%); 1.8 s, p95 3.8 s; $3.65. Claude Opus 5.5: 24 ready (86 to 100%); 3.9 s, p95 5.7 s; $9.35. Gemini 3.5 Flash: 24 ready (86 to 100%); 4.4 s, p95 5.9 s; $10.90, with a median of 974 hidden reasoning tokens per ticket. Gemini 3.1 Pro (preview): 24 ready (86 to 100%); 9.8 s, p95 21.5 s; $14.83, with 950 reasoning tokens. Gemma 3 4B (local): 6 ready (12 to 45%); 4.8 s, p95 5.8 s; no per-token price. Mistral 7B (local): 12 ready (31 to 69%); 4.2 s, p95 8.0 s; no per-token price. Qwen3 4B with a 4,000-token limit (local): 14 ready (39 to 76%); 64.3 s, p95 140.3 s; no per-token price.

What this tells us.

• Quality separates two groups, not eight models. The five hosted models are ready on 22 to 24 tickets. The three local models are ready on 6 to 14. Inside each group, the next section shows that 24 tickets cannot rank them.
• Cost differs by a factor of 55 inside the top group. Claude Haiku 5.5 costs $0.27 per 1,000 tickets; Gemini 3.1 Pro costs $14.83. The two Gemini models cost the most, mostly because of the hidden reasoning tokens. They used a median of 974 (Flash) and 950 (Pro) per ticket, for a reply of about 80 to 90 tokens.
• Bigger or newer is not faster or better here. Claude Sonnet 5.5 had the shortest median time and about the same quality as Claude Opus 5.5. Gemini 3.5 Flash matched Gemini 3.1 Pro at half the time.
• Local saves tokens, not quality. The local models used no paid tokens. But on this computer they were ready on at most 14 of 24 tickets, and Qwen3 needed about a minute per ticket to think.

Omar does the sum for Larkfield, at about 45 tickets a day (1,350 a month). Claude Haiku 5.5 costs about $0.36 a month, and Gemini 3.1 Pro about $20. Both are small next to one hour of an agent’s time. At this volume, quality and waiting time decide, and the price only breaks a tie. At a million tickets a month, the same table would say something else. Volume is an assumption, so write it down.

How sure can 24 tickets be?

Tomás reads the table and says: “Opus and Haiku have 24 of 24, Sonnet only 22. Sonnet is worse.” Is it?

Think of two students who each answer 24 quiz questions. One gets 24 right, the other 22. You would not say that the first one knows the subject better. On a different set of 24 questions, the result could easily turn around. That is selection uncertainty. When you choose a model from a small test, part of the difference that you see is luck in the choice of tickets.

The mathematics course measures this with a 95% confidence interval: a range around the result that the data supports. For a share, its formula is the share ± 1.96 standard errors. For Claude Sonnet 5.5, with 22 of 24:

share = 22 ÷ 24 = 0.917
SE    = √(0.917 × 0.083 ÷ 24) = 0.056
range = 0.917 ± 1.96 × 0.056 = 0.807 to 1.027        → 80.7% to 102.7%

The top of the range is above 100%, which is impossible. For 24 of 24, the formula gives SE = 0 and a range of “100% to 100%”. That is as if 24 tickets could prove that a model never fails. The formula needs larger samples. For small samples, statisticians use the Wilson interval, a corrected version that always stays between 0% and 100%. For 22 of 24 it gives 74% to 98%. For 24 of 24 it gives 86% to 100%.

Top: the share of ready replies for each model run, with its 95% Wilson interval, from 24 tickets. Bottom: the difference between two models on the same 24 tickets, with the paired 95% interval of Compare alternatives in the mathematics course. An interval that crosses zero means the test cannot tell the two models apart.

Text description of the chart

Top panel: eight rows. The five hosted models have dots at 92% (Claude Sonnet 5.5, interval 74 to 98%) or 100% (Claude Haiku 5.5, Claude Opus 5.5, Gemini 3.5 Flash and Gemini 3.1 Pro, intervals 86 to 100%). Their intervals overlap almost completely. The local models: Gemma 3 4B 25% (12 to 45%), Mistral 7B 50% (31 to 69%), Qwen3 4B 58% (39 to 76%). The intervals of the local models overlap each other, but not the intervals of the hosted models. Bottom panel: Sonnet 5.5 − Opus 5.5: −8.3 points (−19.4 to +2.7), crosses zero. Mistral 7B − Gemma 3 4B: +25.0 points (−1.5 to +51.5), crosses zero just. Sonnet 5.5 − Gemma 3 4B: +66.7 points (+47.8 to +85.5), far from zero.

The bottom panel compares two models on the same tickets, as in Compare alternatives in the mathematics course. Only the tickets where the two models disagree count. Sonnet and Opus disagree on 2 tickets, and Opus wins both. The difference is −8.3 points, but its interval runs from −19.4 to +2.7. It crosses zero, so the test cannot tell them apart.

Even a big gap can be uncertain. Mistral 7B was ready on 12 tickets and Gemma 3 4B on 6, a difference of 25 points. They disagree on 12 tickets, 9 to 3 for Mistral. The interval, −1.5 to +51.5, still touches zero. Only the gap between the hosted and the local models, such as Sonnet against Gemma (16 to 0), is clear.

So the honest statements are:

• “On these 24 tickets, on 2026-10-08, the hosted models were ready on 22 to 24 replies, and we cannot rank them by quality.”
• “The local models we tried were clearly behind on this task.”
• Not: “Claude Opus 5.5 is the best model.”

To separate the hosted models, Larkfield would need many more tickets, especially the kinds where they lost points: French judgment calls. As in the mathematics course, four times more tickets give an interval half as wide.

Guided practice: cost per ticket for Claude Haiku 5.5

1. Use the recorded numbers for ticket S01 with Claude Haiku 5.5: 795 input tokens, 277 output tokens (thinking included), no separate reasoning tokens.

2. Use these prices: $0.10 per million input tokens and $0.50 per million output tokens (prompts up to 100,000 tokens).

3. Calculate the input cost: input tokens × input price ÷ 1,000,000.

4. Calculate the output cost in the same way.

5. Add them, then multiply by 1,000 for the cost of 1,000 tickets like this one.

Expected result: about $0.00022 per ticket, about $0.22 per 1,000 tickets.

Example answer

input:   795 × $0.10 ÷ 1,000,000 = $0.0000795
output:  277 × $0.50 ÷ 1,000,000 = $0.0001385
total:                             $0.000218 per ticket
× 1,000:                           $0.22 per 1,000 tickets

The table shows $0.27 per 1,000, because it uses the mean over all 24 tickets: Claude Haiku 5.5 used more output tokens on other tickets (the median is 400). Its output tokens include its thinking, which is why it used more output tokens than Sonnet (median 146) for replies of the same length.

Your turn: when the volume changes

Larkfield signs a contract to handle support for a large garden-centre chain. Volume grows to 2 million tickets a month. Use the cost per 1,000 tickets in the table. What does each hosted model cost per month now? Does the cost still only break ties?

Example answer

2 million tickets = 2,000 × the cost per 1,000 tickets. Per month: Claude Haiku 5.5 about $540, Sonnet about $7,300, Opus about $18,700. Gemini 3.5 Flash about $21,800, Gemini 3.1 Pro about $29,700. The difference between the cheapest and the most expensive is now about $29,000 a month. Cost becomes a real criterion. Now a larger test is worth its price: it checks whether the cheap model is really as good on the hard tickets. The quality ranking did not change; the volume assumption did.

Common mistake

The output limit is set below the hidden reasoning

Symptom. A model returns empty or cut-off replies, but the bill shows many output tokens. In a first run, Gemini 3.5 Flash had a 600-token output limit. It returned about 21 visible tokens per ticket, and no ticket had a complete reply.

Cause. Hidden reasoning tokens count against the output limit. Gemini 3.5 Flash used a median of 974 reasoning tokens per ticket, so a limit of 600 ran out before the answer started. Qwen3 4B did the same with its visible reasoning. At 600 tokens it never answered; at 4,000 tokens it still ran out on 4 of 24 tickets.

Fix. Look at the stop reason and the reasoning tokens of every reply, not only the text. Set the output limit well above the reasoning that you measured; the hosted runs here used 8,000. Count the reasoning tokens in the cost.

For a real AI service. Log the input, output and reasoning tokens and the stop reason of every request. A change of model or of its reasoning setting can change the cost and the cut-off rate without any change in your code.

Live calls are optional. The numbers in this module come from recorded runs, so you need no API key. If you have one, measure your own calls with the same method. Record the seconds, the input, output and reasoning tokens and the stop reason of every call. Look at the median of several calls, never at one call. Expect other numbers than the recorded ones: the network, the load on the provider and the model’s own randomness change them.

Module practice: rank before you look

Grace asks you to do what the team did, on two new tickets. You rank and score the replies blind, then compare your results with the reference scores.

1. Choose ticket D01 in the explorer. Read the four replies.

2. Before you move anything, write one line per reply: what is right, and what is wrong.

3. Put the replies in order, best first.

4. Score each reply with the four criteria of the rubric (0 to 2), and decide: ready or not ready?

5. Click Reveal models and scores. Compare your order and your scores with the reference. For each difference, find the anchor that explains it.

6. Do the same for ticket M02. This ticket mixes French and English, so check the language of each reply.

Explore

Module practice: rank and score before you see the models

D01 · English Two problems: my order is 6 business days late, AND when it finally came the watering can was dented. Delivered today. What now?

Your order, best first: A, B, C, D. Move the replies, then reveal the models and the scores.

1. 1Reply A

   I'm so sorry your order is late and the watering can arrived damaged. According to our policy, orders over 15 business days late are treated as lost, and we will arrange for a replacement or full refund. Please send us a photo of the dented watering can within 14 days of delivery, and we will proceed with either a replacement or a refund as you prefer.

   needs_human: yes · category: delivery
2. 2Reply B

   I apologize for the delay and the damaged watering can. Since your order was more than 5 business days late, we will refund your shipping fee. For the dented watering can, please send us a photo within 14 days of delivery. Once we receive the photo, we will send a replacement at no cost, or a full refund if a replacement is not in stock.

   needs_human: no · category: damaged
3. 3Reply C

   {"language": "en", "category": "delivery", "returns", "needs_human": true, "reply": "Dear customer, we apologize for the delay and the damage to your watering can. As per our policy, your order is more than 5 business days late, so you will receive a refund for the shipping fee. For the damaged item, please send a photo within 14 days of delivery. We will then send a replacement at no cost, or a full refund if no replacement is in stock. Thank you for your patience."}

   needs_human: none · category: invalid JSON (raw output shown)
4. 4Reply D

   Your order arrived 6 business days late, so you are entitled to free shipping and the shipping fee on your order will be refunded. For the dented watering can, please send us a photo of the damage within 14 days of delivery. Once we have it, we will send a replacement at no cost, or a full refund if no replacement is in stock.

   needs_human: no · category: delivery

Real replies to two new tickets, with the model names hidden. Rank them, score them with the rubric on paper, then reveal.

Text description of the explorer

D01 (English; 6 business days late and a dented watering can, delivered today). A, Gemma 3 4B: quotes the 15-day lost-order rule for a 6-day delay and misses the free shipping; asks for a photo (correctness 0, completeness 1, format 1, abstention 1; total 3; not ready). B, Gemini 3.1 Pro: shipping fee refunded, photo within 14 days, replacement or refund (8; ready). C, Mistral 7B: good text, but the raw output is broken JSON ("category": "delivery", "returns"), so the application gets nothing (2, 2, 0, 1; total 5; not ready). D, Claude Haiku 5.5: free shipping, photo within 14 days, replacement or refund (8; ready).

M02 (French and English mixed; a shovel arrived broken, received yesterday). A, Mistral 7B: asks for a photo, promises no replacement, sends it to a person (2, 1, 2, 1; total 6; ready by the rule, but weak). B, Claude Sonnet 5.5: photo of the shovel and the wet box within 14 days, replacement or refund, in French (8; ready). C, Gemma 3 4B: the key instruction is in English, offers “a replacement or a full refund as you prefer” although the policy refunds only when no replacement is in stock (1, 2, 1, 1; total 5; not ready). D, Gemini 3.5 Flash: photo within 14 days, replacement or refund, in French (8; ready).

Expected result: in both tickets, two replies score 8 and are hard to separate. Your order of the other two depends on which error you find worse.

Model answer and what to learn from it

D01. B and D share the top (8 each). Then C (5): the text is right, but format compliance is 0, because the application cannot parse the JSON. A customer would never see this reply. Last, A (3): it applies the wrong rule and misses the free shipping. Many people rank C last, because it looks broken. The rubric ranks A last. A wrong rule that reaches a customer is worse than an error that stops the reply before anyone sends it. Both are not ready.

M02. B and D share the top (8). Then A (6) and C (5). C has the more serious problems: an English instruction in a reply to a French-speaking customer, and an invented choice (“as you prefer”). A is weak but true, and Grace’s rule allows a person to fix it.

What to learn. The reference scores agree with yours on most criteria if you used the anchors. Where you differ, the cause is usually a judgement call of Lesson 2. How bad is a language error? Is a cautious hand-over an error? Write down your reason, as the team did. In the final assignment, the rubric and its reasons are part of your evidence.

Module check

Check your understanding

Retrieval: name the four criteria of Larkfield's rubric and Grace's rule for a ready reply. Then say what the median and the p95 of the seconds per ticket each describe.

Show the answer

Correctness, completeness, format compliance and appropriate abstention, each 0 to 2. A reply is ready to send after a light check when correctness is 2 and no criterion is 0. The median is the typical time: half of the tickets are faster. The p95 is the time that 95% of tickets stay under, so it describes the slow tail that agents feel.

Check your understanding

Predict: Grace grows the test set from 24 to 96 tickets, with the same mix of kinds. Claude Sonnet 5.5 is again ready on 92% of them. What happens to its 95% interval, and could the test now separate it from a model that is ready on 100%?

Show the answer

The interval becomes about half as wide, because the test set is four times larger (as in the mathematics course): roughly 85% to 96% instead of 74% to 98%. A model with 96 of 96 would have an interval of about 96% to 100%. The two intervals would just stop overlapping, so a difference of about 8 points starts to become visible. A difference of 1 or 2 points would still need far more tickets.

Check your understanding

Completion check: Tomás writes 'Claude Opus 5.5 is the best model for Larkfield: it scored 24 of 24.' Explain selection uncertainty, and why 24 tickets cannot prove that one model is better than the others in general.

Show the answer

Selection uncertainty is the part of a difference between models that comes from the particular tickets in the test, not from the models. Another 24 tickets would give other numbers. Four hosted models scored 24 of 24 and one scored 22. The 95% intervals overlap almost completely. The paired difference between Sonnet and Opus (−8.3 points, interval −19.4 to +2.7) includes zero. So the test cannot rank them. The test also covers only 24 synthetic tickets, one prompt, one date and one rubric. It says nothing about other tasks, other languages or later versions of the models. An honest statement is: “On these 24 tickets, on 2026-10-08, the hosted models were ready on 22 to 24 replies; we could not separate them by quality. Claude Haiku 5.5 was the cheapest and among the fastest.”

See it end to endChoose a model for bilingual government answersCase study

The case study builds a test set from real English and French questions and scores the answers with this rubric.

Recap

In this module, you learned to evaluate models on your own task.

• Lesson 1: a representative test set has routine, difficult, multilingual, ambiguous and unanswerable cases, with the answer key written first. Language twins find language problems.
• Lesson 2: a rubric with anchors (correctness, completeness, format compliance, appropriate abstention) makes scores repeatable. Score blind, check the agreement between two readers, and rescore everything when an anchor changes.
• Lesson 3: measure the end-to-end latency (median and p95), the tokens, including hidden reasoning tokens, and the cost per ticket, not the price per token. Put quality, latency and cost in one table. With 24 tickets, a difference of one or two tickets is noise: show the interval.

You now have Larkfield’s evidence: the scores of 192 replies, the times and the costs. Next, you turn this evidence into a decision with must-have requirements and weights.

Reset: nothing to reset. The explorers start again when you reload the page, and the recorded replies never change.

Next: Weighted selection