Resource trade-offs

What you will be able to do

Choose a model and a machine that answer fast enough, fit in memory, and keep up with the number of tickets. Know what you give up when you make a model smaller.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The three ways to get a model and the deployment matrix, from Hosted and self-hosted choices.
• The six questions about licences and data terms, from Licensing and data terms.
• What a token is and how a model writes one token after another, from Predicting continuations.
• The context window, the number of tokens that a model can take in and write in one request, from Context and memory.
• The median, from Describe a dataset in Essential Mathematics and Statistics for AI.

What you need

• A browser. The “Try it” box is optional and needs Ollama.

Measured on a Mac mini M4 with 16 GB of memory and Ollama 0.40.1, with no other model loaded. One example comes from earlier the same day, when other programs used the computer; the lesson marks it as “busy”. Your own numbers will differ.

Three numbers at a restaurant

You order a meal in a restaurant. Two times matter to you: until the first plate arrives, and until the meal is finished. The kitchen watches a third number: how many tables it serves in an hour. A kitchen can serve many tables per hour and still keep each guest waiting a long time.

A model on a server has the same three numbers. This lesson measures them on the Mac mini, with the two things that limit them: memory and model size.

The problem: the same model, fast and slow

Amira sends ticket S02 to Gemma 3 4B, which is already in the Mac mini’s memory. The reply is complete after 4.0 seconds. Then she unloads the model and sends ticket S01. Now the model is a 3.3 GB file on the disk, and Ollama must load it before it can read a single token. The reply comes after 10.0 seconds; the load alone took 6.5 seconds (the median of 8 loads, from 2.7 to 7.4 s).

The model is the same; only its place was different. The machine around the model matters too. Earlier the same day, other programs filled most of the 16 GB (the busy computer). Then the same load took 19 to 342 seconds: the computer first had to make room.

So “how fast is this model?” has no single answer. It depends on which time you measure, and on the machine.

Latency: the first token and the last

Latency is the time that a person or a program waits for an answer. For a language model, two latencies matter:

• The time to first token (TTFT): from the request to the first word of the answer. If the app shows the words as they arrive, the user sees that something happens.
• The total time: from the request to the last token. The assistant returns its answer as one JSON object: a text format that programs read, with named fields such as "language": "fr" between curly brackets. The support app needs the whole object before it can show the draft, so for Larkfield this is the time that counts.

The time line shows where the seconds go, for one cold request and one warm one.

One request to Gemma 3 4B (Q4_K_M) on the Mac mini, to scale: the medians of the measured requests, from Ollama's own timers. A cold request loads the model first. A warm request finds the model already in memory.

Text description of the diagram

Two horizontal bars on the same time scale, with the medians of the measured rounds for Gemma 3 4B (Q4_K_M) on the Mac mini.

• Cold request, ticket S01: load the model (6.5 s, grey), read the prompt of 576 tokens (1.3 s, blue), write the reply of 122 tokens (3.3 s, pale). The first token arrives after 7.8 s; the reply is complete after 11.1 s.
• Warm request, ticket S02: no load. Read the prompt of 573 tokens (1.3 s), then write the reply of 97 tokens (2.6 s). The first token arrives after 1.3 s; the reply is complete after 3.9 s.

A warm request has two parts, and you can calculate each one. Gemma read the prompt at about 438 tokens per second, and wrote at about 37 tokens per second. Reading is faster than writing, because the model reads all the prompt tokens at once but writes one token after another. For ticket S02:

• read: 573 tokens ÷ 438 tokens per second ≈ 1.3 s
• write: 97 tokens ÷ 37 tokens per second ≈ 2.6 s
• total: about 3.9 s. Ollama’s timers measured 3.9 s.

This tells you what makes a request slow. A long policy in the prompt adds reading time. A long reply adds writing time, and writing is about 12 times slower per token. A model that “reasons” before it answers writes hundreds of extra tokens: Qwen3 4B needed about 1,900 tokens and 64 seconds per ticket.

What to do with it: keep the model in memory between requests, with Ollama’s keep_alive setting. Measure the total time on warm requests. Report the cold time separately: it happens after every restart.

Throughput: tickets per hour

Throughput is the amount of work that a system finishes per unit of time: here, tickets per hour.

The two numbers meet in a queue. During one of Amira’s tests, other programs also sent requests to the same Ollama server. Her warm request needed about 4.1 seconds of work. But her own clock showed 12 to 47 seconds: for the rest of the time, it waited behind other requests. When requests arrive faster than the server finishes them, every request waits longer.

One Mac mini that answers one ticket at a time, in about 5 seconds, finishes about 3,600 ÷ 5 = 720 tickets per hour. Larkfield gets about 45 tickets a day. Even a sale day with 100 tickets in one hour uses less than a sixth of that. For Larkfield, throughput is not the limit; latency and quality are.

At a larger shop, throughput becomes the limit. At 45,000 tickets a day, about 1,900 per hour, one machine cannot keep up. You need more machines, or a server that answers several requests at once. A hosted API hides this problem, because the provider adds machines. But you meet the limit in another form: a rate limit. That is the number of requests or tokens that your account may send per minute.

Context size and memory

Think of a desk. A bigger desk holds more open papers at once, but it takes more room in the office, even when most of it is empty.

A model reserves memory for its context window when it loads, like the desk. In Ollama, the setting num_ctx chooses the size of the window. Predict first: if the window is 16 times larger, does the memory grow 16 times?

The memory that Gemma 3 4B (Q4_K_M) used on the Mac mini at three sizes of the context window (Ollama's num_ctx), from ollama ps. A window 16 times larger added about 0.3 GB.

Text description of the chart

A line chart with three points: the memory that Gemma 3 4B (Q4_K_M) used while loaded, at three sizes of the context window. 4,096 tokens: 2.89 GB; 16,384 tokens: 2.93 GB; 65,536 tokens: 3.18 GB.

The memory grows, but far less than 16 times: from 2.89 GB to 3.18 GB. The weights are the same at every size; only the part reserved for the context grows. Other models can reserve much more than Gemma 3, so measure the model you choose. Larkfield’s prompt is about 600 tokens and the reply about 100, so a window of 4,096 tokens is enough, with room to spare.

The opposite mistake is worse. If a prompt is longer than the window, Ollama does not stop with an error. It keeps only the last part of the prompt, and answers anyway. Context and memory shows what that does to the answers. So set num_ctx for your longest real prompt, and check the number of prompt tokens that Ollama reports.

Quantization: a smaller copy of the same model

A photo saved at a lower quality setting makes a smaller file. Usually you cannot see the difference. In a few places, fine detail is gone.

A model’s weights are billions of numbers. Quantization stores each number with fewer bits, so the model file is smaller and needs less memory. (A bit is one 0 or 1; 8 bits make one byte.) The idea is rounding. Suppose a weight is 0.137, and all the weights in its block lie between −1 and 1:

• With 8 bits, there are 256 steps between −1 and 1, each about 0.008 wide. The nearest step is 0.1373: an error of 0.0003.
• With 4 bits, there are only 16 steps, each about 0.13 wide. The nearest step is 0.133: an error of 0.004, more than ten times larger.

The name tells you the bits. Q8_0 keeps about 8 bits per weight. Q4_K_M keeps about 4 to 5, with a separate scale for each small block of weights. The models of this course in Ollama are Q4_K_M by default. Amira downloads the Q8_0 version of Gemma 3 4B (gemma3:4b-it-q8_0) and compares the two on the same computer and the same 24 tickets.

• Q4_K_M
• Q8_0

The same model, Gemma 3 4B, in two quantizations: the size of the file on the disk, and the memory in use while loaded (context window 4,096 tokens).

Text description of the chart

A column chart with four columns. Size on the disk: Q4_K_M 3.34 GB, Q8_0 4.99 GB. Memory while loaded, with a context window of 4,096 tokens: Q4_K_M 2.89 GB, Q8_0 4.53 GB.

Now predict: is the bigger copy faster or slower, and is it better on Larkfield’s tickets?

Writing speed of the two quantizations on the Mac mini, the median of the measured requests, from Ollama's own timers. Reading the prompt was about the same for both (about 440 tokens per second).

Text description of the chart

A column chart with two columns: the writing speed of Gemma 3 4B in output tokens per second. Q4_K_M: 37. Q8_0: 23. Reading the prompt: Q4_K_M 438 and Q8_0 452 tokens per second.

Q8_0 writes about 37% slower. Writing one token means reading all the weights from memory once, and Q8_0 has more bytes to read. A warm ticket took 5.3 seconds instead of 4.0.

The quality is the surprise. Amira runs the 24 tickets through both copies, with the same prompt, and checks the labels automatically:

Quick check (24 tickets)	Q4_K_M	Q8_0
Valid JSON	24	24
Language label right	23	23
Category right	20	20
needs_human right	8	9

(The needs_human row uses the corrected answer key of ticket D02, which Module 4 explains.) The labels are almost the same: only one label differs between the two copies. The replies are not: 23 of 24 replies changed their wording. A label check cannot say which wording is better, so a person must read them.

Camille reads the French replies and finds the main difference. Q4_K_M switched from French to English in the middle of 4 of its 11 French replies. For ticket S12, about a cracked garden bench:

Q4_K_M: Je suis désolé d'apprendre que votre banc de jardin est fissuré. Our warranty covers garden tools and furniture for defects for 2 years, ...
Q8_0:   Je suis désolé d'apprendre que votre banc de jardin a une fissure. Selon notre politique de garantie, les outils de jardinage et meubles ont une garantie de 2 ans contre les défauts. ...

Q8_0 stayed in French in all 11. For a support team in Quebec, that difference matters more than a second per ticket. So “a smaller copy is almost the same” is a guess, not a fact: on this task, the rounding cost quality in the second language. And 24 tickets are few. Module 4 scores replies by reading them with a rubric, and shows how to tell a real difference from chance. It scored the course’s main Gemma 3 4B run, which uses Q4_K_M. The Q8_0 copy has no rubric scores, so this comparison rests on the labels and on Camille’s reading of the French.

Hardware: will it fit?

At decision level, one calculation answers most hardware questions. The memory that a model needs is about:

• the weights: the number of parameters (the weights) × the bits per weight ÷ 8, in bytes;
• plus the context: the memory reserved for the window (the chart above);
• plus room for the operating system and the other programs on the machine.

Try it on Mistral 7B: 7.2 billion parameters at about 4.9 bits per weight (Q4_K_M) is 7.2 × 4.9 ÷ 8 ≈ 4.4 GB. Its file is 4.4 GB, and it used 4.9 GB of memory with a 4,096-token window. A model with 70 billion parameters at the same bits needs about 70 × 4.9 ÷ 8 ≈ 43 GB for the weights alone. It does not fit in 16 GB. On a 16 GB machine, the realistic choice is a model of about 4 to 8 billion parameters, quantized.

If a model does not fit in the memory of the graphics processor, Ollama runs part of it on the main processor. That is slower. The PROCESSOR column of ollama ps shows it: on the Mac mini, every model of this lesson showed 100% GPU. When the whole machine runs out of memory, the computer moves data to the disk, and everything slows down. That is what made the cold load take minutes during Amira’s test.

When each trade-off matters

• Latency matters when a person waits. Throughput matters when the volume is high.
• Context size and quantization matter for memory on your own machine. Measure quality on your own tickets before you choose a smaller copy.
• Hardware sets the upper limit: calculate the memory before you buy or download.
• On a hosted API, most of these are the provider’s problem; you see only latency, rate limits and price.

Try it: measure your own model (optional)

Only do this if you have Ollama and the model gemma3:4b (see the “Try it” box of Hosted and self-hosted choices). Tested with Ollama 0.40.1 on macOS.

1. Open a terminal.
2. Run one request and ask for the timings: ollama run gemma3:4b --verbose "Write one sentence about garden hoses."
3. Read the lines under the answer. load duration is the load, prompt eval rate is the reading speed, and eval rate is the writing speed.
4. Run the same command again. The load duration is now close to zero: the model is warm.

Your speeds depend on your computer. Compare them with the numbers in this lesson.

Common mistake

A benchmark measured the wrong thing

Symptom: Tomás times one request to each local model with a stopwatch, and reports: “Gemma 3 4B took almost six minutes for one ticket. It is far too slow for our agents.”

Cause: his only request was a cold one, on a busy computer: the load alone took 342 seconds, because other programs had filled the memory. On a quiet computer, the same load took about 6.5 seconds, and a warm request took 4.0 seconds. A single measurement also hides how much times vary between runs.

Fix: send one request first to warm the model, and do not count it. Then measure several tickets and report the median. Report the cold time separately, with the state of the machine (other programs, free memory). Always say which time you measured: first token or total.

Consequence in AI work: speed numbers without their conditions mislead. A published “tokens per second” for a model says nothing about your prompt length, your reply length, your hardware or a cold start. Measure on your own task, the same way for every candidate.

Knowledge check

Check your understanding

Larkfield's support app shows the draft reply only when the whole JSON object has arrived. Which latency should Amira report for it: time to first token or total time? And when would the other one matter?

Show the answer

The total time: the app can do nothing with half a JSON object, so the agent waits until the last token. Time to first token matters when an app shows the words as they arrive, as a chat window does. Then the user sees the answer begin early.

Check your understanding

Predict: Grace adds a long section to the support policy, so every prompt grows from about 600 to about 1,800 tokens. The replies stay the same length. What happens to the reading time and the total time of a warm request on the Mac mini?

Show the answer

The reading time grows about three times: 1,800 tokens at about 438 tokens per second is about 4.1 s instead of about 1.4 s. The writing time stays about the same, because the reply has the same length. So the total grows by about 2.7 s, from about 3.9 s to about 6.7 s. The window of 4,096 tokens is still large enough. On a hosted API, the longer prompt also costs three times more input tokens.

Module practice

The practice puts the three lessons together: who runs the model, what the terms allow, and the cost in time and memory.

Guided: two options, one budget, one privacy rule

Grace sets the conditions for the support assistant. All the numbers below are fictional, made for this exercise.

• Budget: at most $60 a month, everything included. Staff time costs $40 an hour.
• Privacy rule: a ticket may leave Larkfield only for a provider that does not train on it and deletes it within 30 days.
• Volume: 1,400 tickets a month; at most 100 in the busiest hour.
• Speed: the draft is ready within 10 seconds (total time, warm).

	Option A: “Northwind API” (hosted)	Option B: “Desk server” (open weights)
Price	$1 per million input tokens, $5 per million output tokens	Mini PC with 32 GB, $1,200, used for 24 months; electricity $4 a month
Tokens per ticket	800 in, 150 out	no price per token
Staff time	1 hour a month (API key, invoice, checks)	4 hours a month (updates, restarts, checks)
Total time per ticket	2.5 s median	6 s warm; 40 s after a restart
Terms	Does not train on API data; deletes inputs and outputs within 30 days; processes in the United States	No provider; the ticket stays on Larkfield’s machine
Memory	the provider’s problem	a 7B model, Q4_K_M: about 4.4 GB, plus the context

1. Calculate the cost of one ticket with option A.
2. Calculate the monthly cost of option A, staff time included.
3. Calculate the monthly cost of option B, staff time included.
4. Check each option against the privacy rule.
5. Check each option against the speed and the volume.
6. Check that option B’s model fits in its memory.
7. Write two sentences for Grace: which option meets every condition, and what is still unknown.

Show a model answer

1. 800 × $1 ÷ 1,000,000 + 150 × $5 ÷ 1,000,000 = $0.0008 + $0.00075 = $0.00155 per ticket.
2. 1,400 × $0.00155 = $2.17 for the tokens, plus 1 hour × $40 = $42.17 a month. Within $60.
3. $1,200 ÷ 24 = $50 for the machine, $4 for electricity, 4 hours × $40 = $160 of staff time: $214 a month. Over the budget.
4. Option A: it does not train on the tickets, and it deletes them within 30 days, so it passes. The rule says nothing about the region; if it did, “processes in the United States” could fail it. Option B: nothing leaves, so it passes.
5. Option A: 2.5 s is within 10 s; the provider handles 100 tickets an hour (check the rate limit of the account). Option B: 6 s warm is within 10 s; one machine finishes 3,600 ÷ 6 = 600 tickets an hour, more than 100. After a restart, the first ticket takes 40 s: keep the model loaded.
6. About 4.4 GB of weights plus the context, in 32 GB: it fits easily.

Note for Grace: “Option A meets the budget, the privacy rule, the speed and the volume, at about $42 a month. Option B meets every condition except the budget: it costs about $214 a month, mostly in staff time. We do not know yet whether either model writes good replies to our tickets. Module 4 measures that.”

Your task: the privacy rule changes

Larkfield’s lawyer reads the privacy rule again and makes it stricter: no ticket may leave Larkfield’s computers. Everything else stays the same. Check both options again. Then write three or four sentences for Grace: what you recommend, and what she must decide.

Show a model answer

Option A now fails the privacy rule: every ticket goes to the provider, whatever its terms say. Option B is the only option that passes the rule, but it costs about $214 a month, against a budget of $60. So no option meets every condition, and the honest answer is to say so. Grace must choose. She can raise the budget to about $215 a month. Or she can lower option B’s cost, for example with a machine that the team already maintains, so that it needs less staff time. The cheapest option, A, is not a choice any more, because a must-have comes before the price.

Module check

Check your understanding

Retrieval: in one sentence each, what are latency, throughput and quantization?

Show the answer

Latency is the time that a person or a program waits for one answer, measured to the first token or to the last. Throughput is the amount of work a system finishes per unit of time, such as tickets per hour. Quantization stores each weight of a model with fewer bits, so the model needs less disk and memory, usually with a small loss of quality.

Check your understanding

Predict, then check with this lesson: Larkfield replaces Gemma 3 4B Q4_K_M with Q8_0 on the Mac mini. What happens to the memory, the writing speed and the quality on the 24 tickets?

Show the answer

Memory: more. The Q8_0 file is 5.0 GB instead of 3.3 GB, and it used 4.5 GB of memory instead of 2.9 GB. It still fits in 16 GB. Writing speed: slower, about 23 instead of 37 tokens per second, because there are more bytes to read for each token. Quality: on the 24 tickets, the labels were almost the same. But Q8_0 never switched from French to English; Q4_K_M did in 4 of 11 French replies. The trade is memory and speed against quality in French, and only a test on your own tickets shows it.

Check your understanding

Completion: Gemini 3.5 Flash is available on the free quota and on the paid Gemini API. Tomás says: 'It is the same model, so our privacy rule gives the same answer for both.' Explain why he is wrong, using the difference between a model's capabilities and a provider's service terms.

Show the answer

A model’s capabilities are what it can do: its answers to Larkfield’s tickets, its speed, its languages. On the free quota and on the paid API, these are the same, because it is the same model. The service terms are what the provider does with the data you send, and they belong to the service, not to the model. The Gemini API terms (last updated 2026-04-28) say that on the free quota, Google uses the content to improve its products and human reviewers may read it; on the paid service, it does not use the prompts to improve its products. So the same model can pass Larkfield’s privacy rule on one service and fail it on the other. Evidence about capability (a test) and evidence about terms (the provider’s pages) answer different questions, and a decision needs both.

End of Module 3

Reset to a known state

This module changes no files. If you downloaded models in a “Try it” box and want the disk space back:

1. Open a terminal.
2. List the models: ollama list
3. Remove each model you no longer need, for example: ollama rm gemma3:4b

Recap

• Latency has two forms: the time to first token and the total time. Measure warm requests, and report a cold start separately. A warm request to Gemma 3 4B took about 4.0 s: reading at about 438 tokens per second, writing at about 37.
• Throughput is work per hour. One machine at 5 s per ticket finishes about 720 tickets an hour, far more than Larkfield needs.
• A larger context window reserves more memory on your own machine. Choose the size from your real prompts.
• Quantization stores weights with fewer bits. Gemma 3 4B at Q4_K_M used 2.9 GB and wrote 37 tokens per second; at Q8_0, 4.5 GB and 23. But Q8_0 kept every French reply in French; Q4_K_M did not.
• Module 3 in short: decide who runs the model, and read the licence and the data terms. Then check that the hardware, the speed and the budget fit. Must-haves come first; quality comes next, in Module 4.

Next, Module 4 builds the test set that measures quality.

Next: Build a representative test set