Choose a model that runs on your own hardware

What you will be able to do

Choose a language model when the text must never leave your computers. You compare two open-weight models and two sizes of the same weights on one computer. You measure what the hardware feels: memory, load time and speed. You score what the users feel: the quality of the answers. Then you recommend one set-up, with the hardware it needs, its limits and a fallback.

Case studyPortfolio pieceSynthetic data

Dataset

Home-care visit notes, 18 notes with an answer key (made for this case study)

Licence

CC0 1.0 (notes); MIT (code)

Time

About 3 hours, of which about 15 minutes of runs

Runs in

Your computer: Ollama and Python 3.12+

Skills used

• Privacy rule as a must-have
• Model licences
• Quantization
• Memory, load time and speed
• Rubric scoring
• Recommendation with a fallback

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is a small project folder: the notes, the instructions for the model, and four scripts. You run the scripts; you do not write code. The finished project has our recorded runs, our scores and our recommendation, for comparison. No suitable computer? Use the finished project. Its runs/ folder has every answer and every time, so you can do Parts 4 and 5 without running a model.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses the whole course. The lessons it leans on most:

• Hosted API or open weights, and who owns what: Hosted and self-hosted choices.
• What a model licence allows: Licensing and data terms.
• Memory, latency and quantization: Resource trade-offs.
• The test set and the rubric: Build a representative test set and Score outputs consistently.
• Medians, and why one run is not enough: Measure practical performance.
• Must-haves before weights, and a fallback: Weighted selection and Fallback and change planning.

What you need on your computer

• Ollama, the free program that runs open-weight models on your computer.
• Python 3.12 or newer. The scripts use only Python’s standard library, so there is nothing to install.
• A text editor such as VS Code, and a spreadsheet program to fill in one CSV file.
• Memory: 16 GB for the three set-ups. With 8 GB, use only gemma3:4b.
• About 13 GB of free disk space for the three model files, and an internet connection to download them once.

Recorded on a Mac mini M4 with 16 GB of memory. Software: macOS, Ollama 0.40.1, Python 3.14.6. New model versions and new quantizations appear often, so your numbers will differ. The method is what lasts. The Windows and Linux commands are not tested.

The situation

Wren Home Care sends care workers to the homes of older clients. After each visit, the worker writes a short note on a tablet. It says what happened: the medicine, any fall, the pain. On a busy day there are 300 notes.

Ines Moreau coordinates the visits. She wants each note turned into a small handover form, so that the urgent cases stand out. Theo Lambert, who runs the agency’s IT, tried a hosted model with an invented note, and it worked well.

Then Ruth Okoye, the agency’s privacy officer, sets a rule: the notes never leave the agency’s computers. They are health information about named people. The agency’s contracts and the law give it little room to send them to another company. So there is no hosted API: only a model that runs on hardware that the agency owns.

The agency has one spare computer for this: a Mac mini M4 with 16 GB of memory, which also runs other office work. Theo must find out which model can do the job on it, and how well. In this case study, you are Theo.

What you will build

A project folder that is the evidence for one decision: the test notes, the recorded answers, the measurements, the scores, and a one-page recommendation.

Everything happens inside one computer. The scripts refuse any address that is not this computer.

Text description of the diagram

The notes and the answer key are the input. measure.py sends one note at a time to Ollama at 127.0.0.1, the address of the computer itself. Ollama loads one model file from the disk into memory. The answers and the times go to the runs folder. score.py checks what a program can check, and a person reads the summaries; together they give the rubric scores. report.py puts everything in one table, which leads to one choice. A hosted model API sits outside the computer; the path to it is refused.

You compare two models and two quantizations (you learn the word in Part 2):

Set-up	Model	Quantization	File on disk	Compares
gemma3:4b	Gemma 3 4B, from Google	Q4_K_M	3.35 GB	the baseline
gemma3:4b-it-q8_0	the same model	Q8_0	4.98 GB	quantization
mistral:7b	Mistral 7B Instruct v0.3, from Mistral AI	Q4_K_M	4.37 GB	another model

Both are popular open-weight models that the course measured in earlier modules. One has 4 billion weights, the other 7 billion, and their licences differ.

We also measured a fourth set-up once: mistral:7b-instruct-v0.3-q8_0, 7.7 GB on disk. Then we dropped it. Part 2 shows why.

The notes and the form

Real visit notes cannot be published. So the 18 notes of this case study are synthetic: written for it, with invented clients and events. They are free to use (CC0). Here is one, with its answer key:

Note R02

C-2210, Mrs Okafor. Evening visit. Dosette box still had the lunchtime tablets in it.
She says she forgot, she was on the phone with her sister. Gave the evening tablets
and she took them. Pain 2/10 in her hip. No falls. Ate soup and toast.

Answer key for R02

{"fall": "no", "medication": "missed", "pain_score": 2, "urgency": "follow_up",
 "must_mention": ["lunchtime tablets missed"]}

The model gets the same instructions for every note (data/task_prompt.md). It must return only a JSON object with five keys:

Key	Allowed values
fall	yes, no, not stated
medication	taken, missed, refused, not stated
pain_score	a number from 0 to 10, or null when the note gives no number
urgency	urgent (a fall with injury, trouble breathing, new confusion), follow_up (a missed dose, pain 6 or more, a safety problem, anything unsure), routine
summary	one sentence in English, at most 30 words, only facts from the note

The 18 notes cover the five kinds of a representative test set:

Kind	Notes	What it tests
Routine	R01–R08	Clear notes: 2 urgent, 4 follow-up, 2 routine
Difficult	D01–D04	An old fall that is not new; a near slip; pain in words, not numbers; new confusion that is easy to miss
Missing information	M01–M03	The note does not say, so the right answer is not stated or null, not a guess
Not a visit note	X01	A message about a shift swap
French	F01–F02	Notes from bilingual workers

Eighteen notes are a start, not proof: a difference of one or two notes between two set-ups is noise.

Set up the project

1. Download the starter project with the button in the box at the top of this page. Unzip it. You get a folder local-model-choice.

2. In VS Code, open the folder: File › Open Folder…, then choose local-model-choice. Open a terminal: Terminal › New Terminal.

3. Install Ollama from ollama.com/download and start it. On macOS and Windows, the app starts the server. On Linux, the installer starts it as a service.

4. Check that Ollama listens only on this computer. This is the privacy rule in practice: a server that listens on the network can receive requests from other machines.

   macOS

   lsof -nP -iTCP:11434 -sTCP:LISTEN

   Windows

   netstat -ano | findstr 11434

   Linux

   ss -ltn | grep 11434

   Output (macOS)

   COMMAND   PID       USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
   ollama  11142 you           3u  IPv4 0x5cb6a0e092f429ed      0t0  TCP 127.0.0.1:11434 (LISTEN)

   The address must be 127.0.0.1:11434. If you see *:11434 or 0.0.0.0:11434, the setting OLLAMA_HOST makes Ollama listen on the network. Remove that setting and restart Ollama before you continue.

5. Run the check script. On Windows, type python instead of python3 in every command on this page.

   python3 check_setup.py

Output (last lines, before the downloads)

  missing  gemma3:4b                        to get it: ollama pull gemma3:4b
  missing  gemma3:4b-it-q8_0                to get it: ollama pull gemma3:4b-it-q8_0
  missing  mistral:7b                       to get it: ollama pull mistral:7b
Pull the missing models, then run this script again.

What this tells us. Python and Ollama work, and no model is on the disk yet. The script sent no note: it only asked Ollama for its version and its list of models.

Now download the three model files. This is the only moment when the project needs the internet, and only the request for the file goes out.

ollama pull gemma3:4b
ollama pull gemma3:4b-it-q8_0
ollama pull mistral:7b

With 8 GB of memory, pull only the first one. Then run python3 check_setup.py again:

Output

Python 3.14.6 on Darwin arm64
Memory: 16 GB
Ollama 0.40.1 answers at http://127.0.0.1:11434
  ready    gemma3:4b                        3.3 GB on disk
  ready    gemma3:4b-it-q8_0                5.0 GB on disk
  ready    mistral:7b                       4.4 GB on disk
Ready.

What this tells us. The Q8 file is half again as large as the Q4 file of the same model. Part 2 explains why.

One more check: the privacy rule is also in the code. Give measure.py an address that is not this computer, and it refuses to send anything:

macOS

OLLAMA_URL=http://example.com:11434 python3 measure.py gemma3:4b --notes R01

Windows

$env:OLLAMA_URL = "http://example.com:11434"; python measure.py gemma3:4b --notes R01; Remove-Item Env:OLLAMA_URL

Linux

OLLAMA_URL=http://example.com:11434 python3 measure.py gemma3:4b --notes R01

Output

Refused: example.com is not this computer. The notes must not leave it.

Part 1: what the rule and the licences allow

The problem: “open” does not mean “any use”

Ruth’s rule removes every hosted model. But a downloaded model comes with a licence, and some licences limit what you may do with it, even on your own computer.

The licences of the two models

Licensing and data terms read both licences for Larkfield. We read them again on the official pages. Read the current pages yourself: licences change. In short:

• Mistral 7B Instruct v0.3 is under the Apache License 2.0, as its model page says: any use, commercial too, with no list of forbidden uses. The model page also warns: “It does not have any moderation mechanisms.”
• Gemma 3 is under the Gemma Terms of Use, last modified April 1, 2026. Commercial use is allowed, and you are “solely responsible for Outputs” (section 3.3). But section 3.2 forbids use “for the restricted uses set forth in the Gemma Prohibited Use Policy”.

For Larkfield’s support replies, that policy did not matter. For health notes, it does. The Gemma Prohibited Use Policy, last modified February 21, 2024, forbids among other uses:

Making automated decisions in domains that affect material or individual rights or well-being (e.g., finance, legal, employment, healthcare, housing, insurance, and social welfare)

Generating, gathering, processing, or inferring sensitive personal or private information about individuals without obtaining all rights, authorizations, and consents required by applicable laws.

What this means for Wren

Home care is healthcare. So with Gemma, the agency must make sure that the model does not decide. In this design, the form only helps a person: the coordinator reads every note and every form, and decides what to do. The agency also needs the legal basis and the consents to process the notes at all, with any model. Both conditions go into the recommendation.

Mistral’s licence adds no use conditions, which gives the agency more freedom later. Today both models pass. This is not legal advice; an agency asks its lawyer to confirm the reading.

Part 2: quantization, or what Q4 and Q8 mean

The problem: the weights do not fit

A model is mostly its weights: billions of numbers learned in training. Mistral 7B has about 7.2 billion of them. Stored with 16 bits (2 bytes) each, the usual format after training, that is about 14.4 GB. Wren’s computer has 16 GB in total, and the operating system and the office programs need some of it. The model would not fit.

The idea: fewer bits per weight

Quantization stores each weight with fewer bits, by rounding it to a step on a short scale (Quantization: a smaller copy of the same model). Q8_0 keeps 8 bits per weight, plus a small scale for each block of 32 weights. Q4_K_M keeps about 4 to 5 bits: most weights get 4 bits, and a few sensitive parts keep more. In Module 3, Q8 kept Gemma’s French replies in French. Will it help on this task? Only a test on the task can tell.

In the running example

The file sizes follow the arithmetic. As in Hardware: will it fit?, Mistral 7B has about 7.2 billion weights. At about 8.5 bits each (8 bits plus the block scales), that is 7.7 GB: the Q8 file that we dropped. At about 4.9 bits (Q4_K_M), it is 4.4 GB: the file is 4.37 GB. And memory follows the file:

Memory of each set-up while loaded, as ollama ps reports it, with a context of 4,096 tokens. Orange: the set-up that we dropped.

Text description of the chart

A column chart of the memory that each set-up uses while it is loaded, in gigabytes, from 0 to about 9. Gemma Q4: 2.89. Gemma Q8: 4.53. Mistral Q4: 4.94. Mistral Q8: 8.21, the tallest column, in another colour.

What this tells us. The three set-ups that we kept need 3 to 5 GB. Mistral Q8 needs 8.2 GB, half of the computer. With one of the three loaded, macOS still had 40% to 50% of its memory free. Earlier that day, other programs used several models on the same computer. The free memory fell to 9%, the computer used the disk as memory (swapping), and everything became slow. On a 16 GB computer that also runs office work, an 8 GB model leaves no room for safety. So we measured Mistral Q8 once, for the record, and dropped it.

Q4_K_M is a common default, at about a third of the 16-bit size. Below 4 bits, quality usually drops fast. The cost of quantization is a risk, not a fixed loss: measure it on your own task.

Part 3: measure the speed

The problem: “fast” means several things

Ines cares how long a care worker waits after pressing Send, and how long the 300 notes of a day take. Four measurements from Resource trade-offs answer that:

Measurement	What it is	Who feels it
Load time	Reading the model file into memory, before the first answer	The first person after a restart, or after a quiet moment
Time to first token	From sending the note to the first piece of the answer. The model reads the whole prompt first.	Anyone who watches the answer appear
Tokens per second	How fast the model writes the answer, once it starts	A long answer
Total time per note	From sending the note to the end of the answer	The daily batch: 300 notes × this time

measure.py takes all of them. In each pass and for each set-up, it unloads the model, loads it again, times the load and reads the memory. Then it sends the 18 notes one by one, at temperature 0, with the same settings for every model. Three passes take turns between the set-ups, so a busy minute does not hit only one set-up. The median ignores one slow moment.

Step 1: one note first

Try one note with one set-up before the long run:

python3 measure.py gemma3:4b --passes 1 --notes R01

Output

pass 1 gemma3:4b: loaded in 2.75 s
  R01   3.19 s  first token  1.25 s  37.0 tokens/s
Wrote runs/gemma3_4b.jsonl

The file has one line for the model, one for the load, and one for each note. The line for R01, shortened:

runs/gemma3_4b.jsonl (shortened)

{"pass": 1, "type": "item", "id": "R01", "total_s": 3.195, "first_token_s": 1.248,
 "input_tokens": 531, "output_tokens": 72, "tokens_per_s": 37.0,
 "ollama_prompt_s": 1.236, "ollama_answer_s": 1.945, "other_models_loaded": [],
 "free_memory_percent": 51, "text": "```json\n{\n  \"fall\": \"no\",\n  \"medication\": \"taken\", ..."}

What this tells us. The load took only 2.75 seconds because the file was already in the computer’s file cache; after a restart, the first load reads the disk and takes longer. Look at the start of the text, too: ```json. We come back to it in Part 4.

Step 2: the full run

Close other heavy programs, and check that no other model is loaded:

ollama ps

The list must be empty. Then run the three set-ups:

python3 measure.py gemma3:4b gemma3:4b-it-q8_0 mistral:7b --passes 3

The run sent 162 notes and took 11 minutes on our computer. The output has one line per note:

Output (the first notes of each set-up, pass 1)

pass 1 gemma3:4b: loaded in 1.24 s
  R01   3.20 s  first token  1.25 s  36.8 tokens/s
  R02   3.25 s  first token  1.15 s  34.7 tokens/s
...
pass 1 gemma3:4b-it-q8_0: loaded in 3.49 s
  R01   4.61 s  first token  1.24 s  22.3 tokens/s
  R02   4.54 s  first token  1.16 s  21.9 tokens/s
...
pass 1 mistral:7b: loaded in 2.53 s
  R01   6.37 s  first token  2.72 s  21.3 tokens/s
  R02   4.59 s  first token  0.47 s  20.9 tokens/s

Common mistake

The same model is a hundred times slower to load an hour earlier

Symptom: Theo’s first test of gemma3:4b, at 14:13, gave a load of 127.6 seconds and a first token after 6 to 13 seconds. In the next attempt, the load took 857 seconds. At 16:23, on the same computer, the same model loaded in 1.2 seconds and gave its first token after 1.2 seconds.

Cause: other programs used Ollama at the same time. On our computer, other course authors ran their own tests on the same Ollama server. Ollama answers one request at a time and keeps only one or a few models in memory. So our requests waited in a queue, and other programs unloaded our model to load theirs. The clock in measure.py counts the waiting too.

Fix: measure on a quiet computer, and check ollama ps before each run. measure.py --wait 10 waits up to 10 minutes before each set-up until no other model is loaded. Ollama also reports its own time to read the prompt and write the answer: report.py shows it as “of which the model worked”. It leaves out every answer that waited in the queue or had to load the model again, and says how many. Repeat the passes and keep the medians.

Consequence in AI work: a speed test on a shared machine compares the other users, not the models. On the agency’s computer, the same queue appears when two care workers send notes at once: the second one waits for the first. Measure that too before you promise a time.

Step 3: the speed results

Run the report script. It reads every run and writes results.md:

python3 report.py

Output (the speed rows)

| | gemma3:4b-it-q8_0 | gemma3:4b | mistral:7b |
|---|---|---|---|
| Size on disk (GB) | 4.98 | 3.35 | 4.37 |
| Memory while loaded (GB) | 4.53 | 2.89 | 4.94 |
| Load time (s) | 3.49 | 2.75 | 2.49 |
| Time to first token (s) | 1.1 | 1.16 | 0.32 |
| Tokens per second | 23.0 | 34.65 | 21.6 |
| Total time per note (s) | 4.32 | 3.29 | 4.1 |
| Of which the model worked (s) | 4.31 | 3.28 | 4.09 |
| Slowest note (s) | 4.73 | 3.546 | 6.374 |

The end of the file says that no answer had to be left out: the computer was quiet.

How fast each set-up writes its answer: median over 3 passes of 18 notes (Mistral Q8: 1 pass). Blue: Q4. Orange: Q8.

Text description of the chart

A column chart of the median tokens per second. Gemma Q4: 34.65, the tallest. Gemma Q8: 23.0. Mistral Q4: 21.6. Mistral Q8: 12.2, the shortest.

What this tells us.

• Q8 is slower than Q4: Gemma writes 23 instead of 35 tokens per second, and Mistral 12 instead of 22. A Q8 file has almost twice the bytes to read for each token (why).
• The larger model is slower too: Mistral Q4 has more weights than Gemma Q4, and writes about 40% fewer tokens per second.
• Time to first token tells another story: Mistral answers after 0.32 seconds, Gemma after 1.1 seconds. Ollama reported that Mistral read the shared instructions again only for the first note after a load (2.7 seconds for R01). After that, it seems to reuse them, and reads only the new note. Gemma read the whole prompt for every note.
• For the daily batch, all three are fine: 300 notes take 16 to 22 minutes, one at a time.

Part 4: score the quality

The problem: a fast wrong answer is worse than no answer

A form that says routine for a client who fell and could not get up is dangerous: the urgent case hides among the routine ones. Speed means nothing until the answers are right.

The rubric

This is Module 4’s rubric: the same four criteria, scored 0, 1 or 2 per note, with the same meaning for each score. The note takes the place of the policy, and “holding back” means writing not stated or null instead of a guess.

Criterion	2	1	0
Correctness	The four labels and the summary agree with the note	One small error that does not change the care (a fall marked yes that did not happen, follow_up for a routine note)	A wrong or invented fact that could change the care: an urgent note not marked urgent, a missed dose marked taken and routine
Completeness	The summary has every fact of must_mention	One needed fact is missing	The main fact is missing, or there is no summary
Format	Exactly one JSON object with the five keys, allowed values, and a summary of at most 30 words	Usable after a small fix: a code fence or text around the object, or one value that is not allowed	The program cannot use it: no JSON object, or a missing key
Abstention	not stated or null for every field that the note does not state, and a value for every field that it states	Too cautious: not stated for a fact that the note states	Not cautious enough: a value that the note does not state, such as an invented “no fall”

It is an adapted rubric in one way: there is no “ready” rule, because missed urgent notes are counted on their own, as a must-have. As in Module 4, abstention counts on every note, because a form can be too cautious as well as not cautious enough. A not stated for a stated fact is not false, so it costs abstention, not correctness. We count one more number on its own, because it is a must-have: urgent notes missed, out of the 4 urgent notes.

Step 1: what a program can check

python3 score.py

Output

gemma3:4b-it-q8_0: 18 answers in pass 1; answers that changed in later passes: 0
gemma3:4b: 18 answers in pass 1; answers that changed in later passes: 0
mistral:7b: 18 answers in pass 1; answers that changed in later passes: 0

Read the summaries: 54 rows of scores/reading.csv need completeness (0, 1 or 2) and false_fact (none, minor or serious). Then run this script again.

What this tells us. At temperature 0, each set-up gave the same answers in all three passes, so the script scores pass 1. (In one earlier run, gemma3:4b changed one word of one summary after Ollama reloaded it: “the same” at temperature 0 is very likely, not guaranteed.) A program cannot tell whether a summary is true to the note: that needs a person.

Step 2: read the summaries

Open scores/reading.csv in a spreadsheet program. Each row has one summary and the facts that it should mention. Fill in two columns:

• completeness: 2 if the summary has every fact of must_mention, 1 if it has some, 0 if none.
• false_fact: none, minor (a wrong detail that does not change the care), or serious (a false fact that could change the care).

Open the note in data/notes.jsonl when you are unsure. Score without looking at the model name if you can: sort the rows by id, not by model. Four real rows, and how we scored them:

Note	Summary	Completeness	False fact
D04 (new confusion)	“First visit since Mr Haddad came home from hospital. Unusual confusion, taking medication with help, no falls, pain score 2/10.”	2	none
D04	“Mr. Haddad took his medication with assistance and reported a pain score of 2/10.”	0: no word about the confusion	none
R06 (breathing)	“Mrs Petrova experienced new shortness of breath, requiring multiple seatings. Called the coordinator for an immediate nurse visit.”	2	minor: the note says the nurse comes this afternoon
R02 (missed dose)	“Mrs. Okafor reported a pain score of 2 in her hip and took her evening medication.”	0: the missed lunchtime dose is not there	none

The first and third rows are Mistral’s; the second and fourth, Gemma Q4’s. Save the file as CSV, and run the script again:

python3 score.py

Output (last lines)

model                              correct complete format abstain  urgent missed
gemma3:4b-it-q8_0                   28/36    28/36   18/36   34/36      2 of 4
gemma3:4b                           28/36    25/36   18/36   34/36      1 of 4
mistral:7b                          30/36    35/36   36/36   33/36      1 of 4
Wrote scores/auto.csv, scores/reading.csv and scores/rubric.csv

Quality against speed: higher is better, further left is faster. The points add up the four criteria, each out of 36 (18 notes × 2). The vertical axis starts at 80.

Text description of the chart

A scatter plot of five set-ups, with the median seconds per note on the horizontal axis and the rubric points on the vertical axis. Gemma Q4: 3.29 seconds, 105 points. Gemma Q8: 4.32 seconds, 108 points. Mistral Q4: 4.1 seconds, 134 points, the highest. Mistral Q8: 6.9 seconds, 133 points. Gemma Q4 with a JSON schema: 3.11 seconds, 125 points.

What this tells us.

• Mistral Q4 gives the best forms: complete summaries (35 of 36), a perfect format, and no invented value.
• Q8 did not help on this task. Gemma Q8 scored 3 points more than Gemma Q4, which is noise, and it missed one more urgent note. Mistral Q8 scored 1 point less than Mistral Q4. Both Q8 files cost about 1.6 times the memory and 1.3 to 1.7 times the time per note.
• The two models fail in opposite directions. Gemma guesses: it marked “no fall” for M01, where the note says nothing about a fall (abstention 0). Mistral is too cautious: it wrote not stated for three notes that say “No falls” (abstention 1 each). Mistral’s worst error is R02: it called the missed lunchtime dose taken and routine.
• Every set-up missed D04: new confusion after a hospital stay, which the rules call urgent. All three wrote follow_up. Mistral’s summary says “Unusual confusion”, so a person who reads it sees the problem. The label alone hides it.

Common mistake

Every Gemma answer fails the program that reads the form

Symptom: the format score of both Gemma set-ups is 18 of 36: one point for each note. A program that reads the forms with json.loads fails on all 18 Gemma answers.

Cause: Gemma wraps every answer in a Markdown code block, ```json ... ```, although the instructions say “Return only a JSON object”. A person does not notice it; a program does. score.py can still find the object inside, so the format gets 1, not 0.

Fix: ask Ollama for structured output: give it the JSON schema of the form, and it allows the model only tokens that fit the schema. measure.py --structured does it:

python3 measure.py gemma3:4b mistral:7b --passes 3 --structured

With the schema, Gemma’s format went from 18 to 36 of 36, at the same speed (3.11 seconds per note). Its summaries also changed, and became more complete (29 instead of 25). But it still missed D04, and now invented “no fall” in X01, the shift-swap message: abstention 32 of 36. Mistral’s answers kept the same scores. A schema guarantees the shape of the answer, not its truth.

Consequence in AI work: a format problem looks small in a demo, where a person reads the answer. In a pipeline, the program that reads the form fails on every note, or a careless clean-up step hides other errors. Score format compliance as its own criterion, and use the strict option that the serving software offers.

Part 5: the recommendation

Must-haves first

Before any trade-off, remove the set-ups that fail a must-have. Wren has four:

Must-have	Gemma Q4	Gemma Q8	Mistral Q4	Mistral Q8
Runs only on the agency’s computer	yes	yes	yes	yes
Licence allows the use, with a person deciding	yes	yes	yes	yes
Fits in 16 GB next to office work	yes (2.9 GB)	yes (4.5 GB)	yes (4.9 GB)	no (8.2 GB)
Misses no urgent note on the test set	no (1 of 4)	no (2 of 4)	no (1 of 4)	no (1 of 4)

No set-up passes the last must-have. That is a finding, not a failure of the method. It means that no model may sort the notes on its own: the form must never lower the priority of a note. The model can still help, as a reading aid next to the note, while the coordinator reads every note. With that design, the last row becomes a known limit, and the choice is between the first three.

Then the trade-off

	Gemma Q4	Gemma Q8	Mistral Q4
Quality (rubric points of 144)	105; 125 with the schema	108	134
Seconds per note	3.3	4.3	4.1
Memory	2.9 GB	4.5 GB	4.9 GB
Licence	Gemma Terms (no automated healthcare decisions)	the same	Apache 2.0

Mistral Q4 costs 0.8 seconds more per note than Gemma Q4 (4 minutes for 300 notes) and 2 GB more memory. For that, the coordinator gets the most complete summaries and clean forms.

The recommendation

RECOMMENDATION.md in the finished project is the one-page record. Its core:

Recommendation: mistral:7b (Mistral 7B Instruct v0.3, Q4_K_M) with structured output, as a reading aid next to each note, not as triage.

Hardware: about 5 GB of memory for the model, so a computer with 16 GB; 4.4 GB of disk. One note at a time, about 4 seconds per note: 300 notes in about 20 minutes.

Limits: no set-up met “miss no urgent note”: all missed new confusion (D04). Mistral marked a missed dose as taken (R02), and wrote not stated for “No falls” three times. 18 invented notes, not checked by a second reader. The form must never lower the priority of a note; the coordinator reads every note.

Fallback: gemma3:4b with structured output, on an 8 GB computer or if Mistral is not available. It needs 2.9 GB and 3.1 seconds per note, with less complete summaries. A person must decide, as its licence requires. If the computer is down, the coordinator reads the notes without forms, as today. No note goes to a hosted model.

Look again when: the agency has a test set of its own notes; the urgency rule for confusion changes in the prompt; a new model or Ollama version arrives; several workers send notes at once; any urgent case is missed in real use.

Responsible use

A local model solves one problem: the notes no longer leave the agency. It does not make the use safe by itself.

The model helps; a person decides. On our test set, every set-up called one or more urgent notes less urgent than they were. The form must never close a case, schedule a visit or skip a call on its own. The coordinator reads the note, not only the form. The Gemma licence requires this in healthcare, and it is good practice with any model.

The computer is now a store of health data. The notes, the runs/ folder and the logs on the Mac mini are as sensitive as the client files. The computer needs disk encryption, user accounts, updates and backups, like any system with health data. Ollama must keep listening only on 127.0.0.1. Never put real notes in a public repository: this project uses synthetic notes for that reason.

Test with your own notes, inside. These 18 notes are invented and short. Before real use, the agency builds a test set from its own notes, on its own computer, with the privacy officer’s approval, and scores it the same way. Real notes are longer, use the agency’s abbreviations, and mix languages.

Errors are not equal across people. Watch the French notes, and notes from workers who write less English. On our two French notes, all three set-ups got the urgent fall (F02) right, and Gemma Q4 called F01 follow_up instead of routine. Two notes cannot show a difference between languages; a real test set needs many more, from every group of workers.

A local model changes too. A new version can arrive under the same name. Note the exact model file (ollama show gives it), and run the comparison again before any update.

Make it a portfolio piece

This project shows that you can choose a model under a real constraint, with evidence. It also shows a decision that someone else can check. Publish it as a repository under your name.

What to put in the repository:

local-model-choice/
  README.md            the situation, the rule, how to run it, the result in one table
  RECOMMENDATION.md    the choice, the hardware, the limits, the fallback, when to look again
  LICENSE              MIT, for the code
  data/                notes.jsonl, task_prompt.md, DATASET.md (synthetic, CC0)
  check_setup.py, measure.py, score.py, report.py
  runs/                every answer and every time, per set-up
  scores/              auto.csv, reading.csv (your reading), rubric.csv
  results.md, results.json

1. Start from your own project folder, with your own runs and your own reading of the summaries.
2. Write RECOMMENDATION.md from the template in the starter. Rewrite the first paragraph of the README in your own words. Add your computer: model, memory, operating system, Ollama version.
3. Make the repository: git init, git add ., git commit -m "Local model choice for visit notes".
4. Push it to GitHub as a public repository. The notes are synthetic and CC0, so you can publish them. Never publish real notes.

How to describe it in an interview, in two sentences: “Health notes had to stay on our own hardware, so I compared two open-weight models and two quantizations with Ollama on memory, speed and rubric quality. No set-up caught every urgent note, so I recommended Mistral 7B at 4-bit as a reading aid, with its hardware, limits and a fallback.”

Go further:

1. Fix the confusion rule. Change the urgency rule in data/task_prompt.md, for example with “does not know the day or the place”. Run all set-ups again, and check that the other notes do not get worse.
2. A third model. Add Qwen3 4B (Apache 2.0). It “thinks” before it answers: measure what that costs per note, as in Reasoning and multimodality.
3. Two requests at once. Send two notes at the same time and measure the time per note. This is the morning rush, when several workers finish their visits together.
4. A smaller computer. Run gemma3:4b on a laptop with 8 GB, and add a column to the table.

Knowledge check

Check your understanding

A colleague says: 'Q8 is more precise than Q4, so it is better. Use the largest file that fits.' What do you answer?

Show the answer

Q8 stores each weight more exactly, but that does not make the answers better on every task. On our 18 notes, Gemma Q8 scored 108 points against 105 for Q4 (noise) and missed one more urgent note; Mistral Q8 scored 133 against 134. Both Q8 files needed about 1.6 times the memory and were slower per note. “The largest that fits” also ignores the other work on the computer: Mistral Q8 needs 8.2 GB of a 16 GB machine. Measure both on your own test set, and take the smaller file unless the larger one is clearly better.

Check your understanding

Your timing run says that mistral:7b needs 12 seconds per note. Yesterday it needed 4. Nothing changed in the project. What do you check, and how?

Show the answer

Check whether the computer was busy. Run ollama ps: another model in the list means another program uses Ollama. Compare the total time with Ollama’s own working time (“of which the model worked” in results.md). Read how many answers report.py left out because they waited in a queue or reloaded the model. Then repeat the passes on a quiet computer and keep the medians. One slow run is not a property of the model.

Check your understanding

Gemma 3's terms forbid 'making automated decisions in domains that affect ... well-being (e.g. ... healthcare ...)'. Can Wren use Gemma for the handover form?

Show the answer

Yes, if the form only helps a person who decides. Here the coordinator reads every note and form, and decides what to do; the model decides nothing on its own. Perhaps the agency later wants the form to trigger actions, for example to skip a call for a “routine” note. That use needs a new legal review, and on our test set no model is good enough for it. With any model, the agency needs the legal basis and the consents to process the notes. This is a reading of the terms, not legal advice.

Recap

• A privacy rule is a must-have that removes every hosted model. Check that the local server listens only on 127.0.0.1.
• Open weights come with a licence: Apache 2.0 adds no use rules; the Gemma terms forbid automated decisions in healthcare.
• Quantization stores each weight with fewer bits (Q8_0 about 8.5 bits, Q4_K_M about 4.9). Memory and time per token shrink with it; here Q8 gave no better forms.
• Measure load time, time to first token, tokens per second, time per note and memory. Repeat, keep the medians, and check that no other program uses the server.
• A program checks labels, format and abstention; a person reads the summaries. Count missed urgent notes on their own.
• Mistral 7B at Q4 gave the best forms in about 4 seconds and 5 GB. No set-up caught every urgent note, so it is a reading aid, never triage. Gemma Q4 with a schema is the fallback.

Next, the final assignment of the course: choose a model for a task of your own, with the same method.

Next: Final assignment