Small local LLMs
- Lesson 21
- 40 min
- Module 5: LLMs as classifiers
Where we are
Our LLM runs on a provider's computers, and every message leaves Larkfield. A small model can run on the laptop instead. In this lesson, we prompt a small local instruction model in the same way, and we measure its accuracy, its memory and its time per message.
Lesson 21 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceSmall local LLMs
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Run the routing on Larkfield’s own computer, so that no message leaves the company and no call has a price. Measure what that costs instead: accuracy, valid answers, memory and seconds per message. On this inbox, the small local models are far from the hosted LLM and the fine-tuned encoder.
NotebookDo this lesson in a notebook
The notebook replays the measured answers of the small local model Qwen3-0.6B on the 2,000 test messages. It compares a 500-message sample with the whole test set and computes a 95% interval for a score. Then it turns seconds per message into a cost. It loads no model, so it needs no download and no key. To run a local model yourself, use the notebook of Ask for a decision, not for text. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The LLM prompts, the schema and the answer check, from Zero-shot classification with an LLM.
- The few-shot prompt and its results, from Few-shot examples and label descriptions.
- The fine-tuned encoder, from Fine-tune a small encoder on CPU.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
The course’s runs were tested on macOS (Apple silicon, 16 GB) with Python 3.12, Transformers 5.22.2 and PyTorch 2.15.0 (CPU). The notebook was tested with pandas 3.1.1. Colab, Kaggle, Windows and Linux were not tested.
A sorting clerk who never leaves the building
Section titled “A sorting clerk who never leaves the building”Some post offices may not send letters outside to be sorted: the letters are private. So they hire their own clerk, who works in a back room. He costs no fee per letter. But he is one person at one desk. He is slower than the big sorting company, and he has read much less.
A local LLM is that clerk. It runs on Larkfield’s own computer. The question is how much slower and how much less accurate it is, and whether that price is worth the privacy.
The problem: every message leaves Larkfield
Section titled “The problem: every message leaves Larkfield”The hosted LLM of the last lesson is accurate on this inbox: 0.970. But every call sends the customer’s whole message, with the order number and sometimes an address, to a provider’s computers. Larkfield’s own data rules may forbid that. The provider also limits the number of calls per minute, and charges per token.
Tomás asks: “Can’t we run an LLM on our own machine?” Yes. Small open-weight models exist: anyone can download the model files and run them. The model never sends anything out. We need to know what it gives up.
The idea: a small instruction model on the CPU
Section titled “The idea: a small instruction model on the CPU”An instruction model is an LLM trained to follow requests like “Choose one route for this message”. Small ones have 0.5 to 4 billion parameters: the numbers that the model learned. The hosted models have many more. Small models fit in a laptop’s memory and run on its CPU, the main processor, with no graphics card.
The course ran two of them, through the Hugging Face Transformers library, on a laptop CPU (Apple M4, 16 GB, 4 threads):
| Model | Parameters | Number format | Licence |
|---|---|---|---|
| Qwen3-0.6B | 0.6 billion | float32: 4 bytes per number | Apache-2.0 |
| Qwen2.5-1.5B-Instruct | 1.5 billion | bfloat16: 2 bytes per number | Apache-2.0 |
A tiny worked example: how much memory?
Section titled “A tiny worked example: how much memory?”The model’s numbers must fit in memory. Multiply the parameters by the bytes per number:
- Qwen3-0.6B: 0.6 billion × 4 bytes = about 2.4 GB for the weights;
- Qwen2.5-1.5B: 1.5 billion × 2 bytes = about 3.0 GB for the weights.
The program needs more than the weights while it runs. The course measured a peak of about 3.8 GB for Qwen2.5-1.5B. For Qwen3-0.6B, the peak was 3.7 to 4.6 GB. So the larger model needs about the same memory, because each of its numbers is half the size. On a 16 GB laptop with a browser and an editor open, one such model at a time is the limit.
Two ways to get a route
Section titled “Two ways to get a route”The course asked each model for a route in two ways:
- A JSON answer, as the hosted LLM: the model writes a route, a confidence and a reason. Same prompt as the hosted LLM with the route names only.
- Route scores: the model writes nothing. For each of the seven route names, the code reads how likely the model finds that name as the answer, and the highest wins. This prompt was short and had the route descriptions. Ask for a decision, not for text explains this way in detail; here, we only look at the results.
Keep in mind: the two ways also used different prompts. A gap between them mixes the two changes.
Larkfield: the local models on the test messages
Section titled “Larkfield: the local models on the test messages”Qwen3-0.6B on all 2,000 test messages
Section titled “Qwen3-0.6B on all 2,000 test messages”The small model ran on every test message:
| Qwen3-0.6B, 2,000 test messages | Valid answers | Accuracy | Takeovers found | Cost per 1,000 | Time per message |
|---|---|---|---|---|---|
| JSON answer | 97.2% | 0.282 | 0 of 12 | US$7,198.3 | 2.1 s |
| Route scores | 100% | 0.194 | 7 of 12 | US$3,410.0 | 453 ms |
It fails on this inbox. With the JSON answer, it chose orders for 1,395 of the 2,000 messages. The other 2.8% of the answers stopped in the middle, at the 60-token limit, and went to review. Both ways cost more than twice “everything to a person” (US$1,500 per 1,000). This model is too small for Larkfield’s seven routes.
Qwen2.5-1.5B on a 500-message sample
Section titled “Qwen2.5-1.5B on a 500-message sample”The larger model took 5 to 8 seconds per message. All 2,000 test messages would have taken more than four hours of a shared laptop for each way. So the course ran it on a 500-message sample of the test set. The sample is the first 500 test messages by message ID, M-100001 to M-102554. That sample is easy to choose, but it is not stratified: nobody fixed the share of each route. It has a few more delivery and orders messages, and fewer account and billing messages, than the whole test set.
A score on 500 messages is less certain than a score on 2,000. A 95% interval gives the range of accuracies that fit the result. Here, we use the Wilson interval, which works well for a share. To compare fairly, the chart shows every method on the same 500 messages:
Text description of the chart
Accuracy on the same 500 test messages, with 95% Wilson intervals: Qwen3-0.6B, route scores 18.2% (15.1 to 21.8); Qwen3-0.6B, JSON answer 27.4% (23.7 to 31.5); Qwen2.5-1.5B, JSON answer 36.8% (32.7 to 41.1); Qwen2.5-1.5B, route scores 73.4% (69.4 to 77.1); Hosted LLM, route names only 75.4% (71.4 to 79); Hosted LLM, few-shot 97% (95.1 to 98.2); Fine-tuned encoder 97.2% (95.4 to 98.3).
On this sample, Qwen2.5-1.5B with route scores gets 367 of 500 right: 0.734 (95% interval 0.694 to 0.771). With a JSON answer, it gets 184: 0.368 (0.327 to 0.411). The intervals do not overlap: the route scores are really better here, with the prompt and the method both changed. Qwen3-0.6B, on the same 500, scores 0.182 and 0.274. The larger model is much better than the smaller one.
But look at the top of the chart. On the same 500 messages, the hosted LLM with examples scores 0.970, and the fine-tuned encoder 0.972. The best local result is about as good as the hosted LLM with the route names only (0.754). It is far from the methods that know Larkfield’s rules. The sample has only 2 of the 12 takeovers, and Qwen2.5-1.5B sent neither of them to account, in either way.
Why the JSON answers fail
Section titled “Why the JSON answers fail”With the JSON answer, 101 of the 500 answers were not valid: only 79.8% were. Most were not JSON at all. Here is a real one, the whole answer:
{l3.invalid_examples[0].answer}The route and the confidence are there, on separate lines, but the check cannot read them. And among the valid answers, one route took over:
Text description of the chart
A column chart of the routes that Qwen2.5-1.5B chose with a JSON answer, on the 500-message sample. Returns_refunds 287, review (answers that were not valid) 101, delivery 50, account 25, billing 17, orders 9, product 6, needs_person 5. The sample really has 103 returns_refunds messages.
The sample has 103 returns_refunds messages, and the model chose that route 287 times. The hosted LLM’s schema prevented broken answers. A local model run this way has no schema unless you add one, and constrained decoding (in the decision-model lesson) is one way.
Time, memory and privacy side by side
Section titled “Time, memory and privacy side by side”Text description of the diagram
The top row is the hosted LLM: the message leaves Larkfield. First, Larkfield’s server sends the message text and the prompt, about 717 tokens, over the internet. Second, on the provider’s computers, chat-small reads it and writes a JSON answer of about 32 tokens. The computers are shared with other customers, so a rate limit caps the calls per minute. Third, the answer comes back in 1.7 seconds per message (median), and 5.3 seconds for the slowest 10%. It costs US$0.088 per 1,000 messages in tokens, and scores 0.970 on the 2,000 test messages. The bottom row is the small local LLM: the message stays at Larkfield. First, Larkfield’s own computer loads Qwen2.5-1.5B-Instruct once, with a peak memory of about 3.8 GB. Second, on the CPU, it scores the 7 route names or writes a JSON answer. Nothing leaves the computer, there is no rate limit and no price per call. Third, the answer takes 7.7 seconds per message with route scores and 5.2 seconds writing JSON. It costs about US$0.11 per 1,000 messages in computer time, an assumption. It scores 0.734 on a 500-message sample with route scores.
Computer time is not free. The course prices it with an assumption of US$0.05 per CPU-hour. At 7.7 seconds per message, that is about US$0.108 per 1,000 messages. That is more than the hosted LLM’s tokens (US$0.088). At Larkfield’s 65 messages a day, the local model needs about 8 minutes of CPU a day. A shop with 50,000 messages a day would need about 107 hours of CPU a day: many machines.
Where a local LLM beats the fine-tuned encoder, and where it does not
Section titled “Where a local LLM beats the fine-tuned encoder, and where it does not”The fine-tuned encoder of Module 4 also runs on Larkfield’s computer. It is the fair comparison for “keep the messages at home”.
| On this inbox | Fine-tuned encoder | Qwen2.5-1.5B, route scores |
|---|---|---|
| Accuracy on the 500-message sample | 0.972 | 0.734 |
| Time per message | 6.7 ms | 7.7 s, about 1,150 times slower |
| Labelled messages needed | 6,000 | none |
| Size of the model files | 470.6 MB (base) + 43.3 MB | about 3 GB of weights |
| Can explain its answer in words | no | yes, with a JSON answer (less accurate here) |
So here, the encoder wins on accuracy, speed and size. A small local LLM can still help:
- With no labels at all. On the first day of a new route, it can sort messages for a person, with no training. The encoder needs labels first.
- With a classification head. A small LLM can also get a trained layer with one output per route, as the encoder has. The course did not train one: the encoder already shows that idea at a much lower cost. If you try it, add it as one more row and measure it the same way.
- When privacy forbids a hosted model and you have few labels. Then a larger local model, or the route-score method of Module 6, may be the best start.
Guided practice: read the local results
Section titled “Guided practice: read the local results”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, replay Qwen3-0.6B’s answers. You see
valid answers: 97.2%and the accuracy0.282. - Read the table of labels and choices. Find the row
orders: it chose that route 1,395 times. - In section 2, predict, then score the model on the first 500 test messages. It gets
0.274. - In section 4, run the cell. You see
0.14 CPU-hours a day; US$0.1076 per 1,000 messages.
Your turn: how sure is a score on 500 messages?
Section titled “Your turn: how sure is a score on 500 messages?”In section 3 of the notebook, complete the function wilson(k, n). It returns the 95% interval for k right answers out of n. Test it on Qwen2.5-1.5B’s 367 right answers out of 500. Predict first: will the interval be narrower or wider than 0.70 to 0.77?
Show the answer
def wilson(k, n, z=1.96): p = k / n centre = (p + z * z / (2 * n)) / (1 + z * z / n) half = z * np.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / (1 + z * z / n) return (round(float(centre - half), 3), round(float(centre + half), 3))
print(wilson(367, 500), wilson(137, 500))(0.694, 0.771) (0.237, 0.315)A little wider: 0.694 to 0.771. With 500 messages, a score of 0.734 could really be anything in that range. On all 2,000 messages, the interval would be about half as wide. The two models’ intervals are far apart, so the larger model is really better than the smaller one.
Common mistake
A score from a sample that nobody chose
Symptom: a teammate runs a local model on “the first 500 messages” and reports 0.734 as the model’s accuracy. Later, on all 2,000, the number is different, and nobody knows which one to trust.
Cause: the first 500 by message ID is a convenient sample, not a designed one. Its mix of routes differs from the whole test set: here, 8.0% account messages instead of 9.8%, and only 2 of the 12 takeovers. A model that is weak on one route scores differently on a sample with fewer of those messages. And 500 messages give a wide interval: about ±4 points here.
Fix: say “on a 500-message sample” every time you quote the number, with its interval. Compare methods only on the same messages, as the chart above does. When you must sample, stratify by route and language, as the course’s splits do. Run the whole test set before you choose a model.
Consequence in AI work: a number from a convenient sample travels without its context. It ends up in a slide as “the local model gets 73%”. Every number needs its sample and its interval next to it.
Knowledge check
Section titled “Knowledge check”Check your understanding
Qwen2.5-1.5B has 2.5 times more parameters than Qwen3-0.6B, but needs about the same memory. Why?
Show the answer
Memory depends on the number of parameters and on the bytes for each number. Qwen3-0.6B ran in float32, with 4 bytes per number: 0.6 billion × 4 = about 2.4 GB of weights. Qwen2.5-1.5B ran in bfloat16, with 2 bytes per number: 1.5 billion × 2 = about 3.0 GB. The measured peaks were about 3.8 GB for the larger model and 3.7 to 4.6 GB for the smaller one.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the 500-message sample, Qwen2.5-1.5B scores 0.734 with route scores and 0.368 with a JSON answer. Give two reasons why the JSON answer does so much worse, and one reason to be careful with the comparison.
Show the answer
First, 101 of the 500 JSON answers were not valid (79.8% valid). The model wrote the route and the confidence on separate lines, not as JSON, so they went to review. Second, among the valid answers, it chose returns_refunds 287 times, for 103 real ones. Be careful: the two ways also used different prompts. The JSON answer had the route names only, and the route scores had the route descriptions. The gap mixes the method and the prompt.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Larkfield's data rules forbid sending messages to a hosted model. Should it use Qwen2.5-1.5B as its router? Use the numbers.
Show the answer
No, not on this inbox. The fine-tuned encoder also runs on Larkfield’s own computer, so it keeps the messages at home too. On the same 500 messages, it scores 0.972 against 0.734. It takes 6.7 ms per message against 7.7 s, and its files are much smaller. On the sample, Qwen2.5-1.5B with route scores costs US$3,232.1 per 1,000 messages. That is more than everything to a person (US$1,500). A local LLM could help on the first day of a new route with no labels, as a sort for a person to check.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A local LLM keeps every message on Larkfield’s computer: no provider, no rate limit, no price per call.
- Memory is about parameters × bytes per number: 1.5 billion × 2 bytes (bfloat16) is about 3 GB; the run peaked at about 3.8 GB.
- Qwen3-0.6B is too small here: 0.282 with a JSON answer, 0.194 with route scores, on all 2,000 test messages.
- Qwen2.5-1.5B, on a 500-message sample: route scores 0.734 (0.694 to 0.771), at 7.7 s per message. JSON answer 0.368, only 79.8% valid. The prompts also differed.
- On the same 500, the hosted LLM with examples and the fine-tuned encoder score about 0.97. The encoder is the better “keep it at home” choice: 6.7 ms per message.
- Quote a sample’s score with its sample and its 95% interval.
Next, we go the other way: a larger, slower model that thinks before it answers, only for the messages that every method finds hard.