Cost per 1,000 messages
- Lesson 23
- 30 min
- Module 5: LLMs as classifiers
Where we are
The comparison table now has hosted, local and reasoning models next to the local classifiers. Their costs are in different units: tokens, seconds and memory. In this lesson, we turn all of them into one cost per 1,000 messages, with a dated price table.
Lesson 23 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceCost per 1,000 messages
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Put every model on one bill: the cost per 1,000 messages. It comes from tokens and a dated price table, or from seconds of computer time, plus the cost of the mistakes. Then add the costs that are not in dollars: the time per message, the rate limit and the privacy of the messages. This is how you choose an LLM option, or none.
NotebookDo this lesson in a notebook
The notebook reads the price table and computes the cost per 1,000 messages from the recorded tokens of three hosted prompts. It puts the running cost next to the cost of the mistakes. Then it checks what a rate limit and the time per message mean for a whole inbox. You need no account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The cost table and “everything to a person”, from The cost of each mistake.
- The costs per model and the per-year arithmetic, from What the extra accuracy costs.
- The three LLM options of this module: hosted, local and reasoning.
What you need on your computer
- Nothing for the notebook route. This module changes no file of your
larkrouteproject.
Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.
Three quotes for one job
Section titled “Three quotes for one job”Larkfield’s post office asks three companies to sort its letters. The first quotes a price per sheet of paper read. The second quotes a price per hour of a clerk’s time. The third rents a machine and charges nothing per letter. The quotes cannot be compared as they are.
The manager turns each one into the same unit: dollars per 1,000 letters. Then she adds what each company’s mistakes would cost, how long each letter would wait, and whether the letters leave the building. Only then does she choose.
The problem: costs in different units
Section titled “The problem: costs in different units”This module added three kinds of LLM to the comparison table, and each comes with its own unit:
- the hosted LLM charges per million tokens, a different price for input and output;
- the reasoning model charges per token too, at a higher price, and adds hidden reasoning tokens;
- the local LLM has no price per call, but uses seconds of a computer and gigabytes of memory.
Tomás compares the price per token only. “The small hosted model costs 10 cents per million input tokens, so it is almost free. The local model is free.” Both sentences skip most of the bill. We need one number for every row, and then the costs that no number holds.
The idea: one cost per 1,000 messages
Section titled “The idea: one cost per 1,000 messages”The cost per 1,000 messages has two parts:
- the running cost: what the model costs to run, from tokens and a price table, or from seconds and a price per CPU-hour;
- the mistake cost: what its wrong routes cost, from the cost table.
Their sum is the column total of the comparison table. Then we look at three costs outside the number: the time per message, the rate limit and privacy.
A tiny worked example: from tokens and seconds to dollars
Section titled “A tiny worked example: from tokens and seconds to dollars”The price table of the course’s recordings is dated, like any price. Chat-small costs US$0.10 per million input tokens and US$0.50 per million output tokens. Chat-strong costs US$2.00 and US$10.00. Prices change often: check the provider’s current price page before you use them. For local models, the course assumes US$0.05 per CPU-hour.
| Model | Mean tokens in | Mean tokens out | Calculation for 1,000 messages | Running cost per 1,000 |
|---|---|---|---|---|
| Hosted LLM, route names only | 186 | 33 | 1,000 × (186 × 0.10 + 33 × 0.50) ÷ 1,000,000 | US$0.035 |
| Hosted LLM, few-shot | 717 | 31.8 | 1,000 × (717 × 0.10 + 31.8 × 0.50) ÷ 1,000,000 | US$0.088 |
| Reasoning model (hard messages) | 732.9 | 47.2 | 1,000 × (732.9 × 2.00 + 47.2 × 10.00) ÷ 1,000,000 | US$1.94 |
| Local Qwen2.5-1.5B, route scores | 7.75 s per message | 1,000 × 7.75 ÷ 3,600 × 0.05 | US$0.108 | |
| Fine-tuned encoder | 6.7 ms per message | 1,000 × 0.0067 ÷ 3,600 × 0.05 | US$0.0001 |
The reasoning model’s output tokens include its hidden reasoning tokens, 13 on average here. The local model’s cost depends entirely on the assumed price of computer time. On a computer that Larkfield already owns and leaves idle, it is close to zero.
Larkfield: the whole bill
Section titled “Larkfield: the whole bill”Now add the mistakes. Here is the bill of the hosted LLM with examples, on the 2,000 test messages:
| Part of the bill | Messages | Cost each | Per 1,000 messages |
|---|---|---|---|
| Running cost (tokens) | 2,000 calls | about US$0.00009 | US$0.088 |
| Team message sent to a person | 22 | US$2 | US$22.0 |
| Wrong queue | 37 | US$4 | US$74.0 |
| Urgent message in the wrong queue | 2 | US$20 | US$20.0 |
| Total | US$116.1 |
The running cost is less than 0.1% of the total. At Larkfield’s volume, about 23,725 messages a year as in What the extra accuracy costs, the tokens cost about US$2 a year. The mistakes cost about US$2,750 a year. On this inbox, the model’s running cost is tiny next to the cost of its mistakes.
Every row so far: cost against accuracy
Section titled “Every row so far: cost against accuracy”Here is the comparison table up to this module, as one chart. Each dot is one method on the test messages. Its accuracy is across, and its total cost per 1,000 messages is up, on a log scale: each step is 10 times more. Before you look, predict: which LLM rows will be cheaper than everything to a person?
Text description of the diagram
The chart places each method by its accuracy, across, and its total cost per 1,000 test messages, up. The cost scale is a log scale from 50 to 10,000 dollars. A red dashed line at 1,500 dollars marks everything to a person. At the bottom right, below the line, are the best methods. They are the fine-tuned encoder (0.980, US$103.0) and the hosted LLM with examples (0.970, US$116.1). TF-IDF with logistic regression is next (0.962, US$296.0). The other trained models are grey dots near them. The hosted LLM with the route names only is at 0.765 and US$1,049.0, just below the line. All the local LLM rows are above the line. Qwen2.5-1.5B is at 0.734 and US$3,232.1 with route scores. With a JSON answer, it is at 0.368 and US$5,223.1. Both are on a 500-message sample. Qwen3-0.6B is at 0.194 and US$3,410.0 with route scores. With a JSON answer, it is at 0.282 and US$7,198.3. The rules-or-a-person baseline and the two zero-shot models without an LLM are also above the line, as grey dots.
Two hosted LLM rows are below the line: few-shot and, just, zero-shot. No local LLM is. The cheapest row is still the fine-tuned encoder, which runs on Larkfield’s own computer. The reasoning model has no row: it ran only on the 130 hard messages.
Time per message
Section titled “Time per message”Money is not the only cost. A customer on chat waits for every answer, and a slow model needs more machines at a busy hour.
Text description of the diagram
One dot per method, on a log scale where each step is 10 times slower. TF-IDF with logistic regression takes 0.16 milliseconds per message. The fine-tuned encoder takes 6.7 ms, and GLiClass 21 ms. Qwen3-0.6B with route scores takes 453 ms. The hosted LLM takes 1.2 seconds with the route names only and 1.7 seconds with examples. The reasoning model takes 2.0 seconds on the hard cases. Qwen3-0.6B writing JSON takes 2.1 seconds. Qwen2.5-1.5B takes 5.2 seconds writing JSON and 7.7 seconds with route scores, both measured on a 500-message sample.
Every LLM is hundreds to thousands of times slower than the trained classifiers. The hosted LLM with examples takes about 250 times longer than the fine-tuned encoder, and about 10,000 times longer than TF-IDF. The hosted times also include the network, and they vary: the slowest 10% of the few-shot calls took 5.3 seconds or more.
Rate limits
Section titled “Rate limits”A hosted provider limits how much one account may send: a number of requests per minute, and a number of tokens per minute. Suppose your account may send 200,000 tokens per minute (an assumption: limits differ by provider, model and account). The few-shot prompt uses about 749 tokens per message, so:
- 200,000 ÷ 749 = about 267 messages a minute at most;
- one call at a time, at 1.71 s each: about 2,107 messages an hour.
Larkfield gets about 65 messages a day, so neither limit matters there. A shop with a sale, a strike or a product launch can get thousands in an hour. Then calls must run in parallel, wait and retry. The LLM API course teaches this in Handle provider limits.
Privacy
Section titled “Privacy”With a hosted model, the whole message leaves Larkfield: the customer’s words, the order number, sometimes an address or a name. Before a hosted model can route real messages, Larkfield needs a contract with the provider that covers this data. Its own privacy rules must also allow it. In this course, only synthetic messages were sent. The local models and the trained classifiers keep every message on Larkfield’s own computer.
Which LLM option belongs in Larkfield’s router?
Section titled “Which LLM option belongs in Larkfield’s router?”Put the numbers together, slice by slice:
| On this inbox | Accuracy | English | French | Total per 1,000 | Time | Data leaves Larkfield |
|---|---|---|---|---|---|---|
| Fine-tuned encoder | 0.980 | 0.980 | 0.978 | US$103.0 | 6.7 ms | no |
| Hosted LLM, few-shot | 0.970 | 0.973 | 0.961 | US$116.1 | 1.7 s | yes |
| Hosted LLM, route names only | 0.765 | 0.757 | 0.784 | US$1,049.0 | 1.2 s | yes |
| Local Qwen2.5-1.5B, route scores (sample) | 0.734 | 0.736 | 0.729 | US$3,232.1 | 7.7 s | no |
| Reasoning model | only on the 130 hard messages: 97 right | US$1.94 running | 2.0 s | yes |
As the only router, no LLM option belongs here. The fine-tuned encoder is more accurate in both languages, cheaper, about 250 times faster, and private. The hosted LLM with examples is close behind, and it has one strength: a written confidence that we checked (ECE 0.008). That makes it a good second step for the messages that a cheap model is unsure about. Module 6 builds that cascade, and finds the lowest total of the course: US$94.3 per 1,000.
Guided practice: one bill for every model
Section titled “Guided practice: one bill for every model”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, read the price table and its warning: prices change often.
- In section 3, run the cell. The few-shot LLM shows
116.1in all and0.088to run. - In section 4, run the cell. You see
at most 267 messages a minuteandabout 2107 messages an hour.
Your turn: tokens times prices
Section titled “Your turn: tokens times prices”In section 2 of the notebook, complete per_1000(mode, model). It returns the cost of 1,000 calls in dollars, from the mean tokens of the recorded answers and the model’s prices. Predict first: how many times more expensive is the reasoning model than the few-shot prompt?
Show the answer
def per_1000(mode, model): answers = pd.read_json(f"data/recordings/llm-{mode}.jsonl", lines=True) price = prices["models"][model] return round(1000 * (answers["tokens_in"].mean() * price["input"] + answers["tokens_out"].mean() * price["output"]) / 1e6, 4){'zero-shot': np.float64(0.0351), 'few-shot': np.float64(0.0876), 'reasoning': np.float64(1.9376)}The reasoning model costs about 22 times more than the few-shot prompt per message (1.9376 ÷ 0.0876). Both prompts have about the same input. The difference is the price per token, 20 times higher, and the extra output tokens.
Common mistake
A cost estimate that counts only tokens
Symptom: a cost review shows “the hosted LLM: US$0.088 per 1,000 messages; the fine-tuned encoder: US$0.0001”. Someone concludes that the choice does not matter: both are almost free. A month later, the support teams report more messages in the wrong queues than before.
Cause: the estimate counts only the running cost. On this inbox, the running cost is less than 0.1% of the few-shot LLM’s total. The mistakes are the real bill: US$116.0 per 1,000 for the LLM against US$103.0 for the encoder. The LLM with the route names only costs US$1,049.0. A cheaper prompt that makes more mistakes costs more.
Fix: always report the total cost per 1,000 messages, from the same test set and the same cost table. Show the running cost as one column. Add the time per message, the rate limit and where the data goes. Date the price table, because prices change.
Consequence in AI work: LLM costs are easy to see on the provider’s invoice. Mistake costs are spread over many teams and never arrive as one bill. If you do not put them in the table, nobody will.
Knowledge check
Section titled “Knowledge check”Check your understanding
Compute the running cost per 1,000 messages of the few-shot prompt from its mean tokens (717 in, 31.8 out) and chat-small's prices (US$0.10 and US$0.50 per million tokens). What share of its total of US$116.1 is it?
Show the answer
1,000 × (717 × 0.10 + 31.8 × 0.50) ÷ 1,000,000 = 1,000 × (71.7 + 15.9) ÷ 1,000,000 = US$0.0876, about US$0.088 per 1,000 messages. That is 0.088 ÷ 116.1 = about 0.08% of the total. The rest, US$116.0, is the cost of the mistakes. They are 22 team messages sent to a person, 37 wrong queues and 2 urgent messages in the wrong queue.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says that the local model is free, because there is no price per call. What does it cost on this inbox? Give three costs.
Show the answer
First, its mistakes. On the 500-message sample, Qwen2.5-1.5B with route scores costs US$3,232.1 per 1,000 messages: more than everything to a person. Second, computer time: 7.7 s per message. At the assumed US$0.05 per CPU-hour, that is about US$0.108 per 1,000, more than the hosted LLM’s tokens. Third, memory and speed: about 3.8 GB while it runs, and only one message every few seconds per machine. Its real advantage is privacy, which the fine-tuned encoder also gives.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Which LLM option, if any, belongs in Larkfield's router? Answer with accuracy by language and cost.
Show the answer
None as the only router. The fine-tuned encoder scores 0.980 in English and 0.978 in French and costs US$103.0 per 1,000 messages. It takes 6.7 ms and keeps the data at Larkfield. The best LLM option, the hosted LLM with examples, scores 0.973 and 0.961 and costs US$116.1. It takes 1.7 s per message and sends every message to a provider. But its written confidence was checked (ECE 0.008). So it can be the second step of a cascade for unsure messages, which Module 6 measures at US$94.3 per 1,000.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The cost per 1,000 messages = running cost + mistake cost. Running cost comes from tokens × a dated price table, or seconds × a price per CPU-hour.
- Running costs per 1,000 here: hosted route names only US$0.035, few-shot US$0.088 and reasoning US$1.94. Local Qwen2.5-1.5B: about US$0.108 (assumed CPU price).
- On this inbox, the running cost is less than 0.1% of the few-shot LLM’s total of US$116.1. The mistakes are the bill.
- Time: every LLM is hundreds to thousands of times slower than the trained classifiers. Rate limits cap the messages per minute. Privacy: a hosted model gets every message.
- No local LLM is cheaper than everything to a person. As the only router, no LLM option beats the fine-tuned encoder (0.980, US$103.0).
- The hosted LLM with examples is a good second step for unsure messages.
Module 6 starts from what a router really needs from a model: a decision and a probability, not a sentence.