Skip to content
Français

Cost per 1,000 messages

  • Lesson 23
  • 30 min
  • Module 5: LLMs as classifiers

Where we are

The comparison table now has hosted, local and reasoning models next to the local classifiers. Their costs are in different units: tokens, seconds and memory. In this lesson, we turn all of them into one cost per 1,000 messages, with a dated price table.

Lesson 23 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceCost per 1,000 messages

What you will be able to do

Put every model on one bill: the cost per 1,000 messages. It comes from tokens and a dated price table, or from seconds of computer time, plus the cost of the mistakes. Then add the costs that are not in dollars: the time per message, the rate limit and the privacy of the messages. This is how you choose an LLM option, or none.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook reads the price table and computes the cost per 1,000 messages from the recorded tokens of three hosted prompts. It puts the running cost next to the cost of the mistakes. Then it checks what a rate limit and the time per message mean for a whole inbox. You need no account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing for the notebook route. This module changes no file of your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter. Colab, Kaggle and Windows were not tested.

Larkfield’s post office asks three companies to sort its letters. The first quotes a price per sheet of paper read. The second quotes a price per hour of a clerk’s time. The third rents a machine and charges nothing per letter. The quotes cannot be compared as they are.

The manager turns each one into the same unit: dollars per 1,000 letters. Then she adds what each company’s mistakes would cost, how long each letter would wait, and whether the letters leave the building. Only then does she choose.

This module added three kinds of LLM to the comparison table, and each comes with its own unit:

  • the hosted LLM charges per million tokens, a different price for input and output;
  • the reasoning model charges per token too, at a higher price, and adds hidden reasoning tokens;
  • the local LLM has no price per call, but uses seconds of a computer and gigabytes of memory.

Tomás compares the price per token only. “The small hosted model costs 10 cents per million input tokens, so it is almost free. The local model is free.” Both sentences skip most of the bill. We need one number for every row, and then the costs that no number holds.

The cost per 1,000 messages has two parts:

  1. the running cost: what the model costs to run, from tokens and a price table, or from seconds and a price per CPU-hour;
  2. the mistake cost: what its wrong routes cost, from the cost table.

Their sum is the column total of the comparison table. Then we look at three costs outside the number: the time per message, the rate limit and privacy.

A tiny worked example: from tokens and seconds to dollars

Section titled “A tiny worked example: from tokens and seconds to dollars”

The price table of the course’s recordings is dated, like any price. Chat-small costs US$0.10 per million input tokens and US$0.50 per million output tokens. Chat-strong costs US$2.00 and US$10.00. Prices change often: check the provider’s current price page before you use them. For local models, the course assumes US$0.05 per CPU-hour.

Model Mean tokens in Mean tokens out Calculation for 1,000 messages Running cost per 1,000
Hosted LLM, route names only 186 33 1,000 × (186 × 0.10 + 33 × 0.50) ÷ 1,000,000 US$0.035
Hosted LLM, few-shot 717 31.8 1,000 × (717 × 0.10 + 31.8 × 0.50) ÷ 1,000,000 US$0.088
Reasoning model (hard messages) 732.9 47.2 1,000 × (732.9 × 2.00 + 47.2 × 10.00) ÷ 1,000,000 US$1.94
Local Qwen2.5-1.5B, route scores 7.75 s per message 1,000 × 7.75 ÷ 3,600 × 0.05 US$0.108
Fine-tuned encoder 6.7 ms per message 1,000 × 0.0067 ÷ 3,600 × 0.05 US$0.0001

The reasoning model’s output tokens include its hidden reasoning tokens, 13 on average here. The local model’s cost depends entirely on the assumed price of computer time. On a computer that Larkfield already owns and leaves idle, it is close to zero.

Now add the mistakes. Here is the bill of the hosted LLM with examples, on the 2,000 test messages:

Part of the bill Messages Cost each Per 1,000 messages
Running cost (tokens) 2,000 calls about US$0.00009 US$0.088
Team message sent to a person 22 US$2 US$22.0
Wrong queue 37 US$4 US$74.0
Urgent message in the wrong queue 2 US$20 US$20.0
Total US$116.1

The running cost is less than 0.1% of the total. At Larkfield’s volume, about 23,725 messages a year as in What the extra accuracy costs, the tokens cost about US$2 a year. The mistakes cost about US$2,750 a year. On this inbox, the model’s running cost is tiny next to the cost of its mistakes.

Here is the comparison table up to this module, as one chart. Each dot is one method on the test messages. Its accuracy is across, and its total cost per 1,000 messages is up, on a log scale: each step is 10 times more. Before you look, predict: which LLM rows will be cheaper than everything to a person?

A scatter chart. Horizontal axis: accuracy on the test messages, from 0.1 to 1.0. Vertical axis: cost per 1,000 test messages in US dollars, log scale from 50 to 10,000. A red dashed line at 1,500 marks everything to a person. Below the line, at the bottom right: the fine-tuned encoder at 0.980 and 103, the hosted LLM few-shot at 0.970 and 116.1, TF-IDF with logistic regression at 0.962 and 296, and grey dots for the other trained models. The hosted LLM zero-shot is at 0.765 and 1,049, just below the line. Above the line, in purple: Qwen2.5-1.5B route scores at 0.734 and 3,232.1 and JSON at 0.368 and 5,223.1, both hollow because they were measured on a 500-message sample; Qwen3-0.6B route scores at 0.194 and 3,410, and JSON at 0.282 and 7,198.3. Grey dots for rules or a person and the two zero-shot models without an LLM are also above the line.
Every row of the comparison table up to this module: accuracy against the total cost per 1,000 test messages (log scale). Hosted LLM rows recorded; the others measured. Hollow dots: a 500-message sample.
Text description of the diagram

The chart places each method by its accuracy, across, and its total cost per 1,000 test messages, up. The cost scale is a log scale from 50 to 10,000 dollars. A red dashed line at 1,500 dollars marks everything to a person. At the bottom right, below the line, are the best methods. They are the fine-tuned encoder (0.980, US$103.0) and the hosted LLM with examples (0.970, US$116.1). TF-IDF with logistic regression is next (0.962, US$296.0). The other trained models are grey dots near them. The hosted LLM with the route names only is at 0.765 and US$1,049.0, just below the line. All the local LLM rows are above the line. Qwen2.5-1.5B is at 0.734 and US$3,232.1 with route scores. With a JSON answer, it is at 0.368 and US$5,223.1. Both are on a 500-message sample. Qwen3-0.6B is at 0.194 and US$3,410.0 with route scores. With a JSON answer, it is at 0.282 and US$7,198.3. The rules-or-a-person baseline and the two zero-shot models without an LLM are also above the line, as grey dots.

Two hosted LLM rows are below the line: few-shot and, just, zero-shot. No local LLM is. The cheapest row is still the fine-tuned encoder, which runs on Larkfield’s own computer. The reasoning model has no row: it ran only on the 130 hard messages.

Money is not the only cost. A customer on chat waits for every answer, and a slow model needs more machines at a busy hour.

A dot chart of time per message, one at a time, on a log scale from 0.1 ms to 10 s. TF-IDF with logistic regression 0.16 ms; the fine-tuned encoder 6.7 ms; GLiClass zero-shot 21 ms; Qwen3-0.6B route scores 453 ms; the hosted LLM zero-shot 1.2 s; the hosted LLM few-shot 1.7 s; the reasoning model on the hard cases 2.0 s; Qwen3-0.6B JSON 2.1 s; Qwen2.5-1.5B JSON 5.2 s and route scores 7.7 s, both on a 500-message sample.
Time per message, one at a time, on a log scale (local models measured on a laptop CPU; hosted models recorded from the provider, so they include the network).
Text description of the diagram

One dot per method, on a log scale where each step is 10 times slower. TF-IDF with logistic regression takes 0.16 milliseconds per message. The fine-tuned encoder takes 6.7 ms, and GLiClass 21 ms. Qwen3-0.6B with route scores takes 453 ms. The hosted LLM takes 1.2 seconds with the route names only and 1.7 seconds with examples. The reasoning model takes 2.0 seconds on the hard cases. Qwen3-0.6B writing JSON takes 2.1 seconds. Qwen2.5-1.5B takes 5.2 seconds writing JSON and 7.7 seconds with route scores, both measured on a 500-message sample.

Every LLM is hundreds to thousands of times slower than the trained classifiers. The hosted LLM with examples takes about 250 times longer than the fine-tuned encoder, and about 10,000 times longer than TF-IDF. The hosted times also include the network, and they vary: the slowest 10% of the few-shot calls took 5.3 seconds or more.

A hosted provider limits how much one account may send: a number of requests per minute, and a number of tokens per minute. Suppose your account may send 200,000 tokens per minute (an assumption: limits differ by provider, model and account). The few-shot prompt uses about 749 tokens per message, so:

  • 200,000 ÷ 749 = about 267 messages a minute at most;
  • one call at a time, at 1.71 s each: about 2,107 messages an hour.

Larkfield gets about 65 messages a day, so neither limit matters there. A shop with a sale, a strike or a product launch can get thousands in an hour. Then calls must run in parallel, wait and retry. The LLM API course teaches this in Handle provider limits.

With a hosted model, the whole message leaves Larkfield: the customer’s words, the order number, sometimes an address or a name. Before a hosted model can route real messages, Larkfield needs a contract with the provider that covers this data. Its own privacy rules must also allow it. In this course, only synthetic messages were sent. The local models and the trained classifiers keep every message on Larkfield’s own computer.

Which LLM option belongs in Larkfield’s router?

Section titled “Which LLM option belongs in Larkfield’s router?”

Put the numbers together, slice by slice:

On this inbox Accuracy English French Total per 1,000 Time Data leaves Larkfield
Fine-tuned encoder 0.980 0.980 0.978 US$103.0 6.7 ms no
Hosted LLM, few-shot 0.970 0.973 0.961 US$116.1 1.7 s yes
Hosted LLM, route names only 0.765 0.757 0.784 US$1,049.0 1.2 s yes
Local Qwen2.5-1.5B, route scores (sample) 0.734 0.736 0.729 US$3,232.1 7.7 s no
Reasoning model only on the 130 hard messages: 97 right US$1.94 running 2.0 s yes

As the only router, no LLM option belongs here. The fine-tuned encoder is more accurate in both languages, cheaper, about 250 times faster, and private. The hosted LLM with examples is close behind, and it has one strength: a written confidence that we checked (ECE 0.008). That makes it a good second step for the messages that a cheap model is unsure about. Module 6 builds that cascade, and finds the lowest total of the course: US$94.3 per 1,000.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, read the price table and its warning: prices change often.
  4. In section 3, run the cell. The few-shot LLM shows 116.1 in all and 0.088 to run.
  5. In section 4, run the cell. You see at most 267 messages a minute and about 2107 messages an hour.

In section 2 of the notebook, complete per_1000(mode, model). It returns the cost of 1,000 calls in dollars, from the mean tokens of the recorded answers and the model’s prices. Predict first: how many times more expensive is the reasoning model than the few-shot prompt?

Show the answer
def per_1000(mode, model):
answers = pd.read_json(f"data/recordings/llm-{mode}.jsonl", lines=True)
price = prices["models"][model]
return round(1000 * (answers["tokens_in"].mean() * price["input"]
+ answers["tokens_out"].mean() * price["output"]) / 1e6, 4)
{'zero-shot': np.float64(0.0351), 'few-shot': np.float64(0.0876), 'reasoning': np.float64(1.9376)}

The reasoning model costs about 22 times more than the few-shot prompt per message (1.9376 ÷ 0.0876). Both prompts have about the same input. The difference is the price per token, 20 times higher, and the extra output tokens.

Common mistake

A cost estimate that counts only tokens

Symptom: a cost review shows “the hosted LLM: US$0.088 per 1,000 messages; the fine-tuned encoder: US$0.0001”. Someone concludes that the choice does not matter: both are almost free. A month later, the support teams report more messages in the wrong queues than before.

Cause: the estimate counts only the running cost. On this inbox, the running cost is less than 0.1% of the few-shot LLM’s total. The mistakes are the real bill: US$116.0 per 1,000 for the LLM against US$103.0 for the encoder. The LLM with the route names only costs US$1,049.0. A cheaper prompt that makes more mistakes costs more.

Fix: always report the total cost per 1,000 messages, from the same test set and the same cost table. Show the running cost as one column. Add the time per message, the rate limit and where the data goes. Date the price table, because prices change.

Consequence in AI work: LLM costs are easy to see on the provider’s invoice. Mistake costs are spread over many teams and never arrive as one bill. If you do not put them in the table, nobody will.

Check your understanding

Compute the running cost per 1,000 messages of the few-shot prompt from its mean tokens (717 in, 31.8 out) and chat-small's prices (US$0.10 and US$0.50 per million tokens). What share of its total of US$116.1 is it?

Show the answer

1,000 × (717 × 0.10 + 31.8 × 0.50) ÷ 1,000,000 = 1,000 × (71.7 + 15.9) ÷ 1,000,000 = US$0.0876, about US$0.088 per 1,000 messages. That is 0.088 ÷ 116.1 = about 0.08% of the total. The rest, US$116.0, is the cost of the mistakes. They are 22 team messages sent to a person, 37 wrong queues and 2 urgent messages in the wrong queue.

Check your understanding

Tomás says that the local model is free, because there is no price per call. What does it cost on this inbox? Give three costs.

Show the answer

First, its mistakes. On the 500-message sample, Qwen2.5-1.5B with route scores costs US$3,232.1 per 1,000 messages: more than everything to a person. Second, computer time: 7.7 s per message. At the assumed US$0.05 per CPU-hour, that is about US$0.108 per 1,000, more than the hosted LLM’s tokens. Third, memory and speed: about 3.8 GB while it runs, and only one message every few seconds per machine. Its real advantage is privacy, which the fine-tuned encoder also gives.

Check your understanding

Which LLM option, if any, belongs in Larkfield's router? Answer with accuracy by language and cost.

Show the answer

None as the only router. The fine-tuned encoder scores 0.980 in English and 0.978 in French and costs US$103.0 per 1,000 messages. It takes 6.7 ms and keeps the data at Larkfield. The best LLM option, the hosted LLM with examples, scores 0.973 and 0.961 and costs US$116.1. It takes 1.7 s per message and sends every message to a provider. But its written confidence was checked (ECE 0.008). So it can be the second step of a cascade for unsure messages, which Module 6 measures at US$94.3 per 1,000.

Recap

  • The cost per 1,000 messages = running cost + mistake cost. Running cost comes from tokens × a dated price table, or seconds × a price per CPU-hour.
  • Running costs per 1,000 here: hosted route names only US$0.035, few-shot US$0.088 and reasoning US$1.94. Local Qwen2.5-1.5B: about US$0.108 (assumed CPU price).
  • On this inbox, the running cost is less than 0.1% of the few-shot LLM’s total of US$116.1. The mistakes are the bill.
  • Time: every LLM is hundreds to thousands of times slower than the trained classifiers. Rate limits cap the messages per minute. Privacy: a hosted model gets every message.
  • No local LLM is cheaper than everything to a person. As the only router, no LLM option beats the fine-tuned encoder (0.980, US$103.0).
  • The hosted LLM with examples is a good second step for unsure messages.

Module 6 starts from what a router really needs from a model: a decision and a probability, not a sentence.

Next: Ask for a decision, not for text