Skip to content
Français

What the extra accuracy costs

  • Lesson 18
  • 30 min
  • Module 4: Embeddings and small fine-tuned models

Where we are

The comparison table now has rules, TF-IDF, embeddings, a fine-tuned encoder and two zero-shot models. Some are more accurate, and all of them cost something. In this lesson, we put training time, speed, size and retraining work next to the gain on the costed measure.

Lesson 18 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceWhat the extra accuracy costs

What you will be able to do

Choose a model the way the business pays for it. A more accurate model saves money on mistakes, and it costs training time, speed, memory and work every time the routes change. Put both sides in one place, in dollars where you can, before you choose.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing. This lesson has no notebook: it uses the numbers that you measured in the earlier lessons.

The post office manager can buy a new sorting machine. The salesman says that it makes one mistake in 50 letters, against one in 26 for the old machine. That sounds like a clear win.

The manager asks other questions. How much does a mistake cost us? How many letters do we sort each day? How long does the machine take per letter? How big is it? And when the post office adds a new kind of mail, who teaches the machine, and how long does that take? The salesman’s number is real, but it is only one line of the bill.

The problem: accuracy is one column of many

Section titled “The problem: accuracy is one column of many”

Tomás has a simple rule: “Use the most accurate model.” On this inbox, that is the fine-tuned encoder, at 0.980. Grace asks the manager’s questions. TF-IDF with logistic regression reaches 0.962, trains in seconds and needs no neural model. Is 1.8 points of accuracy worth a 470 MB model, a training job and a new skill on the team?

Accuracy alone cannot answer, because mistakes do not all cost the same, and because the model has costs that are not mistakes. We need the whole bill.

The comparison table already holds one part of the bill: the total cost per 1,000 messages. It adds the cost of the mistakes, from the cost table, and the running cost. The course assumes US$0.05 per CPU-hour. At that price, every local model of this module costs less than a cent per 1,000 messages to run, except the slow NLI model (about 1.2 cents). Cost per 1,000 messages in Module 5 shows how to compute the running cost of every kind of model, hosted ones too.

The other part comes back each time the model changes:

  • training time, once per version;
  • size: disk, memory, download time;
  • speed: the time per message, one at a time, which decides if the service answers fast enough;
  • the work of retraining: what a person must do when a new route or new labels arrive, and the skills that it needs;
  • explaining: can you show Grace why a message went to a team?

These costs per model need a few lines of judgement next to the number.

A tiny example: from per 1,000 to per year

Section titled “A tiny example: from per 1,000 to per year”

Larkfield’s inbox gets about 65 messages a day: 10,000 messages in 153 days. That is 65 × 365 = 23,725 messages a year, or 23.725 thousands. To turn a cost per 1,000 messages into a cost per year, multiply by 23.725:

Method Cost per 1,000 × 23.725 Mistakes and reviews per year
Everything to a person US$1,500.0 1,500.0 × 23.725 US$35,588
TF-IDF + logistic regression US$296.0 296.0 × 23.725 US$7,023
TF-IDF + linear SVM US$175.0 175.0 × 23.725 US$4,152
Fine-tuned encoder US$103.0 103.0 × 23.725 US$2,444
  • Everything to a person
  • TF-IDF + logistic regression (LR) or linear SVM
  • Fine-tuned encoder
The cost per 1,000 test messages, turned into a cost per year at about 65 messages a day (23,725 a year). The costs are the course's assumptions; the method is what matters.
Text description of the chart

A column chart with four columns. Everything to a person costs US$35,588 a year. TF-IDF with logistic regression costs US$7,023, TF-IDF with a linear SVM US$4,152 and the fine-tuned encoder US$2,444. Every model saves most of the cost of reading everything; the differences between the models are a few thousand dollars a year.

The fine-tuned encoder saves about US$4,579 a year against TF-IDF with logistic regression, and about US$1,708 against the linear SVM. Those are the numbers to compare with the extra work. They rest on two assumptions: the course’s cost table and the volume of the test months. With your own costs and volume, the same arithmetic gives your numbers.

Here is the comparison table as a picture, with every method up to this module. The cost axis is logarithmic: each step is a multiple (100, 1,000, 10,000), so that US$103 and US$6,570 fit on one chart.

A scatter chart of cost per 1,000 test messages, on a log scale from 50 to 10,000, against accuracy from 0.3 to 1.0. A dashed red line at 1,500 marks everything to a person. Below the line on the right: the fine-tuned encoder at 0.980 and 103, the TF-IDF models between 0.932 and 0.966 and 175 to 443, and the two embedding models at 0.920 and 779 and 0.906 and 895. Above the line: rules or a person at 0.555 and 1,757.8, GLiClass zero-shot at 0.516 and 4,431, and NLI zero-shot at 0.329 and 6,570.0.
Cost against accuracy for every method up to this module, on the 2,000 test messages. Lower and further right is better. The fine-tuned encoder is the best point; the zero-shot models and the rules cost more than a person.
Text description of the diagram

The horizontal axis is the accuracy on the test messages, from 0.3 to 1.0. The vertical axis is the cost per 1,000 test messages in US dollars, on a logarithmic scale from 50 to 10,000. A dashed red line at 1,500 marks everything to a person. Below the line, at the right: the fine-tuned encoder at accuracy 0.980 and cost 103, the lowest point. TF-IDF with a linear SVM at 0.966 and 175, TF-IDF with logistic regression at 0.962 and 296, TF-IDF with Naive Bayes at 0.943 and 441, and strong rules with logistic regression at 0.932 and 443. The older embedding model with logistic regression at 0.920 and 779, the newer one at 0.906 and 895. Above the line: rules or a person at 0.555 and 1,757.8, GLiClass zero-shot at 0.516 and 4,431, and NLI zero-shot at 0.329 and 6,570.

The picture has three groups:

  • Above the red line: methods that cost more than a person who reads everything. The rules, and both zero-shot models. They are not routers on this inbox.
  • The frozen embeddings, between US$779 and US$895: better than a person, worse than every TF-IDF model.
  • The bottom right corner: the TF-IDF models and the fine-tuned encoder. Here, small steps in accuracy are large steps in cost, because the last mistakes include the expensive ones.

For the three serious candidates, here are the costs per model:

TF-IDF + logistic regression Frozen embeddings + LR Fine-tuned encoder
Accuracy (test) 0.962 0.920 0.980
Cost per 1,000 (test) US$296.0 US$779.0 US$103.0
Cost per 1,000 (validation) US$472.0 US$955.0 US$204.0
Training time about 3 s about 1 minute to embed, then under 1 s 327 s (3 epochs)
Time per message, one at a time about 0.1 ms 6.8 ms 6.7 ms
Size on disk about 2.1 MB 470.6 MB 470.6 MB + 43.3 MB
A new route arrives add labels, retrain in seconds add labels, retrain in seconds add labels, retrain for about 6 minutes, check the loss, version the checkpoint
Why this route? the top words (route explain) hard to explain hard to explain
Skills on the team scikit-learn sentence-transformers PyTorch and Transformers

Read it column by column:

  • TF-IDF + logistic regression is the cheapest to own: tiny, instant, easy to retrain and to explain. Its mistakes cost almost three times as much as the encoder’s.
  • Frozen embeddings lose on this inbox: more mistakes, a large model, nothing to gain. They earn their place with few labels (Few labels with embeddings), not here.
  • The fine-tuned encoder wins on mistakes on test and on validation (US$204.0 against US$472.0), so the gain is not one lucky test set. It costs a 6-minute training job, a model about 250 times larger than TF-IDF, a slower answer (still only 7 ms), and skills that the team must keep.

Speed is not a problem at 65 messages a day: even 7 ms per message is less than half a second of CPU a day. It would matter for a service that must answer thousands of messages a second.

The fine-tuned encoder saves about US$4,579 a year of mistakes against TF-IDF with logistic regression, at the course’s costs. On the other side, someone needs a few hours once, to set up the training script, the checkpoint and the tests. Then each time the labels change, it takes about 6 minutes of CPU and a check of the scores. At most companies, that work costs less than the saving. At a company with no one who knows PyTorch, it may not.

Two warnings before you decide:

  1. This inbox is easier than a real one. On real messages, every method scores lower, and the gaps can change. The case studies Route banking questions and Find what the bot cannot handle let you check the methods on real data.
  2. The test set is small for rare, expensive messages. It has 12 account takeovers. One takeover more or less moves the cost by US$100 per 1,000. Compare on validation too, as in the table above.
  • Pay when the saving on mistakes, at your costs and your volume, is clearly larger than the work it adds, on validation and on test.
  • Do not pay for a gain that only appears on one data set, or that comes from one or two expensive messages.
  • Do not pay for accuracy that the business does not feel: if every message also goes to a person, a 1-point gain changes nothing.
  • Remember the people: a model that only one person can retrain is a risk, whatever its accuracy.

Grace asks Priya for one recommendation for Larkfield’s inbox, among three: TF-IDF with logistic regression, the frozen embeddings with logistic regression, and the fine-tuned encoder. Write your recommendation in four or five sentences. Give the evidence (accuracy, cost per 1,000 on test and validation, takeovers found) and what it will cost to run and to retrain. Predict first: which one would Grace’s team find easiest to live with?

Show a model answer

Recommend the fine-tuned encoder. On this inbox, it makes 41 mistakes in 2,000 test messages against 76 for TF-IDF and 159 for the frozen embeddings, and it finds all 12 account takeovers. It costs US$103.0 per 1,000 on test against US$296.0 for TF-IDF, and US$204.0 against US$472.0 on validation, so the gain holds on both sets. At about 65 messages a day, that is about US$4,579 a year of mistakes saved, at the course’s costs. It costs a 6-minute training job on a laptop CPU for each new version, 514 MB on disk, about 7 ms per message, and someone who can run PyTorch. Keep TF-IDF + logistic regression as the fallback and as the explanation tool (route explain), and drop the frozen embeddings for this inbox.

Another sound answer is TF-IDF with logistic regression, if no one on the team can maintain a neural model: it costs more in mistakes and much less in work. Either way, the answer names the evidence and the cost.

Common mistake

The most accurate model, with no plan to retrain it

Symptom: a constructed story. Tomás fine-tunes the encoder on his laptop, copies the weights to the server, and the router gets better at once. Three months later, the Grow Hub launches, and a new route is needed. Nobody can retrain the model: the training code was in Tomás’s notebook, he is on leave, and the base model’s revision was never written down.

Cause: the choice looked only at the accuracy and the cost of mistakes. The cost of the next version, the scripts, the pinned versions, the checks and the people who can run them, was never counted.

Fix: before you choose a model, write down how to retrain it. Name the script, the data version and the pinned base model. Write the expected validation loss and scores, and who runs it. Count that work in the decision. Module 9 builds this into the project: route train with a model version and a label version, and route release that compares a new version with the old one.

Consequence in AI work: a model is not a one-time purchase. Routes change, products launch and labels improve. The model that wins today must be easy to rebuild tomorrow, or its accuracy decays while nobody can fix it.

Check your understanding

Convert the fine-tuned encoder's US$103.0 per 1,000 messages and TF-IDF's US$296.0 into a cost per year for Larkfield. Show the arithmetic, and say what the result depends on.

Show the answer

Larkfield gets about 65 messages a day, so 65 × 365 = 23,725 a year, or 23.725 thousands. The encoder: 103.0 × 23.725 ≈ US$2,444 a year. TF-IDF: 296.0 × 23.725 ≈ US$7,023 a year. The difference is about US$4,579 a year. It depends on the course’s cost table (assumptions) and on the volume. It also assumes that the test set is like the future messages, and the test set has only 12 account takeovers.

Check your understanding

GLiClass zero-shot reaches 0.516 and needs no labels and no training. Why does it sit above the red line in the chart, and is there any place for it in Larkfield's router?

Show the answer

Its mistakes cost US$4,431.0 per 1,000 messages, about three times more than everything to a person (US$1,500.0). Half the routes are wrong, and most needs_person messages go to a team. As the router, it has no place. It can help on the first day of a new route: it lists likely Grow Hub messages for Chloé to label first. It found 31 of 80, and 84% of its device answers were right.

Check your understanding

Name three costs of the fine-tuned encoder that the total cost per 1,000 messages does not show, with the numbers of this module.

Show the answer

Training: 327 seconds of CPU for each new version, plus checking the validation loss and the scores. Size: 470.6 MB for the base model plus 43.3 MB of changed weights, against about 2.1 MB for TF-IDF. Speed: 6.7 ms per message, against about 0.1 ms. Also skills (PyTorch and Transformers) and explanation: it has no top words to show Grace, while TF-IDF has route explain.

Recap

  • Judge a model on the whole bill: the cost of mistakes and the running cost per message, then training time, size, speed, retraining work, explanation and skills per model.
  • Turn costs per 1,000 into costs per year with your volume: at about 65 messages a day, multiply by 23.725. The encoder saves about US$4,579 a year against TF-IDF + logistic regression, at the course’s costs.
  • On this inbox, the fine-tuned encoder is the best row so far: 0.980, US$103.0 per 1,000 on test and US$204.0 on validation. It costs a 6-minute training job, 514 MB and a PyTorch skill.
  • The rules and both zero-shot models cost more than a person who reads everything. The frozen embeddings lose to TF-IDF with 6,000 labels.
  • This inbox is easier than a real one, and its test set has only 12 account takeovers. Check on validation and on real data before you promise a saving.

Every model so far was local and trained or tuned by us. Next, we ask a hosted LLM to route the messages from the route names, and we see what an LLM adds and what it costs.

Next: Zero-shot classification with an LLM