The cost of each mistake
- Lesson 3
- 35 min
- Module 1: The routing problem and its cost
Where we are
We have a label scheme: six teams and "needs a person". Not every mistake costs the same: a missed account takeover costs far more than an extra review. In this lesson, we put a cost in each cell of the confusion matrix and get the one measure that the course uses to choose a method.
Lesson 3 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceThe cost of each mistake
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Give each kind of routing mistake its own cost, and add the costs into one number: the cost per 1,000 messages. Accuracy counts every mistake as one, and so it can choose a router that costs more.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s seven routes, the topics and
needs_person, from Design the labels. - A confusion matrix counts, for each true label, how often a model predicted each label. Classification metrics in the machine learning course explains it with two labels. This lesson uses seven.
- A threshold turns a probability into a decision, from The precision–recall trade-off. This lesson explains the one idea that it needs.
What you need on your computer
Nothing. The explorer runs in this page. You run the same numbers yourself in the next lesson.
Two letters that arrive late
Section titled “Two letters that arrive late”Back at the post office. One morning, two letters go into the wrong tray. The first is a birthday card. It arrives one day late, and nobody is much worse off. The second is a letter from a hospital with a date for an operation. It arrives one day late, and the patient misses the date.
Both are “one letter in the wrong tray”. If the post office counts only its mistakes, the two days look the same. But the post office’s real loss is very different. A good manager does not ask “how many letters went wrong?”. She asks “what did our mistakes cost?”. She also knows that some letters should go to the desk of a person, even if that is slower. A mistake on them costs too much.
The problem: accuracy counts every mistake as one
Section titled “The problem: accuracy counts every mistake as one”Accuracy is the share of messages that a router sends to the right route. It is easy to read, and it treats every mistake the same way. Here is a small practice problem we made on purpose: ten messages, and three routers.
| Messages | Router A | Router B | Router C |
|---|---|---|---|
| 4 delivery questions | all right | 1 to orders |
all right |
| 3 returns questions | all right | 1 to product |
all right |
| 2 billing questions | all right | all right | 1 to review |
| 1 account takeover (urgent) | to billing |
right | to review |
| Accuracy | 9 of 10 | 8 of 10 | 8 of 8 decided |
| Cost of the mistakes | US$200 | US$8 | US$3 |
Router A is the most accurate, and the most expensive by far. Its one mistake is an account takeover in the billing queue. While the message waits there, someone else uses the customer’s account. Router B makes two cheap mistakes: two normal messages in the wrong queue, at US$4 each. Router C sends the two messages that it is unsure about to a person, at US$1.50 each, and makes no mistake.
If you choose by accuracy, you choose Router A. If you choose by cost, you choose Router C. This course chooses by cost.
Larkfield’s cost table
Section titled “Larkfield’s cost table”To count costs, Larkfield needs one number for each kind of mistake. Grace and Priya agreed on these values. They are assumptions for the course, with a reason for each, not measurements. You can change them in the project’s config/costs.json, and you do so in the next lesson.
| What happens | Cost (US$) | Why |
|---|---|---|
| A person reviews the message and sends it on | 1.50 | About 1 minute of an agent at US$28 an hour, and a wait of a few hours |
| A message in the wrong team’s queue | 4 | The wrong team reads and moves it (about 2 minutes), and the customer waits about one more day |
| An urgent message in the wrong queue | 20 | One more day for an urgent message: a lost order, a chargeback, or a lost customer in some cases |
| An account takeover that misses the account team | 200 | Someone else uses the account while the message waits: orders, recovery work and the customer’s trust |
A needs_person message sent to a team |
40 | A safety problem, a legal threat or an escalated complaint gets a normal team answer: a worse complaint and a risk |
A team’s message sent to needs_person |
2 | A senior person reads a normal message and forwards it: about 4 minutes |
| The right route | 0 |
Look at the range. The cheapest mistake costs US$2 and the most expensive costs US$200: a factor of 100. So a router that is a little less accurate, but never misses an account takeover, can cost far less.
The cost of one message, step by step
Section titled “The cost of one message, step by step”For each message, the course finds its cost with the same questions, in this order. The first “yes” decides.
Text description of the diagram
The diagram is a column of six questions. Each question has an arrow labelled “yes” to a cost on the right, and an arrow labelled “no” down to the next question. First: is the predicted route the right route? If yes, the cost is US$0. Second: did the message go to review? If yes, US$1.50: a person reads it and sends it on. Third: did it go to needs_person, when a team was right? If yes, US$2: a senior person reads it and forwards it. Fourth: is it an account takeover? If yes, US$200, because someone else uses the account meanwhile. This cost is highlighted. Fifth: was the label needs_person? If yes, US$40: a team answers a safety or legal message. Sixth: is the message urgent? If yes, US$20: an urgent message waits one more day. If every answer is no, the cost is US$4. The wrong team reads and moves the message, and it waits one more day.
Try it on four messages:
| Message | Label | Predicted | Cost |
|---|---|---|---|
M-109856, the refund question of the first lesson (normal) |
returns_refunds |
product |
US$4 |
| An account takeover (urgent) | account |
billing |
US$200 |
| The same account takeover | account |
needs_person |
US$2 |
| “My son cut his hand on the swing seat” | needs_person |
product |
US$40 |
The third row may surprise you. The takeover did not reach the account team, but a senior person read it first and sent it on at once. That costs 4 minutes, not a stolen account. The order of the questions encodes this: a message that reaches a person is safe.
A confusion matrix with a cost in each cell
Section titled “A confusion matrix with a cost in each cell”A confusion matrix has one row for each label (the right route) and one column for each predicted route. The cells on the diagonal are the right answers. Every other cell is one kind of mistake. Now put a cost in each cell.
Here is a second practice problem we made on purpose: 20 messages and three routes.
| Label ↓ / predicted → | delivery | returns_refunds | account | Cost of the row |
|---|---|---|---|---|
| delivery (8) | 7 | 1 normal: US$4 | 0 | US$4 |
| returns_refunds (8) | 1 urgent: US$20 | 7 | 0 | US$20 |
| account (4) | 0 | 1 takeover: US$200 | 3 | US$200 |
The router is right on 17 of 20 messages: an accuracy of 85%. Its three mistakes cost 4 + 20 + 200 = US$224. Per 1,000 messages, that is 224 ÷ 20 × 1,000 = US$11,200. One cell, the takeover, holds almost 90% of the cost.
The one measure: cost per 1,000 messages
Section titled “The one measure: cost per 1,000 messages”The course adds up the cost of every message of the test set and gives it per 1,000 messages:
cost per 1,000 = (review cost + mistake cost) ÷ number of messages × 1,000
Later modules add one more part: the cost to run the router, such as computer time or the price of a language model’s answer. Then the total is the cost of the mistakes, the reviews and the running together. Every row of the course’s comparison table has this total. So you can compare a keyword rule with a large language model on one scale.
Send the unsure messages to a person
Section titled “Send the unsure messages to a person”A router does more than choose a route. Most models also give a probability for each route. The highest probability says how sure the model is. If it is low, the router can send the message to review: a person reads it and sends it on, for US$1.50.
This needs a threshold. If the top probability is at or above the threshold, the router decides. If it is below, the message goes to review. A low threshold decides almost everything. It saves review costs, but it keeps some expensive mistakes. A high threshold sends more messages to review, at US$1.50 each, but it catches more mistakes.
The explorer below uses a real model: TF-IDF with logistic regression. It is a classic model that learns which words go with which route, and Module 3 builds it. It was trained on Larkfield’s 6,000 training messages and scored on the 2,000 test messages, with the cost table above. Move the threshold and watch the costs.
Explore
The cost of the mistakes, and the cost of review
Predict first: The threshold starts at 0.30, where almost every message is decided. Does the cost per 1,000 messages go down if we send 10% more of the messages to review? Does your answer depend on where you start?
- Total cost
- Mistakes
- Reviews by a person
- Decided by the model
- 1,994 (99.7%)
- Sent to review
- 6 (0.3%)
- Accuracy of the decided
- 96.5%
- Review cost
- US$9
- Mistake cost
- US$570
- Total per 1,000 messages
- US$289.5
Decided messages at threshold 0.3: 1,994 decided, 6 sent to review. Each cell: the number of messages and their cost.
| Label ↓ Predicted route → | orders | delivery | returns_ | billing | product | account | needs_ | Row cost |
|---|---|---|---|---|---|---|---|---|
| orders | 275 | 4US$32 | 2US$8 | 1US$4 | 1US$4 | 2US$8 | 0 | US$56 |
| delivery | 8US$32 | 441 | 4US$16 | 2US$8 | 2US$8 | 1US$4 | 1US$2 | US$70 |
| returns_ | 3US$28 | 3US$12 | 388 | 3US$28 | 4US$16 | 2US$8 | 0 | US$92 |
| billing | 1US$4 | 2US$24 | 6US$40 | 256 | 0 | 0 | 0 | US$68 |
| product | 2US$8 | 1US$4 | 6US$40 | 0 | 278 | 1US$4 | 0 | US$56 |
| account | 3US$12 | 3US$12 | 0 | 2US$204 | 0 | 187 | 0 | US$228 |
| needs_ | 0 | 0 | 0 | 0 | 0 | 0 | 99 | US$0 |
| All decided | US$570 | |||||||
At 0.3, the model decides 1,994 of 2,000 messages and sends 6 (0.3%) to review. The reviews cost US$9 and the mistakes that remain cost US$570: US$289.5 per 1,000 messages. The lowest cost on this chart is US$144.2 per 1,000, at 0.57.
Text description of the explorer
The explorer has one slider, the decide threshold, from 0.30 to 1.00 in steps of 0.01. A line chart shows three lines against the threshold. They are the total cost per 1,000 messages, the cost of the mistakes, and the cost of the reviews. The mistakes line falls as the threshold rises. The reviews line rises, slowly at first and then steeply after 0.90. The total line falls from US$289.5 at 0.30 to its lowest point, US$144.2, at 0.57. Then it rises to US$1,500 at 1.00, where every message goes to review. Below the chart, tiles give the exact values at the chosen threshold. They show the messages decided and sent to review, the accuracy of the decided messages, the review cost, the mistake cost and the total. A table shows the confusion matrix of the decided messages: one row per label and one column per predicted route. Each cell gives the number of messages and, for a mistake, its cost; the last column gives each row’s cost. At 0.30, the account row costs US$228, mostly because of one account takeover sent to billing (US$204 in that cell).
What the explorer shows
Section titled “What the explorer shows”Start at 0.30. The model decides 1,994 messages and sends 6 to review. Its 70 mistakes cost US$570, and the total is US$289.5 per 1,000 messages. The comparison table’s row for this model says US$296.0, because there it decides every message, with no review at all. In the matrix, look at the account row. One takeover went to billing, and that cell alone costs US$204: the takeover’s US$200 plus one normal message at US$4.
Now the answer to the prediction: it depends on where you start.
- From 0.30, sending about 10% more to review means a threshold near 0.70. There, 200 messages (10.0%) go to review, and the total falls to US$181.0 per 1,000. The reviews cost more, but they remove many expensive mistakes.
- From 0.57, the lowest point (103 messages to review, US$144.2 per 1,000), another 10% means a threshold near 0.82. The total rises to US$249.2. The extra reviews now catch only a few cheap mistakes.
- At 1.00, every message goes to review: US$1,500 per 1,000, the price of Larkfield today.
Watch one jump in particular: from 0.45 to 0.46, the mistake cost falls from US$494 to US$290. The model gets one of the 12 account takeovers wrong, and sends it to billing. Its top probability is between these two values. At 0.46, it goes to review instead. In the matrix, the cell from account to billing falls from US$204 to US$4.
One caution. This explorer looks at the test messages, to show you the idea. In real work, you choose a threshold on the validation messages and check it once on the test messages, as the machine learning course explains. Module 8 does exactly that, with a limit on how many reviews Grace’s team can carry.
Accuracy can rise while the cost rises
Section titled “Accuracy can rise while the cost rises”The cost table changes which router looks best. Here are three real rows of the course’s comparison table, on the same 2,000 test messages. You run them yourself in the next lesson.
Text description of the chart
A column chart with three columns, on an axis from 0 to 2,000 dollars per 1,000 messages. Everything to a person costs 1,500 dollars; its accuracy is 0, because a review is not a route. Rules or a person costs 1,757.8 dollars, with an accuracy of 0.555. TF-IDF with logistic regression costs 296 dollars, with an accuracy of 0.962. The rules column is the highest. Yet the rules are more accurate than a person who reads everything.
“Everything to a person” has an accuracy of 0: it never chooses a route itself, so it is never “right”. “Rules or a person” lets keyword rules choose a route when one matches, and sends the rest to a person. Its accuracy is 0.555. Yet it costs more: US$1,757.8 against US$1,500 per 1,000 messages. Its rules match 76.4% of the test messages, but only 72.7% of those matches are right. Each wrong match costs at least US$4, more than a review at US$1.50. So higher accuracy and a higher cost come together.
The model in the third column is far better on both counts, here. Larkfield’s inbox is easier than a real one, as the next lesson explains. So do not take 0.962 as what this method gets on any inbox.
Your turn: write the cost table for each route
Section titled “Your turn: write the cost table for each route”Change one thing: instead of the cost of one message, write the cost of each kind of mistake for each label. Make a table with one row for each of the seven routes. For each row, give the cost when the router sends the message to review, to the wrong team, and to needs_person. Use the cost table and the step-by-step rule above. Watch the account and needs_person rows.
Show the answer
| Label | To review | To a wrong team | To needs_person |
|---|---|---|---|
orders |
1.50 | 4 (urgent: 20) | 2 |
delivery |
1.50 | 4 (urgent: 20) | 2 |
returns_refunds |
1.50 | 4 (urgent: 20) | 2 |
billing |
1.50 | 4 (urgent: 20) | 2 |
product |
1.50 | 4 (urgent: 20) | 2 |
account |
1.50 | 4 (urgent: 20); an account takeover: 200 | 2 |
needs_person |
1.50 | 40 | (right: 0) |
Two rows differ from the rest. In the account row, an account takeover costs US$200 in any wrong team, even when the account team is only one step away. In the needs_person row, every team costs US$40, because a team answer to a safety or legal message makes things worse. For both, the review at US$1.50 is cheap insurance. This is why a router should send its unsure messages about accounts and complaints to a person.
Common mistake
100% accurate, and three times more expensive
Symptom: Tomás moves the explorer’s threshold to 0.94 and smiles: “Look, the accuracy of the decided messages is 100%. Not one mistake. Let’s use 0.94.”
Cause: at 0.94, the model decides only 1,361 messages. The other 639 messages, almost a third, go to review at US$1.50 each. The total is US$479.2 per 1,000 messages, against US$144.2 at 0.57. The accuracy of the decided messages hides the work that moved to people.
Fix: choose a threshold by the total cost per 1,000 messages, reviews included. Also check the review load: Grace’s team can carry only so many reviews a day.
Consequence in AI work: a router judged only by the accuracy of what it decides can push its hard cases onto people. The model’s report looks perfect, while the support team drowns in reviews.
Knowledge check
Section titled “Knowledge check”Check your understanding
Explain, with an example from the inbox, how a router can be more accurate and also cost more.
Show the answer
Accuracy counts every mistake as one; the cost table does not. “Rules or a person” has an accuracy of 0.555 and costs US$1,757.8 per 1,000 test messages. “Everything to a person” has an accuracy of 0 and costs US$1,500. The rules choose a route for 76.4% of the messages, and only 72.7% of those choices are right. Each wrong choice costs at least US$4, more than a review at US$1.50, and a missed account takeover costs US$200. So the more accurate router costs more.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
An account takeover goes to needs_person instead of account. What does the course's cost rule charge, and why is that sensible?
Show the answer
US$2, the cost of a team’s message sent to a senior person. The rule asks “did it go to needs_person?” before “is it an account takeover?”. This is sensible because a senior person reads the message first and sends it to the account team at once. The account is not left open for a day, as it would be in the billing queue (US$200).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: in the explorer, moving the threshold from 0.57 to 0.82 sends about 10% more messages to review. Does the cost per 1,000 messages go up or down, and why?
Show the answer
It goes up, from US$144.2 to US$249.2 per 1,000 messages. At 0.57, the most expensive mistakes are already in review. The extra reviews cost US$1.50 each, but the mistakes they remove are mostly cheap ones at US$4. From 0.30, the same extra 10% (to about 0.70) brings the cost down, from US$289.5 to US$181.0. There, the reviews catch expensive mistakes. The answer depends on where you start.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Accuracy counts every mistake as one. Larkfield’s mistakes cost from US$2 to US$200: a factor of 100.
- The cost table (assumptions): review US$1.50, wrong queue US$4, urgent wrong queue US$20 and missed account takeover US$200. A
needs_personmessage to a team costs US$40, and a team’s message toneeds_personUS$2. - A cost-weighted confusion matrix puts a cost in each cell. The course adds the costs into one measure: the cost per 1,000 messages, reviews included, and later the cost to run.
- A threshold sends unsure messages to review. For TF-IDF + logistic regression on the test messages, the total is US$289.5 at 0.30 and US$144.2 at 0.57. At 1.00 it is US$1,500.
- “Rules or a person” is more accurate than “everything to a person”: 0.555 against 0. It also costs more: US$1,757.8 against US$1,500.
You can now judge a router by what its mistakes cost. But every number so far came from data you have not opened yet. Next, you set up the course project, read the Larkfield inbox and its guidelines, and run the first comparison yourself.
Next: Meet the inbox