Skip to content
Français

The cost of each mistake

  • Lesson 3
  • 35 min
  • Module 1: The routing problem and its cost

Where we are

We have a label scheme: six teams and "needs a person". Not every mistake costs the same: a missed account takeover costs far more than an extra review. In this lesson, we put a cost in each cell of the confusion matrix and get the one measure that the course uses to choose a method.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 3 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceThe cost of each mistake

What you will be able to do

Give each kind of routing mistake its own cost, and add the costs into one number: the cost per 1,000 messages. Accuracy counts every mistake as one, and so it can choose a router that costs more.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Larkfield’s seven routes, the topics and needs_person, from Design the labels.
  • A confusion matrix counts, for each true label, how often a model predicted each label. Classification metrics in the machine learning course explains it with two labels. This lesson uses seven.
  • A threshold turns a probability into a decision, from The precision–recall trade-off. This lesson explains the one idea that it needs.

What you need on your computer

Nothing. The explorer runs in this page. You run the same numbers yourself in the next lesson.

Back at the post office. One morning, two letters go into the wrong tray. The first is a birthday card. It arrives one day late, and nobody is much worse off. The second is a letter from a hospital with a date for an operation. It arrives one day late, and the patient misses the date.

Both are “one letter in the wrong tray”. If the post office counts only its mistakes, the two days look the same. But the post office’s real loss is very different. A good manager does not ask “how many letters went wrong?”. She asks “what did our mistakes cost?”. She also knows that some letters should go to the desk of a person, even if that is slower. A mistake on them costs too much.

The problem: accuracy counts every mistake as one

Section titled “The problem: accuracy counts every mistake as one”

Accuracy is the share of messages that a router sends to the right route. It is easy to read, and it treats every mistake the same way. Here is a small practice problem we made on purpose: ten messages, and three routers.

Messages Router A Router B Router C
4 delivery questions all right 1 to orders all right
3 returns questions all right 1 to product all right
2 billing questions all right all right 1 to review
1 account takeover (urgent) to billing right to review
Accuracy 9 of 10 8 of 10 8 of 8 decided
Cost of the mistakes US$200 US$8 US$3

Router A is the most accurate, and the most expensive by far. Its one mistake is an account takeover in the billing queue. While the message waits there, someone else uses the customer’s account. Router B makes two cheap mistakes: two normal messages in the wrong queue, at US$4 each. Router C sends the two messages that it is unsure about to a person, at US$1.50 each, and makes no mistake.

If you choose by accuracy, you choose Router A. If you choose by cost, you choose Router C. This course chooses by cost.

To count costs, Larkfield needs one number for each kind of mistake. Grace and Priya agreed on these values. They are assumptions for the course, with a reason for each, not measurements. You can change them in the project’s config/costs.json, and you do so in the next lesson.

What happens Cost (US$) Why
A person reviews the message and sends it on 1.50 About 1 minute of an agent at US$28 an hour, and a wait of a few hours
A message in the wrong team’s queue 4 The wrong team reads and moves it (about 2 minutes), and the customer waits about one more day
An urgent message in the wrong queue 20 One more day for an urgent message: a lost order, a chargeback, or a lost customer in some cases
An account takeover that misses the account team 200 Someone else uses the account while the message waits: orders, recovery work and the customer’s trust
A needs_person message sent to a team 40 A safety problem, a legal threat or an escalated complaint gets a normal team answer: a worse complaint and a risk
A team’s message sent to needs_person 2 A senior person reads a normal message and forwards it: about 4 minutes
The right route 0

Look at the range. The cheapest mistake costs US$2 and the most expensive costs US$200: a factor of 100. So a router that is a little less accurate, but never misses an account takeover, can cost far less.

For each message, the course finds its cost with the same questions, in this order. The first “yes” decides.

Six questions in a column, each with a yes arrow to a cost and a no arrow down to the next question. Is the predicted route the right route? Yes: US$0. Did the message go to review? Yes: US$1.50, a person reads it and sends it on. Did it go to needs_person, but a team was right? Yes: US$2, a senior person reads it and forwards it. Is it an account takeover? Yes: US$200, someone else uses the account meanwhile, highlighted. Was the label needs_person? Yes: US$40, a team answers a safety or legal message. Is the message urgent? Yes: US$20, an urgent message waits one more day. If no to all: US$4, the wrong team reads and moves it, one more day.
The cost of one message. The questions are asked in order, and the first yes gives the cost. The same rule is in the project's code (larkroute/costs.py) and in the notebooks' helper.
Text description of the diagram

The diagram is a column of six questions. Each question has an arrow labelled “yes” to a cost on the right, and an arrow labelled “no” down to the next question. First: is the predicted route the right route? If yes, the cost is US$0. Second: did the message go to review? If yes, US$1.50: a person reads it and sends it on. Third: did it go to needs_person, when a team was right? If yes, US$2: a senior person reads it and forwards it. Fourth: is it an account takeover? If yes, US$200, because someone else uses the account meanwhile. This cost is highlighted. Fifth: was the label needs_person? If yes, US$40: a team answers a safety or legal message. Sixth: is the message urgent? If yes, US$20: an urgent message waits one more day. If every answer is no, the cost is US$4. The wrong team reads and moves the message, and it waits one more day.

Try it on four messages:

Message Label Predicted Cost
M-109856, the refund question of the first lesson (normal) returns_refunds product US$4
An account takeover (urgent) account billing US$200
The same account takeover account needs_person US$2
“My son cut his hand on the swing seat” needs_person product US$40

The third row may surprise you. The takeover did not reach the account team, but a senior person read it first and sent it on at once. That costs 4 minutes, not a stolen account. The order of the questions encodes this: a message that reaches a person is safe.

A confusion matrix with a cost in each cell

Section titled “A confusion matrix with a cost in each cell”

A confusion matrix has one row for each label (the right route) and one column for each predicted route. The cells on the diagonal are the right answers. Every other cell is one kind of mistake. Now put a cost in each cell.

Here is a second practice problem we made on purpose: 20 messages and three routes.

Label ↓ / predicted → delivery returns_refunds account Cost of the row
delivery (8) 7 1 normal: US$4 0 US$4
returns_refunds (8) 1 urgent: US$20 7 0 US$20
account (4) 0 1 takeover: US$200 3 US$200

The router is right on 17 of 20 messages: an accuracy of 85%. Its three mistakes cost 4 + 20 + 200 = US$224. Per 1,000 messages, that is 224 ÷ 20 × 1,000 = US$11,200. One cell, the takeover, holds almost 90% of the cost.

The course adds up the cost of every message of the test set and gives it per 1,000 messages:

cost per 1,000 = (review cost + mistake cost) ÷ number of messages × 1,000

Later modules add one more part: the cost to run the router, such as computer time or the price of a language model’s answer. Then the total is the cost of the mistakes, the reviews and the running together. Every row of the course’s comparison table has this total. So you can compare a keyword rule with a large language model on one scale.

A router does more than choose a route. Most models also give a probability for each route. The highest probability says how sure the model is. If it is low, the router can send the message to review: a person reads it and sends it on, for US$1.50.

This needs a threshold. If the top probability is at or above the threshold, the router decides. If it is below, the message goes to review. A low threshold decides almost everything. It saves review costs, but it keeps some expensive mistakes. A high threshold sends more messages to review, at US$1.50 each, but it catches more mistakes.

The explorer below uses a real model: TF-IDF with logistic regression. It is a classic model that learns which words go with which route, and Module 3 builds it. It was trained on Larkfield’s 6,000 training messages and scored on the 2,000 test messages, with the cost table above. Move the threshold and watch the costs.

Explore

The cost of the mistakes, and the cost of review

  • Total cost
  • Mistakes
  • Reviews by a person
05001,0001,500Dollars per 1,000 messages0.30.40.50.60.70.80.91Decide threshold (top probability)threshold 0.3: US$289.5
05001,0001,500Dollars per 1,000 messages0.40.60.81Decide threshold (top probability)threshold 0.3: US$289.5
Decided by the model
1,994 (99.7%)
Sent to review
6 (0.3%)
Accuracy of the decided
96.5%
Review cost
US$9
Mistake cost
US$570
Total per 1,000 messages
US$289.5

Decided messages at threshold 0.3: 1,994 decided, 6 sent to review. Each cell: the number of messages and their cost.

Label ↓ Predicted route →ordersdeliveryreturns_refundsbillingproductaccountneeds_personRow cost
orders2754US$322US$81US$41US$42US$80US$56
delivery8US$324414US$162US$82US$81US$41US$2US$70
returns_refunds3US$283US$123883US$284US$162US$80US$92
billing1US$42US$246US$40256000US$68
product2US$81US$46US$4002781US$40US$56
account3US$123US$1202US$20401870US$228
needs_person00000099US$0
All decidedUS$570

At 0.3, the model decides 1,994 of 2,000 messages and sends 6 (0.3%) to review. The reviews cost US$9 and the mistakes that remain cost US$570: US$289.5 per 1,000 messages. The lowest cost on this chart is US$144.2 per 1,000, at 0.57.

TF-IDF + logistic regression on Larkfield's 2,000 test messages (measured). A message whose top probability is under the threshold goes to review at US$1.50. The costs are the course's assumptions.
Text description of the explorer

The explorer has one slider, the decide threshold, from 0.30 to 1.00 in steps of 0.01. A line chart shows three lines against the threshold. They are the total cost per 1,000 messages, the cost of the mistakes, and the cost of the reviews. The mistakes line falls as the threshold rises. The reviews line rises, slowly at first and then steeply after 0.90. The total line falls from US$289.5 at 0.30 to its lowest point, US$144.2, at 0.57. Then it rises to US$1,500 at 1.00, where every message goes to review. Below the chart, tiles give the exact values at the chosen threshold. They show the messages decided and sent to review, the accuracy of the decided messages, the review cost, the mistake cost and the total. A table shows the confusion matrix of the decided messages: one row per label and one column per predicted route. Each cell gives the number of messages and, for a mistake, its cost; the last column gives each row’s cost. At 0.30, the account row costs US$228, mostly because of one account takeover sent to billing (US$204 in that cell).

Start at 0.30. The model decides 1,994 messages and sends 6 to review. Its 70 mistakes cost US$570, and the total is US$289.5 per 1,000 messages. The comparison table’s row for this model says US$296.0, because there it decides every message, with no review at all. In the matrix, look at the account row. One takeover went to billing, and that cell alone costs US$204: the takeover’s US$200 plus one normal message at US$4.

Now the answer to the prediction: it depends on where you start.

  • From 0.30, sending about 10% more to review means a threshold near 0.70. There, 200 messages (10.0%) go to review, and the total falls to US$181.0 per 1,000. The reviews cost more, but they remove many expensive mistakes.
  • From 0.57, the lowest point (103 messages to review, US$144.2 per 1,000), another 10% means a threshold near 0.82. The total rises to US$249.2. The extra reviews now catch only a few cheap mistakes.
  • At 1.00, every message goes to review: US$1,500 per 1,000, the price of Larkfield today.

Watch one jump in particular: from 0.45 to 0.46, the mistake cost falls from US$494 to US$290. The model gets one of the 12 account takeovers wrong, and sends it to billing. Its top probability is between these two values. At 0.46, it goes to review instead. In the matrix, the cell from account to billing falls from US$204 to US$4.

One caution. This explorer looks at the test messages, to show you the idea. In real work, you choose a threshold on the validation messages and check it once on the test messages, as the machine learning course explains. Module 8 does exactly that, with a limit on how many reviews Grace’s team can carry.

The cost table changes which router looks best. Here are three real rows of the course’s comparison table, on the same 2,000 test messages. You run them yourself in the next lesson.

The total cost per 1,000 test messages of three routers (measured, with the course's costs). Everything to a person: accuracy 0. Rules or a person: accuracy 0.555. TF-IDF + logistic regression: accuracy 0.962.
Text description of the chart

A column chart with three columns, on an axis from 0 to 2,000 dollars per 1,000 messages. Everything to a person costs 1,500 dollars; its accuracy is 0, because a review is not a route. Rules or a person costs 1,757.8 dollars, with an accuracy of 0.555. TF-IDF with logistic regression costs 296 dollars, with an accuracy of 0.962. The rules column is the highest. Yet the rules are more accurate than a person who reads everything.

“Everything to a person” has an accuracy of 0: it never chooses a route itself, so it is never “right”. “Rules or a person” lets keyword rules choose a route when one matches, and sends the rest to a person. Its accuracy is 0.555. Yet it costs more: US$1,757.8 against US$1,500 per 1,000 messages. Its rules match 76.4% of the test messages, but only 72.7% of those matches are right. Each wrong match costs at least US$4, more than a review at US$1.50. So higher accuracy and a higher cost come together.

The model in the third column is far better on both counts, here. Larkfield’s inbox is easier than a real one, as the next lesson explains. So do not take 0.962 as what this method gets on any inbox.

Your turn: write the cost table for each route

Section titled “Your turn: write the cost table for each route”

Change one thing: instead of the cost of one message, write the cost of each kind of mistake for each label. Make a table with one row for each of the seven routes. For each row, give the cost when the router sends the message to review, to the wrong team, and to needs_person. Use the cost table and the step-by-step rule above. Watch the account and needs_person rows.

Show the answer
Label To review To a wrong team To needs_person
orders 1.50 4 (urgent: 20) 2
delivery 1.50 4 (urgent: 20) 2
returns_refunds 1.50 4 (urgent: 20) 2
billing 1.50 4 (urgent: 20) 2
product 1.50 4 (urgent: 20) 2
account 1.50 4 (urgent: 20); an account takeover: 200 2
needs_person 1.50 40 (right: 0)

Two rows differ from the rest. In the account row, an account takeover costs US$200 in any wrong team, even when the account team is only one step away. In the needs_person row, every team costs US$40, because a team answer to a safety or legal message makes things worse. For both, the review at US$1.50 is cheap insurance. This is why a router should send its unsure messages about accounts and complaints to a person.

Common mistake

100% accurate, and three times more expensive

Symptom: Tomás moves the explorer’s threshold to 0.94 and smiles: “Look, the accuracy of the decided messages is 100%. Not one mistake. Let’s use 0.94.”

Cause: at 0.94, the model decides only 1,361 messages. The other 639 messages, almost a third, go to review at US$1.50 each. The total is US$479.2 per 1,000 messages, against US$144.2 at 0.57. The accuracy of the decided messages hides the work that moved to people.

Fix: choose a threshold by the total cost per 1,000 messages, reviews included. Also check the review load: Grace’s team can carry only so many reviews a day.

Consequence in AI work: a router judged only by the accuracy of what it decides can push its hard cases onto people. The model’s report looks perfect, while the support team drowns in reviews.

Check your understanding

Explain, with an example from the inbox, how a router can be more accurate and also cost more.

Show the answer

Accuracy counts every mistake as one; the cost table does not. “Rules or a person” has an accuracy of 0.555 and costs US$1,757.8 per 1,000 test messages. “Everything to a person” has an accuracy of 0 and costs US$1,500. The rules choose a route for 76.4% of the messages, and only 72.7% of those choices are right. Each wrong choice costs at least US$4, more than a review at US$1.50, and a missed account takeover costs US$200. So the more accurate router costs more.

Check your understanding

An account takeover goes to needs_person instead of account. What does the course's cost rule charge, and why is that sensible?

Show the answer

US$2, the cost of a team’s message sent to a senior person. The rule asks “did it go to needs_person?” before “is it an account takeover?”. This is sensible because a senior person reads the message first and sends it to the account team at once. The account is not left open for a day, as it would be in the billing queue (US$200).

Check your understanding

Predict: in the explorer, moving the threshold from 0.57 to 0.82 sends about 10% more messages to review. Does the cost per 1,000 messages go up or down, and why?

Show the answer

It goes up, from US$144.2 to US$249.2 per 1,000 messages. At 0.57, the most expensive mistakes are already in review. The extra reviews cost US$1.50 each, but the mistakes they remove are mostly cheap ones at US$4. From 0.30, the same extra 10% (to about 0.70) brings the cost down, from US$289.5 to US$181.0. There, the reviews catch expensive mistakes. The answer depends on where you start.

Recap

  • Accuracy counts every mistake as one. Larkfield’s mistakes cost from US$2 to US$200: a factor of 100.
  • The cost table (assumptions): review US$1.50, wrong queue US$4, urgent wrong queue US$20 and missed account takeover US$200. A needs_person message to a team costs US$40, and a team’s message to needs_person US$2.
  • A cost-weighted confusion matrix puts a cost in each cell. The course adds the costs into one measure: the cost per 1,000 messages, reviews included, and later the cost to run.
  • A threshold sends unsure messages to review. For TF-IDF + logistic regression on the test messages, the total is US$289.5 at 0.30 and US$144.2 at 0.57. At 1.00 it is US$1,500.
  • “Rules or a person” is more accurate than “everything to a person”: 0.555 against 0. It also costs more: US$1,757.8 against US$1,500.

You can now judge a router by what its mistakes cost. But every number so far came from data you have not opened yet. Next, you set up the course project, read the Larkfield inbox and its guidelines, and run the first comparison yourself.

Next: Meet the inbox