Skip to content
Français

Design the labels

  • Lesson 2
  • 30 min
  • Module 1: The routing problem and its cost

Where we are

We have seen what a message in the wrong queue costs. Before any model can help, we must decide what the labels are. In this lesson, we separate teams, topics, priority and actions, and we add a label for the messages that a person must read first.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 2 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceDesign the labels

What you will be able to do

Decide which questions a router must answer, and give each question its own labels. Labels that mix two questions, or that have no place for “a person must read this”, make every later model worse.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code.

Go back to the post office. A letter carries several facts, and each one answers a different question. The address says which van takes it. The street inside that area says where the driver stops. A red express stamp says that it must go out first. A registered sticker says that a person must sign for it.

The worker does not mix these facts into one. She does not make a tray called “north area, express, registered”. She puts the letter in the north tray, and she puts it at the top because of the stamp. A tray for every mix of facts would fill the wall, and most trays would stay empty.

Some letters are different. A parcel with a strange smell does not go in any van’s tray. It goes to the supervisor’s desk, on purpose, before anybody else touches it.

Here is a real message from the Larkfield inbox, M-100006, sent on social media:

@Larkfield, tracking for LK-950908 hasn’t moved in 19 days—is my pizza stone lost? Please look into it urgently, and cancel LK-520899 too; I don’t need it anymore.

This one message answers several questions, and each answer serves a different person:

Question Answer for M-100006 Who uses the answer
Which team must act first? (the route) delivery The router: which queue gets it
What exactly is it about, inside that team? (the topic) late_delivery The team: templates, reports, the right expert
How fast must we answer? (the priority) urgent The queue: what to answer first
Which language, which channel? English, social media Grace: a French-speaking agent; the answer’s style

The route and the topic look alike, but they are different questions. The route says who acts. The topic says what the team must do. The priority is a third question: an urgent message and a normal one can go to the same team. The language is a fourth: a small program can detect it, as Module 7 shows.

So the first rule of label design is: one question, one set of labels. Each question gets its own column in the data, and later its own model or rule. In the Larkfield data, the columns are route, topic, priority, language and channel.

You may also want to know what the team will do: refund, replace, send a courier. That is an action, and it is decided by the team after it reads the message. A router that tries to predict actions takes decisions that belong to people. Larkfield’s router stops at the route, the topic and the priority. The team chooses the action.

Read M-100006 again. It asks for two things: find a lost parcel (delivery) and cancel another order (orders). How should a label scheme handle that? There are three common shapes.

The message M-100006 from social media: tracking for LK-950908 hasn't moved in 19 days, is my pizza stone lost? Please look into it urgently, and cancel LK-520899 too. Three arrows lead to three shapes. One label (single-label): delivery. One queue; the second request waits or is lost. Several labels (multi-label): delivery and orders, with orders highlighted. Two queues, or one queue that passes it on. Two levels (hierarchy): delivery, then an arrow to late_delivery. First the team, then the topic inside it.
Three ways to label the same message. Larkfield uses one route per message (the main request) and keeps the second request in another column.
Text description of the diagram

At the top is the message M-100006, sent on social media. Tracking for order LK-950908 has not moved in 19 days: is the pizza stone lost? Please look into it urgently, and cancel LK-520899 too. Three arrows lead to three shapes side by side (one under the other on a phone). The first shape is one label, called single-label: the message gets only “delivery”. The note says: one queue, and the second request waits or is lost. The second shape is several labels, called multi-label: the message gets “delivery” and “orders”, and “orders” is highlighted. The note says: two queues, or one queue that passes it on. The third shape is two levels, called a hierarchy: first “delivery”, then an arrow to the topic “late_delivery”. The note says: first the team, then the topic inside it.

  • Single-label. Each message gets exactly one route. It is the simplest shape: one message, one queue. The cost is that a second request is easy to miss. The team must notice it and pass it on.
  • Multi-label. A message can get several routes at the same time. It fits messages with two requests, but now the router must decide how many labels to give. Two teams may also answer the same customer twice.
  • Hierarchy. The labels have levels: first the team, then the topic inside the team. A model can choose the team first, and then the topic among only that team’s topics. Two levels also let you report by team or by topic.

Grace chose a single route per message: the main request. The guidelines say that the main request is usually the first one, or the urgent one. The data keeps the full picture in a second column, routes, which lists one or two requests separated by |. For M-100006, route is delivery and routes is delivery|orders.

In the history of 10,000 messages, 1,255 hold two requests. In 106 of them, both requests belong to the same team. For example, M-100979 reports a hacked account and asks to delete the account, so its routes is account|account. The other 1,149 messages need two different teams. Module 7 comes back to these messages and routes some of them to two queues.

Larkfield also uses two levels: 7 routes, and 27 topics inside them. Every topic belongs to exactly one route. That makes the topics a hierarchy under the routes, and Module 7 compares a team-then-topic model with one flat model.

Here are all of Larkfield’s labels, with the number of messages for each one in the history of 10,000 messages.

Seven route boxes on the left, each with its number of messages, and its topics to the right. orders 1,428: change_order 511, cancel_order 491, orders_other 220, missing_confirmation 206. delivery 2,318: late_delivery 1,235, delivered_not_received 448, reschedule_delivery 427, delivery_other 208. returns_refunds 2,015: start_return 715, refund_status 600, damaged_item 509, exchange 191. billing 1,329: double_charge 518, invoice_request 329, payment_failed 284, promo_code 198. product 1,440: product_question 720, stock_availability 299, assembly_help 284, warranty 137. account 979: login_problem 406, change_details 299, delete_account 206, account_takeover 68, highlighted. needs_person 491, highlighted: complaint_escalation 241, safety_issue 159, legal_threat 91.
Larkfield's labels: 7 routes and 27 topics, with the messages of the history. Two labels are highlighted: needs_person, a route on purpose, and account_takeover, the rarest topic.
Text description of the diagram

The diagram has seven rows, one for each route, with the route on the left and its topics to the right. Orders, 1,428 messages: change an order 511, cancel an order 491, other order questions 220, a missing confirmation 206. Delivery, 2,318: late delivery 1,235, delivered but not received 448, a new delivery day 427, other delivery questions 208. Returns and refunds, 2,015: start a return 715, the status of a refund 600, a damaged item 509, an exchange 191. Billing, 1,329: a double charge 518, an invoice 329, a failed payment 284, a promotion code 198. Product, 1,440: a product question 720, stock 299, help with assembly 284, warranty 137. Account, 979: a login problem 406, change personal details 299, delete the account 206, and account takeover 68, which is highlighted. Needs_person, 491, also highlighted: an escalated complaint 241, a safety problem 159, a legal threat 91.

Three things stand out:

  • The routes are not the same size. delivery has 2,318 messages and needs_person has 491. A model sees the large routes far more often.
  • One topic is rare and expensive. account_takeover has 68 messages in the whole history, and 44 in the training part. Someone else has used the customer’s account. It must reach the account team at once. Module 7 is about this kind of rare class.
  • “Other” lives inside a team. orders_other and delivery_other collect the questions that belong to a team but fit no named topic. The team still gets them.

“Other” and “needs a person”, on purpose

Section titled ““Other” and “needs a person”, on purpose”

Two kinds of label are easy to get wrong.

It is tempting to add a top route called other for everything unclear. It quickly becomes the largest route, because every hard message falls into it. Then nobody owns it, and the hard messages wait. Larkfield has no top-level other. An unclear delivery question is still delivery, with the topic delivery_other. When a labeller cannot tell the route at all, the guidelines say: choose your best guess and mark the message for review. Do not use a route as a “not sure” box.

needs_person is a decision about the message

Section titled “needs_person is a decision about the message”

needs_person is not “other”. It has its own clear rule: a safety problem, a legal or press threat, or a complaint that asks for a manager after several tries. When the main request is a safety problem, a legal or press threat, or a request for a manager, needs_person wins over every team. A mention in passing keeps the route of the main request, raises the priority and gets a flag. In the data, it has three topics: complaint_escalation (241), safety_issue (159) and legal_threat (91).

Do not confuse needs_person with the review queue of later modules. needs_person describes the message: any careful reader would send it to a senior person. Review describes the router’s confidence: the model is not sure about this message, so a person checks it. A clear message about a lost parcel can go to review if the model is unsure. A clear legal threat is needs_person even when the model is sure. Module 8 builds the review queue.

Use this list when you design labels for any inbox:

  1. Each label leads to one owner and one action. delivery means “the delivery team acts”. A label that two teams could own causes ping-pong.
  2. One question per set of labels. Route, topic, priority and language each get their own column.
  3. A clear rule and an example for each label, and a tie-break for the pairs that people confuse. Larkfield’s guidelines have a table of hard cases. For example, a refund after a return is returns_refunds, but a wrong charge is billing.
  4. A route for messages that a person must read first, with a written rule. Larkfield’s is needs_person.
  5. Enough examples per label. A label with three examples cannot be learned. Merge it with a neighbour, or plan to collect more.
  6. Stable over time. Labels that change every month make every past label wrong. When a real change comes, add a label on purpose, as Module 9 does for a new product.

Labels are a product decision, not a technical detail. Grace owns them, because the labels mirror her teams. Priya checks that a model can learn them. Chloé, a bilingual support agent, writes the guidelines and labels the messages. Module 2 measures how often two careful labellers agree.

Common mistake

One label for every mix of questions

Symptom: Tomás wants one model and one label: “Let’s make labels such as late_delivery_urgent_fr. Then one prediction answers everything.” It sounds efficient.

Cause: the labels multiply. Larkfield has 27 topics, 3 priorities and 2 languages, so that gives 27 × 3 × 2 = 162 labels. In the 6,000 training messages, 25 of these mixes never appear. Another 32 appear fewer than 10 times. A model cannot learn a label that it never sees. It also cannot learn much from a label that it sees five times. Worse, the language mix is pointless: a French and an English late delivery need the same team.

Fix: one question, one set of labels. Keep route, topic, priority and language in their own columns, and answer each one with its own model or rule.

Consequence in AI work: a combined label scheme looks complete on paper. In practice, it spreads the training examples thin, so the model fails most on the rare and urgent messages, which matter most.

Check your understanding

For the message “Someone changed the email on my account and placed an order I didn’t make: LK-374535”, give the route, a likely topic and a likely priority. Explain why these are three separate labels.

Show the answer

Route account, topic account_takeover, priority urgent. This is message M-100529 in the data, with exactly these labels. They are three questions with three users. The route decides which team’s queue gets the message. The topic tells the account team what to do: lock the account first. The priority decides the order in the queue. An urgent message can belong to any team, so priority is not part of the route.

Check your understanding

A message asks to track a lost parcel and to cancel another order. Compare what single-label, multi-label and a two-level hierarchy do with it. Which does Larkfield use, and where does the second request go?

Show the answer

Single-label gives it one route, so one queue gets it, and the second request is easy to miss. Multi-label gives it both delivery and orders, so two teams can act, but the router must decide how many labels to give. A hierarchy gives a team first and then a topic inside that team, for example delivery and then late_delivery. Larkfield uses one route per message, the main request (usually the first one), and two levels: route and topic. The second request stays in the routes column (delivery|orders), and Module 7 routes some of these messages to two queues.

Check your understanding

What is the difference between the route needs_person and sending a message to review? Give one example of each.

Show the answer

needs_person is a label about the message. A person must read it first, because it is a safety problem, a legal or press threat, or an escalated complaint. “My son cut his hand on the swing seat” is needs_person, however sure the model is. Review is a decision about the router’s confidence: the model is not sure, so a person checks its choice. A clear question about a late parcel can still go to review if the model gives it a low probability.

Recap

  • A message answers several questions: the route (who acts), the topic (what to do), the priority (how fast) and the language. One question, one set of labels.
  • Three shapes: single-label (one route), multi-label (several routes) and a hierarchy (team, then topic). Larkfield uses one route per message, the main request, and 27 topics under 7 routes.
  • 1,255 messages hold two requests; the second one stays in the routes column. account_takeover is rare: 68 messages, 44 in training.
  • No top-level “other” route. needs_person is a route on purpose, with a written rule. Review is not a label: it is a decision about the router’s confidence.
  • Mixing questions into one label gives 162 labels, and 25 of them never appear in training.

You can now design a label scheme. But a label scheme says nothing about which mistakes hurt most. A missed account takeover and a delivery question in the orders queue are both “one error”. Next, you give each kind of mistake its own cost.

Next: The cost of each mistake