Design the labels
- Lesson 2
- 30 min
- Module 1: The routing problem and its cost
Where we are
We have seen what a message in the wrong queue costs. Before any model can help, we must decide what the labels are. In this lesson, we separate teams, topics, priority and actions, and we add a label for the messages that a person must read first.
Lesson 2 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceDesign the labels
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Decide which questions a router must answer, and give each question its own labels. Labels that mix two questions, or that have no place for “a person must read this”, make every later model worse.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s seven routes and the cost of a message in the wrong queue, from A message in the wrong queue.
- A label is the right answer that a model learns from. Supervised learning in the machine learning course explains labels.
What you need on your computer
Nothing. This lesson has no code.
Address, stamp and tray
Section titled “Address, stamp and tray”Go back to the post office. A letter carries several facts, and each one answers a different question. The address says which van takes it. The street inside that area says where the driver stops. A red express stamp says that it must go out first. A registered sticker says that a person must sign for it.
The worker does not mix these facts into one. She does not make a tray called “north area, express, registered”. She puts the letter in the north tray, and she puts it at the top because of the stamp. A tray for every mix of facts would fill the wall, and most trays would stay empty.
Some letters are different. A parcel with a strange smell does not go in any van’s tray. It goes to the supervisor’s desk, on purpose, before anybody else touches it.
Four questions about one message
Section titled “Four questions about one message”Here is a real message from the Larkfield inbox, M-100006, sent on social media:
@Larkfield, tracking for LK-950908 hasn’t moved in 19 days—is my pizza stone lost? Please look into it urgently, and cancel LK-520899 too; I don’t need it anymore.
This one message answers several questions, and each answer serves a different person:
| Question | Answer for M-100006 | Who uses the answer |
|---|---|---|
| Which team must act first? (the route) | delivery |
The router: which queue gets it |
| What exactly is it about, inside that team? (the topic) | late_delivery |
The team: templates, reports, the right expert |
| How fast must we answer? (the priority) | urgent |
The queue: what to answer first |
| Which language, which channel? | English, social media | Grace: a French-speaking agent; the answer’s style |
The route and the topic look alike, but they are different questions. The route says who acts. The topic says what the team must do. The priority is a third question: an urgent message and a normal one can go to the same team. The language is a fourth: a small program can detect it, as Module 7 shows.
So the first rule of label design is: one question, one set of labels. Each question gets its own column in the data, and later its own model or rule. In the Larkfield data, the columns are route, topic, priority, language and channel.
The actions are not labels
Section titled “The actions are not labels”You may also want to know what the team will do: refund, replace, send a courier. That is an action, and it is decided by the team after it reads the message. A router that tries to predict actions takes decisions that belong to people. Larkfield’s router stops at the route, the topic and the priority. The team chooses the action.
One label, several labels, or two levels
Section titled “One label, several labels, or two levels”Read M-100006 again. It asks for two things: find a lost parcel (delivery) and cancel another order (orders). How should a label scheme handle that? There are three common shapes.
Text description of the diagram
At the top is the message M-100006, sent on social media. Tracking for order LK-950908 has not moved in 19 days: is the pizza stone lost? Please look into it urgently, and cancel LK-520899 too. Three arrows lead to three shapes side by side (one under the other on a phone). The first shape is one label, called single-label: the message gets only “delivery”. The note says: one queue, and the second request waits or is lost. The second shape is several labels, called multi-label: the message gets “delivery” and “orders”, and “orders” is highlighted. The note says: two queues, or one queue that passes it on. The third shape is two levels, called a hierarchy: first “delivery”, then an arrow to the topic “late_delivery”. The note says: first the team, then the topic inside it.
- Single-label. Each message gets exactly one route. It is the simplest shape: one message, one queue. The cost is that a second request is easy to miss. The team must notice it and pass it on.
- Multi-label. A message can get several routes at the same time. It fits messages with two requests, but now the router must decide how many labels to give. Two teams may also answer the same customer twice.
- Hierarchy. The labels have levels: first the team, then the topic inside the team. A model can choose the team first, and then the topic among only that team’s topics. Two levels also let you report by team or by topic.
What Larkfield chose
Section titled “What Larkfield chose”Grace chose a single route per message: the main request. The guidelines say that the main request is usually the first one, or the urgent one. The data keeps the full picture in a second column, routes, which lists one or two requests separated by |. For M-100006, route is delivery and routes is delivery|orders.
In the history of 10,000 messages, 1,255 hold two requests. In 106 of them, both requests belong to the same team. For example, M-100979 reports a hacked account and asks to delete the account, so its routes is account|account. The other 1,149 messages need two different teams. Module 7 comes back to these messages and routes some of them to two queues.
Larkfield also uses two levels: 7 routes, and 27 topics inside them. Every topic belongs to exactly one route. That makes the topics a hierarchy under the routes, and Module 7 compares a team-then-topic model with one flat model.
The Larkfield label tree
Section titled “The Larkfield label tree”Here are all of Larkfield’s labels, with the number of messages for each one in the history of 10,000 messages.
Text description of the diagram
The diagram has seven rows, one for each route, with the route on the left and its topics to the right. Orders, 1,428 messages: change an order 511, cancel an order 491, other order questions 220, a missing confirmation 206. Delivery, 2,318: late delivery 1,235, delivered but not received 448, a new delivery day 427, other delivery questions 208. Returns and refunds, 2,015: start a return 715, the status of a refund 600, a damaged item 509, an exchange 191. Billing, 1,329: a double charge 518, an invoice 329, a failed payment 284, a promotion code 198. Product, 1,440: a product question 720, stock 299, help with assembly 284, warranty 137. Account, 979: a login problem 406, change personal details 299, delete the account 206, and account takeover 68, which is highlighted. Needs_person, 491, also highlighted: an escalated complaint 241, a safety problem 159, a legal threat 91.
Three things stand out:
- The routes are not the same size.
deliveryhas 2,318 messages andneeds_personhas 491. A model sees the large routes far more often. - One topic is rare and expensive.
account_takeoverhas 68 messages in the whole history, and 44 in the training part. Someone else has used the customer’s account. It must reach the account team at once. Module 7 is about this kind of rare class. - “Other” lives inside a team.
orders_otheranddelivery_othercollect the questions that belong to a team but fit no named topic. The team still gets them.
“Other” and “needs a person”, on purpose
Section titled ““Other” and “needs a person”, on purpose”Two kinds of label are easy to get wrong.
An “other” route at the top is a trap
Section titled “An “other” route at the top is a trap”It is tempting to add a top route called other for everything unclear. It quickly becomes the largest route, because every hard message falls into it. Then nobody owns it, and the hard messages wait. Larkfield has no top-level other. An unclear delivery question is still delivery, with the topic delivery_other. When a labeller cannot tell the route at all, the guidelines say: choose your best guess and mark the message for review. Do not use a route as a “not sure” box.
needs_person is a decision about the message
Section titled “needs_person is a decision about the message”needs_person is not “other”. It has its own clear rule: a safety problem, a legal or press threat, or a complaint that asks for a manager after several tries. When the main request is a safety problem, a legal or press threat, or a request for a manager, needs_person wins over every team. A mention in passing keeps the route of the main request, raises the priority and gets a flag. In the data, it has three topics: complaint_escalation (241), safety_issue (159) and legal_threat (91).
“Not sure” is not a label at all
Section titled ““Not sure” is not a label at all”Do not confuse needs_person with the review queue of later modules. needs_person describes the message: any careful reader would send it to a senior person. Review describes the router’s confidence: the model is not sure about this message, so a person checks it. A clear message about a lost parcel can go to review if the model is unsure. A clear legal threat is needs_person even when the model is sure. Module 8 builds the review queue.
What makes a good label set
Section titled “What makes a good label set”Use this list when you design labels for any inbox:
- Each label leads to one owner and one action.
deliverymeans “the delivery team acts”. A label that two teams could own causes ping-pong. - One question per set of labels. Route, topic, priority and language each get their own column.
- A clear rule and an example for each label, and a tie-break for the pairs that people confuse. Larkfield’s guidelines have a table of hard cases. For example, a refund after a return is
returns_refunds, but a wrong charge isbilling. - A route for messages that a person must read first, with a written rule. Larkfield’s is
needs_person. - Enough examples per label. A label with three examples cannot be learned. Merge it with a neighbour, or plan to collect more.
- Stable over time. Labels that change every month make every past label wrong. When a real change comes, add a label on purpose, as Module 9 does for a new product.
Labels are a product decision, not a technical detail. Grace owns them, because the labels mirror her teams. Priya checks that a model can learn them. Chloé, a bilingual support agent, writes the guidelines and labels the messages. Module 2 measures how often two careful labellers agree.
Common mistake
One label for every mix of questions
Symptom: Tomás wants one model and one label: “Let’s make labels such as late_delivery_urgent_fr. Then one prediction answers everything.” It sounds efficient.
Cause: the labels multiply. Larkfield has 27 topics, 3 priorities and 2 languages, so that gives 27 × 3 × 2 = 162 labels. In the 6,000 training messages, 25 of these mixes never appear. Another 32 appear fewer than 10 times. A model cannot learn a label that it never sees. It also cannot learn much from a label that it sees five times. Worse, the language mix is pointless: a French and an English late delivery need the same team.
Fix: one question, one set of labels. Keep route, topic, priority and language in their own columns, and answer each one with its own model or rule.
Consequence in AI work: a combined label scheme looks complete on paper. In practice, it spreads the training examples thin, so the model fails most on the rare and urgent messages, which matter most.
Knowledge check
Section titled “Knowledge check”Check your understanding
For the message “Someone changed the email on my account and placed an order I didn’t make: LK-374535”, give the route, a likely topic and a likely priority. Explain why these are three separate labels.
Show the answer
Route account, topic account_takeover, priority urgent. This is message M-100529 in the data, with exactly these labels. They are three questions with three users. The route decides which team’s queue gets the message. The topic tells the account team what to do: lock the account first. The priority decides the order in the queue. An urgent message can belong to any team, so priority is not part of the route.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A message asks to track a lost parcel and to cancel another order. Compare what single-label, multi-label and a two-level hierarchy do with it. Which does Larkfield use, and where does the second request go?
Show the answer
Single-label gives it one route, so one queue gets it, and the second request is easy to miss. Multi-label gives it both delivery and orders, so two teams can act, but the router must decide how many labels to give. A hierarchy gives a team first and then a topic inside that team, for example delivery and then late_delivery. Larkfield uses one route per message, the main request (usually the first one), and two levels: route and topic. The second request stays in the routes column (delivery|orders), and Module 7 routes some of these messages to two queues.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
What is the difference between the route needs_person and sending a message to review? Give one example of each.
Show the answer
needs_person is a label about the message. A person must read it first, because it is a safety problem, a legal or press threat, or an escalated complaint. “My son cut his hand on the swing seat” is needs_person, however sure the model is. Review is a decision about the router’s confidence: the model is not sure, so a person checks its choice. A clear question about a late parcel can still go to review if the model gives it a low probability.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A message answers several questions: the route (who acts), the topic (what to do), the priority (how fast) and the language. One question, one set of labels.
- Three shapes: single-label (one route), multi-label (several routes) and a hierarchy (team, then topic). Larkfield uses one route per message, the main request, and 27 topics under 7 routes.
- 1,255 messages hold two requests; the second one stays in the
routescolumn.account_takeoveris rare: 68 messages, 44 in training. - No top-level “other” route.
needs_personis a route on purpose, with a written rule. Review is not a label: it is a decision about the router’s confidence. - Mixing questions into one label gives 162 labels, and 25 of them never appear in training.
You can now design a label scheme. But a label scheme says nothing about which mistakes hurt most. A missed account takeover and a delivery question in the orders queue are both “one error”. Next, you give each kind of mistake its own cost.
Next: The cost of each mistake