Multi-label messages
- Lesson 29
- 35 min
- Module 7: Harder label shapes
Where we are
Every message so far had one label. Some customers write about two problems at once: a late delivery and a wrong charge. In this lesson, we train one model per label, choose a threshold for each, and send one message to two queues.
Lesson 29 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMulti-label messages
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Send a message with two requests to both teams that must act on it. A router that gives one route per message answers one request and silently drops the other.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It finds the messages with two requests and trains one yes/no model per route. Then it compares one threshold for all routes with a threshold per route, and looks at the messages that pass no threshold. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The
routeandroutescolumns, from Design the labels.routeis the main request;routeslists one or two requests, separated by|. - Thresholds and the F1 score (one number that combines precision and recall), from The precision–recall trade-off in the machine learning course.
- Recall by class and choosing a threshold on validation, from Imbalance and rare classes.
What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.
One envelope, two departments
Section titled “One envelope, two departments”Back at the post office. A customer sends one envelope to the town hall. Inside are two papers: a payment for a parking fine, and a form to change the address of a dog licence. The clerk who opens the mail must choose one tray. She puts the envelope in the “payments” tray, because the payment is on top.
The payment is processed. The dog licence form is not. Nobody in the payments office handles licences, and the form waits in a pile. The customer did everything right. One envelope needed two departments.
A good clerk would make a copy and send the form to the licence office too. To do that, she must notice that the envelope holds two requests, not just one.
The problem: one route drops the second request
Section titled “The problem: one route drops the second request”Many Larkfield customers write the same kind of envelope. In the history of 10,000 messages, 1,255 hold two requests. In the test set, 258 hold two requests, and 229 of them need two different teams. The other 29 name one team twice, such as a hacked account and a request to delete it: both go to account.
Here is one of the 229, written in French on social media (M-100007). In English: “When will the refund of $23.94 for order LK-199513 arrive? I also need a receipt for my purchase of 23/10.” Its routes column says returns_refunds|billing. A refund is a returns question; a receipt is a billing question.
The router of the earlier modules gives each message one route. Its seven scores always add up to 1, so the routes compete. For M-100007, it gives returns_refunds 0.889 and billing only 0.102. It sends the message to the returns team. The billing team never sees the receipt request.
The idea: one yes/no question per route
Section titled “The idea: one yes/no question per route”The clerk’s better way is to ask a separate question for each department: “Does this envelope need the payments office? Does it need the licence office?” Each question gets its own yes or no.
A router can do the same. Instead of one model that chooses among seven routes, we train seven yes/no models, one per route. The billing model answers only “does this message need billing?”. It does not compete with the returns model. This is called one-vs-rest: each model learns one route against all the rest.
A task where one item can have several labels at once is called multi-label classification. A task where each item has exactly one label, as in the earlier modules, is called multi-class.
Every value for one message
Section titled “Every value for one message”Each yes/no model gives a score from 0 to 1. The seven scores no longer need to add up to 1. Each route has a threshold: a route is “yes” when its score reaches its threshold. For M-100007, the seven models give these scores:
| Route | Score | Threshold | Yes? |
|---|---|---|---|
| returns_refunds | 0.968 | 0.30 | yes |
| billing | 0.898 | 0.20 | yes |
| orders | 0.012 | 0.35 | no |
| delivery | 0.010 | 0.35 | no |
| product | 0.008 | 0.25 | no |
| needs_person | 0.002 | 0.40 | no |
| account | 0.000 | 0.35 | no |
Compare with the one-route router. There, billing got 0.102, because returns took most of the share. Here, billing gets 0.898 from its own model. Two routes pass their thresholds, so the message goes to two queues.
Text description of the diagram
On the left, the message M-100007, translated from French: “When will the refund of $23.94 for order LK-199513 arrive? I also need a receipt for my purchase of 23/10.” In the middle, seven horizontal bars, one per route. The length of each bar is the route’s score from 0 to 1. A dark vertical line on each bar marks the route’s threshold. returns_refunds scores 0.968, above its threshold of 0.30, and billing scores 0.898, above 0.20: both bars are green. orders 0.012, delivery 0.010, product 0.008, account 0.000 and needs_person 0.002 are far below their thresholds of 0.25 to 0.40. On the right, two arrows lead to the returns_refunds queue and the billing queue.
Choosing the thresholds
Section titled “Choosing the thresholds”Why not 0.5 for every route? Three numbers show how well a version of the router handles two requests. We report them on the 2,000 test messages, after each version is fixed:
- Both routes found: of the 229 two-team messages, how many get both of their routes?
- A second route by mistake: of the 1,771 one-team messages, how many get a second route that they do not need?
- No route at all: how many messages pass no threshold?
With 0.5 everywhere, 209 of the 229 get both routes. But 95 messages pass no threshold at all. A second request is often short and written at the end (“also, …”), so its model can be less sure. And each route has its own mix of easy and hard messages, so one number may not fit all seven.
So Priya chooses a threshold per route. For each route, she tries 0.10, 0.15, … 0.90 on the validation messages. She keeps the threshold with the best F1 score for that route. The result: billing 0.20, product 0.25, returns_refunds 0.30, orders, delivery and account 0.35, and needs_person 0.40.
Text description of the diagram
Three pairs of horizontal bars. Grey bars are 0.50 for every route; blue bars are a threshold per route, chosen on validation. First pair, two-team messages with both routes found, out of 229 (more is better): 209 grey, 220 blue. Second pair, one-team messages given a second route, out of 1,771 (fewer is better): 11 grey, 43 blue. Third pair, messages above no threshold at all, out of 2,000 (fewer is better): 95 grey, 45 blue.
How good is it?
Section titled “How good is it?”With a threshold per route, on the test messages of this inbox:
- Second route found: 0.961. 220 of the 229 two-team messages reach both teams.
- Exact match: 0.952. For 218 of the 229, the set of routes is exactly right: both teams, and no third one. Exact match is the strictest measure for multi-label: one wrong or missing label makes the whole message wrong.
- A second route by mistake: 2.4%. 43 of the 1,771 one-team messages are sent to a second team that they do not need.
- No route: 2.2%. 45 messages pass no threshold. The failure case below shows what to do with them.
Remember that the Larkfield inbox is easier than a real one: a classic model already routes 96.2% of messages right here. A real inbox may have vaguer second requests. The case study Tag research papers with many labels shows multi-label classification on real data.
Two queues in practice
Section titled “Two queues in practice”Grace has one worry. “If two teams get the message, will the customer get two answers?” A model can find two routes. It cannot decide who replies. That is a team rule, and Larkfield writes it down:
- The main route (the highest score) owns the reply to the customer.
- The second team gets a task, linked to the same message, to do its part. Here: billing sends the receipt.
- The reply says that both requests are handled.
Each second route costs a second team a few minutes. When it is a mistake (2.4% of the one-team messages here), the time is wasted. The course’s cost table has no price for this yet; Grace adds one when she has measured it. Without a second route, the cost is different: a receipt that never comes, and a customer who writes again.
When to use multi-label routing
Section titled “When to use multi-label routing”- Use it when a real share of messages hold two requests that need two teams. Here, 229 of 2,000 test messages is about 11%: worth the work.
- Use it when a missed second request has a real cost, such as a refund that never comes.
- Do not use it when two requests almost never need two teams, or when one team can forward the rare second request. One route per message is simpler to train, to measure and to explain.
- What it costs: seven models instead of one (each trains in seconds here), and seven thresholds to choose and to watch. You also need a team rule for who replies, and labels that list every request.
Guided practice: two routes in the notebook
Section titled “Guided practice: two routes in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, predict how many test messages hold two requests. Then run the cells. You see
history: 1255 of 10000 | test: 258 of 2000andtest messages that need two teams: 229. - In section 2, run the cell. M-100007 gets
returns_refunds 0.968andbilling 0.898. - In section 3, run the cell. With 0.50 everywhere, you see
209,11and95. - In section 4, write the threshold search. The check says
Check passed: thresholds is right. - Run the next cell. You see exact match
0.952, second route found0.961and2.4%.
Your turn: one low threshold for all
Section titled “Your turn: one low threshold for all”Tomás asks: “Why choose seven numbers? Use 0.30 for every route.” In the notebook, run outcome({r: 0.3 for r in ROUTES}). Before you run it, predict: better or worse than the thresholds per route?
Show the answer
pd.Series(outcome({r: 0.3 for r in ROUTES}))two-team messages with both routes found 222two-team messages exactly right 221one-team messages given a second route 42messages with no route 39dtype: int64On these test messages, 0.30 everywhere is a little better than the thresholds per route. It finds both routes for 222 messages against 220, and leaves 39 messages with no route against 45. Most of the gain came from going below 0.50, not from tuning each route. Seven numbers chosen on 2,000 validation messages can fit those messages a little too well.
Be careful with this result, though. We compared the two on the test set, so the test set is no longer a fair judge between them. A fair comparison chooses both on validation. The lesson is still useful. Start with the simplest choice. Keep a threshold per route only if it wins on validation by more than the noise.
Common mistake
A message with no route
Symptom: 45 test messages pass no threshold. The router returns an empty list, and the message sits in no queue at all. Nobody answers it.
Cause: each yes/no model can say “no” on its own. In a one-route router, the highest route always wins, so this cannot happen. Here, a vague message scores low everywhere. For example, M-100173 says, in French, “I wrote to you about LK-519186, see my last email. It annoys me a little.” Its highest score is delivery 0.268, below the delivery threshold of 0.35.
Fix: never drop a message. When no route passes, send it to the route with the highest score, or to a person when that score is low. The next module builds this rule: a review zone for unsure messages. Also count the empty answers in every report.
Consequence in AI work: every multi-label system must answer two new questions. What if nothing passes? What if too much passes? The opposite case also happens here: M-100027 asks to add a gift card to an order. It gets billing 0.858 and also orders 0.438, above 0.35, so the orders team gets a copy that it does not need.
Knowledge check
Section titled “Knowledge check”Check your understanding
The one-route router gives M-100007 returns_refunds 0.889 and billing 0.102. The billing yes/no model gives billing 0.898. Explain why the two numbers for billing are so different.
Show the answer
In the one-route router, the seven scores must add up to 1: the routes compete for one share. Returns takes most of it, so billing gets only 0.102. In one-vs-rest, the billing model answers only its own question, “does this message need billing?”, and does not compete with returns. The message clearly asks for a receipt, so the billing model gives 0.898. Both models read the same message; they answer different questions.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you lower every threshold from 0.50 to 0.10. What happens to the second routes found, the second routes given by mistake, and the messages with no route?
Show the answer
More second routes are found, because more scores pass. More one-team messages also get a second route by mistake, and that grows fast: many weak scores pass 0.10. Fewer messages have no route. On this inbox, going from 0.50 to the thresholds per route (0.20 to 0.40) had three effects. Both routes found rose from 209 to 220. Mistakes rose from 11 to 43. No route fell from 95 to 45. At 0.10, the mistakes would grow further, and each one costs a team time. Choose the thresholds on validation, with the costs in mind.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In practice: your router now sends 11% of messages to two teams. Name two things, apart from the model, that you must decide before you switch it on.
Show the answer
First, who replies to the customer. For example, the main route owns the reply and the second team gets a linked task. Then the customer does not get two answers. Second, what happens when no route passes: send the message to the highest-scoring route or to a person, never drop it. Also useful: how you report results (exact match and second route found, with counts), and labels that list every request, so you can measure it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Multi-label classification gives one message several labels. Here, 258 test messages hold two requests, and 229 need two teams.
- One-vs-rest trains one yes/no model per route, so routes do not compete. M-100007 gets returns_refunds 0.968 and billing 0.898.
- Each route needs a threshold. With thresholds chosen on validation, on this inbox: second route found 0.961 and exact match 0.952. 2.4% of one-team messages get a second route by mistake.
- Some messages pass no threshold: never drop them. A team rule decides who replies.
You can now send one message to two teams. But the teams also need to know how urgent a message is, and in which language to answer. Next, we predict priority and language next to the route.