Skip to content
Français

Multi-label messages

  • Lesson 29
  • 35 min
  • Module 7: Harder label shapes

Where we are

Every message so far had one label. Some customers write about two problems at once: a late delivery and a wrong charge. In this lesson, we train one model per label, choose a threshold for each, and send one message to two queues.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 29 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMulti-label messages

What you will be able to do

Send a message with two requests to both teams that must act on it. A router that gives one route per message answers one request and silently drops the other.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It finds the messages with two requests and trains one yes/no model per route. Then it compares one threshold for all routes with a threshold per route, and looks at the messages that pass no threshold. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.

Back at the post office. A customer sends one envelope to the town hall. Inside are two papers: a payment for a parking fine, and a form to change the address of a dog licence. The clerk who opens the mail must choose one tray. She puts the envelope in the “payments” tray, because the payment is on top.

The payment is processed. The dog licence form is not. Nobody in the payments office handles licences, and the form waits in a pile. The customer did everything right. One envelope needed two departments.

A good clerk would make a copy and send the form to the licence office too. To do that, she must notice that the envelope holds two requests, not just one.

The problem: one route drops the second request

Section titled “The problem: one route drops the second request”

Many Larkfield customers write the same kind of envelope. In the history of 10,000 messages, 1,255 hold two requests. In the test set, 258 hold two requests, and 229 of them need two different teams. The other 29 name one team twice, such as a hacked account and a request to delete it: both go to account.

Here is one of the 229, written in French on social media (M-100007). In English: “When will the refund of $23.94 for order LK-199513 arrive? I also need a receipt for my purchase of 23/10.” Its routes column says returns_refunds|billing. A refund is a returns question; a receipt is a billing question.

The router of the earlier modules gives each message one route. Its seven scores always add up to 1, so the routes compete. For M-100007, it gives returns_refunds 0.889 and billing only 0.102. It sends the message to the returns team. The billing team never sees the receipt request.

The clerk’s better way is to ask a separate question for each department: “Does this envelope need the payments office? Does it need the licence office?” Each question gets its own yes or no.

A router can do the same. Instead of one model that chooses among seven routes, we train seven yes/no models, one per route. The billing model answers only “does this message need billing?”. It does not compete with the returns model. This is called one-vs-rest: each model learns one route against all the rest.

A task where one item can have several labels at once is called multi-label classification. A task where each item has exactly one label, as in the earlier modules, is called multi-class.

Each yes/no model gives a score from 0 to 1. The seven scores no longer need to add up to 1. Each route has a threshold: a route is “yes” when its score reaches its threshold. For M-100007, the seven models give these scores:

Route Score Threshold Yes?
returns_refunds 0.968 0.30 yes
billing 0.898 0.20 yes
orders 0.012 0.35 no
delivery 0.010 0.35 no
product 0.008 0.25 no
needs_person 0.002 0.40 no
account 0.000 0.35 no

Compare with the one-route router. There, billing got 0.102, because returns took most of the share. Here, billing gets 0.898 from its own model. Two routes pass their thresholds, so the message goes to two queues.

Message M-100007 asks about a refund and a receipt. Seven bars show the score of each route's yes/no model, with a dark line at each route's threshold. Only returns_refunds (0.968) and billing (0.898) pass their thresholds, and arrows send the message to both queues.
One message, seven yes/no models. Two scores pass their thresholds, so M-100007 goes to the returns_refunds queue and the billing queue.
Text description of the diagram

On the left, the message M-100007, translated from French: “When will the refund of $23.94 for order LK-199513 arrive? I also need a receipt for my purchase of 23/10.” In the middle, seven horizontal bars, one per route. The length of each bar is the route’s score from 0 to 1. A dark vertical line on each bar marks the route’s threshold. returns_refunds scores 0.968, above its threshold of 0.30, and billing scores 0.898, above 0.20: both bars are green. orders 0.012, delivery 0.010, product 0.008, account 0.000 and needs_person 0.002 are far below their thresholds of 0.25 to 0.40. On the right, two arrows lead to the returns_refunds queue and the billing queue.

Why not 0.5 for every route? Three numbers show how well a version of the router handles two requests. We report them on the 2,000 test messages, after each version is fixed:

  • Both routes found: of the 229 two-team messages, how many get both of their routes?
  • A second route by mistake: of the 1,771 one-team messages, how many get a second route that they do not need?
  • No route at all: how many messages pass no threshold?

With 0.5 everywhere, 209 of the 229 get both routes. But 95 messages pass no threshold at all. A second request is often short and written at the end (“also, …”), so its model can be less sure. And each route has its own mix of easy and hard messages, so one number may not fit all seven.

So Priya chooses a threshold per route. For each route, she tries 0.10, 0.15, … 0.90 on the validation messages. She keeps the threshold with the best F1 score for that route. The result: billing 0.20, product 0.25, returns_refunds 0.30, orders, delivery and account 0.35, and needs_person 0.40.

Grouped bars comparing 0.50 for every route with a threshold per route on the test messages. Both routes found: 209 against 220 of 229. One-team messages given a second route: 11 against 43 of 1,771. Messages with no route: 95 against 45 of 2,000.
On this inbox, thresholds per route find 11 more second routes and halve the messages with no route. The price: 32 more one-team messages get a second route.
Text description of the diagram

Three pairs of horizontal bars. Grey bars are 0.50 for every route; blue bars are a threshold per route, chosen on validation. First pair, two-team messages with both routes found, out of 229 (more is better): 209 grey, 220 blue. Second pair, one-team messages given a second route, out of 1,771 (fewer is better): 11 grey, 43 blue. Third pair, messages above no threshold at all, out of 2,000 (fewer is better): 95 grey, 45 blue.

With a threshold per route, on the test messages of this inbox:

  • Second route found: 0.961. 220 of the 229 two-team messages reach both teams.
  • Exact match: 0.952. For 218 of the 229, the set of routes is exactly right: both teams, and no third one. Exact match is the strictest measure for multi-label: one wrong or missing label makes the whole message wrong.
  • A second route by mistake: 2.4%. 43 of the 1,771 one-team messages are sent to a second team that they do not need.
  • No route: 2.2%. 45 messages pass no threshold. The failure case below shows what to do with them.

Remember that the Larkfield inbox is easier than a real one: a classic model already routes 96.2% of messages right here. A real inbox may have vaguer second requests. The case study Tag research papers with many labels shows multi-label classification on real data.

Grace has one worry. “If two teams get the message, will the customer get two answers?” A model can find two routes. It cannot decide who replies. That is a team rule, and Larkfield writes it down:

  • The main route (the highest score) owns the reply to the customer.
  • The second team gets a task, linked to the same message, to do its part. Here: billing sends the receipt.
  • The reply says that both requests are handled.

Each second route costs a second team a few minutes. When it is a mistake (2.4% of the one-team messages here), the time is wasted. The course’s cost table has no price for this yet; Grace adds one when she has measured it. Without a second route, the cost is different: a receipt that never comes, and a customer who writes again.

  • Use it when a real share of messages hold two requests that need two teams. Here, 229 of 2,000 test messages is about 11%: worth the work.
  • Use it when a missed second request has a real cost, such as a refund that never comes.
  • Do not use it when two requests almost never need two teams, or when one team can forward the rare second request. One route per message is simpler to train, to measure and to explain.
  • What it costs: seven models instead of one (each trains in seconds here), and seven thresholds to choose and to watch. You also need a team rule for who replies, and labels that list every request.

Guided practice: two routes in the notebook

Section titled “Guided practice: two routes in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, predict how many test messages hold two requests. Then run the cells. You see history: 1255 of 10000 | test: 258 of 2000 and test messages that need two teams: 229.
  4. In section 2, run the cell. M-100007 gets returns_refunds 0.968 and billing 0.898.
  5. In section 3, run the cell. With 0.50 everywhere, you see 209, 11 and 95.
  6. In section 4, write the threshold search. The check says Check passed: thresholds is right.
  7. Run the next cell. You see exact match 0.952, second route found 0.961 and 2.4%.

Tomás asks: “Why choose seven numbers? Use 0.30 for every route.” In the notebook, run outcome({r: 0.3 for r in ROUTES}). Before you run it, predict: better or worse than the thresholds per route?

Show the answer
pd.Series(outcome({r: 0.3 for r in ROUTES}))
two-team messages with both routes found 222
two-team messages exactly right 221
one-team messages given a second route 42
messages with no route 39
dtype: int64

On these test messages, 0.30 everywhere is a little better than the thresholds per route. It finds both routes for 222 messages against 220, and leaves 39 messages with no route against 45. Most of the gain came from going below 0.50, not from tuning each route. Seven numbers chosen on 2,000 validation messages can fit those messages a little too well.

Be careful with this result, though. We compared the two on the test set, so the test set is no longer a fair judge between them. A fair comparison chooses both on validation. The lesson is still useful. Start with the simplest choice. Keep a threshold per route only if it wins on validation by more than the noise.

Common mistake

A message with no route

Symptom: 45 test messages pass no threshold. The router returns an empty list, and the message sits in no queue at all. Nobody answers it.

Cause: each yes/no model can say “no” on its own. In a one-route router, the highest route always wins, so this cannot happen. Here, a vague message scores low everywhere. For example, M-100173 says, in French, “I wrote to you about LK-519186, see my last email. It annoys me a little.” Its highest score is delivery 0.268, below the delivery threshold of 0.35.

Fix: never drop a message. When no route passes, send it to the route with the highest score, or to a person when that score is low. The next module builds this rule: a review zone for unsure messages. Also count the empty answers in every report.

Consequence in AI work: every multi-label system must answer two new questions. What if nothing passes? What if too much passes? The opposite case also happens here: M-100027 asks to add a gift card to an order. It gets billing 0.858 and also orders 0.438, above 0.35, so the orders team gets a copy that it does not need.

Check your understanding

The one-route router gives M-100007 returns_refunds 0.889 and billing 0.102. The billing yes/no model gives billing 0.898. Explain why the two numbers for billing are so different.

Show the answer

In the one-route router, the seven scores must add up to 1: the routes compete for one share. Returns takes most of it, so billing gets only 0.102. In one-vs-rest, the billing model answers only its own question, “does this message need billing?”, and does not compete with returns. The message clearly asks for a receipt, so the billing model gives 0.898. Both models read the same message; they answer different questions.

Check your understanding

Predict: you lower every threshold from 0.50 to 0.10. What happens to the second routes found, the second routes given by mistake, and the messages with no route?

Show the answer

More second routes are found, because more scores pass. More one-team messages also get a second route by mistake, and that grows fast: many weak scores pass 0.10. Fewer messages have no route. On this inbox, going from 0.50 to the thresholds per route (0.20 to 0.40) had three effects. Both routes found rose from 209 to 220. Mistakes rose from 11 to 43. No route fell from 95 to 45. At 0.10, the mistakes would grow further, and each one costs a team time. Choose the thresholds on validation, with the costs in mind.

Check your understanding

In practice: your router now sends 11% of messages to two teams. Name two things, apart from the model, that you must decide before you switch it on.

Show the answer

First, who replies to the customer. For example, the main route owns the reply and the second team gets a linked task. Then the customer does not get two answers. Second, what happens when no route passes: send the message to the highest-scoring route or to a person, never drop it. Also useful: how you report results (exact match and second route found, with counts), and labels that list every request, so you can measure it.

Recap

  • Multi-label classification gives one message several labels. Here, 258 test messages hold two requests, and 229 need two teams.
  • One-vs-rest trains one yes/no model per route, so routes do not compete. M-100007 gets returns_refunds 0.968 and billing 0.898.
  • Each route needs a threshold. With thresholds chosen on validation, on this inbox: second route found 0.961 and exact match 0.952. 2.4% of one-team messages get a second route by mistake.
  • Some messages pass no threshold: never drop them. A team rule decides who replies.

You can now send one message to two teams. But the teams also need to know how urgent a message is, and in which language to answer. Next, we predict priority and language next to the route.

Next: Priority, urgency and language