Skip to content
Français

Hierarchical labels

  • Lesson 31
  • 30 min
  • Module 7: Harder label shapes

Where we are

We have teams and topics as two separate questions. Topics live inside teams. In this lesson, we predict the team first and then the topic inside the team, and we see when two steps are better than one flat model.

Lesson 31 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceHierarchical labels

What you will be able to do

Predict a topic in a way that follows the tree of teams and topics, and measure whether it is worth it. A two-step model is not always better than one flat model, and here it is not.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It prints the label tree and trains one flat topic model. Then you build the team-then-topic model and compare where the two go wrong. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.

A big post office does not sort every letter straight to its street. It sorts in two steps. First, the central office puts each letter in the bag of its town. Then each town’s office sorts its own bag by street. A worker in the town office knows only her own 40 streets, and she knows them well.

This has a clear advantage. Each step is a small choice: a few towns, then a few streets. It also has a clear risk. If a letter goes into the wrong town’s bag, the town office cannot fix it. It can only choose among its own streets, and none of them is right.

Larkfield’s labels have the same shape. Each message has a route (one of 7) and a topic inside that route (one of 27). Every topic belongs to exactly one route. Labels like these form a tree, also called a hierarchy: the routes are the branches, and the topics are the leaves.

The label tree: 7 routes, each with 3 or 4 topics, and the number of training messages per topic. The smallest topic, account_takeover under account, has 44 training messages; the largest, late_delivery under delivery, has 742.
Larkfield's labels as a tree: 7 routes, 27 topics. The numbers are training messages per topic.
Text description of the diagram

Seven boxes, one per route, each with its topics below it and the number of training messages. orders (857): cancel_order 294, change_order 305, missing_confirmation 124, orders_other 134. delivery (1,396): delivered_not_received 271, delivery_other 127, late_delivery 742, reschedule_delivery 256. returns_refunds (1,208): damaged_item 305, exchange 116, refund_status 359, start_return 428. billing (795): double_charge 310, invoice_request 197, payment_failed 170, promo_code 118. product (860): assembly_help 170, product_question 432, stock_availability 178, warranty 80. account (591): account_takeover 44, in red, the smallest topic; change_details 180; delete_account 124; login_problem 243. needs_person (293): complaint_escalation 142, legal_threat 56, safety_issue 95.

A flat model ignores the tree. It chooses among all 27 topics at once, and the route follows from the topic. One choice among 27 is harder than one among 7. The model must separate refund_status from late_delivery (different teams) and from start_return (the same team) in the same step. When a tree has hundreds of leaves and few examples per leaf, a flat model can struggle.

The post office’s answer is to follow the tree. A hierarchical model predicts in steps, from the top of the tree down:

  1. A team model chooses among the 7 routes. It learns from all 6,000 training messages.
  2. Each route has its own topic model, trained only on the messages of that route. It chooses among 3 or 4 topics.
  3. For a new message, the team model chooses the route, and that route’s topic model chooses the topic.

That is 8 models: 1 team model and 7 topic models. Each step is a small choice. The topic is always inside the chosen team, so the two answers never contradict each other.

Two ways to predict a topic. Flat: a message goes to one topic model with 27 topics; topic accuracy 0.954, macro-F1 0.955, wrong team 80 of 2,000, 1 model. Team then topic: a team model with 7 routes, then the topic model of that team with 3 or 4 topics; topic accuracy 0.956, macro-F1 0.958, wrong team 76 of 2,000, 8 models.
A flat model against team-then-topic, on the 2,000 test messages of this inbox. The two-step model wins by 3 messages.
Text description of the diagram

Two flows side by side. On the left, flat: a message goes to one topic model with 27 topics, which gives the topic and its team. Under it: topic accuracy 0.954, macro-F1 0.955, wrong team 80 of 2,000, 1 model. On the right, team then topic: a message goes to a team model with 7 routes. Then it goes to the topic model of that team, one of seven, with 3 or 4 topics. That model gives the topic. Under it: topic accuracy 0.956, macro-F1 0.958, wrong team 76 of 2,000, 8 models (1 + 7).

A topic can be wrong in two ways. The model can choose a topic of the wrong team, or a wrong topic inside the right team. Priya counts both, on the 2,000 test messages:

Flat Team, then topic
Right topic 1,908 1,911
Wrong team 80 76
Right team, wrong topic 12 13
Topic accuracy 1,908 ÷ 2,000 = 0.954 1,911 ÷ 2,000 = 0.956
Macro-F1 0.955 0.958

Each column adds up to 2,000: 1,908 + 80 + 12, and 1,911 + 76 + 13. Two things stand out. First, almost every mistake is a mistake of the team, not of the topic inside it. Once the team is right, both models choose the topic well. Second, the two-step model makes 4 fewer team mistakes and 1 more topic mistake. Its gain is 3 messages in 2,000.

The two models disagree on 63 test messages. The flat model is right on 21 of them, the two-step model on 24, and both are wrong on the other 18. That is close to a coin toss.

So, on this inbox, the tree does not help in any way that matters. 0.954 against 0.956 is inside the noise of a 2,000-message test set: a different split could reverse it. Tomás expected the two-step model to win clearly. Priya explains why it does not.

Larkfield has only 27 topics, and every topic has at least 44 training messages, most of them more than 100. One flat logistic regression handles 27 classes easily. The topics of different teams also use different words: “refund” and “parcel” rarely meet. So the hard step is the team step, and both models must take it.

She keeps the flat topic model. It is one model instead of eight: one file to train, to version, to calibrate and to check.

The result here is not a rule for every inbox. A two-step model is worth trying when:

  • There are many labels. With hundreds of topics, one flat choice is hard, and small steps help. The case study Tag research papers with many labels has a deep label tree from real data.
  • There are few examples per label. The team model learns from every message of a team, even when each topic has only a handful. The flat model must learn each rare topic alone.
  • A team owns its own topics. If the billing team adds or splits a topic every month, it can retrain only its own topic model. The team model and the other six topic models do not change. This is often the strongest reason in a company.
  • The answers must agree. A two-step model never gives a topic of another team. Two separate flat models (one for routes, one for topics) can disagree, and then someone must decide.

It costs more: 8 models to train and to watch, two steps of latency, and one more risk, which the failure case shows.

Guided practice: two steps in the notebook

Section titled “Guided practice: two steps in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, run the cell. You see the 7 routes with their topics, and 27 topics.
  4. In section 2, run the cell. You see flat: topic accuracy 0.954, macro-F1 0.955.
  5. In section 3, build the two-step model (see below). The check says Check passed: two_step_accuracy is 0.956.
  6. In section 4, run the cell. You see wrong team 80 for flat and wrong team 76 for two steps, and they differ on 63 messages.

In section 3 of the notebook, train a team model on train["route"] and one topic model per route on that route’s training messages. Then predict each test message’s topic with the topic model of its predicted team. The notebook gives you model(), which returns a new TF-IDF and logistic regression pipeline. Before you run it, predict: more or fewer wrong-team mistakes than the flat model?

Show the answer
team_model = model().fit(train["text"], train["route"])
topic_models = {r: model().fit(train.loc[train["route"] == r, "text"], train.loc[train["route"] == r, "topic"])
for r in train["route"].unique()}
teams = team_model.predict(test["text"])
two_step_pred = np.array([topic_models[team].predict([text])[0] for team, text in zip(teams, test["text"])])
two steps: topic accuracy 0.956, macro-F1 0.958

It makes 76 wrong-team mistakes against 80 for the flat model. The team model looks at all 6,000 messages for one question only, so it makes a few fewer. The topic step then adds 13 mistakes inside the right team. The total gain is 3 messages.

Common mistake

A team mistake that the topic step cannot fix

Symptom: M-100115 says “I’m pretty frustrated about the shelf unit, there’s an issue with it, as I mentioned in my last email.” Its label is returns_refunds, topic damaged_item. The team model says product, and the product topic model then says assembly_help. The topic is wrong, and nothing in the second step noticed.

Cause: in a two-step model, each mistake of the first step passes to the second. The product topic model knows only product topics. It must choose one of them, even when the message is about a damaged item. On the test messages, 76 messages go to the wrong team, and all 76 get a wrong topic. This is called error propagation: an early mistake flows down the tree.

Fix: measure the first step on its own, and improve it first: it decides most of the result. Give the second step a way out: when the topic model is unsure, send the message to review rather than force a topic. Some systems also keep the second-best team and ask its topic model too.

Consequence in AI work: a pipeline of models is only as good as its first step. When you add steps, measure each one, and report where the errors start, not only the final accuracy.

Check your understanding

On this inbox, the flat model reaches 0.954 and team-then-topic 0.956. Tomás says “two steps are better, switch”. What do you answer?

Show the answer

The difference is 3 messages in 2,000, inside the noise of the test set. The two models disagree on 63 messages, and each is right on about a third of them (21 and 24). The two-step model also needs 8 models instead of 1, each to train, version and check. Here the gain does not pay for the cost, so keep the flat model. The answer could change with many more topics, few examples per topic, or teams that change their own topics often.

Check your understanding

Predict: in a two-step model, the team step sends a message to the wrong team. What is the chance that its topic is right?

Show the answer

Zero. The topic model of the wrong team knows only its own topics, and the right topic belongs to another team. This is error propagation: a mistake in the first step cannot be fixed in the second. On Larkfield’s test messages, all 76 wrong-team messages get a wrong topic. So measure and improve the first step first.

Check your understanding

In practice: a bank's support inbox has 12 teams and 300 topics, many with fewer than 20 labelled messages, and each team edits its own topic list every quarter. Flat or hierarchical? Give two reasons.

Show the answer

Try hierarchical first, and compare it with a flat model on the same test set. First reason: 300 topics with fewer than 20 examples each are hard for one flat choice, but the team model learns from all the messages of each team. Second reason: each team can retrain its own topic model when it changes its list, without touching the others. Measure the team step on its own, because its mistakes pass to the topic step.

Recap

  • Routes and topics form a tree. A flat model chooses among all 27 topics; a hierarchical model chooses the team first, then the topic inside the team.
  • On this inbox, flat 0.954 against team-then-topic 0.956 (macro-F1 0.955 / 0.958): 3 messages in 2,000, no real gain. Almost every mistake is a team mistake.
  • A hierarchy helps with many labels, few examples per label, teams that own their topics, or answers that must agree. It costs more models and error propagation.

You can now handle rare classes, two requests, priority, language and a label tree. But every one of these answers came with a score, and we used thresholds as if the scores were true probabilities. Next, we check whether they are.

Next: Is the confidence true?