Hierarchical labels
- Lesson 31
- 30 min
- Module 7: Harder label shapes
Where we are
We have teams and topics as two separate questions. Topics live inside teams. In this lesson, we predict the team first and then the topic inside the team, and we see when two steps are better than one flat model.
Lesson 31 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceHierarchical labels
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Predict a topic in a way that follows the tree of teams and topics, and measure whether it is worth it. A two-step model is not always better than one flat model, and here it is not.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It prints the label tree and trains one flat topic model. Then you build the team-then-topic model and compare where the two go wrong. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Topics and routes, from Design the labels. Each topic belongs to one team.
- Accuracy and macro-F1 (the F1 of each class, averaged with the same weight), from Imbalance and rare classes.
- TF-IDF with logistic regression, from Text as numbers: bag of words and TF-IDF.
What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.
Sorting by town, then by street
Section titled “Sorting by town, then by street”A big post office does not sort every letter straight to its street. It sorts in two steps. First, the central office puts each letter in the bag of its town. Then each town’s office sorts its own bag by street. A worker in the town office knows only her own 40 streets, and she knows them well.
This has a clear advantage. Each step is a small choice: a few towns, then a few streets. It also has a clear risk. If a letter goes into the wrong town’s bag, the town office cannot fix it. It can only choose among its own streets, and none of them is right.
The problem: 27 topics in one choice
Section titled “The problem: 27 topics in one choice”Larkfield’s labels have the same shape. Each message has a route (one of 7) and a topic inside that route (one of 27). Every topic belongs to exactly one route. Labels like these form a tree, also called a hierarchy: the routes are the branches, and the topics are the leaves.
Text description of the diagram
Seven boxes, one per route, each with its topics below it and the number of training messages. orders (857): cancel_order 294, change_order 305, missing_confirmation 124, orders_other 134. delivery (1,396): delivered_not_received 271, delivery_other 127, late_delivery 742, reschedule_delivery 256. returns_refunds (1,208): damaged_item 305, exchange 116, refund_status 359, start_return 428. billing (795): double_charge 310, invoice_request 197, payment_failed 170, promo_code 118. product (860): assembly_help 170, product_question 432, stock_availability 178, warranty 80. account (591): account_takeover 44, in red, the smallest topic; change_details 180; delete_account 124; login_problem 243. needs_person (293): complaint_escalation 142, legal_threat 56, safety_issue 95.
A flat model ignores the tree. It chooses among all 27 topics at once, and the route follows from the topic. One choice among 27 is harder than one among 7. The model must separate refund_status from late_delivery (different teams) and from start_return (the same team) in the same step. When a tree has hundreds of leaves and few examples per leaf, a flat model can struggle.
The idea: team first, then topic
Section titled “The idea: team first, then topic”The post office’s answer is to follow the tree. A hierarchical model predicts in steps, from the top of the tree down:
- A team model chooses among the 7 routes. It learns from all 6,000 training messages.
- Each route has its own topic model, trained only on the messages of that route. It chooses among 3 or 4 topics.
- For a new message, the team model chooses the route, and that route’s topic model chooses the topic.
That is 8 models: 1 team model and 7 topic models. Each step is a small choice. The topic is always inside the chosen team, so the two answers never contradict each other.
Text description of the diagram
Two flows side by side. On the left, flat: a message goes to one topic model with 27 topics, which gives the topic and its team. Under it: topic accuracy 0.954, macro-F1 0.955, wrong team 80 of 2,000, 1 model. On the right, team then topic: a message goes to a team model with 7 routes. Then it goes to the topic model of that team, one of seven, with 3 or 4 topics. That model gives the topic. Under it: topic accuracy 0.956, macro-F1 0.958, wrong team 76 of 2,000, 8 models (1 + 7).
Every value: where each model goes wrong
Section titled “Every value: where each model goes wrong”A topic can be wrong in two ways. The model can choose a topic of the wrong team, or a wrong topic inside the right team. Priya counts both, on the 2,000 test messages:
| Flat | Team, then topic | |
|---|---|---|
| Right topic | 1,908 | 1,911 |
| Wrong team | 80 | 76 |
| Right team, wrong topic | 12 | 13 |
| Topic accuracy | 1,908 ÷ 2,000 = 0.954 | 1,911 ÷ 2,000 = 0.956 |
| Macro-F1 | 0.955 | 0.958 |
Each column adds up to 2,000: 1,908 + 80 + 12, and 1,911 + 76 + 13. Two things stand out. First, almost every mistake is a mistake of the team, not of the topic inside it. Once the team is right, both models choose the topic well. Second, the two-step model makes 4 fewer team mistakes and 1 more topic mistake. Its gain is 3 messages in 2,000.
The two models disagree on 63 test messages. The flat model is right on 21 of them, the two-step model on 24, and both are wrong on the other 18. That is close to a coin toss.
Larkfield: no real gain here
Section titled “Larkfield: no real gain here”So, on this inbox, the tree does not help in any way that matters. 0.954 against 0.956 is inside the noise of a 2,000-message test set: a different split could reverse it. Tomás expected the two-step model to win clearly. Priya explains why it does not.
Larkfield has only 27 topics, and every topic has at least 44 training messages, most of them more than 100. One flat logistic regression handles 27 classes easily. The topics of different teams also use different words: “refund” and “parcel” rarely meet. So the hard step is the team step, and both models must take it.
She keeps the flat topic model. It is one model instead of eight: one file to train, to version, to calibrate and to check.
When a hierarchy helps
Section titled “When a hierarchy helps”The result here is not a rule for every inbox. A two-step model is worth trying when:
- There are many labels. With hundreds of topics, one flat choice is hard, and small steps help. The case study Tag research papers with many labels has a deep label tree from real data.
- There are few examples per label. The team model learns from every message of a team, even when each topic has only a handful. The flat model must learn each rare topic alone.
- A team owns its own topics. If the billing team adds or splits a topic every month, it can retrain only its own topic model. The team model and the other six topic models do not change. This is often the strongest reason in a company.
- The answers must agree. A two-step model never gives a topic of another team. Two separate flat models (one for routes, one for topics) can disagree, and then someone must decide.
It costs more: 8 models to train and to watch, two steps of latency, and one more risk, which the failure case shows.
Guided practice: two steps in the notebook
Section titled “Guided practice: two steps in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see the 7 routes with their topics, and
27 topics. - In section 2, run the cell. You see
flat: topic accuracy 0.954, macro-F1 0.955. - In section 3, build the two-step model (see below). The check says
Check passed: two_step_accuracy is 0.956. - In section 4, run the cell. You see
wrong team 80for flat andwrong team 76for two steps, andthey differ on 63 messages.
Your turn: build the two-step model
Section titled “Your turn: build the two-step model”In section 3 of the notebook, train a team model on train["route"] and one topic model per route on that route’s training messages. Then predict each test message’s topic with the topic model of its predicted team. The notebook gives you model(), which returns a new TF-IDF and logistic regression pipeline. Before you run it, predict: more or fewer wrong-team mistakes than the flat model?
Show the answer
team_model = model().fit(train["text"], train["route"])topic_models = {r: model().fit(train.loc[train["route"] == r, "text"], train.loc[train["route"] == r, "topic"]) for r in train["route"].unique()}teams = team_model.predict(test["text"])two_step_pred = np.array([topic_models[team].predict([text])[0] for team, text in zip(teams, test["text"])])two steps: topic accuracy 0.956, macro-F1 0.958It makes 76 wrong-team mistakes against 80 for the flat model. The team model looks at all 6,000 messages for one question only, so it makes a few fewer. The topic step then adds 13 mistakes inside the right team. The total gain is 3 messages.
Common mistake
A team mistake that the topic step cannot fix
Symptom: M-100115 says “I’m pretty frustrated about the shelf unit, there’s an issue with it, as I mentioned in my last email.” Its label is returns_refunds, topic damaged_item. The team model says product, and the product topic model then says assembly_help. The topic is wrong, and nothing in the second step noticed.
Cause: in a two-step model, each mistake of the first step passes to the second. The product topic model knows only product topics. It must choose one of them, even when the message is about a damaged item. On the test messages, 76 messages go to the wrong team, and all 76 get a wrong topic. This is called error propagation: an early mistake flows down the tree.
Fix: measure the first step on its own, and improve it first: it decides most of the result. Give the second step a way out: when the topic model is unsure, send the message to review rather than force a topic. Some systems also keep the second-best team and ask its topic model too.
Consequence in AI work: a pipeline of models is only as good as its first step. When you add steps, measure each one, and report where the errors start, not only the final accuracy.
Knowledge check
Section titled “Knowledge check”Check your understanding
On this inbox, the flat model reaches 0.954 and team-then-topic 0.956. Tomás says “two steps are better, switch”. What do you answer?
Show the answer
The difference is 3 messages in 2,000, inside the noise of the test set. The two models disagree on 63 messages, and each is right on about a third of them (21 and 24). The two-step model also needs 8 models instead of 1, each to train, version and check. Here the gain does not pay for the cost, so keep the flat model. The answer could change with many more topics, few examples per topic, or teams that change their own topics often.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: in a two-step model, the team step sends a message to the wrong team. What is the chance that its topic is right?
Show the answer
Zero. The topic model of the wrong team knows only its own topics, and the right topic belongs to another team. This is error propagation: a mistake in the first step cannot be fixed in the second. On Larkfield’s test messages, all 76 wrong-team messages get a wrong topic. So measure and improve the first step first.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In practice: a bank's support inbox has 12 teams and 300 topics, many with fewer than 20 labelled messages, and each team edits its own topic list every quarter. Flat or hierarchical? Give two reasons.
Show the answer
Try hierarchical first, and compare it with a flat model on the same test set. First reason: 300 topics with fewer than 20 examples each are hard for one flat choice, but the team model learns from all the messages of each team. Second reason: each team can retrain its own topic model when it changes its list, without touching the others. Measure the team step on its own, because its mistakes pass to the topic step.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Routes and topics form a tree. A flat model chooses among all 27 topics; a hierarchical model chooses the team first, then the topic inside the team.
- On this inbox, flat 0.954 against team-then-topic 0.956 (macro-F1 0.955 / 0.958): 3 messages in 2,000, no real gain. Almost every mistake is a team mistake.
- A hierarchy helps with many labels, few examples per label, teams that own their topics, or answers that must agree. It costs more models and error propagation.
You can now handle rare classes, two requests, priority, language and a label tree. But every one of these answers came with a score, and we used thresholds as if the scores were true probabilities. Next, we check whether they are.
Next: Is the confidence true?