Skip to content
Français

Priority, urgency and language

  • Lesson 30
  • 30 min
  • Module 7: Harder label shapes

Where we are

We route each message to the right team. The team also needs to know how urgent it is and which language it is in. In this lesson, we train priority as a separate model, detect the language, and add rules for words that always mean urgent.

Lesson 30 of 37 · Text Classification and Routing: Send Each Message to the Right PlacePriority, urgency and language

What you will be able to do

Give each message a priority and a language next to its route, so the right person answers it first and in the right language. The route alone says who answers, not how soon or in which language.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It trains a priority model, tests a rule for urgent words and combines the two. Then it trains a language detector, reads its disagreements, and compares the router on English and French messages. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.

Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.

At the post office, the clerk asks three questions about each letter, and each has its own sign. The address says which van takes it. A red express stamp says that it goes out first. In a town where people speak two languages, the language on the envelope says which counter calls the customer back.

The three questions are independent. An express letter can go to any part of the town. A letter in French can be urgent or not. So the clerk does not look for one combined sign, such as “north van, express, French”. She answers each question on its own.

The problem: the route says who, not how soon

Section titled “The problem: the route says who, not how soon”

Larkfield’s router answers the first question: which team. The teams need the other two answers as well.

  • Priority. Each message is urgent, normal or low. In the test set, 372 messages are urgent, 1,275 normal and 353 low. An urgent message that waits in a normal pile is expensive. The cost table prices an urgent message in the wrong queue at US$20, five times a normal one.
  • Language. 29% of the test messages are in French. Larkfield has French-speaking agents, such as Chloé. A French message that reaches an agent who reads only English waits, or gets a poor answer.

Why not one combined label, such as delivery-urgent-fr? Seven routes × three priorities × two languages make 42 labels. Many of them would have very few training messages: the test set has only one urgent product message. Three small questions, each with plenty of examples, are easier to learn.

So each question gets its own model, trained on the same messages with a different label column. The team model reads route, the priority model reads priority, and the language detector reads language. Their three answers then decide the queue together.

One French message goes to three models. The team model answers delivery, the priority model with the urgent words answers urgent, and the language detector answers fr. Together they send the message to the top of the French-speaking delivery agents' queue.
Three separate answers for one message decide where it goes and how soon. The message is an example we wrote for this picture.
Text description of the diagram

On the left, a message in French: “Bonjour, mon colis LK-… devait arriver hier, j’en ai besoin aujourd’hui pour un cadeau” (“Hello, my parcel LK-… should have arrived yesterday, I need it today for a gift”). Three arrows lead to three models. The team model, with 7 routes, answers delivery. The priority model, with the urgent-words rule, answers urgent. The language detector answers fr. The three answers lead to one box: French-speaking delivery agents, top of their queue.

Priya trains a priority model: TF-IDF with logistic regression, as for the route, but on the priority column. Its accuracy on the test messages is 0.714. The router reaches 0.962 on this inbox, so priority is much harder. Why? The confusion table shows where the mistakes are.

A 3 by 3 confusion table of the priority model on the test messages. Rows are the label, columns the prediction. Urgent: 229 urgent, 134 normal, 9 low. Normal: 23 urgent, 1,114 normal, 138 low. Low: 4 urgent, 265 normal, 84 low. Recall: urgent 0.616, normal 0.874, low 0.238. Precision: urgent 0.895, normal 0.736, low 0.364.
The priority model's confusion table on the 2,000 test messages. Most mistakes are between neighbours: urgent read as normal, and low read as normal.
Text description of the diagram

A table with three rows and three columns. The rows are the label; the columns are the model’s answer, in the order urgent, normal, low. Row urgent (372 messages): 229 answered urgent, 134 normal, 9 low; recall 0.616. Row normal (1,275): 23 urgent, 1,114 normal, 138 low; recall 0.874. Row low (353): 4 urgent, 265 normal, 84 low; recall 0.238. Under the columns, the precision of each answer: urgent 0.895, normal 0.736, low 0.364. The cells on the diagonal are right answers. The two largest mistakes are 134 urgent messages read as normal, and 265 low messages read as normal.

Read it row by row. Almost all mistakes are between neighbours. An urgent message is read as normal (134), or a low one as normal (265). Urgent and low are almost never swapped (9 and 4). That makes sense: urgency is a matter of degree. “I need it by Friday” sits between normal and urgent, and two people could label it differently.

For Grace, one row matters most: the urgent one. Out of 372 urgent messages, the model finds 229: urgent recall 0.616. When it says “urgent”, it is right 229 times out of 256: precision 0.895. So it is careful, and it misses more than a third of the urgent messages.

Some words almost always mean urgent: “urgent”, “today”, “immediately”, “right now”, and in French aujourd’hui, tout de suite, urgence. A regular expression finds them:

URGENT_WORDS = re.compile(r"\b(?:urgent\w*|asap|today|tonight|immediately|right now|urgence|aujourd'hui|"
r"imm[ée]diatement|tout de suite|vite)\b", re.I)
rule_urgent = test["text"].str.contains(URGENT_WORDS).to_numpy()

The rule matches 132 test messages. 127 of them are urgent: precision 0.962. But it finds only 127 of the 372 urgent messages: recall 0.341. Many urgent customers never write “urgent”. They write “my order is missing and it was a gift for tomorrow”.

Grace asks: “Mark a message urgent if the model says so or the rule matches.” Count it:

  • The model marks 256 messages, 229 right. The rule matches 132, 127 right.
  • Most rule matches are already marked by the model. Together, they mark 266 messages, of which 235 are urgent.
  • Recall: 235 ÷ 372 = 0.632. Precision: 235 ÷ 266 = 0.883.
Pairs of bars for three ways to find urgent messages. Urgent-words rule: recall 0.341, precision 0.962. Priority model: recall 0.616, precision 0.895. Model or rule: recall 0.632, precision 0.883.
On this inbox, the rule adds only 6 urgent messages to the model's 229. Its value is that it is certain and easy to explain.
Text description of the diagram

Three pairs of horizontal bars. Blue bars are recall, out of the 372 urgent test messages; orange bars are precision. Urgent-words rule: recall 0.341 (127 of 372), precision 0.962 (127 of 132). Priority model: recall 0.616 (229 of 372), precision 0.895 (229 of 256). Model or rule: recall 0.632 (235 of 372), precision 0.883 (235 of 266).

The rule adds only 6 urgent messages that the model missed. So why keep it? Because it is certain and easy to explain. If a customer writes “today”, the message goes first, and an agent can see why. A model can lose a word like that after a retraining. A rule cannot.

The Labelling guidelines set one more rule. A safety problem, a legal threat or an account takeover that a message mentions only in passing does not change its route. It raises the message’s priority, and a person flags it. So priority carries part of the safety net. The takeover flag of Imbalance and rare classes belongs here too: a flagged message goes to the top of its queue.

The language question is much easier. A language detector does not need to understand a message. It looks at letter groups: ou, é, qu', the, ing. These groups are very different in English and French. A Naive Bayes model on groups of 1 to 3 letters is enough. Priya trains it on the language column of the 6,000 training messages.

For the sentence “Mon colis n’est pas arrivé” (“My parcel has not arrived”), the detector gives fr 0.985 and en 0.015. The letters é, n', est and pas almost never appear in English messages.

On the 2,000 test messages, the detector agrees with the label 1,995 times: accuracy 0.9975. Priya reads the 5 disagreements, and the failure case below shows what she finds.

The language column also lets Priya check the router on each language. This is called a slice: the same measure, on one part of the data. On the test messages of this inbox, the router reaches 0.966 on English (1,411 messages) and 0.952 on French (589). The French slice is a little weaker. The difference is 1.4 points on 589 messages, about 8 messages. Small, but she keeps watching it with every new model.

With the language known, the queue rule is simple: a French message goes to the French-speaking agents of its team. Chloé reads the French messages that need a person.

  • Use a separate model for each question whose answer does not depend on the others: route, priority, language. Each one has more training examples than a combined label.
  • Add a rule for words that must always count, even if the model misses them: urgent words, a product recall, a legal word. Keep the rules few and easy to read.
  • Do not trust priority as much as route. On this inbox, priority reaches 0.714 against 0.962 for the route. It is a matter of degree. Use it to order a queue, not to hide a message.
  • A language detector is cheap and very accurate between two languages that look different. For languages that share most letters, or for very short messages, it can fail: test it on your own data.
  • What it costs: three models instead of one, each with its own checks, and a few rules to maintain. Each trains in seconds here.

Guided practice: three answers in the notebook

Section titled “Guided practice: three answers in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The data cell ends with Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  3. In section 1, run the cell. You see accuracy 0.714 and the confusion table.
  4. In section 2, run the cell. You see urgent recall 0.616 and precision 0.895.
  5. In section 3, run the rule. You see recall 0.341 and precision 0.962.
  6. In section 5, run the language detector. You see language accuracy 0.9975, 5 disagreements.
  7. In section 6, run the cell. You see English 0.966 and French 0.952.

In section 4 of the notebook, mark a message urgent when the model says urgent or the rule matches. Compute the recall and the precision of this combination, rounded to 3 decimals. Before you run it, predict: will the recall rise a lot?

Show the answer
both = model_urgent | rule_urgent
both_recall = round(float(both[urgent].mean()), 3)
both_precision = round(float(urgent[both].mean()), 3)
print(both_recall, both_precision)
0.632 0.883

The recall rises only from 0.616 to 0.632, and the precision falls a little, from 0.895 to 0.883. The model already finds most messages that contain an urgent word. The rule is still worth keeping for its certainty, not for its recall.

Common mistake

The detector's mistakes are the labels' mistakes

Symptom: the language detector disagrees with the label on 5 test messages. Tomás wants to report “5 errors” and try a larger model.

Cause: Priya reads the 5 messages. Four are labelled en but written in French, such as M-100856: “Bonjour, je n’ai toujours pas reçu mon toaster…”. One is labelled fr but written in English (M-108940). The detector is right every time. The labels are wrong. Here they are slips of the program that made this practice inbox. In a real inbox, they come from a language field that the customer chose, or from a default value.

Fix: correct the 5 labels before you trust the language slices. Then use the detector, not the stored field, to choose the queue.

Consequence in AI work: when a strong, simple model “fails” on a few examples, read them before you change the model. A disagreement is often a label error. This is the same idea as finding noisy labels in Find noisy labels.

Check your understanding

The priority model is right on 0.714 of the test messages, and almost all its mistakes are between neighbours (urgent ↔ normal, normal ↔ low). Explain why this pattern is less bad than the accuracy suggests, and which mistake still matters most.

Show the answer

Priority is a matter of degree, so a message between normal and urgent can fairly go either way; two people could label it differently. A low message read as normal (265) only moves up a queue a little. Urgent and low are almost never swapped (9 and 4). The mistake that still matters most is an urgent message read as normal: 134 of 372. Each one waits like a normal message, and the cost table prices an urgent message in the wrong queue at US$20.

Check your understanding

Predict: you add the urgent-words rule to the priority model (model or rule). Will recall rise a lot? What does the rule add, if not recall?

Show the answer

No. On this inbox, recall rises only from 0.616 to 0.632. The rule finds 127 urgent messages, and the model already marks all but 6 of them. The rule adds certainty and a clear reason. A message that says “today” always goes first, even after a retraining, and an agent can see why. Its precision is high (0.962), so it adds very few false alarms.

Check your understanding

In practice: your language detector disagrees with the stored language on 0.25% of messages. What do you do before you report it as the detector's error rate?

Show the answer

Read the messages where they disagree. On Larkfield’s test set, all 5 disagreements were label errors: four messages labelled en were written in French, and one labelled fr was in English. If the detector is right, correct the labels and report that the stored field is wrong, not the detector. Then use the detector to choose the French or English queue.

Recap

  • Route, priority and language are three separate questions, each with its own model and the same messages.
  • The priority model reaches 0.714 on this inbox; urgent recall 0.616, precision 0.895. Its mistakes are mostly between neighbours.
  • An urgent-words rule has precision 0.962 and recall 0.341. Model or rule: recall 0.632, precision 0.883. The rule adds certainty more than recall.
  • A language detector on letter groups agrees with the label on 0.9975 of test messages, and its 5 disagreements are label errors. The router reaches 0.966 on English and 0.952 on French.

You can now answer who, how soon and in which language. But each team also has its own topics, and the route and topic labels form a tree. Next, we ask whether a model should follow that tree.

Next: Hierarchical labels