Priority, urgency and language
- Lesson 30
- 30 min
- Module 7: Harder label shapes
Where we are
We route each message to the right team. The team also needs to know how urgent it is and which language it is in. In this lesson, we train priority as a separate model, detect the language, and add rules for words that always mean urgent.
Lesson 30 of 37 · Text Classification and Routing: Send Each Message to the Right PlacePriority, urgency and language
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Give each message a priority and a language next to its route, so the right person answers it first and in the right language. The route alone says who answers, not how soon or in which language.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It trains a priority model, tests a rule for urgent words and combines the two. Then it trains a language detector, reads its disagreements, and compares the router on English and French messages. Its setup cells install pandas and scikit-learn and download the practice data, so it needs no local setup. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Precision and recall, from Classification metrics in the machine learning course. A confusion table counts, for each true label, how often the model gave each answer.
- Letter groups (character n-grams), from Text as numbers: bag of words and TF-IDF.
- Regular expressions, from Keyword rules and regular expressions.
- Naive Bayes, from Naive Bayes, logistic regression and linear SVM.
- The guideline for passing mentions, from Labelling guidelines.
What you need on your computer: nothing for this lesson. The notebook runs in Colab, Kaggle or VS Code. The project does not change in this module.
Tested on macOS with Python 3.12, pandas 3.1.1 and scikit-learn 1.9.1.
Three questions about one letter
Section titled “Three questions about one letter”At the post office, the clerk asks three questions about each letter, and each has its own sign. The address says which van takes it. A red express stamp says that it goes out first. In a town where people speak two languages, the language on the envelope says which counter calls the customer back.
The three questions are independent. An express letter can go to any part of the town. A letter in French can be urgent or not. So the clerk does not look for one combined sign, such as “north van, express, French”. She answers each question on its own.
The problem: the route says who, not how soon
Section titled “The problem: the route says who, not how soon”Larkfield’s router answers the first question: which team. The teams need the other two answers as well.
- Priority. Each message is
urgent,normalorlow. In the test set, 372 messages are urgent, 1,275 normal and 353 low. An urgent message that waits in a normal pile is expensive. The cost table prices an urgent message in the wrong queue at US$20, five times a normal one. - Language. 29% of the test messages are in French. Larkfield has French-speaking agents, such as Chloé. A French message that reaches an agent who reads only English waits, or gets a poor answer.
Why not one combined label, such as delivery-urgent-fr? Seven routes × three priorities × two languages make 42 labels. Many of them would have very few training messages: the test set has only one urgent product message. Three small questions, each with plenty of examples, are easier to learn.
The idea: one model per question
Section titled “The idea: one model per question”So each question gets its own model, trained on the same messages with a different label column. The team model reads route, the priority model reads priority, and the language detector reads language. Their three answers then decide the queue together.
Text description of the diagram
On the left, a message in French: “Bonjour, mon colis LK-… devait arriver hier, j’en ai besoin aujourd’hui pour un cadeau” (“Hello, my parcel LK-… should have arrived yesterday, I need it today for a gift”). Three arrows lead to three models. The team model, with 7 routes, answers delivery. The priority model, with the urgent-words rule, answers urgent. The language detector answers fr. The three answers lead to one box: French-speaking delivery agents, top of their queue.
Priority as its own model
Section titled “Priority as its own model”Priya trains a priority model: TF-IDF with logistic regression, as for the route, but on the priority column. Its accuracy on the test messages is 0.714. The router reaches 0.962 on this inbox, so priority is much harder. Why? The confusion table shows where the mistakes are.
Text description of the diagram
A table with three rows and three columns. The rows are the label; the columns are the model’s answer, in the order urgent, normal, low. Row urgent (372 messages): 229 answered urgent, 134 normal, 9 low; recall 0.616. Row normal (1,275): 23 urgent, 1,114 normal, 138 low; recall 0.874. Row low (353): 4 urgent, 265 normal, 84 low; recall 0.238. Under the columns, the precision of each answer: urgent 0.895, normal 0.736, low 0.364. The cells on the diagonal are right answers. The two largest mistakes are 134 urgent messages read as normal, and 265 low messages read as normal.
Read it row by row. Almost all mistakes are between neighbours. An urgent message is read as normal (134), or a low one as normal (265). Urgent and low are almost never swapped (9 and 4). That makes sense: urgency is a matter of degree. “I need it by Friday” sits between normal and urgent, and two people could label it differently.
For Grace, one row matters most: the urgent one. Out of 372 urgent messages, the model finds 229: urgent recall 0.616. When it says “urgent”, it is right 229 times out of 256: precision 0.895. So it is careful, and it misses more than a third of the urgent messages.
A rule for words that always mean urgent
Section titled “A rule for words that always mean urgent”Some words almost always mean urgent: “urgent”, “today”, “immediately”, “right now”, and in French aujourd’hui, tout de suite, urgence. A regular expression finds them:
URGENT_WORDS = re.compile(r"\b(?:urgent\w*|asap|today|tonight|immediately|right now|urgence|aujourd'hui|" r"imm[ée]diatement|tout de suite|vite)\b", re.I)rule_urgent = test["text"].str.contains(URGENT_WORDS).to_numpy()The rule matches 132 test messages. 127 of them are urgent: precision 0.962. But it finds only 127 of the 372 urgent messages: recall 0.341. Many urgent customers never write “urgent”. They write “my order is missing and it was a gift for tomorrow”.
Every value: the model or the rule
Section titled “Every value: the model or the rule”Grace asks: “Mark a message urgent if the model says so or the rule matches.” Count it:
- The model marks 256 messages, 229 right. The rule matches 132, 127 right.
- Most rule matches are already marked by the model. Together, they mark 266 messages, of which 235 are urgent.
- Recall: 235 ÷ 372 = 0.632. Precision: 235 ÷ 266 = 0.883.
Text description of the diagram
Three pairs of horizontal bars. Blue bars are recall, out of the 372 urgent test messages; orange bars are precision. Urgent-words rule: recall 0.341 (127 of 372), precision 0.962 (127 of 132). Priority model: recall 0.616 (229 of 372), precision 0.895 (229 of 256). Model or rule: recall 0.632 (235 of 372), precision 0.883 (235 of 266).
The rule adds only 6 urgent messages that the model missed. So why keep it? Because it is certain and easy to explain. If a customer writes “today”, the message goes first, and an agent can see why. A model can lose a word like that after a retraining. A rule cannot.
Passing mentions raise the priority
Section titled “Passing mentions raise the priority”The Labelling guidelines set one more rule. A safety problem, a legal threat or an account takeover that a message mentions only in passing does not change its route. It raises the message’s priority, and a person flags it. So priority carries part of the safety net. The takeover flag of Imbalance and rare classes belongs here too: a flagged message goes to the top of its queue.
Detect the language
Section titled “Detect the language”The language question is much easier. A language detector does not need to understand a message. It looks at letter groups: ou, é, qu', the, ing. These groups are very different in English and French. A Naive Bayes model on groups of 1 to 3 letters is enough. Priya trains it on the language column of the 6,000 training messages.
Every value for one sentence
Section titled “Every value for one sentence”For the sentence “Mon colis n’est pas arrivé” (“My parcel has not arrived”), the detector gives fr 0.985 and en 0.015. The letters é, n', est and pas almost never appear in English messages.
On the 2,000 test messages, the detector agrees with the label 1,995 times: accuracy 0.9975. Priya reads the 5 disagreements, and the failure case below shows what she finds.
Do French messages get the same service?
Section titled “Do French messages get the same service?”The language column also lets Priya check the router on each language. This is called a slice: the same measure, on one part of the data. On the test messages of this inbox, the router reaches 0.966 on English (1,411 messages) and 0.952 on French (589). The French slice is a little weaker. The difference is 1.4 points on 589 messages, about 8 messages. Small, but she keeps watching it with every new model.
With the language known, the queue rule is simple: a French message goes to the French-speaking agents of its team. Chloé reads the French messages that need a person.
When to use separate models
Section titled “When to use separate models”- Use a separate model for each question whose answer does not depend on the others: route, priority, language. Each one has more training examples than a combined label.
- Add a rule for words that must always count, even if the model misses them: urgent words, a product recall, a legal word. Keep the rules few and easy to read.
- Do not trust priority as much as route. On this inbox, priority reaches 0.714 against 0.962 for the route. It is a matter of degree. Use it to order a queue, not to hide a message.
- A language detector is cheap and very accurate between two languages that look different. For languages that share most letters, or for very short messages, it can fail: test it on your own data.
- What it costs: three models instead of one, each with its own checks, and a few rules to maintain. Each trains in seconds here.
Guided practice: three answers in the notebook
Section titled “Guided practice: three answers in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The data cell ends with
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - In section 1, run the cell. You see accuracy
0.714and the confusion table. - In section 2, run the cell. You see urgent recall
0.616and precision0.895. - In section 3, run the rule. You see recall
0.341and precision0.962. - In section 5, run the language detector. You see
language accuracy 0.9975, 5 disagreements. - In section 6, run the cell. You see English
0.966and French0.952.
Your turn: the model or the rule
Section titled “Your turn: the model or the rule”In section 4 of the notebook, mark a message urgent when the model says urgent or the rule matches. Compute the recall and the precision of this combination, rounded to 3 decimals. Before you run it, predict: will the recall rise a lot?
Show the answer
both = model_urgent | rule_urgentboth_recall = round(float(both[urgent].mean()), 3)both_precision = round(float(urgent[both].mean()), 3)print(both_recall, both_precision)0.632 0.883The recall rises only from 0.616 to 0.632, and the precision falls a little, from 0.895 to 0.883. The model already finds most messages that contain an urgent word. The rule is still worth keeping for its certainty, not for its recall.
Common mistake
The detector's mistakes are the labels' mistakes
Symptom: the language detector disagrees with the label on 5 test messages. Tomás wants to report “5 errors” and try a larger model.
Cause: Priya reads the 5 messages. Four are labelled en but written in French, such as M-100856: “Bonjour, je n’ai toujours pas reçu mon toaster…”. One is labelled fr but written in English (M-108940). The detector is right every time. The labels are wrong. Here they are slips of the program that made this practice inbox. In a real inbox, they come from a language field that the customer chose, or from a default value.
Fix: correct the 5 labels before you trust the language slices. Then use the detector, not the stored field, to choose the queue.
Consequence in AI work: when a strong, simple model “fails” on a few examples, read them before you change the model. A disagreement is often a label error. This is the same idea as finding noisy labels in Find noisy labels.
Knowledge check
Section titled “Knowledge check”Check your understanding
The priority model is right on 0.714 of the test messages, and almost all its mistakes are between neighbours (urgent ↔ normal, normal ↔ low). Explain why this pattern is less bad than the accuracy suggests, and which mistake still matters most.
Show the answer
Priority is a matter of degree, so a message between normal and urgent can fairly go either way; two people could label it differently. A low message read as normal (265) only moves up a queue a little. Urgent and low are almost never swapped (9 and 4). The mistake that still matters most is an urgent message read as normal: 134 of 372. Each one waits like a normal message, and the cost table prices an urgent message in the wrong queue at US$20.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you add the urgent-words rule to the priority model (model or rule). Will recall rise a lot? What does the rule add, if not recall?
Show the answer
No. On this inbox, recall rises only from 0.616 to 0.632. The rule finds 127 urgent messages, and the model already marks all but 6 of them. The rule adds certainty and a clear reason. A message that says “today” always goes first, even after a retraining, and an agent can see why. Its precision is high (0.962), so it adds very few false alarms.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In practice: your language detector disagrees with the stored language on 0.25% of messages. What do you do before you report it as the detector's error rate?
Show the answer
Read the messages where they disagree. On Larkfield’s test set, all 5 disagreements were label errors: four messages labelled en were written in French, and one labelled fr was in English. If the detector is right, correct the labels and report that the stored field is wrong, not the detector. Then use the detector to choose the French or English queue.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Route, priority and language are three separate questions, each with its own model and the same messages.
- The priority model reaches 0.714 on this inbox; urgent recall 0.616, precision 0.895. Its mistakes are mostly between neighbours.
- An urgent-words rule has precision 0.962 and recall 0.341. Model or rule: recall 0.632, precision 0.883. The rule adds certainty more than recall.
- A language detector on letter groups agrees with the label on 0.9975 of test messages, and its 5 disagreements are label errors. The router reaches 0.966 on English and 0.952 on French.
You can now answer who, how soon and in which language. But each team also has its own topics, and the route and topic labels form a tree. Next, we ask whether a model should follow that tree.
Next: Hierarchical labels