Skip to content
Français

Labelling guidelines

  • Lesson 5
  • 30 min
  • Module 2: Labels you can trust

Where we are

We ran the first comparison, and every model learned from the labels we gave it. If two people label the same message in two ways, the model learns both. In this lesson, we write labelling guidelines: a rule and an example for each label, and for the hard cases between two labels. A post office with two new sorters shows the problem.

Lesson 5 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceLabelling guidelines

What you will be able to do

Write down how each message gets its label. Then two people give the same label to the same message, and the model learns one rule instead of two.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

  • Nothing new. If you want to read the full guidelines in a text editor, open data/guidelines.md in your project folder. The file comes with the data that python get_data.py downloaded.

This lesson has no code. You label ten real messages from the inbox by hand.

Think of a small post office. Two new workers sort the letters into bags, one bag for each town. On the first day, nobody gives them a rulebook. Most letters are easy: the town is on the envelope.

Then a letter arrives with two addresses: a home address in one town, and “please forward to” another town. The first worker puts it in the bag of the home town. The second worker puts the same kind of letter in the bag of the forward town. Both workers are careful. Both have a reason. But the post office now sends the same kind of letter to two places.

The head of the post office does not fix this by telling the workers to try harder. She writes one line in a rulebook: “Two addresses: use the forward address.” From the next day, both workers do the same thing.

Larkfield’s models learn from labelled messages. A label is the right answer for one message: here, its route. A model looks at thousands of labelled messages and learns which words lead to which route. So the labels are the model’s only teacher.

Now imagine that Chloé and Grace label messages without a rulebook. A customer writes: “I returned the chair two weeks ago. I still see the charge on my card.” Chloé reads “returned” and labels it returns_refunds. Grace reads “charge” and labels it billing. Over a few weeks, they each label dozens of messages like this one.

One message at the top: I returned the chair two weeks ago, I still see the charge on my card. An arrow goes to Chloé, who labels it returns_refunds, and an arrow goes to Grace, who labels it billing. Both arrows lead to the training data, which has the same kind of message with two different labels. An arrow leads to the model, which learns a coin flip: about half go to each team.
A practice example we made on purpose. Two careful people, no rule: the training data teaches the model both answers.
Text description of the diagram

The diagram shows one constructed message at the top: “I returned the chair two weeks ago. I still see the charge on my card.” Two arrows go down. One goes to Chloé, who labels it returns_refunds. The other goes to Grace, who labels it billing. Both labels go into the training data, which now has the same kind of message with two different labels. The last arrow goes to the model. It learns a coin flip: about half of these messages go to each team.

What does the model learn from this? It learns that such messages are sometimes returns_refunds and sometimes billing. When a new message of this kind arrives, it guesses. Half of the customers wait in the wrong queue. Each wrong queue costs at least US$4 in the course’s cost table (an assumption). No change to the model fixes this. The confusion is in the labels, so it must be fixed in the labels.

This is not a rare case. In the next lesson, you measure it: when Chloé and Grace labelled the same 800 messages, they gave different routes to 16% of the billing messages. The commonest disagreement was exactly this pair, billing and returns_refunds.

Labelling guidelines are the rulebook for the people who label. They say, for each label, which messages it covers, with an example. Most importantly, they say what to do in the hard cases: the messages that fit two labels.

Good guidelines have four parts:

  1. One rule above all the others. It says what a label means. For Larkfield, the label is the team that must act first. So the first rule is: label the request, not the words. Ask what the customer wants Larkfield to do.
  2. A rule and an example for each label. The rule says what the label covers. The example is a real message, so the reader sees the rule at work.
  3. A rule for each hard case between two labels. It names the two labels, says which one to choose, and gives the reason.
  4. What to do when you are not sure. Without this part, each person invents their own habit, and the habits differ.

Take the hard case from the diagram above. Here are three short messages. Before you read on, decide on a route for each one.

Message Words that pull What the customer wants
A. “I returned the chair two weeks ago. I still see the charge on my card.” returned, charge Money back for a return
B. “There are two charges of $89.99 on my card for one order.” charges A wrong charge fixed
C. “Can I return the lamp if the colour is wrong? I have not ordered yet.” return Information before buying

Message A has the word “charge”, but the customer wants the refund for a return. The returns team tracks returned items, so the route is returns_refunds. Message B is a charge that is wrong: two charges for one order. The billing team fixes charges, so it is billing. Message C has the word “return”, but nothing was bought yet. It is a question about the return policy, so it is product.

The words pulled A toward billing and C toward returns_refunds. The rule “label the request, not the words” gave a different answer for both. Chloé turned this into a short decision path, the same for everyone:

A decision flow. First question: is the main request about safety, a lawyer, the press or a manager? Yes leads to needs_person. No leads to the second question: did the customer send an item back, or want to? Yes leads to returns_refunds, refund after a return. No leads to the third question: is a charge wrong, twice, too much, or a failed payment? Yes leads to billing. No leads to another route: read the route table.
The hard case between returns_refunds and billing, as three questions in a fixed order. The first question is about the main request.
Text description of the diagram

The flow starts with a question: is the main request about safety, a lawyer, the press or a manager? If yes, the route is needs_person. If no, the second question asks: did the customer send an item back, or want to? If yes, the route is returns_refunds, for example a refund after a return. If no, the third question asks: is a charge wrong, for example twice, too much, or a failed payment? If yes, the route is billing. If no, the message belongs to another route, and the labeller reads the route table.

The order of the questions is part of the rule. The needs_person question comes first, because a team answer to a safety problem or a legal threat makes things worse. But it asks about the main request, not about every sentence. The section on passing mentions below explains this choice.

Chloé wrote Larkfield’s guidelines with Grace, the support lead. Grace owns the six teams, so she knows which team must act first. The full text is data/guidelines.md in your project. Here it is on one page:

A one-page card titled Larkfield labelling guidelines: one route per message. The first rule: label the request, not the words. Two requests: label the main one. Then seven rows, one per route, each with a rule and an example: orders, delivery, returns_refunds, billing, product, account and needs_person. Then six hard cases between two routes, and at the bottom a warning: not sure, write your best guess and mark the message for review.
Larkfield's labelling guidelines as one card: the first rule, one rule and one example per route, the hard cases, and what to do when you are not sure.
Text description of the diagram

The card has a blue title: Larkfield labelling guidelines, one route per message. Under it, the first rule: label the request, not the words; what does the customer want Larkfield to do? A second line says: two requests, label the main one, usually the first or the urgent one.

Then seven rows, one for each route, with a rule and an example. orders: change or cancel an order before it ships; “Cancel LK-123456, I ordered it twice.” delivery: where a parcel is, late, lost, delivered but not received; “Tracking says delivered, nothing at my door.” returns_refunds: return, exchange, arrived damaged or wrong, refund after a return; “I sent the chair back. When do I get my money?” billing: a double or wrong charge, a failed payment, an invoice, a code or a gift card; “Two charges of $89.99 for one order.” product: a product question before or after buying, and the warranty; “Can the bench stay outside in winter?” account: login, details, emails, and someone else using the account; “Someone changed my email and placed an order.” needs_person: the main request is about safety, a legal or press threat, or a manager; “The heater started smoking.”

Then the hard cases between two routes. Refund after a return goes to returns_refunds; a wrong charge goes to billing. Arrived damaged or wrong goes to returns_refunds; did not arrive goes to delivery. A question about the return policy before buying goes to product; a return after buying goes to returns_refunds. Broke after weeks of use goes to product; broken on arrival goes to returns_refunds. Not shipped yet goes to orders; shipped goes to delivery. When the main request is about safety, a lawyer, the press or a manager, it goes to needs_person. When the main request is that someone else used the account, it goes to account. When one of them is only mentioned in passing, the message keeps the main request’s route. It gets a higher priority and a flag for a person.

At the bottom, on an orange background: not sure? Write your best guess and mark the message for review. Never use a route as a “not sure” label.

Four choices on this card need a word of explanation.

About one message in eight in the inbox asks for two things. For example: “My card was charged more than the order total. I would also like to return the sun lounger.” The course gives each message one route: the team that acts first. The guidelines say to label the main request, usually the first one, or the urgent one. That sends this message to billing. The second request is not lost: the data keeps it in the column routes. Later, Multi-label messages in Module 7 routes some of these messages to two teams.

Section titled “Safety, legal threats and takeovers mentioned in passing”

Some messages mention a serious problem in passing. Here are two from the inbox:

Message (shortened) Route in the data
“I’m still waiting for order LK-940779, and it’s been 10 days. If this isn’t resolved, I may need to contact a consumer protection office.” delivery
“Order LK-861033 was due on Nov 14 and still hasn’t arrived. Also, my son got hurt on a sharp edge; and SPRING15 didn’t work on LK-189512.” delivery

Both mention something from the needs_person list: a legal threat, a child who got hurt. But the main request of both is a late parcel. Larkfield’s team made a decision here, and wrote it on the card:

  • The route follows the main request. A message whose main request is safety, a legal threat or an account takeover goes to needs_person or account.
  • A passing mention does not change the route. It raises the message’s priority, and the labeller flags it, so a person reads it.

Why this choice? Grace wants each team to own the requests it can solve, and the delivery team can find the parcel. The rule also keeps one meaning for every label: the team that acts first.

The decision has a risk, and the team accepted it knowingly. A model trained on these labels sends such a message to a team, not to a senior person. If the flag or the higher priority is lost on the way, a team answers a message that mentions an injury. The cost table prices a needs_person message sent to a team at US$40, ten times a normal wrong queue. So the flag is part of the rule, not an extra. Later in the course, the router uses priority and a review zone to catch these messages.

Without a rule, a person who is not sure often picks the route with the most messages, or the one the team can move later. Both habits teach the model something false. So the guidelines say: write your best guess, and mark the message for review. The mark is separate from the label. Later lessons use these marks to see where the guidelines are still unclear.

The inbox is in English and French. The guidelines do not change with the language: a refund after a return is returns_refunds in both. Product names stay in English, because they are names from Larkfield’s catalogue. Chloé is bilingual, so she labels both.

Here are ten messages from the inbox. They are synthetic: we made them for this course. Label each one with the card above. Write your ten routes down before you open the answer.

# Message (shortened)
1 “My corner sofa broke after 13 days. One part gave way. Would that be covered by the warranty?”
2 “Which courier delivers large items like the garden hose?”
3 “The password reset says my email isn’t registered, even though I’ve placed orders before.”
4 “I’m upset to see a second charge of $232.35 on my statement. Why was I charged twice? I was looking at your return policy before buying.”
5 “J’ai retourné le produit water bottle il y a 6 jours, mais je n’ai pas encore reçu le remboursement.” (I returned the water bottle 6 days ago, but I have not received the refund yet.)
6 “My son cut his hand on the folding chair because the edge is very sharp. Could you let me know what can be done?”
7 “My card was charged $204.52, more than the order total. Please fix this. I’d also like to return the sun lounger: it doesn’t fit on my balcony.”
8 “Can the courier leave my parcel LK-808134 with my neighbour at number 12? Also, please update my phone number and my default address.”
9 “Ça fait trois courriels et deux clavardages sans réponse. Je veux parler à un superviseur. Aussi, ma commande LK-559315 n’est toujours pas arrivée.” (Three emails and two chats with no answer. I want to speak to a supervisor. Also, my order has still not arrived.)
10 “Could you please take care of the thing I mentioned in my last email for LK-174207? Also, do you know when that item will be back in stock?”
Show the reviewed routes
# Reviewed route The rule that decides
1 product Broke after some days of use, and the customer asks about the warranty. Broken on arrival would be returns_refunds.
2 delivery The guidelines put delivery costs and areas in delivery. A question about the courier is the same kind of question, even before an order.
3 account Login and password. No sign of someone else in the account, so it is not a takeover, but it is still account.
4 billing A double charge. “Return policy” is in the message, but the customer does not want a return.
5 returns_refunds A refund after a return. The rules are the same in French.
6 needs_person The main request is about a child who was hurt: a safety problem.
7 billing Two requests. The first one, a wrong charge, is the main request. The data keeps the second route, returns_refunds, in routes.
8 delivery Two requests. The first one is about the parcel. The second route, account, is kept in routes.
9 needs_person The main request, written first, is a supervisor after several tries. The late order is the second request.
10 billing This is the reviewed label, but nothing in the text points to billing. The message refers to an earlier email. A careful labeller writes a best guess and marks it for review.

If you agreed on 8 or more, the card works for you. Message 10 is the honest case: no guideline can decide it from the text alone. That is why the guidelines say what to do when you are not sure.

Where you disagreed, find the line on the card that decides. If no line decides, you found a gap in the guidelines. Real guidelines grow this way: each disagreement that the card cannot settle becomes a new hard case.

Message 8 shows a pair that the card does not cover: delivery and account. A customer asks the courier to do something, and also asks to change the default address in the account.

Write one hard-case line for the card in the same shape as the others: the two routes, which one to choose, and why. Use the cost table to decide which mistake is worse.

Show one possible answer

“A request about a parcel that has shipped and a change to the saved address → delivery. The parcel is moving now; the account change can wait. If the customer says that someone else changed the address, it is an account takeover → account.”

The second sentence matters most. A missed account takeover costs US$200 in the course’s cost table, the highest cost of all. So the rule names the one sign that changes the answer.

Write guidelines before anyone labels, and keep them next to the labels in the same project. Change them when a disagreement shows a gap, and give each version a date or a number. A model trained on labels from version 1 learned version 1’s rules.

Guidelines cost time. Someone must write the first version with the team leads, and each new hard case costs another short talk. A rule that nobody reads costs the same and helps nothing, so keep the guidelines short: one page, with the full text for reference.

Guidelines do not replace a check. They make agreement possible; they do not prove it. The next lesson measures how often two people who follow these guidelines really agree.

Common mistake

A guideline that is a keyword rule

Symptom: To save time, a team writes its guideline as a list of words: “If the message says refund or return, label it returns_refunds.” Labelling is fast, and two people who follow it agree on almost everything.

Cause: The guideline labels the words, not the request. It is Tomás’s keyword rule from A message in the wrong queue, written down for people. A question such as “Can I return the lamp if the colour is wrong?” before an order gets returns_refunds. People who follow a keyword guideline copy the mistake into the labels, and the model learns it.

Fix: Start each rule from what the customer wants Larkfield to do. Use the words only as examples. Add the hard case “a question about the return policy before buying → product”.

Consequence in AI work: High agreement between people does not prove that the labels are right. Two people can agree perfectly on a wrong rule. Then every model, from keyword rules to an LLM, learns the wrong rule from the labels, and the score on those labels looks good.

Check your understanding

Two careful people label the same kind of message in two different ways. Why can no change to the model fix this, and what fixes it?

Show the answer

The model learns only from the labels. If the training data has the same kind of message with two labels, the model learns that both are possible. Then it guesses on new messages. A larger or newer model learns the same confusion. The fix is in the labels: a written rule for that hard case, so that both people choose the same route, and then labels that follow the rule.

Check your understanding

Label this message with Larkfield's guidelines, and name the rule that decides: “I want to return the garden bench, it arrived with a broken leg. Also, my daughter scratched her arm on a loose screw.”

Show the answer

returns_refunds, with a higher priority and a flag. The main request is a return of an item that arrived broken: broken on arrival is returns_refunds. The injury is mentioned in passing, so the route follows the main request. The team’s guideline says that a passing mention of safety raises the priority and is flagged, so a person reads the message. If the main request were the injury, the route would be needs_person.

Check your understanding

A labeller is not sure about a vague message and labels it with the biggest route, delivery, “because it can be moved later”. What is wrong with this habit, and what do the guidelines say instead?

Show the answer

The label teaches the model that vague messages are delivery messages, which is false. Another labeller may use another habit, so the labels disagree too. The guidelines say: write your best guess from the facts in the message, and mark the message for review. The mark is separate from the label, so it shows where the guidelines are still unclear.

Recap

  • A model learns only from its labels. If two careful people label the same kind of message in two ways, the model learns both and guesses.
  • Labelling guidelines fix this: one rule above the others (“label the request, not the words”), a rule and an example for each route, a rule for each hard case between two routes, and what to do when you are not sure.
  • The hard case returns_refunds or billing becomes three questions in a fixed order. The first one asks if the main request needs a person.
  • The route follows the main request. A safety problem, legal threat or takeover mentioned only in passing raises the priority and is flagged; this is a team decision with a known risk.
  • A message with two requests gets the main one; the second route stays in routes. “Not sure” is a mark for review, never a route.
  • A guideline written as a list of words is a keyword rule. People agree on it, and it is still wrong.

You can now write guidelines that two people can follow. But do they really give the same labels? Next, Chloé and Grace label the same 800 messages, and we measure how often they agree.

Next: Measure agreement