Skip to content
Français

A message in the wrong queue

  • Lesson 1
  • 25 min
  • Module 1: The routing problem and its cost

Where we are

In the machine learning course, we predicted whether a ticket would be escalated, from a table of numbers. Larkfield's support inbox asks a different question: which team must read this message? In this lesson, we follow one refund request that went to the wrong queue, and we see what each day of delay cost.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 1 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceA message in the wrong queue

What you will be able to do

See why sorting support messages is a business problem with a price, not only a text problem. One message in the wrong queue costs work, time and sometimes a customer.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. You open the data in Meet the inbox, the last lesson of this module.

Think of a small post office early in the morning. A worker stands in front of a wall of trays. Each tray belongs to one delivery van, and each van serves one part of the town. The worker takes a letter, reads the address, and drops the letter in the right tray. She does this hundreds of times before the vans leave.

Now suppose she drops one letter in the wrong tray. Nothing breaks. The letter goes out on the wrong van. The driver sees the address, cannot deliver it, and brings it back in the evening. Tomorrow it goes out again on the right van. The letter arrives one day late, and two people spent time on it.

Most letters are easy: the address is clear. Some are not. The writing is bad, or the street name exists in two parts of the town. A good post office has a small desk for these letters. A person with more experience reads them first and decides.

Keep this picture in mind. A support inbox works in the same way, and the rest of this course builds the worker who sorts.

Larkfield is an online shop for home and garden products in Canada. You may know it from other courses. Its customers write to the support inbox by email, by chat and on social media, in English or in French. In this course’s data, the inbox receives about 65 messages a day. In this course, every Larkfield message is synthetic: we made the data for the course, and it has no real people.

On Thursday 29 January 2026, at 16:54, a customer wrote this in the chat:

hi, about the cushion set I sent back weeks ago… could you check what’s happening with that? see my last email, maybe it got missed.

This is message M-109856 in the course data. The customer returned a cushion set and wants to know about her refund. So the right team is the returns and refunds team. But Larkfield’s old system put the message in the product team’s queue. A queue is the list of messages that one team must answer, oldest first.

Here is what happened next. The data tells us the queue; the days are a story that we made up to show the cost.

What happened (a story): on Thursday at 16:54, the customer asks about her refund by chat. An arrow leads to the product team's queue, where the message waits for three days. An arrow leads to Sunday: a product agent reads it, sees a refund question, and moves it in 2 minutes. An arrow leads to the returns team's queue: an answer on Monday, but the customer has already left. A dashed arrow, what should happen, goes from the first box straight to the right queue: the returns team reads it on Friday and answers. A note says that the cost table counts US$4 for a message in the wrong queue: about 2 minutes of work and one more day of waiting, on average. This story is a bad case.
The trip of message M-109856. The wrong queue is real (the old system's queue in the data); the days are a story. The cost table counts an average wrong queue as US$4.
Text description of the diagram

The diagram has two rows. The top row, “What happened (a story)”, has four boxes joined by arrows. First box: Thursday at 16:54, the customer asks about her refund by chat. Second box, with a thick border: the product team’s queue, where the message waits for three days. Third box: on Sunday, a product agent reads it, sees a refund question, and moves it to the returns team in 2 minutes. Fourth box, with a thick border: the returns team’s queue, with an answer on Monday, when the customer has already left. The bottom row, “What should happen”, has a dashed arrow from the first box to the right queue. There, the returns team reads the message on Friday and answers. A note under the diagram says that the course’s cost table counts US$4 for a message in the wrong queue. That is about 2 minutes of work and one more day of waiting, on average. This story is a bad case.

Who touches the message, and what each touch costs

Section titled “Who touches the message, and what each touch costs”

Count the people. The customer writes once and waits. A product agent opens the message, reads it, sees that it is not for her team, and moves it. A returns agent reads it again and answers. One person’s work became two. The customer waited four days for an answer that the right team could give the next day.

Each step has a cost:

  • The extra reading. The product agent spent about 2 minutes on a message that her team cannot answer.
  • The extra waiting. The message lost at least one day in a queue where nobody could help.
  • The customer. In our story, she asked her bank to get the money back, and she does not buy from Larkfield again. That is the most expensive part, and the hardest to count.

Grace, Larkfield’s support lead, owns the six teams. Priya is the data scientist who builds the classifiers in this course. Grace and Priya agreed on one number for a message in the wrong queue: US$4. It covers about 2 minutes of work and one more day of waiting, on average. It is an assumption: a value they chose for the course, with a reason, not a measurement. Most wrong-queue messages cost little. Some, like our story, cost a customer. The average hides that spread, so the course also gives more expensive mistakes their own, higher cost. The lesson The cost of each mistake shows them all.

Read the message again. It names a product, “the cushion set”. It does not say “refund” or “return”. It says “sent back”. A reader who looks for product names sees a product question. A reader who asks “what does the customer want us to do?” sees a refund question.

This is the first rule of Larkfield’s labelling guidelines: label the request, not the words. In the old system’s data, 12 training messages of the returns team ended up in the product queue, like this one. The rule sounds simple. Most of this course is about how hard it is to follow for every message.

Larkfield’s inbox has seven trays, called routes. Six of them are teams. The seventh is the small desk of the post office: a person must read the message first.

A new message, by email, chat or social media, in English or French, goes to a box: sort it, which team must act first? From there, arrows lead to seven trays: orders, change or cancel an order before it ships; delivery, where is my parcel; returns_refunds, a return, a damaged item, a refund after a return; billing, a wrong charge, an invoice, a promotion code; product, a question about a product or its warranty; account, login, personal details, someone else in my account; needs_person, a person reads it first: safety, legal, a manager. The needs_person tray has a thick border. A note says: 6 teams and 1 route for a person.
The seven routes of Larkfield's inbox: six teams, and needs_person for the messages that a person must read first.
Text description of the diagram

On the left, a new message arrives by email, chat or social media, in English or French. An arrow leads to a box that asks: which team must act first? From that box, seven arrows lead to seven trays. Orders: change or cancel an order before it ships. Delivery: where is my parcel? Returns and refunds: a return, a damaged item, or a refund after a return. Billing: a wrong charge, an invoice or a promotion code. Product: a question about a product or its warranty. Account: login, personal details, or someone else in my account. The last tray, needs_person, has a thick border. A person reads these messages first: a safety problem, a legal threat or a request for a manager. A note under the sorting box says: 6 teams and 1 route for a person.

The route names in the data are orders, delivery, returns_refunds, billing, product, account and needs_person. The last one is special. A message whose main request is about an injured child or a lawyer must not get a standard team answer. It goes to a senior person first.

Sorting messages into routes is text classification: a model reads a text and gives it one label from a fixed list. The machine learning course classified tickets from a table of numbers. Here, the input is the words that a customer wrote, in two languages, with typos and stories. When the label decides where the text goes next, we call the job routing, and the model a router.

Larkfield today: a person reads everything

Section titled “Larkfield today: a person reads everything”

Today, Larkfield does not trust a program to sort. A person reads every new message first and moves it to a team. This works, and it costs money. Grace’s number for one such reading is US$1.50. That pays about 1 minute of an agent at US$28 an hour. Larkfield pays its staff in Canadian dollars, but the course gives every cost in US dollars, like the model prices. The message also waits a few hours before it reaches its team. It is an assumption too.

A tiny calculation shows the scale:

Messages Cost
One message read by a person 1 US$1.50
One day at Larkfield about 65 about US$98
Per 1,000 messages 1,000 US$1,500

The course always gives costs per 1,000 messages, so that you can compare methods that were tested on different numbers of messages. US$1,500 per 1,000 messages is the row called “everything to a person”. It is the price of Larkfield today, and every method in this course must beat it.

The reader also makes mistakes, as the old queue showed. In the cost of “everything to a person”, the course counts every reading as right. This makes Larkfield today look a little better than it is. Module 2 measures how often two careful people disagree.

Tomás, Larkfield’s AI engineer, has an idea at once: “Send every message to a large language model and let it choose the team.” A large language model (LLM) is a model trained on a huge amount of text: it reads a request and writes an answer. Grace asks a different question: “When it is wrong, what does that cost me?” Omar, the product manager, adds a third one: “And what happens when we launch a new product that no team has seen?”

All three questions are fair, and the course answers each of them with numbers. You build a router that:

  • sends most messages straight to the right team;
  • sends the unsure ones to a person, on purpose;
  • never gives a standard team answer to a message that needs a person;
  • costs less per 1,000 messages than a person reading everything.

You compare many kinds of router on the same messages and the same costs. They range from keyword rules and classical models to embedding models, a small fine-tuned model and large language models. Each one adds a row to one comparison table. The cheapest row wins, not the most famous method.

When an automatic router is not the answer

Section titled “When an automatic router is not the answer”

A router is a system that people must build, test and watch. It is not always worth it:

  • Few messages. With 20 messages a day, one person can read them all in half an hour. A router saves too little to pay for its work.
  • Every message needs a person. A legal team that must read every letter gains nothing from a router.
  • The teams change every month. A router learns from past labels. If the labels change often, it is always out of date.

Larkfield has about 65 messages a day, six stable teams, and a clear cost for each mistake. That is a good case for a router.

Common mistake

A keyword that looks obvious

Symptom: Tomás writes one quick rule: “If a message contains refund or return, send it to returns_refunds.” He tries it on a few messages, and it looks right.

Cause: the customer of M-109856 wrote “sent back”, not “return”. The rule does not match, so her message still needs a person. The rule also fails in the other direction. A customer who asks “If I buy the hammock and don’t like the colour, can I return it?” has not bought anything yet. The guidelines say that this is a product question. In the course’s test messages, 36 of the 144 product questions mention a return or a refund. The course’s keyword rules send 32 of them to returns_refunds.

Fix: label the request, not the words. A rule can help when a word always means one thing, such as an order number or “charged twice”. Module 3 measures where rules help and where they break.

Consequence in AI work: a rule tried on five messages looks perfect. On thousands, it moves whole groups of messages to the wrong team, and nobody notices until customers complain.

The Larkfield inbox lets you learn each method on one familiar set of messages. Real work has other data and other traps. The course ends with case studies on real, public datasets, and each one goes from the raw data to a recommendation.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

  • Route banking questionsCompare TF-IDF, embeddings, a small fine-tuned encoder and a replayed LLM on 77 fine-grained banking intents, and build a cascade with a cost target.
  • Find what the bot cannot handleDetect out-of-scope requests and send them to a person, on an intent dataset with an out-of-scope set, and measure how many wrong automatic answers this prevents.
  • Sort requests with few labelsStart with 20 labels per intent on real open requests in English and French, add labels with active learning and LLM-assisted labelling with review, and show the learning curve and the labelling cost.
  • Tag research papers with many labelsOptional: give each paper abstract all the subject labels that fit, with one model and one threshold per label, and report the results per label.

Read the list as a map of where the course goes. The case studies use the whole course, so most learners start them after Module 9.

Check your understanding

Message M-109856 says “the cushion set I sent back weeks ago … could you check what’s happening with that?” Which route is right, and why did the old system get it wrong?

Show the answer

The right route is returns_refunds. The customer returned an item and asks about the refund, so the returns team must act. The message names a product (“the cushion set”) and does not use the words “refund” or “return”. A reader or a program that looks at the words, not at the request, sees a product question. Larkfield’s guidelines say: label the request, not the words.

Check your understanding

Larkfield receives about 65 messages a day, and a person reads each one first at US$1.50. What does this cost per day and per 1,000 messages? Why does the course give costs per 1,000 messages?

Show the answer

About 65 × 1.50 = US$98 a day, and 1,000 × 1.50 = US$1,500 per 1,000 messages. A cost per 1,000 messages does not depend on how many messages a test used. So you can compare two methods even if one was tested on 2,000 messages and another on 130. US$1,500 per 1,000 is the “everything to a person” row: the price of Larkfield today, which every router must beat.

Check your understanding

Give one situation where an automatic router is not worth building, and say why.

Show the answer

Any of these. Very few messages: a person reads 20 messages a day in half an hour, so a router saves too little. Every message must be read by a person anyway, for example legal letters. Or the teams and labels change so often that a router trained on past labels is always out of date. A router has a cost too: people must build it, test it and watch it.

Recap

  • Routing is text classification with a consequence: the label decides which team reads the message. Larkfield has seven routes: six teams and needs_person.
  • Message M-109856 asked about a refund, and the old system put it in the product team’s queue. In the course’s cost table, a message in the wrong queue costs US$4 (an assumption). That is about 2 minutes of work and one more day, on average.
  • Label the request, not the words. The message said “sent back”, not “refund”.
  • Larkfield today: a person reads every message first, at US$1.50 each. That is US$1,500 per 1,000 messages. Every router in the course must beat this row.
  • A keyword rule tried on a few messages looks perfect, and fails on whole groups of messages.

You can now say what a routing mistake costs Larkfield. But “which team?” is only one of the questions a message asks: it also has a topic, a priority and a language. Next, you design the labels, and you decide which questions get their own label.

Next: Design the labels