Meet the inbox
- Lesson 4
- 35 min
- Module 1: The routing problem and its cost
Where we are
We have labels and a cost for each mistake, on paper. We have not opened the data yet. In this lesson, we set up the project, read the Larkfield inbox and its labelling guidelines, run the first comparison, and learn how to reset everything.
Lesson 4 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMeet the inbox
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Open the Larkfield inbox, check that your setup works, and run the course’s first comparison yourself. Every later module compares its methods on these same messages, with these same costs.
NotebookDo this lesson in a notebook
The notebook has every code cell, task and check of this lesson. It reads the inbox, the cost table and the first comparison. Its setup cells install pandas, scikit-learn and matplotlib and download the data (about 9 MB), so it needs no local setup. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The seven routes, the topics and the priority, from Design the labels.
- The cost table and the cost per 1,000 messages, from The cost of each mistake.
- How to run a cell in a notebook and read a pandas table. Machine Learning: From Problem to Reliable Model uses both from its first lessons.
What you need on your computer
- The notebook route (recommended for this lesson): a browser and a free Google account for Colab, or a free Kaggle account.
- The project route (needed from Module 2): Python 3.12, a terminal, and Git or a browser to download a ZIP file. The steps are in The project on your computer below.
Tested on macOS (Apple silicon) with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter and in the terminal. The Windows commands, Linux, Colab and Kaggle were not tested.
The first morning in the mail room
Section titled “The first morning in the mail room”A new worker joins the post office. Before she sorts a single letter, a colleague shows her around. How many letters come in each day? Which trays are full, and which are almost empty? Where is the sheet with the sorting rules? And how does the manager count her mistakes at the end of the week?
None of this is sorting. But a worker who skips it sorts by guesswork, and nobody can tell if she is good. This lesson is your first morning. You count the messages, read the rules, and run the scoreboard that every module uses.
Open the notebook and run the setup
Section titled “Open the notebook and run the setup”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the first setup cell. It installs the packages and prints
pandas 3.1.1andscikit-learn 1.9.1. - If you see other versions, restart the kernel, then run the cell again.
- Run the second setup cell. It downloads the inbox into a folder
larkfield-inbox/data, with the course’s helperrouting.py, and checks each file. - Check its last line:
Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic. - Run the helpers cell. It loads the messages into
messages,train,validationandtest, and the helper asrt. It prints nothing.
The setup checks each file with a checksum: a short fingerprint of the file. If one byte is wrong, the fingerprint changes, and the cell stops with wrong checksum. Then delete the folder larkfield-inbox and run the cell again. If it says Could not download, check your internet connection. In Kaggle, also turn on Internet in the notebook settings.
What is in the inbox
Section titled “What is in the inbox”Each row of messages.csv is one message that a customer sent to Larkfield. The columns that the course uses most:
| Column | What it holds | Example |
|---|---|---|
text |
What the customer wrote | “hi, about the cushion set I sent back weeks ago…” |
channel, language |
email, chat or social; en or fr |
chat, en |
route |
The label: the team that must act first, or needs_person |
returns_refunds |
topic, priority |
The second and third questions | refund_status, normal |
routes |
One or two requests, separated by ` | ` |
split |
train, validation or test |
train |
Section 1 of the notebook asks you to predict the largest and the smallest route, then counts them:
print(len(messages), "messages from", messages["created_at"].min()[:10], "to", messages["created_at"].max()[:10])print(messages["route"].value_counts().to_string())10000 messages from 2025-09-01 to 2026-01-31routedelivery 2318returns_refunds 2015product 1440orders 1428billing 1329account 979needs_person 491The history covers five months, about 65 messages a day. The split is fixed for the whole course: 6,000 messages to train on, 2,000 for validation and 2,000 for the test. Each route and each language has the same share in the three parts.
Text description of the diagram
A horizontal bar chart with one bar per route, from the largest to the smallest. Each bar has an English part and a French part. Delivery: 2,318 messages, 1,643 in English and 675 in French. Returns and refunds: 2,015, with 1,430 and 585. Product: 1,440, with 1,018 and 422. Orders: 1,428, with 1,006 and 422. Billing: 1,329, with 958 and 371. Account: 979, with 683 and 296. Needs_person: 491, with 333 and 158. About 3 messages in 10 are in French in every route. The chart covers 10,000 messages from 1 September 2025 to 31 January 2026.
- Language
- Channel
Text description of the chart
A column chart in two groups, on an axis from 0 to 8,000 messages. Language: 7,071 messages in English and 2,929 in French. Channel: 4,538 by email, 3,438 by chat and 2,024 on social media. The median message has 28 words: 33 by email, 25 by chat and 24 on social media.
Section 3 counts the other labels. Every route has 4 topics, except needs_person, which has 3: 27 topics in all. In the training part, 3,955 messages are normal, 1,028 urgent and 1,017 low. Only 44 training messages are account takeovers, the rare and expensive topic.
Read the guidelines
Section titled “Read the guidelines”The labels came from people who followed written rules. They are in data/guidelines.md, which Chloé, a bilingual support agent, wrote for the labellers. Open it in the notebook’s file panel, or on GitHub. It has three parts:
- The one rule before all others: label the request, not the words. If a customer asks for two things, label the main request: usually the first one, or the urgent one.
- A rule and an example for each route. For example,
deliverycovers a parcel that is late, lost, or “marked delivered but not received”. - The hard cases between two routes. A refund after a return is
returns_refunds; a wrong charge isbilling. An item that arrived damaged isreturns_refunds; one that broke after weeks of use isproduct(warranty).
Read the hard cases before Module 2. They are where careful people disagree, and so they are where models disagree too.
This inbox is easier than a real one
Section titled “This inbox is easier than a real one”Before you see any score, know this about the data. The Larkfield inbox is easier than a real support inbox, and the course says so on purpose.
Here is why. A program built each message from one of 27 topic templates, with facts such as an order number and a product name. Then one small language model rewrote every draft in a customer’s words, with styles, tones and typos. The wording varies, but it comes from one writer and 27 patterns. So the same words come back again and again, and a simple model finds them easily. On this inbox, a classic model with no deep learning, TF-IDF with logistic regression, routes 0.962 of the test messages right. The first version of the data gave 0.993. Two rounds of harder wording brought it down only to 0.962.
So when the course quotes a score, it says “on this inbox” or “here”. Do not take it as what a method gets on real messages. The dataset card, data/dataset-card.md, says the same under Limitations. The case studies at the end of the course use real, public data, such as customers’ banking questions and requests that a bot cannot handle. There, the scores are lower and the differences between methods are larger.
The inbox still has real difficulty in places. We planted hard messages on purpose, and they are where methods differ:
Text description of the diagram
The chart has five horizontal bars. Its title says: hard messages, 3,660 of 10,000, or 36.6%. The first bar, in grey, shows the plain messages, which have none of these difficulties: 6,340, or 63.4%. Then come the four hard kinds. Vague messages, such as “see my last email”: 1,266, or 12.7%. Messages with two requests: 1,255, or 12.6%. Policy edge messages, which use another team’s words, such as a product question that mentions a return: 960, or 9.6%. Sarcasm: 179, or 1.8%.
A vague message points to an earlier message instead of saying what it wants. The hook message of the first lesson, “see my last email, maybe it got missed”, is one. A policy edge message uses another team’s words, like the product question that mentions a return. These are the messages to read when two methods disagree.
The cost table in code
Section titled “The cost table in code”Section 4 of the notebook loads the cost table and prices two mistakes with the helper rt.message_cost(label, predicted, priority, topic):
costs = rt.load_costs()for name, dollars in costs.items(): print(f"{name:30} {dollars:8.2f} dollars")print(rt.message_cost("returns_refunds", "product", "normal", "refund_status"))print(rt.message_cost("account", "billing", "urgent", "account_takeover"))review 1.50 dollarswrong_queue 4.00 dollarswrong_queue_urgent 20.00 dollarsmissed_account_takeover 200.00 dollarsneeds_person_sent_to_team 40.00 dollarsteam_message_sent_to_person 2.00 dollars4.0200.0The first priced mistake is the hook: a refund question in the product team’s queue, US$4. The second is an account takeover in the billing queue, US$200. The helper follows the step-by-step rule of the last lesson.
The first comparison
Section titled “The first comparison”Section 5 runs the course’s first comparison on the 2,000 test messages. It has three rows:
- Everything to a person: Larkfield today.
- Rules or a person: the project’s keyword rules decide when one matches, and a person reads the rest.
- TF-IDF + logistic regression: a first model, trained on the 6,000 training messages. Module 3 explains how it works. For now, it is a model that learns which words go with which route.
Predict first: which row has the lowest total cost per 1,000 messages? Is it also the most accurate? The cell ends with rt.show(rows), which shows this table:
| method | accuracy | macro_f1 | takeover_recall | needs_person_recall | accuracy_en | accuracy_fr | mistakes_per_1000 | run_per_1000 | total_per_1000 | ms_per_message | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | TF-IDF + logistic regression | 0.962 | 0.965 | 0.917 | 1.000 | 0.966 | 0.952 | 296.0 | 0.0 | 296.0 | None |
| 1 | everything to a person | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 1500.0 | 0.0 | 1500.0 | None |
| 2 | rules or a person | 0.555 | 0.625 | 0.833 | 0.818 | 0.577 | 0.503 | 1757.8 | 0.0 | 1757.8 | None |
This is the comparison table of the course, in its first form. Read the columns from left to right:
accuracy: the share of messages sent to the right route. A review counts as not right.macro_f1: a score that gives every route the same weight, so that the small routes count as much asdelivery. The machine learning course explains the F1 score.takeover_recall: the share of the 12 account takeovers in the test messages that reach the account team. 0.917 is 11 of 12.needs_person_recall: the share of the 99needs_personmessages that reach a person.accuracy_en,accuracy_fr: the accuracy in English and in French.mistakes_per_1000,run_per_1000,total_per_1000: the cost of the mistakes and reviews, the cost to run, and their sum, per 1,000 messages.ms_per_message: the time to route one message. The notebook does not time the methods, so it showsNone.
On this inbox, the model costs US$296 per 1,000 messages, about a fifth of Larkfield today. The rules cost more than a person, as the last lesson showed. The French messages are a little harder for the model than the English ones: 0.952 against 0.966.
Your turn: the French messages
Section titled “Your turn: the French messages”Section 6 asks you to count the French test messages that the model sends to the wrong route. Put the number in wrong_fr. Use model.predict(...) on the French test messages, and compare with their route. Then run the check cell.
Show the answer
fr = test[test["language"] == "fr"]wrong_fr = int((model.predict(fr["text"]) != fr["route"]).sum())print(wrong_fr, "of", len(fr))28 of 589The check cell prints Check passed: wrong_fr is 28. That is 561 of 589 right, the 0.952 of the accuracy_fr column.
Common mistake
Accuracy without the cost
Symptom: Tomás reads the table and says: “The model is right on 96% of the messages. Ship it.” Section 7 of the notebook groups the model’s 76 wrong test messages by their label and adds their costs:
count sumrouteaccount 9 232.0returns_refunds 16 96.0delivery 20 76.0billing 10 72.0product 11 60.0orders 10 56.0Cause: the 9 wrong account messages cost US$232, which is 39% of the US$592 that all the mistakes cost. Most of it comes from one account takeover in the billing queue. The 20 wrong delivery messages, more than twice as many, cost only US$76. “96% right” says nothing about which 4% are wrong.
Fix: read the cost by route, not only the accuracy. Module 7 raises the recall of the account takeovers on purpose, and Module 8 sends the unsure ones to a person.
Consequence in AI work: a router that is “96% accurate” can still let its most expensive mistakes through. The team learns about them from an angry customer, not from the report.
The project on your computer
Section titled “The project on your computer”From Module 2, you also work in the course project, larkroute. It is a small Python package with a command, route, that runs the same comparison on your computer. The notebook taught you the data. The project gives you a tool that you change module by module.
The project lives in the course’s labs repository, in C36/snapshots/. Each snapshot is a copy of the project at one point in the course. You start from the snapshot start.
-
Open a terminal and go to the folder where you keep projects, for example
~/projects. -
Get the labs repository, with Git:
Terminal window git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.gitWithout Git, download the ZIP file of the repository and unzip it here. Its folder is then
nextia-learning-labs-main: use that name in the next step. -
Copy the snapshot
startto a new folderlarkroute, and go into it. On macOS and Linux:Terminal window cp -R nextia-learning-labs/C36/snapshots/start larkroutecd larkrouteOn Windows (PowerShell):
Terminal window Copy-Item -Recurse nextia-learning-labs\C36\snapshots\start larkroutecd larkroute -
Make a virtual environment and turn it on. On macOS and Linux:
Terminal window python3.12 -m venv .venvsource .venv/bin/activateOn Windows (PowerShell):
Terminal window py -3.12 -m venv .venv.venv\Scripts\Activate.ps1 -
Install the pinned packages and the project itself:
Terminal window pip install -r requirements.txt -
Download the data into
data/. The command is the same on every system:Terminal window python get_data.pyYou see
Ready: 35 files in data/, every checksum is right. -
Run the tests. You see
10 passed.Terminal window python -m pytest -q
A virtual environment is a folder (.venv) with its own Python packages, so this project’s versions do not change other projects. When it is on, your prompt starts with (.venv). Turn it on again in each new terminal.
Now run the first two commands:
route info10000 messages (6000 train, 2000 test, 2000 validation), from 2025-09-01 to 2026-01-31. All synthetic.Routes: delivery 2318, returns_refunds 2015, product 1440, orders 1428, billing 1329, account 979, needs_person 491Languages: en 7071, fr 2929 | Channels: email 4538, chat 3438, social 2024route compare --quickScoring 3 methods on the test messages (trained on the reviewed labels)... everything to a person: done rules or a person: done TF-IDF + logistic regression: done method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msgTF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.14 everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00 rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.03These are the notebook’s three rows. The project sorts them from the cheapest to the most expensive. It took about 6 seconds here. The last column, milliseconds per message, depends on your computer, so your value will differ. Every other number must be the same. “Trained on the reviewed labels” means the labels in messages.csv. Module 2 meets a noisier set.
Change one thing: the price of a review
Section titled “Change one thing: the price of a review”The cost table is a file, config/costs.json. Before you change it, predict: if a review costs US$1.00 instead of US$1.50, which of the three rows change?
- Open
config/costs.jsonin your editor. - Change the line
"review": 1.5,to"review": 1.0,and save the file. - Run
route compare --quickagain.
TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.17 everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1000.0 0.0 1000.0 0.00 rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1639.5 0.0 1639.5 0.04The two rows that use a person get cheaper. Everything to a person falls to US$1,000, and the rules fall to US$1,639.5. The model’s row does not change, because it decides every message and sends none to review. A cheaper review makes review more attractive. Module 8 uses exactly this trade-off.
route reset puts the project back to its start: it deletes what the commands wrote (models/, runs/, queue/), restores the configuration files, and checks the data.
route resetReset: removed nothing; config restored: costs.json; data: all files are right.The project restored your changed costs.json. Run it whenever you are unsure what state the project is in. It deletes only those three folders, so keep your own files elsewhere.
Common mistake
A data file that is missing or changed
Symptom: route info stops with this line:
These data files are missing or changed: messages.csv. Run: route resetCause: a file in data/ was deleted, moved, or changed by hand, so its checksum no longer matches. Commands that read it would give numbers that nobody else can compare.
Fix: run route reset. It downloads every file that is missing or changed, then checks them again. The first line of its output is the line of get_data.py: Ready: 35 files in data/, every checksum is right. If you see Could not download, check your internet connection and run it again.
Consequence in AI work: a comparison table means something only when every row used the same data. A checksum is a cheap way to know that it did.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does the course say “on this inbox” when it quotes a score such as 0.962? What makes the Larkfield inbox easier than a real one?
Show the answer
One small language model wrote every message from 27 topic templates. The wording varies, but the same words come back again and again. So even a simple TF-IDF model routes 0.962 of the test messages right. A real inbox has many writers, new topics and messier language, so the same method scores lower there. The dataset card says this under its limitations, and the case studies use real public data, where the scores are lower.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In route compare --quick, “rules or a person” has an accuracy of 0.555 and “everything to a person” has 0. Predict what happens to both rows if a review costs US$1.00 instead of US$1.50, and explain why the TF-IDF row does not change.
Show the answer
Both rows get cheaper, because both send messages to a person. Everything to a person falls from US$1,500 to US$1,000 per 1,000 messages. Rules or a person falls from US$1,757.8 to US$1,639.5. The accuracies do not change, because a review is never counted as a right route. The TF-IDF row stays at US$296, because the model decides every message and sends none to review.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
You changed config/costs.json and deleted data/messages.csv by mistake. Which single command brings the project back, and what does it do?
Show the answer
route reset. It deletes what the commands wrote (models/, runs/, queue/) and puts back the course’s configuration files, such as costs.json. Then it downloads every data file that is missing or has the wrong checksum. It prints what it did, for example config restored: costs.json; data: all files are right.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The inbox has 10,000 messages from September 2025 to January 2026: 6,000 to train, 2,000 for validation and 2,000 for the test. They are in English (7,071) and French (2,929), by email, chat and social media.
- The routes are not the same size:
delivery2,318,needs_person491. Only 44 training messages are account takeovers. - This inbox is easier than a real one: one model wrote every message from 27 templates. Say “on this inbox” when you quote a score. 36.6% of the messages are planted hard cases.
- The first comparison, on the test messages: TF-IDF + logistic regression 0.962 and US$296 per 1,000. Everything to a person costs US$1,500, and rules or a person 0.555 and US$1,757.8.
- The project
larkrouteruns it withroute compare --quick.config/costs.jsonholds the costs, androute resetbrings everything back.
You can now open the data and run the scoreboard. But every score so far trusts the labels in messages.csv, and labels are written by people who sometimes disagree. Next, in Module 2, you read the labelling guidelines closely and write a rule for each hard case.
Next: Labelling guidelines