Skip to content
Français

Meet the inbox

  • Lesson 4
  • 35 min
  • Module 1: The routing problem and its cost

Where we are

We have labels and a cost for each mistake, on paper. We have not opened the data yet. In this lesson, we set up the project, read the Larkfield inbox and its labelling guidelines, run the first comparison, and learn how to reset everything.

Lesson 4 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceMeet the inbox

What you will be able to do

Open the Larkfield inbox, check that your setup works, and run the course’s first comparison yourself. Every later module compares its methods on these same messages, with these same costs.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every code cell, task and check of this lesson. It reads the inbox, the cost table and the first comparison. Its setup cells install pandas, scikit-learn and matplotlib and download the data (about 9 MB), so it needs no local setup. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • The notebook route (recommended for this lesson): a browser and a free Google account for Colab, or a free Kaggle account.
  • The project route (needed from Module 2): Python 3.12, a terminal, and Git or a browser to download a ZIP file. The steps are in The project on your computer below.

Tested on macOS (Apple silicon) with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1, in Jupyter and in the terminal. The Windows commands, Linux, Colab and Kaggle were not tested.

A new worker joins the post office. Before she sorts a single letter, a colleague shows her around. How many letters come in each day? Which trays are full, and which are almost empty? Where is the sheet with the sorting rules? And how does the manager count her mistakes at the end of the week?

None of this is sorting. But a worker who skips it sorts by guesswork, and nobody can tell if she is good. This lesson is your first morning. You count the messages, read the rules, and run the scoreboard that every module uses.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the first setup cell. It installs the packages and prints pandas 3.1.1 and scikit-learn 1.9.1.
  3. If you see other versions, restart the kernel, then run the cell again.
  4. Run the second setup cell. It downloads the inbox into a folder larkfield-inbox/data, with the course’s helper routing.py, and checks each file.
  5. Check its last line: Ready: 10000 messages (6000 train, 2000 validation, 2000 test), all synthetic.
  6. Run the helpers cell. It loads the messages into messages, train, validation and test, and the helper as rt. It prints nothing.

The setup checks each file with a checksum: a short fingerprint of the file. If one byte is wrong, the fingerprint changes, and the cell stops with wrong checksum. Then delete the folder larkfield-inbox and run the cell again. If it says Could not download, check your internet connection. In Kaggle, also turn on Internet in the notebook settings.

Each row of messages.csv is one message that a customer sent to Larkfield. The columns that the course uses most:

Column What it holds Example
text What the customer wrote “hi, about the cushion set I sent back weeks ago…”
channel, language email, chat or social; en or fr chat, en
route The label: the team that must act first, or needs_person returns_refunds
topic, priority The second and third questions refund_status, normal
routes One or two requests, separated by ` `
split train, validation or test train

Section 1 of the notebook asks you to predict the largest and the smallest route, then counts them:

print(len(messages), "messages from", messages["created_at"].min()[:10], "to", messages["created_at"].max()[:10])
print(messages["route"].value_counts().to_string())
10000 messages from 2025-09-01 to 2026-01-31
route
delivery 2318
returns_refunds 2015
product 1440
orders 1428
billing 1329
account 979
needs_person 491

The history covers five months, about 65 messages a day. The split is fixed for the whole course: 6,000 messages to train on, 2,000 for validation and 2,000 for the test. Each route and each language has the same share in the three parts.

A horizontal bar chart of the messages per route, each bar split into English and French. delivery 2,318 (1,643 English, 675 French); returns_refunds 2,015 (1,430 and 585); product 1,440 (1,018 and 422); orders 1,428 (1,006 and 422); billing 1,329 (958 and 371); account 979 (683 and 296); needs_person 491 (333 and 158). 10,000 messages from 2025-09-01 to 2026-01-31.
Messages per route in the history, in English and in French. The largest route has almost five times as many messages as needs_person.
Text description of the diagram

A horizontal bar chart with one bar per route, from the largest to the smallest. Each bar has an English part and a French part. Delivery: 2,318 messages, 1,643 in English and 675 in French. Returns and refunds: 2,015, with 1,430 and 585. Product: 1,440, with 1,018 and 422. Orders: 1,428, with 1,006 and 422. Billing: 1,329, with 958 and 371. Account: 979, with 683 and 296. Needs_person: 491, with 333 and 158. About 3 messages in 10 are in French in every route. The chart covers 10,000 messages from 1 September 2025 to 31 January 2026.

  • Language
  • Channel
The 10,000 messages of the history by language and by channel. Each message has one language and one channel.
Text description of the chart

A column chart in two groups, on an axis from 0 to 8,000 messages. Language: 7,071 messages in English and 2,929 in French. Channel: 4,538 by email, 3,438 by chat and 2,024 on social media. The median message has 28 words: 33 by email, 25 by chat and 24 on social media.

Section 3 counts the other labels. Every route has 4 topics, except needs_person, which has 3: 27 topics in all. In the training part, 3,955 messages are normal, 1,028 urgent and 1,017 low. Only 44 training messages are account takeovers, the rare and expensive topic.

The labels came from people who followed written rules. They are in data/guidelines.md, which Chloé, a bilingual support agent, wrote for the labellers. Open it in the notebook’s file panel, or on GitHub. It has three parts:

  • The one rule before all others: label the request, not the words. If a customer asks for two things, label the main request: usually the first one, or the urgent one.
  • A rule and an example for each route. For example, delivery covers a parcel that is late, lost, or “marked delivered but not received”.
  • The hard cases between two routes. A refund after a return is returns_refunds; a wrong charge is billing. An item that arrived damaged is returns_refunds; one that broke after weeks of use is product (warranty).

Read the hard cases before Module 2. They are where careful people disagree, and so they are where models disagree too.

Before you see any score, know this about the data. The Larkfield inbox is easier than a real support inbox, and the course says so on purpose.

Here is why. A program built each message from one of 27 topic templates, with facts such as an order number and a product name. Then one small language model rewrote every draft in a customer’s words, with styles, tones and typos. The wording varies, but it comes from one writer and 27 patterns. So the same words come back again and again, and a simple model finds them easily. On this inbox, a classic model with no deep learning, TF-IDF with logistic regression, routes 0.962 of the test messages right. The first version of the data gave 0.993. Two rounds of harder wording brought it down only to 0.962.

So when the course quotes a score, it says “on this inbox” or “here”. Do not take it as what a method gets on real messages. The dataset card, data/dataset-card.md, says the same under Limitations. The case studies at the end of the course use real, public data, such as customers’ banking questions and requests that a bot cannot handle. There, the scores are lower and the differences between methods are larger.

The inbox still has real difficulty in places. We planted hard messages on purpose, and they are where methods differ:

A horizontal bar chart. Hard messages: 3,660 of 10,000 (36.6%). Plain messages, none of these: 6,340 (63.4%). Vague: 1,266 (12.7%). Two requests: 1,255 (12.6%). Policy edge: 960 (9.6%). Sarcasm: 179 (1.8%).
Hard messages in the history: 36.6% of the 10,000 messages are vague, hold two requests, sit on a policy edge, or use sarcasm. Each message has at most one of these kinds.
Text description of the diagram

The chart has five horizontal bars. Its title says: hard messages, 3,660 of 10,000, or 36.6%. The first bar, in grey, shows the plain messages, which have none of these difficulties: 6,340, or 63.4%. Then come the four hard kinds. Vague messages, such as “see my last email”: 1,266, or 12.7%. Messages with two requests: 1,255, or 12.6%. Policy edge messages, which use another team’s words, such as a product question that mentions a return: 960, or 9.6%. Sarcasm: 179, or 1.8%.

A vague message points to an earlier message instead of saying what it wants. The hook message of the first lesson, “see my last email, maybe it got missed”, is one. A policy edge message uses another team’s words, like the product question that mentions a return. These are the messages to read when two methods disagree.

Section 4 of the notebook loads the cost table and prices two mistakes with the helper rt.message_cost(label, predicted, priority, topic):

costs = rt.load_costs()
for name, dollars in costs.items():
print(f"{name:30} {dollars:8.2f} dollars")
print(rt.message_cost("returns_refunds", "product", "normal", "refund_status"))
print(rt.message_cost("account", "billing", "urgent", "account_takeover"))
review 1.50 dollars
wrong_queue 4.00 dollars
wrong_queue_urgent 20.00 dollars
missed_account_takeover 200.00 dollars
needs_person_sent_to_team 40.00 dollars
team_message_sent_to_person 2.00 dollars
4.0
200.0

The first priced mistake is the hook: a refund question in the product team’s queue, US$4. The second is an account takeover in the billing queue, US$200. The helper follows the step-by-step rule of the last lesson.

Section 5 runs the course’s first comparison on the 2,000 test messages. It has three rows:

  1. Everything to a person: Larkfield today.
  2. Rules or a person: the project’s keyword rules decide when one matches, and a person reads the rest.
  3. TF-IDF + logistic regression: a first model, trained on the 6,000 training messages. Module 3 explains how it works. For now, it is a model that learns which words go with which route.

Predict first: which row has the lowest total cost per 1,000 messages? Is it also the most accurate? The cell ends with rt.show(rows), which shows this table:

method accuracy macro_f1 takeover_recall needs_person_recall accuracy_en accuracy_fr mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 None
1 everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 None
2 rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 None

This is the comparison table of the course, in its first form. Read the columns from left to right:

  • accuracy: the share of messages sent to the right route. A review counts as not right.
  • macro_f1: a score that gives every route the same weight, so that the small routes count as much as delivery. The machine learning course explains the F1 score.
  • takeover_recall: the share of the 12 account takeovers in the test messages that reach the account team. 0.917 is 11 of 12.
  • needs_person_recall: the share of the 99 needs_person messages that reach a person.
  • accuracy_en, accuracy_fr: the accuracy in English and in French.
  • mistakes_per_1000, run_per_1000, total_per_1000: the cost of the mistakes and reviews, the cost to run, and their sum, per 1,000 messages.
  • ms_per_message: the time to route one message. The notebook does not time the methods, so it shows None.

On this inbox, the model costs US$296 per 1,000 messages, about a fifth of Larkfield today. The rules cost more than a person, as the last lesson showed. The French messages are a little harder for the model than the English ones: 0.952 against 0.966.

Section 6 asks you to count the French test messages that the model sends to the wrong route. Put the number in wrong_fr. Use model.predict(...) on the French test messages, and compare with their route. Then run the check cell.

Show the answer
fr = test[test["language"] == "fr"]
wrong_fr = int((model.predict(fr["text"]) != fr["route"]).sum())
print(wrong_fr, "of", len(fr))
28 of 589

The check cell prints Check passed: wrong_fr is 28. That is 561 of 589 right, the 0.952 of the accuracy_fr column.

Common mistake

Accuracy without the cost

Symptom: Tomás reads the table and says: “The model is right on 96% of the messages. Ship it.” Section 7 of the notebook groups the model’s 76 wrong test messages by their label and adds their costs:

count sum
route
account 9 232.0
returns_refunds 16 96.0
delivery 20 76.0
billing 10 72.0
product 11 60.0
orders 10 56.0

Cause: the 9 wrong account messages cost US$232, which is 39% of the US$592 that all the mistakes cost. Most of it comes from one account takeover in the billing queue. The 20 wrong delivery messages, more than twice as many, cost only US$76. “96% right” says nothing about which 4% are wrong.

Fix: read the cost by route, not only the accuracy. Module 7 raises the recall of the account takeovers on purpose, and Module 8 sends the unsure ones to a person.

Consequence in AI work: a router that is “96% accurate” can still let its most expensive mistakes through. The team learns about them from an angry customer, not from the report.

From Module 2, you also work in the course project, larkroute. It is a small Python package with a command, route, that runs the same comparison on your computer. The notebook taught you the data. The project gives you a tool that you change module by module.

The project lives in the course’s labs repository, in C36/snapshots/. Each snapshot is a copy of the project at one point in the course. You start from the snapshot start.

  1. Open a terminal and go to the folder where you keep projects, for example ~/projects.

  2. Get the labs repository, with Git:

    Terminal window
    git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

    Without Git, download the ZIP file of the repository and unzip it here. Its folder is then nextia-learning-labs-main: use that name in the next step.

  3. Copy the snapshot start to a new folder larkroute, and go into it. On macOS and Linux:

    Terminal window
    cp -R nextia-learning-labs/C36/snapshots/start larkroute
    cd larkroute

    On Windows (PowerShell):

    Terminal window
    Copy-Item -Recurse nextia-learning-labs\C36\snapshots\start larkroute
    cd larkroute
  4. Make a virtual environment and turn it on. On macOS and Linux:

    Terminal window
    python3.12 -m venv .venv
    source .venv/bin/activate

    On Windows (PowerShell):

    Terminal window
    py -3.12 -m venv .venv
    .venv\Scripts\Activate.ps1
  5. Install the pinned packages and the project itself:

    Terminal window
    pip install -r requirements.txt
  6. Download the data into data/. The command is the same on every system:

    Terminal window
    python get_data.py

    You see Ready: 35 files in data/, every checksum is right.

  7. Run the tests. You see 10 passed.

    Terminal window
    python -m pytest -q

A virtual environment is a folder (.venv) with its own Python packages, so this project’s versions do not change other projects. When it is on, your prompt starts with (.venv). Turn it on again in each new terminal.

Now run the first two commands:

Terminal window
route info
10000 messages (6000 train, 2000 test, 2000 validation), from 2025-09-01 to 2026-01-31. All synthetic.
Routes: delivery 2318, returns_refunds 2015, product 1440, orders 1428, billing 1329, account 979, needs_person 491
Languages: en 7071, fr 2929 | Channels: email 4538, chat 3438, social 2024
Terminal window
route compare --quick
Scoring 3 methods on the test messages (trained on the reviewed labels)...
everything to a person: done
rules or a person: done
TF-IDF + logistic regression: done
method accuracy macro-F1 takeover needs person EN FR mistakes $/1k run $/1k total $/1k ms/msg
TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.14
everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1500.0 0.0 1500.0 0.00
rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1757.8 0.0 1757.8 0.03

These are the notebook’s three rows. The project sorts them from the cheapest to the most expensive. It took about 6 seconds here. The last column, milliseconds per message, depends on your computer, so your value will differ. Every other number must be the same. “Trained on the reviewed labels” means the labels in messages.csv. Module 2 meets a noisier set.

The cost table is a file, config/costs.json. Before you change it, predict: if a review costs US$1.00 instead of US$1.50, which of the three rows change?

  1. Open config/costs.json in your editor.
  2. Change the line "review": 1.5, to "review": 1.0, and save the file.
  3. Run route compare --quick again.
TF-IDF + logistic regression 0.962 0.965 0.917 1.000 0.966 0.952 296.0 0.0 296.0 0.17
everything to a person 0.000 0.000 0.000 0.000 0.000 0.000 1000.0 0.0 1000.0 0.00
rules or a person 0.555 0.625 0.833 0.818 0.577 0.503 1639.5 0.0 1639.5 0.04

The two rows that use a person get cheaper. Everything to a person falls to US$1,000, and the rules fall to US$1,639.5. The model’s row does not change, because it decides every message and sends none to review. A cheaper review makes review more attractive. Module 8 uses exactly this trade-off.

route reset puts the project back to its start: it deletes what the commands wrote (models/, runs/, queue/), restores the configuration files, and checks the data.

Terminal window
route reset
Reset: removed nothing; config restored: costs.json; data: all files are right.

The project restored your changed costs.json. Run it whenever you are unsure what state the project is in. It deletes only those three folders, so keep your own files elsewhere.

Common mistake

A data file that is missing or changed

Symptom: route info stops with this line:

These data files are missing or changed: messages.csv. Run: route reset

Cause: a file in data/ was deleted, moved, or changed by hand, so its checksum no longer matches. Commands that read it would give numbers that nobody else can compare.

Fix: run route reset. It downloads every file that is missing or changed, then checks them again. The first line of its output is the line of get_data.py: Ready: 35 files in data/, every checksum is right. If you see Could not download, check your internet connection and run it again.

Consequence in AI work: a comparison table means something only when every row used the same data. A checksum is a cheap way to know that it did.

Check your understanding

Why does the course say “on this inbox” when it quotes a score such as 0.962? What makes the Larkfield inbox easier than a real one?

Show the answer

One small language model wrote every message from 27 topic templates. The wording varies, but the same words come back again and again. So even a simple TF-IDF model routes 0.962 of the test messages right. A real inbox has many writers, new topics and messier language, so the same method scores lower there. The dataset card says this under its limitations, and the case studies use real public data, where the scores are lower.

Check your understanding

In route compare --quick, “rules or a person” has an accuracy of 0.555 and “everything to a person” has 0. Predict what happens to both rows if a review costs US$1.00 instead of US$1.50, and explain why the TF-IDF row does not change.

Show the answer

Both rows get cheaper, because both send messages to a person. Everything to a person falls from US$1,500 to US$1,000 per 1,000 messages. Rules or a person falls from US$1,757.8 to US$1,639.5. The accuracies do not change, because a review is never counted as a right route. The TF-IDF row stays at US$296, because the model decides every message and sends none to review.

Check your understanding

You changed config/costs.json and deleted data/messages.csv by mistake. Which single command brings the project back, and what does it do?

Show the answer

route reset. It deletes what the commands wrote (models/, runs/, queue/) and puts back the course’s configuration files, such as costs.json. Then it downloads every data file that is missing or has the wrong checksum. It prints what it did, for example config restored: costs.json; data: all files are right.

Recap

  • The inbox has 10,000 messages from September 2025 to January 2026: 6,000 to train, 2,000 for validation and 2,000 for the test. They are in English (7,071) and French (2,929), by email, chat and social media.
  • The routes are not the same size: delivery 2,318, needs_person 491. Only 44 training messages are account takeovers.
  • This inbox is easier than a real one: one model wrote every message from 27 templates. Say “on this inbox” when you quote a score. 36.6% of the messages are planted hard cases.
  • The first comparison, on the test messages: TF-IDF + logistic regression 0.962 and US$296 per 1,000. Everything to a person costs US$1,500, and rules or a person 0.555 and US$1,757.8.
  • The project larkroute runs it with route compare --quick. config/costs.json holds the costs, and route reset brings everything back.

You can now open the data and run the scoreboard. But every score so far trusts the labels in messages.csv, and labels are written by people who sometimes disagree. Next, in Module 2, you read the labelling guidelines closely and write a rule for each hard case.

Next: Labelling guidelines