Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

Each module added one piece and tried it alone: the labels, the methods, the costs, the cascade, the zones and the service. In this assignment, you use them all at once. Larkfield adds a marketplace channel and two new products, and you deliver the routing system for the next quarter.

What you will be able to do

Use the whole course on one new quarter at Larkfield. You label and price the mistakes, compare methods on one fair test, and choose a router with its zones. You run it as a service, watch April for drift and add a new topic safely. Then you write the decision for Grace and Omar, with its cost and its limits.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

  • Modules 1 to 9. Each part below names the lessons that it uses. The LLM lessons of Module 5 are optional here.
  • The certificate also asks for at least one case study of Module 10. To see the same steps on real data first, open Route banking questions.

What you need on your computer

  • Python 3.12, a terminal, and a notebook if you like one. Parts 1 to 3 work well in a notebook. Parts 4 and 5 need the project and a terminal.
  • The course’s labs repository with the folder C36, as in Meet the inbox. Run git -C nextia-learning-labs pull to get the folder C36/final.
  • The project snapshot C36/snapshots/end-of-m09: the route command with the service, the drift check and the release check.
  • About 300 MB of disk for the virtual environment, and about 500 MB more if you use the embedding model of Module 4. No account, key or money. An LLM row is optional and needs your own key.

Tested on macOS (Apple silicon), with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. We wrote the Windows (PowerShell) commands, but we did not run them.

The situation: a new channel and two new products

Section titled “The situation: a new channel and two new products”

In March, Larkfield starts to sell on an online marketplace. Customers now also write from there: about 3 messages in 10. Two products are new. The Patio heater is on sale from March. The Robot mower R2 goes on sale on 6 April. The Grow Hub, which arrived in the launch month of Module 9, has its own route, devices, for setup questions about smart products.

Grace, the support lead, writes to Priya:

“I need the router for the next quarter. My team can read at most 1 message in 10. I want the right team for the rest, and I want to know what the mistakes cost us. Please also tell me what you will do when the next product arrives.”

Omar, who owns the mower launch, adds: “The mower questions must not wait in the wrong queue.” Tomás asks: “Why not send every message to an LLM?” Your answer to each of them must come with evidence.

We, the course team, made the data with the same generator as the course inbox. We know the right routes of the April messages. We do not give them to you: the labs folder has no answer key. You find the problems in April with the course’s methods. The data is synthetic, like every Larkfield message.

Use the course’s cost table without changes, so that your numbers compare with other learners’ numbers. These are assumptions, the same as in The cost of each mistake:

Assumption Value (US$ per message)
A review by a person 1.50
A message in the wrong queue 4
An urgent message in the wrong queue 20
A missed account takeover 200
A needs_person message sent to a team 40
A team message sent to a senior person 2
Everything to a person (Larkfield before the router) 1,500 per 1,000 messages
Grace’s review limit the review and person zones together: at most 10% of the messages
The volume about 1,300 messages a month (April: 1,120 messages in 26 days, about 43 a day)

If you want to try a second cost table, add it as an extra and say why. Keep the first one as the main result.

The data is in labs C36/final/data/: three files and a list of their checksums. A checksum is a long number computed from the bytes of a file. If one byte changes, the checksum changes.

File Rows One row Columns
messages.csv (334 KB) 1,280 one message of March 2026, with its labels message_id, created_at, channel, language, text, route, topic, priority, routes, split, synthetic
double_labelled.csv (3 KB) 100 one March training message, labelled by Chloé and by Grace message_id, label_chloe, label_grace
stream.csv (238 KB) 1,120 one message of 1 to 26 April 2026, with no labels message_id, created_at, channel, language, text, synthetic

What the columns mean:

  • route: the team that must act first, or needs_person. There are eight routes: the course’s seven and devices.
  • topic: the kind of request inside the route, for example grow_hub_setup or account_takeover. priority is urgent, normal or low.
  • routes: one route, or two routes with | between them when a message asks for two things. The first one is the label.
  • channel: email, chat, social or, new, marketplace. language: en or fr.
  • split: train (768), validation (256) or test (256).

You also use the course data, labs C36/data/: the 6,000 training messages of the history, the guidelines and the cost table. Here is March by route. Look at the small routes before you read any score:

The 1,280 March messages by route: delivery (DEL), returns and refunds (RET), product (PRO), orders (ORD), billing (BIL), account (ACC), devices (DEV) and needs_person (PER). The test split has about one fifth of each: 15 device messages and 12 for needs_person, so one message changes their recall by 7 to 8 points.
Text description of the chart

A column chart with eight columns, one per route, from the largest to the smallest. Delivery 280, returns and refunds 245, product 171, orders 168, billing 159, account 118, devices 83 and needs_person 56. Devices and needs_person have their own colours. The account route includes only 10 account takeovers: 6 in train, 3 in validation and 1 in test.

Only one test message is an account takeover. So a takeover recall of 1.0 on the test set means “one message was right”, not “every takeover is found”. Say so wherever you quote it.

  1. Copy the finished project into a new folder, routing-q2, and install it:

    Terminal window
    cp -R nextia-learning-labs/C36/snapshots/end-of-m09 routing-q2
    cd routing-q2
    python3.12 -m venv .venv
    source .venv/bin/activate
    pip install -r requirements.txt
    python get_data.py

    get_data.py puts the course data in data/. The project’s 42 tests use it.

  2. Save this script as make_data.py in routing-q2. It checks the checksums, then writes a data folder for the quarter, q2-data/. The models learn from the course’s 6,000 training messages and March’s 768. Validation and test are March only, because the quarter looks like March, not like last year.

    import hashlib
    import shutil
    import sys
    from pathlib import Path
    import pandas as pd
    labs = Path(sys.argv[1])
    final = labs / "final" / "data"
    for line in (final / "SHA256SUMS").read_text().splitlines():
    digest, name = line.split()
    ok = hashlib.sha256((final / name).read_bytes()).hexdigest() == digest
    print("OK " if ok else "BAD", name)
    if not ok:
    raise SystemExit("A file is not the published one: download it again.")
    history = pd.read_csv(labs / "data" / "messages.csv", keep_default_na=False, na_values=[""])
    march = pd.read_csv(final / "messages.csv", keep_default_na=False, na_values=[""])
    history = history[history["split"] == "train"]
    out = Path("q2-data")
    out.mkdir(exist_ok=True)
    messages = pd.concat([history, march], ignore_index=True)
    messages.to_csv(out / "messages.csv", index=False)
    shutil.copyfile(final / "stream.csv", out / "launch_month.csv")
    shutil.copyfile(final / "double_labelled.csv", out / "double_labelled.csv")
    print(messages.groupby("split").size().to_string())
    print(f"Wrote {out}/: {len(messages)} labelled messages, "
    f"{len(pd.read_csv(out / 'launch_month.csv'))} April messages to replay.")

    The April stream gets the name launch_month.csv, so that route drift replays April without a change to the code.

  3. Run it, then tell the route command to use the new folder:

    Terminal window
    python make_data.py ../nextia-learning-labs/C36
    export LARKROUTE_DATA=q2-data

    You see:

    OK double_labelled.csv
    OK messages.csv
    OK stream.csv
    split
    test 256
    train 6768
    validation 256
    Wrote q2-data/: 7280 labelled messages, 1120 April messages to replay.

    Run every route command of this assignment with LARKROUTE_DATA set, in the routing-q2 folder. Run the tests without it: they check the course data. Do not run route reset during the assignment: it deletes models/.

You never choose a setting on the messages that grade it. So each part of the data has one job:

Four boxes in time order. Inside a frame, March, 1,280 messages with labels: train, 768, with the course's 6,000, the models learn. Validation, 256, every choice: method, thresholds, zones. Test, 256, used once, after the choices are frozen. Then a second frame, April, the replay: 1 to 26 April, 1,120 messages, no labels: route them, watch the drift, the Robot mower R2 from 6 April.
March trains, chooses and tests. April is the replay, with no labels: you find its problems yourself.
Text description of the diagram

The diagram shows four boxes in time order. The first three are in a frame named “March: 1,280 messages with labels”. Box 1, train, 768 messages: together with the course’s 6,000, the models learn from them. Box 2, validation, 256 messages: every choice, the method, the thresholds and the zones. Box 3, test, 256 messages: used once, after the choices are frozen. An arrow leads to box 4, in a frame named “April: the replay”. From 1 to 26 April, 1,120 messages with no labels: you route them and watch the drift. The Robot mower R2 goes on sale on 6 April.

Uses Design the labels, Labelling guidelines, Measure agreement and The cost of each mistake.

Write LABELS.md, one or two pages:

  1. Run route agreement. Write the agreement and Cohen’s kappa on the 100 double-labelled messages. Compare them with the course’s 800 messages (87.9%, kappa 0.854).
  2. Name the two routes where Chloé and Grace disagree most. Give the counts next to the rates: on 100 messages, a route may have only a few messages.
  3. Read five of the messages where they disagree. For each one, say which rule of the guidelines decides it, or that no rule does.
  4. Add two rules to the guidelines for the quarter. The first is for devices: what counts as a setup question, and what goes to product or returns_refunds instead. The second is for the marketplace: does the channel change the route? Give each rule one example message.
  5. Copy the cost table above. Under it, say in two sentences which mistake costs Larkfield most this quarter, and why accuracy alone cannot see it.

Uses Keyword rules and regular expressions, Naive Bayes, logistic regression and linear SVM, Sentence embeddings for classification, Fine-tune a small encoder on CPU, Zero-shot without an LLM, What the extra accuracy costs and Cost per 1,000 messages.

Compare two baselines and at least four method families, all trained on the same messages and scored with the same cost table:

Family Rows
Baselines “everything to a person” (the cost of Larkfield before the router), and the simplest model you trust, for example TF-IDF + logistic regression
Rules “rules or a person”
Classical TF-IDF with logistic regression, a linear SVM or Naive Bayes
Embeddings or an encoder frozen embeddings + logistic regression, or the small encoder fine-tuned on CPU
Zero-shot or LLM GLiClass or NLI with one phrase per route (no labels), or an LLM with your own key (optional)

The project’s rows run with one command. The other rows need a few lines of your own. In a notebook, score them with larkroute.metrics.score(). Or add them to the project as in Purpose-built decision models: Add a new model as one more row. For a zero-shot model, write a phrase for devices too, for example “setting up a smart device, its app or its Wi-Fi”.

  1. First answer one question: which messages should the models learn from? Train TF-IDF + logistic regression three times: on March’s 768 messages only, on the course’s 6,000 only, and on both. Score each one on the validation set. Keep the best for the rest of the work, and write the reason in one sentence.
  2. Score every row on the validation set: route compare --split validation --methods "…" for the project’s methods, your own code for the others. Choose every setting there (for example C of the logistic regression on the embeddings).
  3. Write these columns for each row:
    • accuracy, and macro-F1 over the eight routes;
    • the recall of devices and of needs_person;
    • the accuracy on the marketplace, on the other channels, on English and on French;
    • the cost of the mistakes per 1,000, the cost to run per 1,000, the total, and the milliseconds per message.
  4. Freeze your choices. Then score every row on the test set once, and write the same columns next to the validation ones.
  5. Explain the difference between validation and test for the two cheapest rows. On 256 messages, one urgent message in the wrong queue adds 20 ÷ 256 × 1,000 = US$78.1 per 1,000. A missed takeover adds US$781.3.
  6. For your best row, write the recall of each route on the test set. Add the accuracy of each channel and language. Put the number of messages next to every rate.

Put the results in comparison.csv, with one row per method and split. Do not sort by accuracy: sort by the total cost per 1,000.

Uses Is the confidence true?, Decide, review or send to a person, Cascades and Imbalance and rare classes.

  1. Choose one method, or a cascade of two. Give the reason in three to five sentences: the total cost, the review load, the speed, and how easily Grace’s team can trust it.
  2. Check its confidence on the validation set: the reliability table and the ECE. A threshold on a confidence that is not true moves the wrong messages.
  3. Choose the decide, review and person zones on the validation set, with the review and person zones together at most 10%. For TF-IDF + logistic regression, route zones --review-max 0.10 does it. For another method, use larkroute.zones.choose().
  4. Report the zones once on the test set: the share in each zone, the errors in the decide zone and the cost per 1,000. Compare the cost with the same method without zones.
  5. Turn the shares into work. Write the messages a day for Grace’s team, at about 43 messages a day. Write the cost per month, at about 1,300 messages.
  6. Run the same choice with a limit of 5% and of 20%. Say if the limit binds: does the chosen threshold change?

Uses A routing service.

  1. Train the first version and make it current:

    Terminal window
    route train --version v1 --make-current

    route train trains TF-IDF + logistic regression and chooses its zones on validation. If you chose another method in Part 3, change the model line of larkroute/commands/train.py and say so in the report.

  2. Start the service with route serve. In a second terminal, ask /health, then send one marketplace message (PowerShell: Invoke-RestMethod, as in the service lesson):

    Terminal window
    curl -s http://127.0.0.1:8036/health
    curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \
    -d '{"message_id": "Q-100001", "text": "My order LK-123456 from the marketplace has not arrived.", "channel": "marketplace"}'
  3. Send an empty text and check that the answer is HTTP 422.

  4. Stop the service (Ctrl+C). Run the tests without LARKROUTE_DATA: python -m pytest -q must say 42 passed. Add one test of your own for something you changed, and run them again.

Uses Drift and new topics, When you have few labels, Few labels with embeddings and Add a label or retrain safely.

  1. Run route drift. Write the weekly table and the alarms. For each alarm, say what moved: the route shares (PSI), the unsure share, or both.
  2. Run route drift --topics --since 2026-04-06. Read the messages of each group. Name each group in a few words, and say which groups are a new topic and which are not. If two topics share a group, try --k 8.
  3. Decide, with evidence, whether the new topic needs a new route or belongs to a route that exists. Use your guideline rule from Part 1.
  4. Find where the current model sends the new topic’s messages, and in which zones. A message in the decide zone of a wrong queue costs more than a message in review.
  1. Choose April messages to label. Take the messages since 6 April with the new topic’s words. Add a random sample of about 60 other April messages, so that the model does not learn “April means the new product”.
  2. Label them yourself with your guidelines, as Chloé would, in a file my_april_labels.csv with the columns message_id, route, split. Put the messages of 20 April and later in split = check, the earlier ones in train. Keep at least 20 messages of the new topic in check.
  3. Train a candidate: route train --version v2 --extra my_april_labels.csv. Without --make-current, the service still loads v1.
  4. Check the candidate on the check messages, which it did not learn from. Compare v1 and v2: the recall and precision of the route that receives the new topic. Set the bar before you look, for example a recall of 0.80.
  5. Run route release --candidate v2. Read every slice. Write which slices are too small to judge.
  6. If both checks pass, restart the service and check /health. If one fails, the current model stays v1: label more messages and try again. Write each try.

Uses What the extra accuracy costs, Cascades: When to use a cascade, and what it costs and Learn from corrections.

Write DECISION.md, one page, for Grace and Omar, who do not code. It must say:

  1. The router for the quarter: the method, the zones and the model version.
  2. Its cost per 1,000 messages and per month, against “everything to a person”. Give the test number and the validation number, and say why they differ.
  3. The work for Grace’s team: messages a day in review and to a senior person.
  4. What happened in April: what you found, what you changed, and the evidence that the change is safe.
  5. An answer to Tomás, with numbers: what would an LLM add, and what would it cost in time and privacy?
  6. What the router cannot do. Give at least one weak place with a number, for example a route, a language or a channel.
  7. When to look again: the alarms you watch, who reads them, and what happens when the next product arrives.
Six boxes in two rows of three. 1. Labels and costs: agreement on 100, guidelines, cost table. 2. Comparison: 2 baselines, 4 families, validation, then test once. 3. The choice: method or cascade, zones, review load. 4. The service: v1 running, tests pass. 5. April: drift, new topic, a few labels, v2 released. 6. The decision: cost per month, limits, next steps.
The routing system, in the order that you build it. Each part uses the one before.
Text description of the diagram

The diagram shows six boxes, in two rows of three. An arrow joins the rows. 1, Labels and costs: agreement on 100 messages, the guidelines and the cost table. 2, Comparison: two baselines and four families, on validation, then on test once. 3, The choice: a method or a cascade, its zones and the review load. 4, The service: v1 running, and the tests pass. 5, April: the drift, the new topic, a few labels and v2 released. 6, The decision: the cost per month, the limits and the next steps.

routing-q2/
LABELS.md Part 1
REPORT.md Parts 2 to 5, with the commands that make every number
comparison.csv Part 2
my_april_labels.csv Part 5
make_data.py and every script or notebook that you used
models/ v1, v2 and their cards (models/current names the version in use)
tests/ with your own test
DECISION.md Part 6
requirements.txt exact versions

REPORT.md has these sections:

# Routing for the next quarter: report
## Data checks (checksums, rows, splits, the small routes)
## Labels and agreement (Part 1, short; the full text is in LABELS.md)
## Which messages to learn from (Part 2, step 1)
## Comparison (Part 2: validation and test, sorted by total cost, slices with counts)
## The router and its zones (Part 3: reason, calibration, zones on validation, test once, work a day)
## The service (Part 4: commands and outputs, tests)
## April (Part 5: drift table, topics, labels, checks of v2, release)
## Labels and limits (which numbers are measured, constructed or assumed)

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own work, or ask a colleague. Each item has points: all of them when demonstrated, half when partially demonstrated, none when not yet demonstrated. The points show where your work is weakest. But the course is complete only when all six items are demonstrated: 100 points. You can improve an item and send it again.

Required item Points Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Labels and costs (Measure agreement) 10 Agreement and kappa on the 100 messages, the two weakest routes with counts, five disagreements explained, a rule for devices and one for the marketplace, and the cost table with the costliest mistake named. No counts next to the rates. No guidelines for the new route or the new channel.
2. Comparison (Modules 3 to 5) 25 The two baselines and four families, trained on the same messages, chosen on validation, tested once, with the columns above, sorted by total cost. The training question is answered with numbers. The validation-test difference is explained. One family is scored on other messages, or with another cost table. No simple baseline. Or a setting chosen on the test set.
3. The router and its zones (Module 8) 20 A reason beyond accuracy, the calibration checked, zones chosen on validation under the 10% limit, test once, and the work a day and the cost a month. No calibration check, or no work a day. Zones chosen on the test set. Or a review load above the limit.
4. The service (Module 9) 10 v1 runs, answers a marketplace message with its zone and versions, refuses an empty text, and the tests pass with one test of your own. The tests pass, but no test of your own. The service does not run from the instructions in the report.
5. April (Modules 2 and 9) 20 The drift found with the weekly table and the topics, the new topic named and placed, a few labels with a time split, the new topic checked on messages the model did not learn from, the release check read slice by slice, and v2 current only after both checks. The check of the new topic uses messages that v2 learned from. No check of the new topic. Or v2 made current without the release check.
6. The decision (all modules) 15 One page in plain words: the router, its cost a month against everything to a person, the work a day, April, an answer to Tomás with numbers, a weak place with a number, and when to look again. The limits are general, with no number. No cost. Or a claim that the router is right for every message.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • A method chosen by accuracy alone, with no cost per 1,000 messages. (Items 2 and 3.)
  • A threshold or a method chosen on the test set, or changed after you looked at the test results. (Items 2 and 3.)
  • A comparison without a simple baseline. (Item 2.)
  • A new model released without a check of what it added on messages it did not learn from. (Item 5.)

A good routing system is honest before it is impressive. It has three things.

  • Honest measures under the course’s costs. The cost per 1,000 messages, the review load and the recall of the small routes, each with its count. The assumptions are next to them, and each number has a label: measured, constructed or assumed.
  • A simple baseline in the comparison. TF-IDF + logistic regression trains in seconds. If a bigger model does not beat it on cost, the report says so and keeps the simple one.
  • A decision with its cost. “We route with this model and these zones. It costs about this much a month, against this much for a person who reads everything. Grace’s team reads about this many messages a day. It is weak here. We look again when this alarm fires.”
Show our results, to check your method

Open this only after you have your own numbers. We ran the assignment once, from the public data only, with the steps above. If your choices differ, your numbers differ. Then compare the method, not the numbers. We labelled the April messages ourselves, by reading them, as you do.

  • Agreement: 82.0% on the 100 messages, Cohen’s kappa 0.787: lower than the course’s 0.854. The weakest routes are returns_refunds and billing (24 messages each, about half of them disagreed). The commonest pair is a refund or a charge, as in the course.
  • Which messages to learn from (TF-IDF + logistic regression, cost per 1,000 on validation): March only US$2,062.5, the course’s 6,000 only US$539.1, both US$171.9. The 6,000 never saw a devices message, so they found none of them. March alone has too few messages.
The same model, trained on three sets of messages. Both sets together cost the least, by far: the old messages give the volume, and March gives the new route and the new channel.
Text description of the chart

A column chart with three columns: the cost per 1,000 validation messages of TF-IDF with logistic regression, trained on three sets. March only, 768 messages: 2,062.5 dollars. The course history only, 6,000 messages: 539.1 dollars. Both: 171.9 dollars, in a different colour. On the test set, the order is the same: 1,367.2, 859.4 and 296.9 dollars.

  • Comparison (trained on both; validation, then test once; total cost per 1,000):

    Method Validation Test Test accuracy
    TF-IDF words only + LR 125.0 281.2 0.945
    TF-IDF + linear SVM 140.6 273.4 0.945
    TF-IDF + logistic regression 171.9 296.9 0.941
    strong rules + logistic regression 250.0 453.1 0.910
    TF-IDF + Naive Bayes 328.1 210.9 0.945
    embeddings MiniLM-L12 + LR 593.8 382.8 0.938
    everything to a person 1,500.0 1,500.0 0
    rules or a person 1,658.2 2,279.3 0.477
    GLiClass zero-shot (no labels) 4,007.8 4,562.5 0.535

    The TF-IDF family is the cheapest, as in the course. Naive Bayes is the most expensive TF-IDF row on validation and the cheapest on test. On 256 messages, a few urgent mistakes more or less swap the order. That is why you choose on validation and test once. On test, French is weaker than English for the chosen model (0.899 on 69 messages against 0.957 on 187). The marketplace is close to the other channels (0.933 on 75 messages). GLiClass finds 11 of the 15 device messages without any label, but it costs more than a person who reads everything.

  • Zones (review and person at most 10%, chosen on validation): TF-IDF + logistic regression decides at 0.49 or more. Validation: decide 94.1%, cost US$132.8 per 1,000. Test: decide 92.6%, review 2.7%, person 4.7%, cost US$259.8. The linear SVM decides at 0.69: US$113.3 on validation and US$238.3 on test, with 8.6% in review. The limit of 10% does not bind for logistic regression: 5%, 15% and 20% give nearly the same thresholds. Its person zone is the messages that the model sends to needs_person.

  • The service: v1 learned from 6,768 messages and 8 routes, with a validation accuracy of 0.957. All 42 tests pass.

  • Drift in April (v1; validation unsure share 5.9%): the unsure share is 8%, 11.4%, 13.8% and 11.5% by week, with an alarm in the last three weeks. The PSI of the route shares stays under 0.06: the shares hardly move. The new topic hides inside other queues. Of 113 unsure messages since 6 April, 4 of the 6 groups are about the mower: the charging station, the boundary wire (in English and in French) and the mowing schedule.

  • Where the mower went: we found 104 messages since 6 April with the mower’s words. v1 sent them to returns_refunds (36), product (27), delivery (17), billing (13) and others, never to devices. 42 of them were in the decide zone. We read and labelled 164 messages: 84 of the 104 are setup questions, so they belong to devices. The others are returns, orders, deliveries, a charge, a takeover or a legal threat that mention a mower.

  • The release: v2 learned from 6,882 messages (114 of ours). On our 50 check messages from 20 April, devices recall went from 0.000 to 0.962 (25 of 26), with a precision of 0.893. route release passed: no slice got worse. But three route slices have fewer than 30 messages: account 21, devices 15 and needs_person 12. The check shows them and does not judge them.

Common mistake

The router never learned the new route

Symptom: the device messages of March go to product or returns_refunds, and devices recall is 0.

Cause: the model learned from the course’s 6,000 messages only. None of them is a devices message.

Fix: learn from the history and March together. Check that every route of the test set is in the training set before you score anything.

Common mistake

The cheapest row changes between validation and test

Symptom: the cheapest method on validation is not the cheapest on test, and you want to change your choice.

Cause: 256 messages are few. One urgent message in the wrong queue adds US$78.1 per 1,000. Four of them change the order of the rows.

Fix: keep the choice that you made on validation. Report both numbers, explain the difference, and say how many messages it rests on.

Common mistake

No alarm on the route shares, so no drift

Symptom: the PSI stays low in April, and the report says “no drift”.

Cause: the new topic does not move the route shares much. Its messages spread over five old queues.

Fix: watch the unsure share too, and read the unsure messages by group. A new topic often shows first as more unsure messages, not as new shares.

Common mistake

The release passes, but the new topic was never checked

Symptom: route release says RELEASED, and the report says the mower is fixed.

Cause: devices already existed in v1, so the release check saw no new route. It checked only the old slices of March.

Fix: check the new topic yourself, on labelled April messages that the candidate did not learn from, before the release.

Common mistake

Every April message is labelled as the new topic

Symptom: after the new version, ordinary April messages, for example a mower return, go to devices.

Cause: the labels came only from the new topic’s words, and every one of them got the new route.

Fix: read each message before you label it: a return of a mower is still a return. Add a random sample of other April messages, so that the model sees the old routes in April too.

Recap

You can now take a changing inbox to a router that a team can trust. You write guidelines and measure agreement, and you price each mistake. You compare methods on one fair test, with a simple baseline, and choose by cost. You set zones under the team’s limit and run the router as a service. You find a new topic without labels, add it with a few labels, and release it only after a check of what changed. And you tell the owners what the router costs and what it cannot do. To keep going, Evaluating and Testing AI Systems goes deeper into datasets, slices and regression checks. Building Reliable Applications with LLM APIs builds the prompts and structured outputs of an LLM router.