Final assignment
- After the case studies
- about 3 hr
Where we are
Each module added one piece and tried it alone: the labels, the methods, the costs, the cascade, the zones and the service. In this assignment, you use them all at once. Larkfield adds a marketplace channel and two new products, and you deliver the routing system for the next quarter.
What you will be able to do
Use the whole course on one new quarter at Larkfield. You label and price the mistakes, compare methods on one fair test, and choose a router with its zones. You run it as a service, watch April for drift and add a new topic safely. Then you write the decision for Grace and Omar, with its cost and its limits.
Before you startWhat to know and to have ready
What you need to know
- Modules 1 to 9. Each part below names the lessons that it uses. The LLM lessons of Module 5 are optional here.
- The certificate also asks for at least one case study of Module 10. To see the same steps on real data first, open Route banking questions.
What you need on your computer
- Python 3.12, a terminal, and a notebook if you like one. Parts 1 to 3 work well in a notebook. Parts 4 and 5 need the project and a terminal.
- The course’s labs repository with the folder
C36, as in Meet the inbox. Rungit -C nextia-learning-labs pullto get the folderC36/final. - The project snapshot
C36/snapshots/end-of-m09: theroutecommand with the service, the drift check and the release check. - About 300 MB of disk for the virtual environment, and about 500 MB more if you use the embedding model of Module 4. No account, key or money. An LLM row is optional and needs your own key.
Tested on macOS (Apple silicon), with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. We wrote the Windows (PowerShell) commands, but we did not run them.
The situation: a new channel and two new products
Section titled “The situation: a new channel and two new products”In March, Larkfield starts to sell on an online marketplace. Customers now also write from there: about 3 messages in 10. Two products are new. The Patio heater is on sale from March. The Robot mower R2 goes on sale on 6 April. The Grow Hub, which arrived in the launch month of Module 9, has its own route, devices, for setup questions about smart products.
Grace, the support lead, writes to Priya:
“I need the router for the next quarter. My team can read at most 1 message in 10. I want the right team for the rest, and I want to know what the mistakes cost us. Please also tell me what you will do when the next product arrives.”
Omar, who owns the mower launch, adds: “The mower questions must not wait in the wrong queue.” Tomás asks: “Why not send every message to an LLM?” Your answer to each of them must come with evidence.
We, the course team, made the data with the same generator as the course inbox. We know the right routes of the April messages. We do not give them to you: the labs folder has no answer key. You find the problems in April with the course’s methods. The data is synthetic, like every Larkfield message.
The costs and the limits
Section titled “The costs and the limits”Use the course’s cost table without changes, so that your numbers compare with other learners’ numbers. These are assumptions, the same as in The cost of each mistake:
| Assumption | Value (US$ per message) |
|---|---|
| A review by a person | 1.50 |
| A message in the wrong queue | 4 |
| An urgent message in the wrong queue | 20 |
| A missed account takeover | 200 |
A needs_person message sent to a team |
40 |
| A team message sent to a senior person | 2 |
| Everything to a person (Larkfield before the router) | 1,500 per 1,000 messages |
| Grace’s review limit | the review and person zones together: at most 10% of the messages |
| The volume | about 1,300 messages a month (April: 1,120 messages in 26 days, about 43 a day) |
If you want to try a second cost table, add it as an extra and say why. Keep the first one as the main result.
The data
Section titled “The data”The data is in labs C36/final/data/: three files and a list of their checksums. A checksum is a long number computed from the bytes of a file. If one byte changes, the checksum changes.
| File | Rows | One row | Columns |
|---|---|---|---|
messages.csv (334 KB) |
1,280 | one message of March 2026, with its labels | message_id, created_at, channel, language, text, route, topic, priority, routes, split, synthetic |
double_labelled.csv (3 KB) |
100 | one March training message, labelled by Chloé and by Grace | message_id, label_chloe, label_grace |
stream.csv (238 KB) |
1,120 | one message of 1 to 26 April 2026, with no labels | message_id, created_at, channel, language, text, synthetic |
What the columns mean:
route: the team that must act first, orneeds_person. There are eight routes: the course’s seven anddevices.topic: the kind of request inside the route, for examplegrow_hub_setuporaccount_takeover.priorityisurgent,normalorlow.routes: one route, or two routes with|between them when a message asks for two things. The first one is the label.channel:email,chat,socialor, new,marketplace.language:enorfr.split:train(768),validation(256) ortest(256).
You also use the course data, labs C36/data/: the 6,000 training messages of the history, the guidelines and the cost table. Here is March by route. Look at the small routes before you read any score:
Text description of the chart
A column chart with eight columns, one per route, from the largest to the smallest. Delivery 280, returns and refunds 245, product 171, orders 168, billing 159, account 118, devices 83 and needs_person 56. Devices and needs_person have their own colours. The account route includes only 10 account takeovers: 6 in train, 3 in validation and 1 in test.
Only one test message is an account takeover. So a takeover recall of 1.0 on the test set means “one message was right”, not “every takeover is found”. Say so wherever you quote it.
Set up the folder
Section titled “Set up the folder”-
Copy the finished project into a new folder,
routing-q2, and install it:Terminal window cp -R nextia-learning-labs/C36/snapshots/end-of-m09 routing-q2cd routing-q2python3.12 -m venv .venvsource .venv/bin/activatepip install -r requirements.txtpython get_data.pyTerminal window Copy-Item -Recurse nextia-learning-labs\C36\snapshots\end-of-m09 routing-q2cd routing-q2py -3.12 -m venv .venv.venv\Scripts\Activate.ps1pip install -r requirements.txtpython get_data.pyget_data.pyputs the course data indata/. The project’s 42 tests use it. -
Save this script as
make_data.pyinrouting-q2. It checks the checksums, then writes a data folder for the quarter,q2-data/. The models learn from the course’s 6,000 training messages and March’s 768. Validation and test are March only, because the quarter looks like March, not like last year.import hashlibimport shutilimport sysfrom pathlib import Pathimport pandas as pdlabs = Path(sys.argv[1])final = labs / "final" / "data"for line in (final / "SHA256SUMS").read_text().splitlines():digest, name = line.split()ok = hashlib.sha256((final / name).read_bytes()).hexdigest() == digestprint("OK " if ok else "BAD", name)if not ok:raise SystemExit("A file is not the published one: download it again.")history = pd.read_csv(labs / "data" / "messages.csv", keep_default_na=False, na_values=[""])march = pd.read_csv(final / "messages.csv", keep_default_na=False, na_values=[""])history = history[history["split"] == "train"]out = Path("q2-data")out.mkdir(exist_ok=True)messages = pd.concat([history, march], ignore_index=True)messages.to_csv(out / "messages.csv", index=False)shutil.copyfile(final / "stream.csv", out / "launch_month.csv")shutil.copyfile(final / "double_labelled.csv", out / "double_labelled.csv")print(messages.groupby("split").size().to_string())print(f"Wrote {out}/: {len(messages)} labelled messages, "f"{len(pd.read_csv(out / 'launch_month.csv'))} April messages to replay.")The April stream gets the name
launch_month.csv, so thatroute driftreplays April without a change to the code. -
Run it, then tell the
routecommand to use the new folder:Terminal window python make_data.py ../nextia-learning-labs/C36export LARKROUTE_DATA=q2-dataTerminal window python make_data.py ..\nextia-learning-labs\C36$env:LARKROUTE_DATA = "q2-data"You see:
OK double_labelled.csvOK messages.csvOK stream.csvsplittest 256train 6768validation 256Wrote q2-data/: 7280 labelled messages, 1120 April messages to replay.Run every
routecommand of this assignment withLARKROUTE_DATAset, in therouting-q2folder. Run the tests without it: they check the course data. Do not runroute resetduring the assignment: it deletesmodels/.
Two months, four jobs
Section titled “Two months, four jobs”You never choose a setting on the messages that grade it. So each part of the data has one job:
Text description of the diagram
The diagram shows four boxes in time order. The first three are in a frame named “March: 1,280 messages with labels”. Box 1, train, 768 messages: together with the course’s 6,000, the models learn from them. Box 2, validation, 256 messages: every choice, the method, the thresholds and the zones. Box 3, test, 256 messages: used once, after the choices are frozen. An arrow leads to box 4, in a frame named “April: the replay”. From 1 to 26 April, 1,120 messages with no labels: you route them and watch the drift. The Robot mower R2 goes on sale on 6 April.
Part 1: labels and costs
Section titled “Part 1: labels and costs”Uses Design the labels, Labelling guidelines, Measure agreement and The cost of each mistake.
Write LABELS.md, one or two pages:
- Run
route agreement. Write the agreement and Cohen’s kappa on the 100 double-labelled messages. Compare them with the course’s 800 messages (87.9%, kappa 0.854). - Name the two routes where Chloé and Grace disagree most. Give the counts next to the rates: on 100 messages, a route may have only a few messages.
- Read five of the messages where they disagree. For each one, say which rule of the guidelines decides it, or that no rule does.
- Add two rules to the guidelines for the quarter. The first is for
devices: what counts as a setup question, and what goes toproductorreturns_refundsinstead. The second is for the marketplace: does the channel change the route? Give each rule one example message. - Copy the cost table above. Under it, say in two sentences which mistake costs Larkfield most this quarter, and why accuracy alone cannot see it.
Part 2: the comparison
Section titled “Part 2: the comparison”Uses Keyword rules and regular expressions, Naive Bayes, logistic regression and linear SVM, Sentence embeddings for classification, Fine-tune a small encoder on CPU, Zero-shot without an LLM, What the extra accuracy costs and Cost per 1,000 messages.
The rows
Section titled “The rows”Compare two baselines and at least four method families, all trained on the same messages and scored with the same cost table:
| Family | Rows |
|---|---|
| Baselines | “everything to a person” (the cost of Larkfield before the router), and the simplest model you trust, for example TF-IDF + logistic regression |
| Rules | “rules or a person” |
| Classical | TF-IDF with logistic regression, a linear SVM or Naive Bayes |
| Embeddings or an encoder | frozen embeddings + logistic regression, or the small encoder fine-tuned on CPU |
| Zero-shot or LLM | GLiClass or NLI with one phrase per route (no labels), or an LLM with your own key (optional) |
The project’s rows run with one command. The other rows need a few lines of your own. In a notebook, score them with larkroute.metrics.score(). Or add them to the project as in Purpose-built decision models: Add a new model as one more row. For a zero-shot model, write a phrase for devices too, for example “setting up a smart device, its app or its Wi-Fi”.
One procedure for every row
Section titled “One procedure for every row”- First answer one question: which messages should the models learn from? Train TF-IDF + logistic regression three times: on March’s 768 messages only, on the course’s 6,000 only, and on both. Score each one on the validation set. Keep the best for the rest of the work, and write the reason in one sentence.
- Score every row on the validation set:
route compare --split validation --methods "…"for the project’s methods, your own code for the others. Choose every setting there (for exampleCof the logistic regression on the embeddings). - Write these columns for each row:
- accuracy, and macro-F1 over the eight routes;
- the recall of
devicesand ofneeds_person; - the accuracy on the marketplace, on the other channels, on English and on French;
- the cost of the mistakes per 1,000, the cost to run per 1,000, the total, and the milliseconds per message.
- Freeze your choices. Then score every row on the test set once, and write the same columns next to the validation ones.
- Explain the difference between validation and test for the two cheapest rows. On 256 messages, one urgent message in the wrong queue adds 20 ÷ 256 × 1,000 = US$78.1 per 1,000. A missed takeover adds US$781.3.
- For your best row, write the recall of each route on the test set. Add the accuracy of each channel and language. Put the number of messages next to every rate.
Put the results in comparison.csv, with one row per method and split. Do not sort by accuracy: sort by the total cost per 1,000.
Part 3: choose the router and its zones
Section titled “Part 3: choose the router and its zones”Uses Is the confidence true?, Decide, review or send to a person, Cascades and Imbalance and rare classes.
- Choose one method, or a cascade of two. Give the reason in three to five sentences: the total cost, the review load, the speed, and how easily Grace’s team can trust it.
- Check its confidence on the validation set: the reliability table and the ECE. A threshold on a confidence that is not true moves the wrong messages.
- Choose the decide, review and person zones on the validation set, with the review and person zones together at most 10%. For TF-IDF + logistic regression,
route zones --review-max 0.10does it. For another method, uselarkroute.zones.choose(). - Report the zones once on the test set: the share in each zone, the errors in the decide zone and the cost per 1,000. Compare the cost with the same method without zones.
- Turn the shares into work. Write the messages a day for Grace’s team, at about 43 messages a day. Write the cost per month, at about 1,300 messages.
- Run the same choice with a limit of 5% and of 20%. Say if the limit binds: does the chosen threshold change?
Part 4: the service
Section titled “Part 4: the service”Uses A routing service.
-
Train the first version and make it current:
Terminal window route train --version v1 --make-currentroute traintrains TF-IDF + logistic regression and chooses its zones on validation. If you chose another method in Part 3, change the model line oflarkroute/commands/train.pyand say so in the report. -
Start the service with
route serve. In a second terminal, ask/health, then send one marketplace message (PowerShell:Invoke-RestMethod, as in the service lesson):Terminal window curl -s http://127.0.0.1:8036/healthcurl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \-d '{"message_id": "Q-100001", "text": "My order LK-123456 from the marketplace has not arrived.", "channel": "marketplace"}' -
Send an empty text and check that the answer is HTTP 422.
-
Stop the service (Ctrl+C). Run the tests without
LARKROUTE_DATA:python -m pytest -qmust say42 passed. Add one test of your own for something you changed, and run them again.
Part 5: April
Section titled “Part 5: April”Uses Drift and new topics, When you have few labels, Few labels with embeddings and Add a label or retrain safely.
Find what changed
Section titled “Find what changed”- Run
route drift. Write the weekly table and the alarms. For each alarm, say what moved: the route shares (PSI), the unsure share, or both. - Run
route drift --topics --since 2026-04-06. Read the messages of each group. Name each group in a few words, and say which groups are a new topic and which are not. If two topics share a group, try--k 8. - Decide, with evidence, whether the new topic needs a new route or belongs to a route that exists. Use your guideline rule from Part 1.
- Find where the current model sends the new topic’s messages, and in which zones. A message in the decide zone of a wrong queue costs more than a message in review.
A few labels, then a safe release
Section titled “A few labels, then a safe release”- Choose April messages to label. Take the messages since 6 April with the new topic’s words. Add a random sample of about 60 other April messages, so that the model does not learn “April means the new product”.
- Label them yourself with your guidelines, as Chloé would, in a file
my_april_labels.csvwith the columnsmessage_id, route, split. Put the messages of 20 April and later insplit=check, the earlier ones intrain. Keep at least 20 messages of the new topic incheck. - Train a candidate:
route train --version v2 --extra my_april_labels.csv. Without--make-current, the service still loadsv1. - Check the candidate on the
checkmessages, which it did not learn from. Comparev1andv2: the recall and precision of the route that receives the new topic. Set the bar before you look, for example a recall of 0.80. - Run
route release --candidate v2. Read every slice. Write which slices are too small to judge. - If both checks pass, restart the service and check
/health. If one fails, the current model staysv1: label more messages and try again. Write each try.
Part 6: the decision
Section titled “Part 6: the decision”Uses What the extra accuracy costs, Cascades: When to use a cascade, and what it costs and Learn from corrections.
Write DECISION.md, one page, for Grace and Omar, who do not code. It must say:
- The router for the quarter: the method, the zones and the model version.
- Its cost per 1,000 messages and per month, against “everything to a person”. Give the test number and the validation number, and say why they differ.
- The work for Grace’s team: messages a day in review and to a senior person.
- What happened in April: what you found, what you changed, and the evidence that the change is safe.
- An answer to Tomás, with numbers: what would an LLM add, and what would it cost in time and privacy?
- What the router cannot do. Give at least one weak place with a number, for example a route, a language or a channel.
- When to look again: the alarms you watch, who reads them, and what happens when the next product arrives.
What to hand in
Section titled “What to hand in”Text description of the diagram
The diagram shows six boxes, in two rows of three. An arrow joins the rows. 1, Labels and costs: agreement on 100 messages, the guidelines and the cost table. 2, Comparison: two baselines and four families, on validation, then on test once. 3, The choice: a method or a cascade, its zones and the review load. 4, The service: v1 running, and the tests pass. 5, April: the drift, the new topic, a few labels and v2 released. 6, The decision: the cost per month, the limits and the next steps.
routing-q2/ LABELS.md Part 1 REPORT.md Parts 2 to 5, with the commands that make every number comparison.csv Part 2 my_april_labels.csv Part 5 make_data.py and every script or notebook that you used models/ v1, v2 and their cards (models/current names the version in use) tests/ with your own test DECISION.md Part 6 requirements.txt exact versionsREPORT.md has these sections:
# Routing for the next quarter: report
## Data checks (checksums, rows, splits, the small routes)## Labels and agreement (Part 1, short; the full text is in LABELS.md)## Which messages to learn from (Part 2, step 1)## Comparison (Part 2: validation and test, sorted by total cost, slices with counts)## The router and its zones (Part 3: reason, calibration, zones on validation, test once, work a day)## The service (Part 4: commands and outputs, tests)## April (Part 5: drift table, topics, labels, checks of v2, release)## Labels and limits (which numbers are measured, constructed or assumed)Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own work, or ask a colleague. Each item has points: all of them when demonstrated, half when partially demonstrated, none when not yet demonstrated. The points show where your work is weakest. But the course is complete only when all six items are demonstrated: 100 points. You can improve an item and send it again.
| Required item | Points | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|---|
| 1. Labels and costs (Measure agreement) | 10 | Agreement and kappa on the 100 messages, the two weakest routes with counts, five disagreements explained, a rule for devices and one for the marketplace, and the cost table with the costliest mistake named. |
No counts next to the rates. | No guidelines for the new route or the new channel. |
| 2. Comparison (Modules 3 to 5) | 25 | The two baselines and four families, trained on the same messages, chosen on validation, tested once, with the columns above, sorted by total cost. The training question is answered with numbers. The validation-test difference is explained. | One family is scored on other messages, or with another cost table. | No simple baseline. Or a setting chosen on the test set. |
| 3. The router and its zones (Module 8) | 20 | A reason beyond accuracy, the calibration checked, zones chosen on validation under the 10% limit, test once, and the work a day and the cost a month. | No calibration check, or no work a day. | Zones chosen on the test set. Or a review load above the limit. |
| 4. The service (Module 9) | 10 | v1 runs, answers a marketplace message with its zone and versions, refuses an empty text, and the tests pass with one test of your own. |
The tests pass, but no test of your own. | The service does not run from the instructions in the report. |
| 5. April (Modules 2 and 9) | 20 | The drift found with the weekly table and the topics, the new topic named and placed, a few labels with a time split, the new topic checked on messages the model did not learn from, the release check read slice by slice, and v2 current only after both checks. |
The check of the new topic uses messages that v2 learned from. |
No check of the new topic. Or v2 made current without the release check. |
| 6. The decision (all modules) | 15 | One page in plain words: the router, its cost a month against everything to a person, the work a day, April, an answer to Tomás with numbers, a weak place with a number, and when to look again. | The limits are general, with no number. | No cost. Or a claim that the router is right for every message. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- A method chosen by accuracy alone, with no cost per 1,000 messages. (Items 2 and 3.)
- A threshold or a method chosen on the test set, or changed after you looked at the test results. (Items 2 and 3.)
- A comparison without a simple baseline. (Item 2.)
- A new model released without a check of what it added on messages it did not learn from. (Item 5.)
What good looks like
Section titled “What good looks like”A good routing system is honest before it is impressive. It has three things.
- Honest measures under the course’s costs. The cost per 1,000 messages, the review load and the recall of the small routes, each with its count. The assumptions are next to them, and each number has a label: measured, constructed or assumed.
- A simple baseline in the comparison. TF-IDF + logistic regression trains in seconds. If a bigger model does not beat it on cost, the report says so and keeps the simple one.
- A decision with its cost. “We route with this model and these zones. It costs about this much a month, against this much for a person who reads everything. Grace’s team reads about this many messages a day. It is weak here. We look again when this alarm fires.”
Show our results, to check your method
Open this only after you have your own numbers. We ran the assignment once, from the public data only, with the steps above. If your choices differ, your numbers differ. Then compare the method, not the numbers. We labelled the April messages ourselves, by reading them, as you do.
- Agreement: 82.0% on the 100 messages, Cohen’s kappa 0.787: lower than the course’s 0.854. The weakest routes are
returns_refundsandbilling(24 messages each, about half of them disagreed). The commonest pair is a refund or a charge, as in the course. - Which messages to learn from (TF-IDF + logistic regression, cost per 1,000 on validation): March only US$2,062.5, the course’s 6,000 only US$539.1, both US$171.9. The 6,000 never saw a
devicesmessage, so they found none of them. March alone has too few messages.
Text description of the chart
A column chart with three columns: the cost per 1,000 validation messages of TF-IDF with logistic regression, trained on three sets. March only, 768 messages: 2,062.5 dollars. The course history only, 6,000 messages: 539.1 dollars. Both: 171.9 dollars, in a different colour. On the test set, the order is the same: 1,367.2, 859.4 and 296.9 dollars.
-
Comparison (trained on both; validation, then test once; total cost per 1,000):
Method Validation Test Test accuracy TF-IDF words only + LR 125.0 281.2 0.945 TF-IDF + linear SVM 140.6 273.4 0.945 TF-IDF + logistic regression 171.9 296.9 0.941 strong rules + logistic regression 250.0 453.1 0.910 TF-IDF + Naive Bayes 328.1 210.9 0.945 embeddings MiniLM-L12 + LR 593.8 382.8 0.938 everything to a person 1,500.0 1,500.0 0 rules or a person 1,658.2 2,279.3 0.477 GLiClass zero-shot (no labels) 4,007.8 4,562.5 0.535 The TF-IDF family is the cheapest, as in the course. Naive Bayes is the most expensive TF-IDF row on validation and the cheapest on test. On 256 messages, a few urgent mistakes more or less swap the order. That is why you choose on validation and test once. On test, French is weaker than English for the chosen model (0.899 on 69 messages against 0.957 on 187). The marketplace is close to the other channels (0.933 on 75 messages). GLiClass finds 11 of the 15 device messages without any label, but it costs more than a person who reads everything.
-
Zones (review and person at most 10%, chosen on validation): TF-IDF + logistic regression decides at 0.49 or more. Validation: decide 94.1%, cost US$132.8 per 1,000. Test: decide 92.6%, review 2.7%, person 4.7%, cost US$259.8. The linear SVM decides at 0.69: US$113.3 on validation and US$238.3 on test, with 8.6% in review. The limit of 10% does not bind for logistic regression: 5%, 15% and 20% give nearly the same thresholds. Its person zone is the messages that the model sends to
needs_person. -
The service:
v1learned from 6,768 messages and 8 routes, with a validation accuracy of 0.957. All 42 tests pass. -
Drift in April (
v1; validation unsure share 5.9%): the unsure share is 8%, 11.4%, 13.8% and 11.5% by week, with an alarm in the last three weeks. The PSI of the route shares stays under 0.06: the shares hardly move. The new topic hides inside other queues. Of 113 unsure messages since 6 April, 4 of the 6 groups are about the mower: the charging station, the boundary wire (in English and in French) and the mowing schedule. -
Where the mower went: we found 104 messages since 6 April with the mower’s words.
v1sent them toreturns_refunds(36),product(27),delivery(17),billing(13) and others, never todevices. 42 of them were in the decide zone. We read and labelled 164 messages: 84 of the 104 are setup questions, so they belong todevices. The others are returns, orders, deliveries, a charge, a takeover or a legal threat that mention a mower. -
The release:
v2learned from 6,882 messages (114 of ours). On our 50 check messages from 20 April,devicesrecall went from 0.000 to 0.962 (25 of 26), with a precision of 0.893.route releasepassed: no slice got worse. But three route slices have fewer than 30 messages: account 21, devices 15 and needs_person 12. The check shows them and does not judge them.
Common mistakes
Section titled “Common mistakes”Common mistake
The router never learned the new route
Symptom: the device messages of March go to product or returns_refunds, and devices recall is 0.
Cause: the model learned from the course’s 6,000 messages only. None of them is a devices message.
Fix: learn from the history and March together. Check that every route of the test set is in the training set before you score anything.
Common mistake
The cheapest row changes between validation and test
Symptom: the cheapest method on validation is not the cheapest on test, and you want to change your choice.
Cause: 256 messages are few. One urgent message in the wrong queue adds US$78.1 per 1,000. Four of them change the order of the rows.
Fix: keep the choice that you made on validation. Report both numbers, explain the difference, and say how many messages it rests on.
Common mistake
No alarm on the route shares, so no drift
Symptom: the PSI stays low in April, and the report says “no drift”.
Cause: the new topic does not move the route shares much. Its messages spread over five old queues.
Fix: watch the unsure share too, and read the unsure messages by group. A new topic often shows first as more unsure messages, not as new shares.
Common mistake
The release passes, but the new topic was never checked
Symptom: route release says RELEASED, and the report says the mower is fixed.
Cause: devices already existed in v1, so the release check saw no new route. It checked only the old slices of March.
Fix: check the new topic yourself, on labelled April messages that the candidate did not learn from, before the release.
Common mistake
Every April message is labelled as the new topic
Symptom: after the new version, ordinary April messages, for example a mower return, go to devices.
Cause: the labels came only from the new topic’s words, and every one of them got the new route.
Fix: read each message before you label it: a return of a mower is still a return. Add a random sample of other April messages, so that the model sees the old routes in April too.
Recap
You can now take a changing inbox to a router that a team can trust. You write guidelines and measure agreement, and you price each mistake. You compare methods on one fair test, with a simple baseline, and choose by cost. You set zones under the team’s limit and run the router as a service. You find a new topic without labels, add it with a few labels, and release it only after a check of what changed. And you tell the owners what the router costs and what it cannot do. To keep going, Evaluating and Testing AI Systems goes deeper into datasets, slices and regression checks. Building Reliable Applications with LLM APIs builds the prompts and structured outputs of an LLM router.