Add a label or retrain safely
- Lesson 37
- 40 min
- Module 9: Run it in production
Where we are
We found a new topic that has no label. Adding a label changes the model, and a new model can break what worked. In this lesson, we version the labels and the data, compare the new model with the old one by slice, and release it only if it passes.
Lesson 37 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceAdd a label or retrain safely
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Add the new route to the model without breaking what works. You train a new version on versioned labels and compare it with the current model, slice by slice. You check the new route on messages kept apart, and you release it only if it passes. If it fails in use, you go back in one command.
Before you startWhat to know and to have ready
What you need to know
- The new topic, Grow Hub setup, and Chloé’s 250 labelled launch messages, from Drift and new topics.
- Model versions, label versions and the service, from A routing service.
- Slices: accuracy for each language, channel or route, from Naive Bayes, logistic regression and linear SVM. Uncertainty and slices in the evaluation course goes further.
What you need on your computer
The project folder larkroute from the last lesson, with its environment active and v1 as the current model. Tested on macOS with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. The commands are the same in PowerShell, but we did not run them there.
A new edition of the rulebook
Section titled “A new edition of the rulebook”Think of the post office once more. The new housing estate needs its streets in the rulebook, so the office prints a second edition. Before the sorters use it, the supervisor tests it. She takes a large pile of old letters whose right bins she knows. Each letter is sorted twice, once with each edition.
She does not only count the total mistakes. She checks each district on its own, because a new page can confuse the sorters about one old street. She also sorts a few letters for the new estate, which she kept apart for the test. Only if every district is as good as before, and the new streets are found, does edition 2 go to the counter. Edition 1 stays on the shelf, in case a problem appears later.
The new rulebook has an edition number, and so do the lists it came from. If a letter goes to the wrong street next month, the logbook says which edition sorted it.
The problem: one number hides what broke
Section titled “The problem: one number hides what broke”Priya can train a model with the new route in a few seconds. The hard part is to know that it is better. Here is a real trap. Chloé labelled 50 Grow Hub messages for training. Suppose she had labelled only 10. Priya trains two candidates, one with each set of labels:
$ route train --version v2 --extra data/launch_labels_sample.csvSaved models/v2: 6150 messages, 8 routes, labels-441842bea4; validation accuracy 0.960; decide at 0.56, person below 0.00.$ route train --version try10 --extra devices_10.csvSaved models/try10: 6110 messages, 8 routes, labels-c8f05e2568; validation accuracy 0.960; decide at 0.56, person below 0.00.Both have validation accuracy 0.960, higher than v1 (0.958). By that number, they are the same, and both look ready. But the validation messages come from before the launch: not one of them is about the Grow Hub. The number cannot see the new route at all. Later in this lesson, the release check finds that try10 misses one Grow Hub message in three.
The opposite trap exists too. A new model can be better overall and worse on one group: French messages, social media, or the account route with its rare takeovers. An overall score of 0.96 can hide a drop from 0.95 to 0.90 for 589 French customers.
The idea: a release check, slice by slice
Section titled “The idea: a release check, slice by slice”The answer has four parts. Together, they are the release check, sometimes called a regression check: a test that a new version did not make things worse. (“Regression” here means going backwards, not the statistics method.)
Text description of the diagram
The diagram has three groups. The first group makes a candidate. New labels come in: Chloé’s 150 launch messages for training, 50 of them about devices. Priya trains a model with a new version name, v2, and v1 stays as it was. The second group is the release check, with two questions. First: is no slice worse by more than 2 points? The slices are the 7 routes, the 2 languages and the 3 channels. If yes, the second question: is the new route found on its 30 check messages, with a recall of 0.80 or more? If the answer to either question is no, the result is NOT RELEASED, and the current model stays. If both answers are yes, the third group starts. The result is RELEASED: the current model is v2, and the previous one is v1. Restart the service, and /health says v2. Then watch the drift check, with a new reference. A dashed arrow leads from there to route release --rollback, which goes back to v1 if a problem appears.
- Version the labels and the data. The candidate gets a new version name, and
v1stays untouched. The candidate’s card records its label version, the fingerprint of every message ID and route it learned from. The extra labels are a file with a checksum (launch_labels_sample.csv), so anyone can train the same model again. - Compare by slice. A slice is a group of test messages that share one value: a route, a language or a channel. The check scores the current model and the candidate (the new model that wants to replace it) on each slice. A slice fails if it drops by more than the tolerance: 2 points (0.02) here.
- Check each new route on messages kept apart. The test set has no Grow Hub messages, so it cannot judge the new route. Chloé’s 250 labels are split: 150 for training and 100 for checking. The check needs a recall of at least 0.80 on the new route’s check messages. Recall is the share of the route’s messages that the model finds.
- Keep the way back. On release,
models/currentnames the new version, andmodels/previousnames the old one.route release --rollbackgoes back in one command.
A worked slice check
Section titled “A worked slice check”Here are three slices, with every value. The first two rows are real rows of the check below. The third is constructed, to show a failure:
| Slice | Messages | Current | Candidate | Change | Rule | Result |
|---|---|---|---|---|---|---|
| route=product | 289 | 0.962 | 0.955 | −0.7 points | 0.955 ≥ 0.962 − 0.02 = 0.942 | ok |
| language=fr | 589 | 0.952 | 0.946 | −0.6 points | 0.946 ≥ 0.952 − 0.02 = 0.932 | ok |
| a constructed slice | 120 | 0.950 | 0.920 | −3.0 points | 0.920 < 0.950 − 0.02 = 0.930 | WORSE |
Small drops are allowed, because a slice of a few hundred messages moves by chance. One message in 289 is 0.35 points. A slice with fewer than 30 messages is shown but not judged: one message would be more than 3 points.
Train and check the new route
Section titled “Train and check the new route”-
Look at Chloé’s labels. Run this in the
larkroutefolder (the same command works in PowerShell):Terminal window python -c "import pandas as pd; print(pd.read_csv('data/launch_labels_sample.csv').groupby(['split', 'route']).size().unstack(0))"split check trainrouteaccount 6 13billing 11 12delivery 19 30devices 30 50needs_person 5 5orders 12 15product 9 13returns_refunds 8 12There are 150 training labels, 50 of them
devices, and 100 check labels, 30 of themdevices. Chloé labelled the other routes too, so the model also learns how the old routes look in the launch month. -
Train the candidate
v2on the 6,000 training messages and the 150 launch messages:Terminal window route train --version v2 --extra data/launch_labels_sample.csvSaved models/v2: 6150 messages, 8 routes, labels-441842bea4; validation accuracy 0.960; decide at 0.56, person below 0.00.The label version changed from
labels-4bad2e4ef7tolabels-441842bea4, because the labels changed. The decide threshold moved from 0.58 to 0.56:route trainchooses it again on validation for each version. Without--make-current, the service still loadsv1. -
Make the trap of the problem section: a label file with only 10 of the 50 device messages. Create
fewer_devices.pyin thelarkroutefolder:"""Keep only the first N device messages of Chloé's training labels (the other labels stay)."""import sysimport pandas as pdn = int(sys.argv[1])labels = pd.read_csv("data/launch_labels_sample.csv")is_device = (labels["split"] == "train") & (labels["route"] == "devices")fewer = pd.concat([labels[~is_device], labels[is_device].head(n)])fewer.to_csv(f"devices_{n}.csv", index=False)print(f"devices_{n}.csv: {len(fewer)} labels, {n} of them devices for training") -
Run it with 10, and train the candidate
try10:Terminal window python fewer_devices.py 10route train --version try10 --extra devices_10.csvYou see
devices_10.csv: 210 labels, 10 of them devices for training, then the lineSaved models/try10: …of the problem section. -
Run the release check on
try10:Terminal window route release --candidate try10Current v1 against candidate try10 on the test messages (a slice fails if it drops more than 2%; slices under 30 messages are shown, not judged):slice messages current candidate resultroute=account 196 0.954 0.954 okroute=billing 266 0.962 0.955 okroute=delivery 461 0.957 0.957 okroute=needs_person 99 1.000 1.000 okroute=orders 285 0.965 0.965 okroute=product 289 0.962 0.962 okroute=returns_refunds 404 0.960 0.963 oklanguage=en 1411 0.966 0.967 oklanguage=fr 589 0.952 0.947 okchannel=chat 687 0.956 0.953 okchannel=email 898 0.969 0.970 okchannel=social 415 0.957 0.957 oknew route=devices (launch check) 30 - 0.667 WORSENOT RELEASED: 1 slice(s) got worse. The current model stays v1.Every old slice is fine. But the new route finds only 20 of the 30 check messages (0.667), below 0.80. The command ends with an error code, so a script that runs it stops here too.
models/currentstill saysv1. -
Run the release check on
v2:Terminal window route release --candidate v2Current v1 against candidate v2 on the test messages (a slice fails if it drops more than 2%; slices under 30 messages are shown, not judged):slice messages current candidate resultroute=account 196 0.954 0.959 okroute=billing 266 0.962 0.959 okroute=delivery 461 0.957 0.957 okroute=needs_person 99 1.000 1.000 okroute=orders 285 0.965 0.961 okroute=product 289 0.962 0.955 okroute=returns_refunds 404 0.960 0.963 oklanguage=en 1411 0.966 0.967 oklanguage=fr 589 0.952 0.946 okchannel=chat 687 0.956 0.955 okchannel=email 898 0.969 0.969 okchannel=social 415 0.957 0.954 oknew route=devices (launch check) 30 - 0.900 okRELEASED: the current model is now v2. Restart the service to load it.Now
models/currentsaysv2, andmodels/previoussaysv1.
Read the check of v2
Section titled “Read the check of v2”No slice got worse by more than the tolerance: 0 of 12. Six slices dropped by less than one point; the largest drops are 0.7 points for route=product and 0.6 points for French. Three stayed the same, and three got a little better, for example account (0.954 → 0.959). The new route finds 27 of the 30 check messages: recall 0.900.
How much does the new route depend on Chloé’s work? We trained candidates with 5, 10, 20 and all 50 device labels and ran the same check against v1:
Text description of the chart
A column chart with four columns. The height is the recall of devices on the 30 check messages. With 5 device labels for training, it is 0.467. With 10 labels, it is 0.667. With 20 labels, it is 0.767, and with 50 labels, 0.900. Only the last column is above the release level of 0.80. The first three are drawn in a different colour.
More labels for the new route help a lot here: from 0.467 to 0.900. The old slices did not care: they passed with 5 labels as well as with 50.
The hidden truth gives one more check, which only a simulated inbox allows. On all 167 Grow Hub setup messages of the launch month, v2 has a devices recall of 0.952 and a precision of 0.988. These 167 include the 50 messages that v2 learned from, so the recall is a little flattering. Still, the 30 check messages (0.900) did not overstate the new route. In real work, you would not have this number; Chloé’s check messages are the evidence.
Use the new version, and keep the way back
Section titled “Use the new version, and keep the way back”-
Start the service again in the first terminal. It loads the current model when it starts:
Terminal window route serve -
In the second terminal, ask for its health and send the Grow Hub message of the first lesson (PowerShell: as in A routing service):
Terminal window curl -s http://127.0.0.1:8036/healthcurl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \-d '{"message_id": "M-999007", "text": "My new Grow Hub keeps blinking orange and will not connect to the Wi-Fi.", "channel": "social"}'{"status":"ok","model_version":"v2","label_version":"labels-441842bea4","routes":["account","billing","delivery","devices","needs_person","orders","product","returns_refunds"]}{"message_id":"M-999007","route":"devices","zone":"decide","confidence":0.9962,"model_version":"v2","label_version":"labels-441842bea4"}With
v1, this message went to review asproduct, with confidence 0.4671. Now it goes straight todevices, with 0.9962. The answer namesv2, so the log shows which model decided each message. -
Stop the service with Ctrl+C. Run the drift check, which now uses
v2:Terminal window route driftModel v2; validation: unsure (review or person) 10.0%. Launch month, by week:week messages delivery product account mean_conf unsure psi2026-02-02 533 0.206 0.152 0.113 0.914 0.101 0.0162026-02-09 477 0.191 0.140 0.094 0.919 0.092 0.8182026-02-16 510 0.461 0.096 0.075 0.904 0.069 0.8392026-02-23 480 0.138 0.154 0.092 0.903 0.096 1.085ALARM week of 2026-02-09: route shares moved (PSI 0.82). Read a sample of messages before you retrain.ALARM week of 2026-02-16: route shares moved (PSI 0.84). Read a sample of messages before you retrain.ALARM week of 2026-02-23: route shares moved (PSI 1.08). Read a sample of messages before you retrain. -
Practise the way back. Go back to
v1:Terminal window route release --rollbackRolled back: the current model is v1.
Read the drift table of step 3 with care. The good news first: the unsure share fell back to 9.2%, 6.9% and 9.6% in the last three weeks, close to validation. The Grow Hub messages are no longer unsure. But the PSI jumped to 0.82, 0.84 and 1.08. Why?
The drift check compares each week with the validation messages, and validation has no devices messages. The share of devices goes from almost 0 to about 10%. In the PSI formula, the log of “10% divided by almost 0” is very large. So these alarms do not mean that something new happened. They mean that the reference is out of date. After you add a route, the drift check needs a new reference: a set of messages that includes the new route. A labelled sample of the weeks after the launch would do. Until then, read the unsure share and the mean confidence, not the PSI.
The rollback works because nothing was deleted. route release --rollback swaps the names in models/current and models/previous: now current says v1 and previous says v2. A rollback also needs a service restart, like a release.
Your turn: 20 device labels
Section titled “Your turn: 20 device labels”You are on v1 again, after the rollback. Make a label file with 20 device messages, train the candidate try20, and run the release check against v1. Before you run the check, predict: will it pass?
Show the answer
python fewer_devices.py 20route train --version try20 --extra devices_20.csvroute release --candidate try20The last two lines of the check:
new route=devices (launch check) 30 - 0.767 WORSENOT RELEASED: 1 slice(s) got worse. The current model stays v1.It fails: 23 of 30 check messages found (0.767), below 0.80. Every old slice passed again. The chart above shows the same: the new route needs more than 20 examples here. Now release v2 again, so the rest of the course has the new route:
route release --candidate v2The last line is RELEASED: the current model is now v2. Restart the service to load it. Your models/ folder keeps every version you trained: v1, v2, try10 and try20. Delete the try folders when you no longer need them.
When to use a release check, and what it costs
Section titled “When to use a release check, and what it costs”Use a release check for every new model, not only for a new label. Retraining on fresh data, a new threshold or a new feature can each break a slice. The check took a few seconds here, so there is no reason to skip it.
The real cost is the labels. The launch check needs labelled messages of the new route that the model never trained on. And each slice needs enough messages to judge. Chloé labelled 250 launch messages. The course assumes that a person’s review of one message costs US$1.50. So her labels cost about US$375 of her time. It is a small price next to a route that misses one Grow Hub message in three.
Know its limits. The tolerance of 2 points and the recall of 0.80 are choices that Grace and Priya made, not laws. Write them down with their reasons. The check reads the test messages each time. That is fine for a pass-or-fail gate. But if you train twenty candidates and keep the one with the best test slices, the test set has become a validation set. Regression checks in the evaluation course shows how a team keeps its checks honest over many releases. The case study Regression-test a classifier release there does the same work on real data.
Common mistake
Released on validation accuracy
Symptom: this is a constructed case, built on the real numbers above. Tomás compares the two candidates by their training lines. Both say validation accuracy 0.960, better than v1 (0.958). He makes try10 current with route train --version try10b --extra devices_10.csv --make-current, without the release check. A week later, Chloé finds that a third of the Grow Hub messages still go to product or returns_refunds.
Cause: the validation messages are from before the launch, so none of them is about the Grow Hub. The validation score cannot see the new route. The candidate with 10 device labels finds only 0.667 of the check messages, and --make-current skipped the check that would have said so.
Fix: make a candidate current only through route release. The check scores every old slice on the test messages and every new route on the check messages that were kept apart. If a team needs to skip it in an emergency, they should write down why, and run the check afterwards.
Consequence in AI work: trust in one good overall number is a common way for a model that “passed” to fail in use. Evidence for each slice and each new class lets the team trust a release. When a release fails, the same evidence shows what to fix.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why can the validation accuracy (0.960 for both try10 and v2) not decide which candidate to release? What evidence does the release check use instead?
Show the answer
The validation messages come from before the launch, so none of them is about the Grow Hub. A score on them cannot see the new route, and both candidates get 0.960. The release check uses two kinds of evidence. For the old routes, it compares the two models on each slice of the test messages, with a tolerance of 2 points. The slices are 7 routes, 2 languages and 3 channels. For each new route, it measures recall on labelled messages that were kept apart: 30 Grow Hub check messages. try10 found 0.667 and failed; v2 found 0.900 and passed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a candidate scores 0.952 → 0.935 on French messages (589 messages) and 0.940 → 0.900 on a slice of 20 messages. Every other slice is fine. Is it released? Explain each slice.
Show the answer
Yes, it is released. French drops by 1.7 points. The limit is 0.952 − 0.02 = 0.932, and 0.935 is above it, so the slice is ok. It is still worth watching. The slice of 20 messages drops by 4 points. But it has fewer than 30 messages, so the check shows it and does not judge it: one message in 20 is 5 points. A person should still read the errors in that small slice before the release.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
After the release of v2, route drift shows PSI 0.82 to 1.08 every week, but the unsure share is back near 10%. Is the new model broken? What do you change?
Show the answer
No. The model routes the launch month well: the unsure share fell from 17.0% and 20.8% to 9.2% and 9.6%. The PSI is high because the reference, the validation messages, has no devices messages. A new route’s share against a reference share of almost 0 makes the log in the PSI formula very large. Change the reference to a set of messages that includes the new route, such as a labelled sample of the weeks after the launch. Until then, watch the unsure share and the mean confidence.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Version the labels and the data. A candidate gets a new name (
v2), the old version stays, and the label version (labels-441842bea4) records exactly which labels it learned. - Compare by slice, not by one number.
v2was worse by more than 2 points on 0 of 12 slices. Validation accuracy (0.960) could not tellv2fromtry10. - Check each new route on messages kept apart.
devicesrecall on the 30 check messages: 0.900 forv2; 0.767, 0.667 and 0.467 with 20, 10 and 5 labels, which failed. On the hidden truth, which only this simulated inbox has,v2has recall 0.952 and precision 0.988. - Release, restart, watch, and keep the way back.
route releasemakes the candidate current only if it passes;--rollbackswaps back in one command. After a new route, give the drift check a new reference.
You can now run, watch and update a router on Larkfield’s synthetic inbox. But this inbox is easier than a real one: a classic model reaches 0.962 here. Next, the case studies test the same methods on real messages, starting with 77 kinds of banking questions.
Next: Route banking questions