Skip to content
Français

Add a label or retrain safely

  • Lesson 37
  • 40 min
  • Module 9: Run it in production

Where we are

We found a new topic that has no label. Adding a label changes the model, and a new model can break what worked. In this lesson, we version the labels and the data, compare the new model with the old one by slice, and release it only if it passes.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 37 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceAdd a label or retrain safely

What you will be able to do

Add the new route to the model without breaking what works. You train a new version on versioned labels and compare it with the current model, slice by slice. You check the new route on messages kept apart, and you release it only if it passes. If it fails in use, you go back in one command.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

The project folder larkroute from the last lesson, with its environment active and v1 as the current model. Tested on macOS with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. The commands are the same in PowerShell, but we did not run them there.

Think of the post office once more. The new housing estate needs its streets in the rulebook, so the office prints a second edition. Before the sorters use it, the supervisor tests it. She takes a large pile of old letters whose right bins she knows. Each letter is sorted twice, once with each edition.

She does not only count the total mistakes. She checks each district on its own, because a new page can confuse the sorters about one old street. She also sorts a few letters for the new estate, which she kept apart for the test. Only if every district is as good as before, and the new streets are found, does edition 2 go to the counter. Edition 1 stays on the shelf, in case a problem appears later.

The new rulebook has an edition number, and so do the lists it came from. If a letter goes to the wrong street next month, the logbook says which edition sorted it.

Priya can train a model with the new route in a few seconds. The hard part is to know that it is better. Here is a real trap. Chloé labelled 50 Grow Hub messages for training. Suppose she had labelled only 10. Priya trains two candidates, one with each set of labels:

$ route train --version v2 --extra data/launch_labels_sample.csv
Saved models/v2: 6150 messages, 8 routes, labels-441842bea4; validation accuracy 0.960; decide at 0.56, person below 0.00.
$ route train --version try10 --extra devices_10.csv
Saved models/try10: 6110 messages, 8 routes, labels-c8f05e2568; validation accuracy 0.960; decide at 0.56, person below 0.00.

Both have validation accuracy 0.960, higher than v1 (0.958). By that number, they are the same, and both look ready. But the validation messages come from before the launch: not one of them is about the Grow Hub. The number cannot see the new route at all. Later in this lesson, the release check finds that try10 misses one Grow Hub message in three.

The opposite trap exists too. A new model can be better overall and worse on one group: French messages, social media, or the account route with its rare takeovers. An overall score of 0.96 can hide a drop from 0.95 to 0.90 for 589 French customers.

The answer has four parts. Together, they are the release check, sometimes called a regression check: a test that a new version did not make things worse. (“Regression” here means going backwards, not the statistics method.)

Three groups. Make a candidate: new labels, Chloé's 150 launch messages for training with 50 devices; then train with a new name, route train --version v2, and v1 stays as it was. The release check: first question, no slice worse by more than 2 points, over 7 routes, 2 languages and 3 channels? If yes, second question: the new route found on its 30 check messages, with recall 0.80 or more? If no to either question: NOT RELEASED, the current model stays. If yes to both: use it. RELEASED: current v2, previous v1. Restart the service, and /health says v2. Watch the drift check with a new reference. A dashed arrow, a problem: route release --rollback, back to v1.
The release checklist. A candidate gets a new version name, passes two checks, and only then becomes current. The old version stays, so a rollback is one command.
Text description of the diagram

The diagram has three groups. The first group makes a candidate. New labels come in: Chloé’s 150 launch messages for training, 50 of them about devices. Priya trains a model with a new version name, v2, and v1 stays as it was. The second group is the release check, with two questions. First: is no slice worse by more than 2 points? The slices are the 7 routes, the 2 languages and the 3 channels. If yes, the second question: is the new route found on its 30 check messages, with a recall of 0.80 or more? If the answer to either question is no, the result is NOT RELEASED, and the current model stays. If both answers are yes, the third group starts. The result is RELEASED: the current model is v2, and the previous one is v1. Restart the service, and /health says v2. Then watch the drift check, with a new reference. A dashed arrow leads from there to route release --rollback, which goes back to v1 if a problem appears.

  1. Version the labels and the data. The candidate gets a new version name, and v1 stays untouched. The candidate’s card records its label version, the fingerprint of every message ID and route it learned from. The extra labels are a file with a checksum (launch_labels_sample.csv), so anyone can train the same model again.
  2. Compare by slice. A slice is a group of test messages that share one value: a route, a language or a channel. The check scores the current model and the candidate (the new model that wants to replace it) on each slice. A slice fails if it drops by more than the tolerance: 2 points (0.02) here.
  3. Check each new route on messages kept apart. The test set has no Grow Hub messages, so it cannot judge the new route. Chloé’s 250 labels are split: 150 for training and 100 for checking. The check needs a recall of at least 0.80 on the new route’s check messages. Recall is the share of the route’s messages that the model finds.
  4. Keep the way back. On release, models/current names the new version, and models/previous names the old one. route release --rollback goes back in one command.

Here are three slices, with every value. The first two rows are real rows of the check below. The third is constructed, to show a failure:

Slice Messages Current Candidate Change Rule Result
route=product 289 0.962 0.955 −0.7 points 0.955 ≥ 0.962 − 0.02 = 0.942 ok
language=fr 589 0.952 0.946 −0.6 points 0.946 ≥ 0.952 − 0.02 = 0.932 ok
a constructed slice 120 0.950 0.920 −3.0 points 0.920 < 0.950 − 0.02 = 0.930 WORSE

Small drops are allowed, because a slice of a few hundred messages moves by chance. One message in 289 is 0.35 points. A slice with fewer than 30 messages is shown but not judged: one message would be more than 3 points.

  1. Look at Chloé’s labels. Run this in the larkroute folder (the same command works in PowerShell):

    Terminal window
    python -c "import pandas as pd; print(pd.read_csv('data/launch_labels_sample.csv').groupby(['split', 'route']).size().unstack(0))"
    split check train
    route
    account 6 13
    billing 11 12
    delivery 19 30
    devices 30 50
    needs_person 5 5
    orders 12 15
    product 9 13
    returns_refunds 8 12

    There are 150 training labels, 50 of them devices, and 100 check labels, 30 of them devices. Chloé labelled the other routes too, so the model also learns how the old routes look in the launch month.

  2. Train the candidate v2 on the 6,000 training messages and the 150 launch messages:

    Terminal window
    route train --version v2 --extra data/launch_labels_sample.csv
    Saved models/v2: 6150 messages, 8 routes, labels-441842bea4; validation accuracy 0.960; decide at 0.56, person below 0.00.

    The label version changed from labels-4bad2e4ef7 to labels-441842bea4, because the labels changed. The decide threshold moved from 0.58 to 0.56: route train chooses it again on validation for each version. Without --make-current, the service still loads v1.

  3. Make the trap of the problem section: a label file with only 10 of the 50 device messages. Create fewer_devices.py in the larkroute folder:

    """Keep only the first N device messages of Chloé's training labels (the other labels stay)."""
    import sys
    import pandas as pd
    n = int(sys.argv[1])
    labels = pd.read_csv("data/launch_labels_sample.csv")
    is_device = (labels["split"] == "train") & (labels["route"] == "devices")
    fewer = pd.concat([labels[~is_device], labels[is_device].head(n)])
    fewer.to_csv(f"devices_{n}.csv", index=False)
    print(f"devices_{n}.csv: {len(fewer)} labels, {n} of them devices for training")
  4. Run it with 10, and train the candidate try10:

    Terminal window
    python fewer_devices.py 10
    route train --version try10 --extra devices_10.csv

    You see devices_10.csv: 210 labels, 10 of them devices for training, then the line Saved models/try10: … of the problem section.

  5. Run the release check on try10:

    Terminal window
    route release --candidate try10
    Current v1 against candidate try10 on the test messages (a slice fails if it drops more than 2%; slices under 30 messages are shown, not judged):
    slice messages current candidate result
    route=account 196 0.954 0.954 ok
    route=billing 266 0.962 0.955 ok
    route=delivery 461 0.957 0.957 ok
    route=needs_person 99 1.000 1.000 ok
    route=orders 285 0.965 0.965 ok
    route=product 289 0.962 0.962 ok
    route=returns_refunds 404 0.960 0.963 ok
    language=en 1411 0.966 0.967 ok
    language=fr 589 0.952 0.947 ok
    channel=chat 687 0.956 0.953 ok
    channel=email 898 0.969 0.970 ok
    channel=social 415 0.957 0.957 ok
    new route=devices (launch check) 30 - 0.667 WORSE
    NOT RELEASED: 1 slice(s) got worse. The current model stays v1.

    Every old slice is fine. But the new route finds only 20 of the 30 check messages (0.667), below 0.80. The command ends with an error code, so a script that runs it stops here too. models/current still says v1.

  6. Run the release check on v2:

    Terminal window
    route release --candidate v2
    Current v1 against candidate v2 on the test messages (a slice fails if it drops more than 2%; slices under 30 messages are shown, not judged):
    slice messages current candidate result
    route=account 196 0.954 0.959 ok
    route=billing 266 0.962 0.959 ok
    route=delivery 461 0.957 0.957 ok
    route=needs_person 99 1.000 1.000 ok
    route=orders 285 0.965 0.961 ok
    route=product 289 0.962 0.955 ok
    route=returns_refunds 404 0.960 0.963 ok
    language=en 1411 0.966 0.967 ok
    language=fr 589 0.952 0.946 ok
    channel=chat 687 0.956 0.955 ok
    channel=email 898 0.969 0.969 ok
    channel=social 415 0.957 0.954 ok
    new route=devices (launch check) 30 - 0.900 ok
    RELEASED: the current model is now v2. Restart the service to load it.

    Now models/current says v2, and models/previous says v1.

No slice got worse by more than the tolerance: 0 of 12. Six slices dropped by less than one point; the largest drops are 0.7 points for route=product and 0.6 points for French. Three stayed the same, and three got a little better, for example account (0.954 → 0.959). The new route finds 27 of the 30 check messages: recall 0.900.

How much does the new route depend on Chloé’s work? We trained candidates with 5, 10, 20 and all 50 device labels and ran the same check against v1:

The launch check needs a recall of 0.80. With 5, 10 or 20 device labels for training, the candidate fails; with 50, it passes with 0.900. Every old slice passed in all four runs.
Text description of the chart

A column chart with four columns. The height is the recall of devices on the 30 check messages. With 5 device labels for training, it is 0.467. With 10 labels, it is 0.667. With 20 labels, it is 0.767, and with 50 labels, 0.900. Only the last column is above the release level of 0.80. The first three are drawn in a different colour.

More labels for the new route help a lot here: from 0.467 to 0.900. The old slices did not care: they passed with 5 labels as well as with 50.

The hidden truth gives one more check, which only a simulated inbox allows. On all 167 Grow Hub setup messages of the launch month, v2 has a devices recall of 0.952 and a precision of 0.988. These 167 include the 50 messages that v2 learned from, so the recall is a little flattering. Still, the 30 check messages (0.900) did not overstate the new route. In real work, you would not have this number; Chloé’s check messages are the evidence.

Use the new version, and keep the way back

Section titled “Use the new version, and keep the way back”
  1. Start the service again in the first terminal. It loads the current model when it starts:

    Terminal window
    route serve
  2. In the second terminal, ask for its health and send the Grow Hub message of the first lesson (PowerShell: as in A routing service):

    Terminal window
    curl -s http://127.0.0.1:8036/health
    curl -s http://127.0.0.1:8036/route -H 'content-type: application/json' \
    -d '{"message_id": "M-999007", "text": "My new Grow Hub keeps blinking orange and will not connect to the Wi-Fi.", "channel": "social"}'
    {"status":"ok","model_version":"v2","label_version":"labels-441842bea4","routes":["account","billing","delivery","devices","needs_person","orders","product","returns_refunds"]}
    {"message_id":"M-999007","route":"devices","zone":"decide","confidence":0.9962,"model_version":"v2","label_version":"labels-441842bea4"}

    With v1, this message went to review as product, with confidence 0.4671. Now it goes straight to devices, with 0.9962. The answer names v2, so the log shows which model decided each message.

  3. Stop the service with Ctrl+C. Run the drift check, which now uses v2:

    Terminal window
    route drift
    Model v2; validation: unsure (review or person) 10.0%. Launch month, by week:
    week messages delivery product account mean_conf unsure psi
    2026-02-02 533 0.206 0.152 0.113 0.914 0.101 0.016
    2026-02-09 477 0.191 0.140 0.094 0.919 0.092 0.818
    2026-02-16 510 0.461 0.096 0.075 0.904 0.069 0.839
    2026-02-23 480 0.138 0.154 0.092 0.903 0.096 1.085
    ALARM week of 2026-02-09: route shares moved (PSI 0.82). Read a sample of messages before you retrain.
    ALARM week of 2026-02-16: route shares moved (PSI 0.84). Read a sample of messages before you retrain.
    ALARM week of 2026-02-23: route shares moved (PSI 1.08). Read a sample of messages before you retrain.
  4. Practise the way back. Go back to v1:

    Terminal window
    route release --rollback
    Rolled back: the current model is v1.

Read the drift table of step 3 with care. The good news first: the unsure share fell back to 9.2%, 6.9% and 9.6% in the last three weeks, close to validation. The Grow Hub messages are no longer unsure. But the PSI jumped to 0.82, 0.84 and 1.08. Why?

The drift check compares each week with the validation messages, and validation has no devices messages. The share of devices goes from almost 0 to about 10%. In the PSI formula, the log of “10% divided by almost 0” is very large. So these alarms do not mean that something new happened. They mean that the reference is out of date. After you add a route, the drift check needs a new reference: a set of messages that includes the new route. A labelled sample of the weeks after the launch would do. Until then, read the unsure share and the mean confidence, not the PSI.

The rollback works because nothing was deleted. route release --rollback swaps the names in models/current and models/previous: now current says v1 and previous says v2. A rollback also needs a service restart, like a release.

You are on v1 again, after the rollback. Make a label file with 20 device messages, train the candidate try20, and run the release check against v1. Before you run the check, predict: will it pass?

Show the answer
Terminal window
python fewer_devices.py 20
route train --version try20 --extra devices_20.csv
route release --candidate try20

The last two lines of the check:

new route=devices (launch check) 30 - 0.767 WORSE
NOT RELEASED: 1 slice(s) got worse. The current model stays v1.

It fails: 23 of 30 check messages found (0.767), below 0.80. Every old slice passed again. The chart above shows the same: the new route needs more than 20 examples here. Now release v2 again, so the rest of the course has the new route:

Terminal window
route release --candidate v2

The last line is RELEASED: the current model is now v2. Restart the service to load it. Your models/ folder keeps every version you trained: v1, v2, try10 and try20. Delete the try folders when you no longer need them.

When to use a release check, and what it costs

Section titled “When to use a release check, and what it costs”

Use a release check for every new model, not only for a new label. Retraining on fresh data, a new threshold or a new feature can each break a slice. The check took a few seconds here, so there is no reason to skip it.

The real cost is the labels. The launch check needs labelled messages of the new route that the model never trained on. And each slice needs enough messages to judge. Chloé labelled 250 launch messages. The course assumes that a person’s review of one message costs US$1.50. So her labels cost about US$375 of her time. It is a small price next to a route that misses one Grow Hub message in three.

Know its limits. The tolerance of 2 points and the recall of 0.80 are choices that Grace and Priya made, not laws. Write them down with their reasons. The check reads the test messages each time. That is fine for a pass-or-fail gate. But if you train twenty candidates and keep the one with the best test slices, the test set has become a validation set. Regression checks in the evaluation course shows how a team keeps its checks honest over many releases. The case study Regression-test a classifier release there does the same work on real data.

Common mistake

Released on validation accuracy

Symptom: this is a constructed case, built on the real numbers above. Tomás compares the two candidates by their training lines. Both say validation accuracy 0.960, better than v1 (0.958). He makes try10 current with route train --version try10b --extra devices_10.csv --make-current, without the release check. A week later, Chloé finds that a third of the Grow Hub messages still go to product or returns_refunds.

Cause: the validation messages are from before the launch, so none of them is about the Grow Hub. The validation score cannot see the new route. The candidate with 10 device labels finds only 0.667 of the check messages, and --make-current skipped the check that would have said so.

Fix: make a candidate current only through route release. The check scores every old slice on the test messages and every new route on the check messages that were kept apart. If a team needs to skip it in an emergency, they should write down why, and run the check afterwards.

Consequence in AI work: trust in one good overall number is a common way for a model that “passed” to fail in use. Evidence for each slice and each new class lets the team trust a release. When a release fails, the same evidence shows what to fix.

Check your understanding

Why can the validation accuracy (0.960 for both try10 and v2) not decide which candidate to release? What evidence does the release check use instead?

Show the answer

The validation messages come from before the launch, so none of them is about the Grow Hub. A score on them cannot see the new route, and both candidates get 0.960. The release check uses two kinds of evidence. For the old routes, it compares the two models on each slice of the test messages, with a tolerance of 2 points. The slices are 7 routes, 2 languages and 3 channels. For each new route, it measures recall on labelled messages that were kept apart: 30 Grow Hub check messages. try10 found 0.667 and failed; v2 found 0.900 and passed.

Check your understanding

Predict: a candidate scores 0.952 → 0.935 on French messages (589 messages) and 0.940 → 0.900 on a slice of 20 messages. Every other slice is fine. Is it released? Explain each slice.

Show the answer

Yes, it is released. French drops by 1.7 points. The limit is 0.952 − 0.02 = 0.932, and 0.935 is above it, so the slice is ok. It is still worth watching. The slice of 20 messages drops by 4 points. But it has fewer than 30 messages, so the check shows it and does not judge it: one message in 20 is 5 points. A person should still read the errors in that small slice before the release.

Check your understanding

After the release of v2, route drift shows PSI 0.82 to 1.08 every week, but the unsure share is back near 10%. Is the new model broken? What do you change?

Show the answer

No. The model routes the launch month well: the unsure share fell from 17.0% and 20.8% to 9.2% and 9.6%. The PSI is high because the reference, the validation messages, has no devices messages. A new route’s share against a reference share of almost 0 makes the log in the PSI formula very large. Change the reference to a set of messages that includes the new route, such as a labelled sample of the weeks after the launch. Until then, watch the unsure share and the mean confidence.

Recap

  • Version the labels and the data. A candidate gets a new name (v2), the old version stays, and the label version (labels-441842bea4) records exactly which labels it learned.
  • Compare by slice, not by one number. v2 was worse by more than 2 points on 0 of 12 slices. Validation accuracy (0.960) could not tell v2 from try10.
  • Check each new route on messages kept apart. devices recall on the 30 check messages: 0.900 for v2; 0.767, 0.667 and 0.467 with 20, 10 and 5 labels, which failed. On the hidden truth, which only this simulated inbox has, v2 has recall 0.952 and precision 0.988.
  • Release, restart, watch, and keep the way back. route release makes the candidate current only if it passes; --rollback swaps back in one command. After a new route, give the drift check a new reference.

You can now run, watch and update a router on Larkfield’s synthetic inbox. But this inbox is easier than a real one: a classic model reaches 0.962 here. Next, the case studies test the same methods on real messages, starting with 77 kinds of banking questions.

Next: Route banking questions