Drift and new topics
- Lesson 36
- 40 min
- Module 9: Run it in production
Where we are
Our routing service runs and keeps a log of its decisions. The world does not stay still: Omar launches a new product, and a courier strike fills the inbox. In this lesson, we watch label shares, confidence and the review rate, and we find a new topic in the unsure messages.
Lesson 36 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceDrift and new topics
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Notice when the inbox changes, before the customers do. You watch three signals that need no labels: the share of each route, the confidence and the share of unsure messages. Then you decide, with evidence, whether the world changed or the model broke, and you find a new topic in the unsure messages.
Before you startWhat to know and to have ready
What you need to know
- The routing service, the model
v1and its zones, from A routing service. This lesson continues in the same project folder. - The validation messages and their unsure share, from Decide, review or send to a person. This lesson teaches PSI, the Population Stability Index, from the start. Has the distribution changed? in the anomaly detection course uses it on other data (optional).
What you need on your computer
The project folder larkroute from the last lesson, with its environment active and the model v1 as the current model. The launch month is already in data/launch_month.csv. Tested on macOS with Python 3.12.13, pandas 3.1.1 and scikit-learn 1.9.1. The commands are the same in PowerShell, but we did not run them there.
Two changes at the post office
Section titled “Two changes at the post office”Think of the sorting counter again. One December, the post office gets twice as many parcels as usual. The sorter’s work changes a lot: the parcel bin fills up every hour. But her rules are still right. Every parcel goes into the parcel bin, as it should. The world changed, and the sorting did not break.
In the same month, a new housing estate opens at the edge of town. Its streets are not in her rulebook. She does not know where to put those letters, so they pile up in the supervisor’s tray. This time, the rulebook is missing something.
Both changes show up at the counter as “something is different”. But they need opposite answers. For the parcels, she needs more help at the bin, not new rules. For the new estate, she needs new rules. The supervisor must tell the two apart, and the only way is to look at the letters.
The problem: a model never says “I do not know this”
Section titled “The problem: a model never says “I do not know this””In February 2026, Larkfield had both changes in one month. The launch month has 2,000 messages, from 2 February to 1 March. Omar launched a new product, the Grow Hub: an indoor garden with a Wi-Fi controller and soil sensors. From 9 February, customers wrote about setting it up. And in the week of 16 February, a courier strike delayed parcels.
These are constructed events: we planted them in the synthetic inbox to practise on. The data also has a hidden truth, the right route for every launch message. It exists only because the inbox is simulated; a real help desk does not have it. This lesson uses it at the end, to check our conclusions. During the month, Larkfield sees only what you see.
The model v1 has seven routes. None of them is right for a Grow Hub setup question. A model has no way to say “this is new”. It must pick one of the routes it knows. The hidden truth shows what it did with the 167 Grow Hub setup messages:
Text description of the chart
A column chart with four columns, for the 167 Grow Hub setup messages and the route that model v1 chose for them. Product 79, returns_refunds 31, account 19, and the four other routes together 38: billing 15, delivery 14, orders 7 and needs_person 2. The columns get shorter from left to right.
No error message appeared. Each of these messages got a route and a confidence. Most of them (77.8%) were unsure, so they went to Chloé’s review queue. The others went straight to a wrong team.
And accuracy cannot warn you. To measure accuracy, you need the right route of each message. In real use, nobody labels the inbox as it arrives.
The idea: watch what you can see without labels
Section titled “The idea: watch what you can see without labels”Even without labels, the service’s log shows three things every week:
- The share of each route. If 20% of messages usually go to delivery, and suddenly 47% do, something changed.
- The mean confidence. If the model is less sure on average, the messages may be less like the ones it learned from.
- The unsure share. This is the share of messages in the review or person zone. On the validation messages, it was 9.8%.
A change in the data that a model meets, compared with the data it learned from, is called drift. A change in the route shares is prediction drift: the model’s answers move. Drift is a signal to look, not a verdict. It does not say whether the model is wrong.
One number for “the shares moved”: PSI
Section titled “One number for “the shares moved”: PSI”To watch seven route shares at once, the project sums their changes into one number: the Population Stability Index, or PSI. For each route, it takes the new share minus the old share, times the natural log of new share divided by old share. Then it adds the terms. Big changes in either direction add a lot, and small changes add almost nothing.
Here is a constructed example with three groups of routes. The old shares come from a normal week, and the new shares from a busy week:
| Route | Old share | New share | New − old | ln(new / old) | Term |
|---|---|---|---|---|---|
| delivery | 0.20 | 0.47 | +0.27 | +0.854 | 0.231 |
| product | 0.15 | 0.13 | −0.02 | −0.143 | 0.003 |
| everything else | 0.65 | 0.40 | −0.25 | −0.486 | 0.121 |
| PSI | 0.355 |
Each term is positive, because the two factors always have the same sign. A common rule of thumb says that a PSI below 0.10 means no real change. From 0.10 to 0.25 is a moderate change, and above 0.25 is a large change. These limits are conventions, not laws.
The drift check of the project
Section titled “The drift check of the project”route drift replays the launch month through the current model, one week at a time. It compares each week with the validation messages. It raises an alarm for a week when one of two things happens:
- the PSI of the route shares is 0.10 or more, or
- the unsure share is at least 1.5 times the validation share (9.8% × 1.5 = 14.7%).
An alarm does not retrain anything. It says: “Read a sample of messages before you retrain.”
Run the drift check
Section titled “Run the drift check”-
In the
larkroutefolder, with the environment active, run the drift check:Terminal window route driftModel v1; validation: unsure (review or person) 9.8%. Launch month, by week:week messages delivery product account mean_conf unsure psi2026-02-02 533 0.206 0.152 0.109 0.911 0.111 0.0162026-02-09 477 0.199 0.193 0.111 0.873 0.170 0.0252026-02-16 510 0.469 0.129 0.080 0.875 0.125 0.2842026-02-23 480 0.144 0.231 0.106 0.844 0.208 0.083ALARM week of 2026-02-09: unsure messages 17%. Read a sample of messages before you retrain.ALARM week of 2026-02-16: route shares moved (PSI 0.28). Read a sample of messages before you retrain.ALARM week of 2026-02-23: unsure messages 21%. Read a sample of messages before you retrain. -
Read the table one column at a time.
delivery,productandaccountare the shares of three routes.mean_confis the mean confidence,unsureis the unsure share andpsiis the PSI against validation.
The chart shows the two columns that raised the alarms:
- Unsure (review or person)
- Delivery (predicted)
- Validation: unsure 9.8%
- Alarm: unsure 14.7%
Text description of the chart
The chart has four weeks on the horizontal axis: 2, 9, 16 and 23 February. The unsure share is 11.1%, 17.0%, 12.5% and 20.8%. A dashed line at 9.8% shows the validation share, and a dashed line at 14.7% shows the alarm level. The unsure share is above the alarm line in the second and fourth weeks. The predicted delivery share is 20.6%, 19.9%, 46.9% and 14.4%. It more than doubles in the third week, the strike week, while the unsure share that week is 12.5%, below the alarm line.
Text description of the chart
A column chart with four columns, one per week. The PSI is 0.016 in the week of 2 February and 0.025 in the week of 9 February. It is 0.284 in the week of 16 February and 0.083 in the week of 23 February. Only the third column is above the alarm level of 0.10. It is drawn in a different colour.
The three alarms tell two different stories:
- The week of 16 February: the route shares moved (PSI 0.284), because the delivery share more than doubled. But the model stays sure: its unsure share, 12.5%, is lower than the week before.
- The weeks of 9 and 23 February: the route shares barely moved (PSI 0.025 and 0.083). But the unsure share rose to 17.0% and 20.8%. The mean confidence fell from 0.911 to 0.873 and 0.844. The model meets more messages that it cannot place.
Did the world change, or did the model break?
Section titled “Did the world change, or did the model break?”An alarm asks one question: should we change the model? To answer it, you need a person who reads messages. The diagram shows the decision:
Text description of the diagram
The diagram starts with an alarm. Either the route shares moved, with a PSI of 0.10 or more, or the unsure share reached 1.5 times the validation share. The next step is to read and label a sample of that week’s messages. Then a first question: do the messages fit the routes we have? If not, it is a new topic: group the unsure messages, name the topic and add a label. The example is Grow Hub setup. If they fit, a second question: is the model still right on the labelled sample? If yes, the world changed: keep the model and tell the team. The example is the courier strike. If not, the model is wrong for these messages: label more of them, retrain, and check the new model before release.
The strike week: read and check a sample
Section titled “The strike week: read and check a sample”Chloé labelled 250 launch messages, about 60 a week: data/launch_labels_sample.csv. Of these, 80 are about the Grow Hub, which she labelled devices, a route that does not exist yet. You can compare v1 with her labels, week by week. Create the file week_check.py in the larkroute folder:
"""Check model v1 against Chloé's labels, week by week, and print three delivery messages of the strike week."""import pandas as pd
from larkroute import data, model_storefrom larkroute.drift import route_messages
model, card = model_store.load("v1")stream = route_messages(model, card, data.launch_month())labels = pd.read_csv(data.data_dir() / "launch_labels_sample.csv")checked = stream.merge(labels, on="message_id")checked["week"] = pd.to_datetime(checked["created_at"]).dt.to_period("W-SUN").dt.start_time.dt.dateold = checked[checked["route"] != "devices"] # only the routes that v1 knowsweeks = checked.groupby("week")print(pd.DataFrame({ "labelled": weeks.size(), "delivery_label": weeks["route"].apply(lambda r: (r == "delivery").mean()).round(3), "delivery_predicted": weeks["predicted"].apply(lambda p: (p == "delivery").mean()).round(3), "old_routes": old.groupby("week").size(), "right_on_old_routes": old.groupby("week").apply(lambda g: (g["predicted"] == g["route"]).mean()).round(3),}).to_string())strike = stream[(stream["created_at"] >= "2026-02-16") & (stream["created_at"] < "2026-02-23")]for text in strike[strike["predicted"] == "delivery"]["text"].head(3): print("-", " ".join(text.split())[:110])route_messages() is the function that route drift uses: it adds the predicted route, the confidence and the zone to each message. Run the file:
python week_check.py labelled delivery_label delivery_predicted old_routes right_on_old_routesweek2026-02-02 47 0.191 0.191 47 0.9572026-02-09 71 0.141 0.155 37 1.0002026-02-16 60 0.400 0.417 43 0.9302026-02-23 72 0.083 0.097 43 0.953- Could you please check the status of order LK-781053? It was due on Jan 28, but it has not arrived. My payment- Bonjour, Ma commande LK-917032 devait arriver le 15/01, mais je ne l’ai toujours pas reçue. Pourriez-vous véri- Tracking for LK-310229 hasn’t moved in 3 days. Have my solar lights been lost? MorganNow put the evidence for the strike week together:
- The messages fit an existing route. The sample messages are ordinary late-parcel questions. Chloé labelled 40.0% of the week’s sample as delivery; the model said 41.7%.
- The model is still right. On the routes it knows, it was right on 0.930 of the week’s labelled messages: 40 of 43. With only 43 messages, one or two errors move this number a lot. So 0.930 is close to the 0.957 and 0.953 of the other weeks.
- The model is still sure. The unsure share that week was 12.5%, below the alarm level.
- Something outside explains it. Grace knows that a courier strike delayed parcels that week.
So the world changed, and the model did not break. More customers wrote about late parcels, and the model sent them to delivery, as it should. The right answer is not a new model. It is more agents on the delivery team that week, and perhaps a notice on the website about the strike.
The hidden truth confirms it. That week, 46.3% of the messages were really about delivery, and the model said 46.9%. On the old routes, it was right on 0.953 of all 510 messages. Remember that only a simulated inbox has this check.
Find the new topic in the unsure messages
Section titled “Find the new topic in the unsure messages”The weeks of 9 and 23 February are a different story: more messages that the model cannot place. If many of them are about the same new subject, it is a new topic: a kind of message that has no route yet. To find it, you need to group the unsure messages by what they say.
Group messages with similar words
Section titled “Group messages with similar words”Clustering puts items into groups so that the items in one group look alike, without any labels. The project uses k-means on the TF-IDF numbers of the messages. You choose k, the number of groups. k-means starts with k centres and puts each message in the group of its nearest centre. Then it moves each centre to the middle of its group, and repeats until the groups stop changing. For each group, the project prints its size and its six top words. These are the words with the largest weights at the group’s centre.
Think of three short messages and k = 2: “Grow Hub light blinking orange”, “the orange light keeps blinking” and “where is my parcel”. The first two share “light”, “blinking” and “orange”, so their TF-IDF numbers are close, and they end in one group. The parcel question shares no word with them, so it ends in the other group. The top words of the first group would be its shared words: “blinking”, “light” and “orange”.
k-means always makes exactly k groups, even when the data has fewer real topics. It finds groups of words, not meanings. So a person must read each group and say what it is.
-
Group the unsure messages since 9 February, the first day of Grow Hub messages:
Terminal window route drift --topics245 unsure messages since 2026-02-09, in 6 groups:group messages top_words5 73 order, lk, larkfield, really, hello, mentioned2 66 je, est, vous, la, le, bonjour4 37 orange, fi, wi, light, blinking, keeps0 27 update, firmware, schedule, watering, stopped, grow1 23 offline, soil, app, right, sensor, says3 19 pair, second, sensor, grow, explain, helloMessages of the largest group: M-200580, M-200582, M-200584, M-200615, M-200620 -
Create the file
read_groups.pyin thelarkroutefolder. It prints two messages from each group:"""Print two unsure messages from each group, so that a person can read them."""from larkroute import data, model_storefrom larkroute.drift import route_messages, topicsmodel, card = model_store.load()stream = route_messages(model, card, data.launch_month())unsure = stream[(stream["zone"] != "decide") & (stream["created_at"] >= "2026-02-09")]table, groups = topics(unsure["text"], k=6)for g in table["group"]:print(f"--- group {g}")for text in unsure["text"][groups == g].head(2):print(" ", " ".join(text.split())[:110]) -
Run it:
Terminal window python read_groups.py--- group 5i’m a journalist writing about delivery problems and need someone to call me today, please—this is really frusHi, I’m a journalist writing about delivery problems. I’m not sure if you can arrange a call, but could someon--- group 2Bonjour, je vous ai écrit au sujet de la commande LK-328749 : le produit planter est arrivé brisé. Pouvez-vousBonjour, je suis vraiment déçue : personne ne répond après trois courriels et deux clavardages. Je veux parler--- group 4Hi, my Grow Hub won’t connect to my Wi-Fi, and the light keeps blinking orange. I’m getting frustrated becauseHi @Larkfield, it’s still doing the same thing as in my last email—the orange light keeps blinking 😕--- group 0@Larkfield, since the firmware update, my Grow Hub has stopped watering on schedule. Could you please advise hDear Larkfield, Since the firmware update, the Grow Hub has stopped watering according to its schedule. Could--- group 1Could you please help check this? The Larkfield app says the soil sensor is offline, even though it is right bhi, the app says my soil sensor is offline. It’s right next to the hub, though, so I’m not sure what’s gone wr--- group 3Hello, how do I pair a second sensor with the Grow Hub? Thank you, MorganHi, how do I pair a second sensor with the Grow Hub? Thanks, Riley
Chloé reads the groups with Omar:
- Groups 4, 0, 1 and 3 are all the Grow Hub. Group 4 is the Wi-Fi light that blinks orange (37 messages). Group 0 is the schedule that stopped after a firmware update (27). Group 1 is a soil sensor that shows offline (23), and group 3 is pairing a second sensor (19).
- Group 5 is not a topic. It holds messages that are hard for other reasons, such as vague follow-ups (“as I mentioned”) and a journalist who wants a call. Its top words are common words.
- Group 2 is the French messages. k-means put them together because they share French words like “je” and “vous”, not because they share a subject.
The hidden truth agrees: every message in groups 0, 1, 3 and 4 is a Grow Hub setup message. In group 2, 28.8% are, and in group 5, 6.8%. So the new topic is Grow Hub setup, with four kinds of problem. Omar and Grace agree on a new route, devices, and a small team that knows the product. The next lesson adds the route to the model.
Clustering is one way to find unusual messages. Unusual text and logs with embeddings in the anomaly detection course shows others, such as distances in an embedding space.
Your turn: more groups
Section titled “Your turn: more groups”Group 2 holds French messages, and 28.8% of them are about the Grow Hub. Run the topic step with 8 groups instead of 6. Before you run it, predict: what happens to the French group? Do the four Grow Hub groups stay?
route drift --topics --k 8Show the answer
245 unsure messages since 2026-02-09, in 8 groups: group messages top_words 0 73 order, lk, larkfield, really, hello, mentioned 2 50 je, est, la, le, vous, ai 1 27 update, firmware, schedule, watering, stopped, grow 4 25 light, blinking, orange, keeps, fi, wi 3 23 offline, soil, app, right, sensor, says 7 19 pair, second, sensor, grow, explain, hello 5 16 deuxième, comment, capteur, au, associer, vous 6 12 clignote, voyant, se, pas, le, neMessages of the largest group: M-200580, M-200582, M-200584, M-200615, M-200620The four English Grow Hub groups stay (27, 25, 23 and 19 messages). The French group shrinks from 66 to 50. Two new French groups appear. Group 5 is “comment associer un deuxième capteur”: how to pair a second sensor. Group 6 is “le voyant clignote”: the light blinks. The orange-light group shrinks from 37 to 25, because its French messages moved to group 6. So French customers have the same Grow Hub problems, and the new route needs French examples too.
With --k 4, the opposite happens: the pairing messages join the follow-ups in one large group of 115. Fewer groups hide small topics. There is no right k; try two or three values and read the groups each time.
When to watch drift, and what it costs
Section titled “When to watch drift, and what it costs”Run a drift check on a schedule, for example every Monday for the week before. It is cheap: route drift took about 1.5 seconds on the test laptop for 2,000 messages. It needs no labels, so it works from the first day.
The expensive part is the person who reads. Every alarm needs someone who reads a sample and decides. Give the alarms an owner (here, Chloé with Grace), or people will learn to ignore them.
Know the limits. A weekly window of about 500 messages is small: shares move by chance from week to week. PSI says that the shares moved, not why. And drift checks find new topics only if their messages are unsure or shift the shares. The labelled sample is your only direct measure of accuracy, so keep labelling a few messages every week.
Do not retrain automatically when an alarm fires. In the strike week, a new model would have learned nothing useful: the old one was right.
Common mistake
The largest group is not the new topic
Symptom: Tomás runs route drift --topics, sees that group 5 is the largest (73 messages), and proposes a new route called “follow-ups”. He does not read the messages. Chloé finds that the new route would collect vague messages about orders, delivery, refunds and a journalist’s request.
Cause: k-means always makes k groups. Its largest group often holds messages that share common words (“order”, “hello”, “mentioned”), not one subject. Its top words describe the words, not the meaning. Only 6.8% of group 5 are Grow Hub messages, and the rest belong to routes that already exist.
Fix: read a few messages from every group, and let a person name each group, or say “not a topic”. Here, the four smaller groups were the real topic. Check your reading against a labelled sample when you have one.
Consequence in AI work: a route built from a misread group sends messages away from the teams that can answer them. It also teaches the next model a label that means nothing. Clustering finds candidates; people decide what they are.
Knowledge check
Section titled “Knowledge check”Check your understanding
Completion check: the delivery share rose from about 20% to 46.9% in the week of 16 February. Is this drift in the model or a real change in the world? Give your evidence.
Show the answer
It is a real change in the world: a courier strike delayed parcels, and more customers wrote about them. The evidence:
- The messages fit an existing route. The sample messages are ordinary late-parcel questions.
- On Chloé’s labelled sample, the delivery share was 40.0%, and the model said 41.7%.
- The model was still right on the routes it knows. It got 0.930 of the week’s labelled messages right (40 of 43), close to the other weeks.
- The model was still sure: the unsure share was 12.5%, below the alarm level of 14.7%. Only the PSI (0.284) raised the alarm.
In this simulated inbox, the hidden truth confirms it. The true delivery share was 46.3%, against 46.9% predicted, and the accuracy on the old routes was 0.953. The answer is more delivery agents that week, not a new model.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: a week has PSI 0.02 and an unsure share of 21%. What has probably happened, and what do you do first?
Show the answer
The route shares hardly moved, but the model is unsure much more often than on validation (9.8%). This looks like messages that the model cannot place: a new topic, or new wording for an old one. It is the pattern of the weeks of 9 and 23 February. First, read a sample of the unsure messages, for example with route drift --topics and read_groups.py. Then decide with Chloé what the groups are. Do not retrain before you know.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Compute the PSI of two routes whose shares move from 0.50 and 0.50 to 0.60 and 0.40. Is it a large change by the rule of thumb?
Show the answer
- First route: (0.60 − 0.50) × ln(0.60 / 0.50) = 0.10 × 0.182 = 0.018.
- Second route: (0.40 − 0.50) × ln(0.40 / 0.50) = −0.10 × −0.223 = 0.022.
- PSI = 0.018 + 0.022 = 0.040.
That is below 0.10, so by the rule of thumb it is no real change. Yet each share moved by ten points.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A model never says “this is new”.
v1spread the 167 Grow Hub setup messages over every route, and 77.8% of them were unsure. - Without labels, watch three signals each week: the route shares (summed into PSI), the mean confidence and the unsure share.
route driftraises an alarm at PSI 0.10 or an unsure share of 14.7% (1.5 × 9.8%). - An alarm means “read”, not “retrain”. In the strike week (PSI 0.284), the messages fit, the model was right on a labelled sample and it stayed sure: the world changed.
- In the weeks of 9 and 23 February, the unsure share rose to 17.0% and 20.8%. k-means on the 245 unsure messages found four groups that are all the Grow Hub. A person must read every group: the two largest were not the topic.
You now know that Larkfield needs a new route, devices. Adding a label changes the model, and a new model can break what worked. Next, you train it with Chloé’s labels and release it only if it passes a check on every slice.