Monitor and retrain
- Lesson 28
- 40 min
- Module 9: Run it in production
Where we are
In the last lesson, we served suggestions from a nightly batch, with a fallback that always answers. But the shop changes every week: new products, a new season, new customers. In this lesson, we watch the recommender week by week and by segment, see what a model that is never retrained loses, and compare a new model with the current one before we release it.
Lesson 28 of 29 · Recommendation and Personalisation: Show Each Person What MattersMonitor and retrain
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Watch a running recommender week by week, for each kind of customer. Decide how often to retrain it, and check that a retrained model is really better before it replaces the current one.
NotebookDo this lesson in a notebook
The notebook measures a trending list over the last eight weeks of the log, by segment. It compares a list computed again every week with one computed once and never again. You compute what the stale list costs. Then you apply a simple release rule. Its setup cells download about 7 MB. About 15 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Batch serving and the fallback, from Batch or real time.
- Customer segments (new, occasional, loyal) and coverage, from Evaluate by segment and Beyond accuracy.
- ALS, from ALS and BPR with implicit.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject after the last lesson, with its virtual environment active. - The practice files in
../nextia-learning-labs, beside the project folder. You copy one file from its snapshotend-of-m09.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.
A shop assistant who stopped learning
Section titled “A shop assistant who stopped learning”Think of the good shop assistant from the start of this course. In January, she learned what each regular customer likes. Winter coats for one, bird food for another. Her suggestions were excellent.
Then she stopped paying attention. It is now June. She still suggests winter coats. She has never heard of the new summer range. And she does not know the customers who came for the first time in spring, so she treats them all the same.
Nothing dramatic happened. She is still polite, still fast, still at work every day. Her suggestions just became a little worse each week. The manager only notices if she checks every Monday: how many suggestions customers took, for regulars and for newcomers.
The problem: a model that goes wrong quietly
Section titled “The problem: a model that goes wrong quietly”Larkfield’s recommender can do the same. Here is a real example from the course’s measurements. An ALS model was trained on every event before 11 May 2026. Then it was never trained again.
- The service still answers in under a millisecond.
/healthstill says"status":"ok".- No error appears in any log.
But the model’s recall@10 fell from 0.131 in the week of 11 May to 0.093 in the week of 29 June. That is about 29% less. Over the same weeks, its lists contained almost no products of the new summer range: 0.1% of the places or less. Nothing crashed. The suggestions just got worse.
A crash tells you that something is wrong. A slow decline does not. You only see it if you measure it.
The idea: monitoring, a retrain schedule and a release check
Section titled “The idea: monitoring, a retrain schedule and a release check”Monitoring means measuring the running system again and again, and comparing each number with what you expect. A model that has not been trained on recent data is called stale, like stale bread.
The world behind the data changes: seasons, new products, new customers. This is called drift. The MLOps course explains it in Data and concept drift. For a recommender, the usual answer is simple: train the model again on the newest data, on a fixed retrain schedule, for example every week.
But a retrained model can also be worse, for example after a bug in the night job. So before it replaces the current model, a release check compares the two on the same recent customers. If the new model is worse, the current one stays. Keeping the old files also lets you roll back: go back to the previous model in one step.
What to watch
Section titled “What to watch”Larkfield watches three groups of numbers. They come at different speeds.
| Group | Numbers | How soon you know |
|---|---|---|
| Health of the service | Latency p50 and p95; share of answers from the fallback, and why; empty or short lists; errors | At once, at every request |
| Quality of the lists | Recall@10 by segment; coverage; share of new products in the lists | After a week: you must wait for the orders |
| Business | Click and order rates by segment; added revenue per visit; returns; unsubscribes | After a week or more, from the shop’s logs |
The middle group has a delay because recall needs to know what customers ordered. The MLOps course calls this a delayed label, in Delayed labels.
Larkfield’s project measures the first two groups. Click and order rates come from the shop’s own logs of what each visit saw and did. The project has no live shop. The only rates it has are the simulated ones from Online tests and guardrails.
Why “by segment”
Section titled “Why “by segment””One average can hide a group that is doing badly. Here is a constructed week with five customers:
| Customer | Segment | Products ordered | Of them, in the list | Recall@10 |
|---|---|---|---|---|
| A | new | 2 | 1 | 0.50 |
| B | new | 1 | 0 | 0.00 |
| C | loyal | 1 | 1 | 1.00 |
| D | loyal | 2 | 0 | 0.00 |
| E | loyal | 2 | 1 | 0.50 |
The mean of all five is (0.5 + 0 + 1 + 0 + 0.5) ÷ 5 = 0.40. New customers alone: (0.5 + 0) ÷ 2 = 0.25. Loyal customers: (1 + 0 + 0.5) ÷ 3 = 0.50.
Now imagine that next week the loyal customers stay at 0.50 and the new ones fall to 0. With the same mix of customers, the mean falls only to 0.30. A team that watches only the mean sees a small dip. A team that watches segments sees that new customers get nothing useful.
Guided practice: eight weeks of monitoring
Section titled “Guided practice: eight weeks of monitoring”recs monitor replays the last eight weeks of the log, as if the recommender had been running. For each week, it trains the model on every event before that week (retrained weekly). Then it measures the lists on the orders of the week. Beside it, it measures the model of the first week, never trained again (the stale model).
-
In your
larkfield-recsfolder, with the virtual environment active, copy the command:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m09cp $S/larkfield_recs/commands/monitor.py larkfield_recs/commands/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m09"Copy-Item "$S\larkfield_recs\commands\monitor.py" larkfield_recs\commands\ -
Run eight weeks with ALS. It trains nine models, so it takes about half a minute:
Terminal window recs monitor --weeks 8als: retrained every week; 'stale model' = the model of the first week, never retrainedweek from customers recall new occasional loyal coverage new prod. short lists stale model2026-05-11 1,208 0.146 0.118 0.123 31.2% 0.1% 0.0% 0.1312026-05-18 1,215 0.140 0.129 0.138 31.3% 2.7% 0.0% 0.1192026-05-25 1,209 0.213 0.131 0.171 31.3% 2.7% 0.0% 0.1492026-06-01 1,176 0.139 0.133 0.165 31.1% 3.2% 0.0% 0.0952026-06-08 1,151 0.186 0.141 0.172 31.1% 6.0% 0.0% 0.1162026-06-15 1,170 0.155 0.145 0.164 31.2% 6.6% 0.0% 0.0902026-06-22 1,162 0.183 0.127 0.161 31.4% 9.4% 0.0% 0.0932026-06-29 1,111 0.146 0.152 0.138 31.2% 10.2% 0.0% 0.093 -
Read the columns from left to right:
customers: the customers who ordered in that week.recall new,occasional,loyal: recall@10 of the retrained model, by segment.coverage: the share of the products on sale that appear in at least one list.new prod.: the share of list places that hold products added in the last 60 days.short lists: the share of lists with fewer than 10 products.stale model: recall@10 of the stale model, for all customers.
If recs monitor stops with invalid choice: 'monitor', read the first common mistake below.
Reading the dashboard
Section titled “Reading the dashboard”A dashboard is a page of charts that a team looks at regularly. Here are Larkfield’s first two charts, drawn from the same run.
- new
- occasional
- loyal
Text description of the chart
A line chart over eight weeks, from the week of 11 May 2026 (week 1) to the week of 29 June (week 8). The vertical axis is recall@10, from 0 to 0.25. New customers: 0.146, 0.140, 0.213, 0.139, 0.186, 0.155, 0.183, 0.146. Occasional customers: 0.118, 0.129, 0.131, 0.133, 0.141, 0.145, 0.127, 0.152. Loyal customers: 0.123, 0.138, 0.171, 0.165, 0.172, 0.164, 0.161, 0.138. Each segment has between 309 and 504 customers in a week.
The retrained model holds steady. The lines move up and down a lot, but they do not fall. Each segment has only 309 to 504 customers in a week. So a jump like new customers’ 0.213 in week 3 is mostly chance. Do not act on one week.
- retrained every week
- stale: trained once, before 11 May
Text description of the chart
A line chart over the same eight weeks. The vertical axis is recall@10 for all customers, from 0 to 0.2. Retrained every week: 0.131, 0.136, 0.175, 0.145, 0.169, 0.155, 0.161, 0.145. Stale, trained once before 11 May: 0.131, 0.119, 0.149, 0.095, 0.116, 0.090, 0.093, 0.093. In week 1 the two models are the same. From week 4, the stale model is between 0.050 and 0.069 lower every week.
This second chart is the one that matters. Both lines are measured on the same customers in the same week. So the week’s chance, a good week or a bad week, affects both lines in the same way. The gap between them is the cost of not retraining. It grows quickly: from week 4, the stale model is about a third worse.
Who loses most from a stale model?
Section titled “Who loses most from a stale model?”The course’s chart script also measured the stale model by segment (it is in the chart data, not in the command’s output). In the last week:
| Segment | Customers | Retrained | Stale |
|---|---|---|---|
| New | 465 | 0.146 | 0.088 |
| Occasional | 312 | 0.152 | 0.088 |
| Loyal | 334 | 0.138 | 0.105 |
Every segment loses. New customers lost most in six of the seven weeks after the first. In the week of 1 June, their recall with the stale model was only 0.059, against 0.139 with the retrained one. A likely reason: many of them first visited after 11 May. The stale model has never seen them, like the assistant who never met the spring customers.
The other columns
Section titled “The other columns”Three columns stay flat, and that is good news. Coverage stays at about 31% every week. Short lists stay at 0.0%: every customer gets 10 products. The speed of the service does not depend on the week. You measured it in the last lesson: a lookup takes 0.72 ms at p50.
The new products column moves. The summer range launched on 1 May. The retrained model shows it more and more: 0.1% of the list places in the first week, 10.2% in the last. The stale model never learns it: 0.1% or less every week. A flat line here would be a warning even if recall looked fine.
Compare before you release
Section titled “Compare before you release”Retraining every week does not mean releasing every week without a look. Larkfield’s rule is the release check in the picture.
Text description of the diagram
The picture shows Larkfield’s weekly release check. A new week of events arrives. The model is retrained on every event so far. Then the new model and the current model are compared on the last week, with recall by segment, coverage, short lists and speed. If the new model is as good or better for every segment, it is released. The old files are kept, so that the shop can roll back. If it is worse for a segment, or too slow, the current model stays, and a person looks at why.
Apply the rule to the last week of the run. The new model is the one retrained for the week of 29 June. The current model is the stale one:
- New customers: 0.146 against 0.088. Better.
- Occasional customers: 0.152 against 0.088. Better.
- Loyal customers: 0.138 against 0.105. Better.
- Coverage 31.2%, short lists 0.0%, speed unchanged. No problem.
So the rule says: release. In a normal week, the two models are only one week apart, and the difference is small. Then the rule mainly catches a broken retrain: a model trained on half the data, or on the wrong dates.
The comparison needs a little tolerance. Week-to-week noise can make a good model look slightly worse in one segment. A practical rule allows a small loss, for example 0.01. The MLOps course builds such checks in Evaluation gates.
How often to retrain
Section titled “How often to retrain”In this run, the stale model fell behind within three weeks. Retraining ALS takes about 4 seconds, and the nightly two-stage batch about 2 minutes. Under the course’s server price, that is a fraction of a cent. So the computing cost is not the limit. The real costs are the checks and the people who look at a failed check. For Larkfield, a weekly retrain with a release check, and the nightly batch that uses the released model, is a sensible start.
Your turn: does a trending list go stale too?
Section titled “Your turn: does a trending list go stale too?”Trending is the fallback of the service. It is also computed from the last 28 days, so it can go stale. Run the same monitoring for trending:
recs monitor --method trending --weeks 8Predict first. Does a trending list that was computed once in early May lose more or less than ALS by the end of June? What happens to the new prod. column? Then open the answer.
Show the answer
The course team’s run (about one second):
trending: retrained every week; 'stale model' = the model of the first week, never retrainedweek from customers recall new occasional loyal coverage new prod. short lists stale model2026-05-11 1,208 0.155 0.153 0.118 0.3% 0.0% 0.0% 0.1442026-05-18 1,215 0.150 0.184 0.162 0.3% 10.0% 0.0% 0.1442026-05-25 1,209 0.246 0.145 0.189 0.3% 10.0% 0.0% 0.1632026-06-01 1,176 0.148 0.173 0.150 0.3% 20.0% 0.0% 0.0912026-06-08 1,151 0.192 0.157 0.112 0.3% 20.0% 0.0% 0.0902026-06-15 1,170 0.165 0.153 0.156 0.3% 20.0% 0.0% 0.0942026-06-22 1,162 0.198 0.138 0.143 0.3% 30.0% 0.0% 0.0752026-06-29 1,111 0.148 0.191 0.122 0.3% 30.0% 0.0% 0.098The frozen trending list starts at 0.144 and falls to 0.075 to 0.098 from June. That is about as bad as the stale ALS model. The trending list recomputed every week stays high, and by the end 3 of its 10 products are new products (30%). Coverage is 0.3% because everybody gets the same 10 products.
So the fallback needs a schedule too. In the project, recs batch writes a new fallback.json every time it runs. A service whose night job stopped weeks ago serves a stale fallback as well as stale lists.
When to monitor what, and what it costs
Section titled “When to monitor what, and what it costs”- Health numbers (speed, fallback share, short lists): watch them all the time, with an alert. They are cheap, and a problem there is urgent.
- Quality by segment (recall, coverage, new products): measure them every week, when the orders of the week are known. Compare the retrained and the current model on the same customers.
- Business numbers (click and order rates by segment, returns, unsubscribes): read them weekly from the shop’s logs, and in online tests.
Do not set an alert on one week of one segment: a few hundred customers make a noisy number. Look at several weeks, or at the paired gap between two models.
The costs are small in computing and real in people. Somebody must read the dashboard. Somebody must decide when a check fails. Without that person, monitoring is a page of charts that nobody opens.
Common mistakes
Section titled “Common mistakes”Common mistake
recs: error: argument COMMAND: invalid choice: 'monitor'
Symptom: recs monitor prints:
recs: error: argument COMMAND: invalid choice: 'monitor' (choose from batch, bench, candidates, check, compare, evaluate, explain, info, leakage, llm, loop, matrix, online, rank, reset, rules, serve, session, similar, tune)Cause: larkfield_recs/commands/monitor.py is not in your project yet. You see this in a project at the end of the last lesson.
Fix: copy the file as in step 1, from the folder larkfield-recs.
Common mistake
'New customers fell by a third: roll back!'
Symptom: Tomás reads the retrained model’s recall for new customers: 0.213 in the week of 25 May, 0.139 the next week. He wants to roll back.
Cause: one segment in one week is a few hundred customers, so the number jumps by chance. The week after, it was 0.186 again. And the stale model, measured on the same customers, was lower in both weeks.
Fix: compare the two models on the same week and the same customers, and look at several weeks before you act.
Knowledge check
Section titled “Knowledge check”Check your understanding
The service answers in 0.7 ms and /health says ok every day. Why can the recommender still be failing?
Show the answer
Speed and health only show that the service answers. They do not show whether the suggestions are good. A stale model answers just as fast. In Larkfield’s run, an ALS model that was never retrained fell from recall@10 0.131 to 0.093 in seven weeks. No error appeared anywhere. Only quality measures, such as recall by segment and the share of new products, show that kind of failure.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why is the gap between the retrained and the stale model a better signal than the retrained model's recall alone?
Show the answer
Both models are measured on the same customers in the same week. A good or bad week moves both numbers in the same way, so the gap removes most of that noise. The retrained model alone jumps from week to week (for example 0.175, then 0.145). The gap was between 0.050 and 0.069 from week 4 on: a steady signal that retraining pays.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In your project, run recs monitor --method trending --weeks 8. What does the new prod. column show for the retrained list, and why does it matter for the service?
Show the answer
It rises from 0.0% in the week of 11 May to 30.0% in the last two weeks. The retrained trending list takes in new products. The stale trending list never does, and its recall falls to 0.075–0.098. It matters because trending is the service’s fallback. If the night job stops, the fallback goes stale too. So the fallback needs the same schedule as the model.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Monitoring measures the running system again and again. Watch three groups: service health (at once), list quality by segment (after a week), business numbers (from the shop’s logs).
- A stale model fails quietly. Larkfield’s ALS, never retrained, fell from 0.131 to 0.093 in seven weeks. Its lists held 0.1% new products or less; the retrained model reached 10.2%.
- Compare models on the same customers and week. The gap (0.050 to 0.069 from week 4) is steadier than one model’s weekly value.
- A release check compares the retrained and the current model by segment, with coverage, short lists and speed. Keep the old files to roll back.
- One segment in one week is noisy: do not act on a single week. The fallback needs a retrain schedule too.
The recommender is now served, monitored and retrained. But every list is built from what customers did, and that is personal data. Next, you handle a customer who asks to be forgotten, a customer who opts out, and a family who shares one inbox.