Skip to content
Français

Monitor and retrain

  • Lesson 28
  • 40 min
  • Module 9: Run it in production

Where we are

In the last lesson, we served suggestions from a nightly batch, with a fallback that always answers. But the shop changes every week: new products, a new season, new customers. In this lesson, we watch the recommender week by week and by segment, see what a model that is never retrained loses, and compare a new model with the current one before we release it.

What you will be able to do

Watch a running recommender week by week, for each kind of customer. Decide how often to retrain it, and check that a retrained model is really better before it replaces the current one.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook measures a trending list over the last eight weeks of the log, by segment. It compares a list computed again every week with one computed once and never again. You compute what the stale list costs. Then you apply a simple release rule. Its setup cells download about 7 MB. About 15 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project after the last lesson, with its virtual environment active.
  • The practice files in ../nextia-learning-labs, beside the project folder. You copy one file from its snapshot end-of-m09.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

Think of the good shop assistant from the start of this course. In January, she learned what each regular customer likes. Winter coats for one, bird food for another. Her suggestions were excellent.

Then she stopped paying attention. It is now June. She still suggests winter coats. She has never heard of the new summer range. And she does not know the customers who came for the first time in spring, so she treats them all the same.

Nothing dramatic happened. She is still polite, still fast, still at work every day. Her suggestions just became a little worse each week. The manager only notices if she checks every Monday: how many suggestions customers took, for regulars and for newcomers.

The problem: a model that goes wrong quietly

Section titled “The problem: a model that goes wrong quietly”

Larkfield’s recommender can do the same. Here is a real example from the course’s measurements. An ALS model was trained on every event before 11 May 2026. Then it was never trained again.

  • The service still answers in under a millisecond.
  • /health still says "status":"ok".
  • No error appears in any log.

But the model’s recall@10 fell from 0.131 in the week of 11 May to 0.093 in the week of 29 June. That is about 29% less. Over the same weeks, its lists contained almost no products of the new summer range: 0.1% of the places or less. Nothing crashed. The suggestions just got worse.

A crash tells you that something is wrong. A slow decline does not. You only see it if you measure it.

The idea: monitoring, a retrain schedule and a release check

Section titled “The idea: monitoring, a retrain schedule and a release check”

Monitoring means measuring the running system again and again, and comparing each number with what you expect. A model that has not been trained on recent data is called stale, like stale bread.

The world behind the data changes: seasons, new products, new customers. This is called drift. The MLOps course explains it in Data and concept drift. For a recommender, the usual answer is simple: train the model again on the newest data, on a fixed retrain schedule, for example every week.

But a retrained model can also be worse, for example after a bug in the night job. So before it replaces the current model, a release check compares the two on the same recent customers. If the new model is worse, the current one stays. Keeping the old files also lets you roll back: go back to the previous model in one step.

Larkfield watches three groups of numbers. They come at different speeds.

Group Numbers How soon you know
Health of the service Latency p50 and p95; share of answers from the fallback, and why; empty or short lists; errors At once, at every request
Quality of the lists Recall@10 by segment; coverage; share of new products in the lists After a week: you must wait for the orders
Business Click and order rates by segment; added revenue per visit; returns; unsubscribes After a week or more, from the shop’s logs

The middle group has a delay because recall needs to know what customers ordered. The MLOps course calls this a delayed label, in Delayed labels.

Larkfield’s project measures the first two groups. Click and order rates come from the shop’s own logs of what each visit saw and did. The project has no live shop. The only rates it has are the simulated ones from Online tests and guardrails.

One average can hide a group that is doing badly. Here is a constructed week with five customers:

Customer Segment Products ordered Of them, in the list Recall@10
A new 2 1 0.50
B new 1 0 0.00
C loyal 1 1 1.00
D loyal 2 0 0.00
E loyal 2 1 0.50

The mean of all five is (0.5 + 0 + 1 + 0 + 0.5) ÷ 5 = 0.40. New customers alone: (0.5 + 0) ÷ 2 = 0.25. Loyal customers: (1 + 0 + 0.5) ÷ 3 = 0.50.

Now imagine that next week the loyal customers stay at 0.50 and the new ones fall to 0. With the same mix of customers, the mean falls only to 0.30. A team that watches only the mean sees a small dip. A team that watches segments sees that new customers get nothing useful.

Guided practice: eight weeks of monitoring

Section titled “Guided practice: eight weeks of monitoring”

recs monitor replays the last eight weeks of the log, as if the recommender had been running. For each week, it trains the model on every event before that week (retrained weekly). Then it measures the lists on the orders of the week. Beside it, it measures the model of the first week, never trained again (the stale model).

  1. In your larkfield-recs folder, with the virtual environment active, copy the command:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m09
    cp $S/larkfield_recs/commands/monitor.py larkfield_recs/commands/
  2. Run eight weeks with ALS. It trains nine models, so it takes about half a minute:

    Terminal window
    recs monitor --weeks 8
    als: retrained every week; 'stale model' = the model of the first week, never retrained
    week from customers recall new occasional loyal coverage new prod. short lists stale model
    2026-05-11 1,208 0.146 0.118 0.123 31.2% 0.1% 0.0% 0.131
    2026-05-18 1,215 0.140 0.129 0.138 31.3% 2.7% 0.0% 0.119
    2026-05-25 1,209 0.213 0.131 0.171 31.3% 2.7% 0.0% 0.149
    2026-06-01 1,176 0.139 0.133 0.165 31.1% 3.2% 0.0% 0.095
    2026-06-08 1,151 0.186 0.141 0.172 31.1% 6.0% 0.0% 0.116
    2026-06-15 1,170 0.155 0.145 0.164 31.2% 6.6% 0.0% 0.090
    2026-06-22 1,162 0.183 0.127 0.161 31.4% 9.4% 0.0% 0.093
    2026-06-29 1,111 0.146 0.152 0.138 31.2% 10.2% 0.0% 0.093
  3. Read the columns from left to right:

    • customers: the customers who ordered in that week.
    • recall new, occasional, loyal: recall@10 of the retrained model, by segment.
    • coverage: the share of the products on sale that appear in at least one list.
    • new prod.: the share of list places that hold products added in the last 60 days.
    • short lists: the share of lists with fewer than 10 products.
    • stale model: recall@10 of the stale model, for all customers.

If recs monitor stops with invalid choice: 'monitor', read the first common mistake below.

A dashboard is a page of charts that a team looks at regularly. Here are Larkfield’s first two charts, drawn from the same run.

  • new
  • occasional
  • loyal
ALS retrained every week, by segment. The lines jump from week to week, but no segment falls steadily.
Text description of the chart

A line chart over eight weeks, from the week of 11 May 2026 (week 1) to the week of 29 June (week 8). The vertical axis is recall@10, from 0 to 0.25. New customers: 0.146, 0.140, 0.213, 0.139, 0.186, 0.155, 0.183, 0.146. Occasional customers: 0.118, 0.129, 0.131, 0.133, 0.141, 0.145, 0.127, 0.152. Loyal customers: 0.123, 0.138, 0.171, 0.165, 0.172, 0.164, 0.161, 0.138. Each segment has between 309 and 504 customers in a week.

The retrained model holds steady. The lines move up and down a lot, but they do not fall. Each segment has only 309 to 504 customers in a week. So a jump like new customers’ 0.213 in week 3 is mostly chance. Do not act on one week.

  • retrained every week
  • stale: trained once, before 11 May
The same weeks and the same customers, two models. In week 1 they are the same model. From week 4, the stale model stays more than 0.04 below the retrained one.
Text description of the chart

A line chart over the same eight weeks. The vertical axis is recall@10 for all customers, from 0 to 0.2. Retrained every week: 0.131, 0.136, 0.175, 0.145, 0.169, 0.155, 0.161, 0.145. Stale, trained once before 11 May: 0.131, 0.119, 0.149, 0.095, 0.116, 0.090, 0.093, 0.093. In week 1 the two models are the same. From week 4, the stale model is between 0.050 and 0.069 lower every week.

This second chart is the one that matters. Both lines are measured on the same customers in the same week. So the week’s chance, a good week or a bad week, affects both lines in the same way. The gap between them is the cost of not retraining. It grows quickly: from week 4, the stale model is about a third worse.

The course’s chart script also measured the stale model by segment (it is in the chart data, not in the command’s output). In the last week:

Segment Customers Retrained Stale
New 465 0.146 0.088
Occasional 312 0.152 0.088
Loyal 334 0.138 0.105

Every segment loses. New customers lost most in six of the seven weeks after the first. In the week of 1 June, their recall with the stale model was only 0.059, against 0.139 with the retrained one. A likely reason: many of them first visited after 11 May. The stale model has never seen them, like the assistant who never met the spring customers.

Three columns stay flat, and that is good news. Coverage stays at about 31% every week. Short lists stay at 0.0%: every customer gets 10 products. The speed of the service does not depend on the week. You measured it in the last lesson: a lookup takes 0.72 ms at p50.

The new products column moves. The summer range launched on 1 May. The retrained model shows it more and more: 0.1% of the list places in the first week, 10.2% in the last. The stale model never learns it: 0.1% or less every week. A flat line here would be a warning even if recall looked fine.

Retraining every week does not mean releasing every week without a look. Larkfield’s rule is the release check in the picture.

A new week of events leads to: retrain the model on every event so far. Then a decision: compare the new and the current model on the last week, using recall by segment, coverage, short lists and speed. If the new one is as good or better for every segment, release it and keep the old one to roll back. If it is worse for a segment or too slow, keep the current model, and a person looks at why.
Larkfield's weekly release check: retrain, compare on the same week, then release or keep.
Text description of the diagram

The picture shows Larkfield’s weekly release check. A new week of events arrives. The model is retrained on every event so far. Then the new model and the current model are compared on the last week, with recall by segment, coverage, short lists and speed. If the new model is as good or better for every segment, it is released. The old files are kept, so that the shop can roll back. If it is worse for a segment, or too slow, the current model stays, and a person looks at why.

Apply the rule to the last week of the run. The new model is the one retrained for the week of 29 June. The current model is the stale one:

  • New customers: 0.146 against 0.088. Better.
  • Occasional customers: 0.152 against 0.088. Better.
  • Loyal customers: 0.138 against 0.105. Better.
  • Coverage 31.2%, short lists 0.0%, speed unchanged. No problem.

So the rule says: release. In a normal week, the two models are only one week apart, and the difference is small. Then the rule mainly catches a broken retrain: a model trained on half the data, or on the wrong dates.

The comparison needs a little tolerance. Week-to-week noise can make a good model look slightly worse in one segment. A practical rule allows a small loss, for example 0.01. The MLOps course builds such checks in Evaluation gates.

In this run, the stale model fell behind within three weeks. Retraining ALS takes about 4 seconds, and the nightly two-stage batch about 2 minutes. Under the course’s server price, that is a fraction of a cent. So the computing cost is not the limit. The real costs are the checks and the people who look at a failed check. For Larkfield, a weekly retrain with a release check, and the nightly batch that uses the released model, is a sensible start.

Section titled “Your turn: does a trending list go stale too?”

Trending is the fallback of the service. It is also computed from the last 28 days, so it can go stale. Run the same monitoring for trending:

Terminal window
recs monitor --method trending --weeks 8

Predict first. Does a trending list that was computed once in early May lose more or less than ALS by the end of June? What happens to the new prod. column? Then open the answer.

Show the answer

The course team’s run (about one second):

trending: retrained every week; 'stale model' = the model of the first week, never retrained
week from customers recall new occasional loyal coverage new prod. short lists stale model
2026-05-11 1,208 0.155 0.153 0.118 0.3% 0.0% 0.0% 0.144
2026-05-18 1,215 0.150 0.184 0.162 0.3% 10.0% 0.0% 0.144
2026-05-25 1,209 0.246 0.145 0.189 0.3% 10.0% 0.0% 0.163
2026-06-01 1,176 0.148 0.173 0.150 0.3% 20.0% 0.0% 0.091
2026-06-08 1,151 0.192 0.157 0.112 0.3% 20.0% 0.0% 0.090
2026-06-15 1,170 0.165 0.153 0.156 0.3% 20.0% 0.0% 0.094
2026-06-22 1,162 0.198 0.138 0.143 0.3% 30.0% 0.0% 0.075
2026-06-29 1,111 0.148 0.191 0.122 0.3% 30.0% 0.0% 0.098

The frozen trending list starts at 0.144 and falls to 0.075 to 0.098 from June. That is about as bad as the stale ALS model. The trending list recomputed every week stays high, and by the end 3 of its 10 products are new products (30%). Coverage is 0.3% because everybody gets the same 10 products.

So the fallback needs a schedule too. In the project, recs batch writes a new fallback.json every time it runs. A service whose night job stopped weeks ago serves a stale fallback as well as stale lists.

  • Health numbers (speed, fallback share, short lists): watch them all the time, with an alert. They are cheap, and a problem there is urgent.
  • Quality by segment (recall, coverage, new products): measure them every week, when the orders of the week are known. Compare the retrained and the current model on the same customers.
  • Business numbers (click and order rates by segment, returns, unsubscribes): read them weekly from the shop’s logs, and in online tests.

Do not set an alert on one week of one segment: a few hundred customers make a noisy number. Look at several weeks, or at the paired gap between two models.

The costs are small in computing and real in people. Somebody must read the dashboard. Somebody must decide when a check fails. Without that person, monitoring is a page of charts that nobody opens.

Common mistake

recs: error: argument COMMAND: invalid choice: 'monitor'

Symptom: recs monitor prints:

recs: error: argument COMMAND: invalid choice: 'monitor' (choose from batch, bench, candidates, check, compare, evaluate, explain, info, leakage, llm, loop, matrix, online, rank, reset, rules, serve, session, similar, tune)

Cause: larkfield_recs/commands/monitor.py is not in your project yet. You see this in a project at the end of the last lesson.

Fix: copy the file as in step 1, from the folder larkfield-recs.

Common mistake

'New customers fell by a third: roll back!'

Symptom: Tomás reads the retrained model’s recall for new customers: 0.213 in the week of 25 May, 0.139 the next week. He wants to roll back.

Cause: one segment in one week is a few hundred customers, so the number jumps by chance. The week after, it was 0.186 again. And the stale model, measured on the same customers, was lower in both weeks.

Fix: compare the two models on the same week and the same customers, and look at several weeks before you act.

Check your understanding

The service answers in 0.7 ms and /health says ok every day. Why can the recommender still be failing?

Show the answer

Speed and health only show that the service answers. They do not show whether the suggestions are good. A stale model answers just as fast. In Larkfield’s run, an ALS model that was never retrained fell from recall@10 0.131 to 0.093 in seven weeks. No error appeared anywhere. Only quality measures, such as recall by segment and the share of new products, show that kind of failure.

Check your understanding

Why is the gap between the retrained and the stale model a better signal than the retrained model's recall alone?

Show the answer

Both models are measured on the same customers in the same week. A good or bad week moves both numbers in the same way, so the gap removes most of that noise. The retrained model alone jumps from week to week (for example 0.175, then 0.145). The gap was between 0.050 and 0.069 from week 4 on: a steady signal that retraining pays.

Check your understanding

In your project, run recs monitor --method trending --weeks 8. What does the new prod. column show for the retrained list, and why does it matter for the service?

Show the answer

It rises from 0.0% in the week of 11 May to 30.0% in the last two weeks. The retrained trending list takes in new products. The stale trending list never does, and its recall falls to 0.075–0.098. It matters because trending is the service’s fallback. If the night job stops, the fallback goes stale too. So the fallback needs the same schedule as the model.

Recap

  • Monitoring measures the running system again and again. Watch three groups: service health (at once), list quality by segment (after a week), business numbers (from the shop’s logs).
  • A stale model fails quietly. Larkfield’s ALS, never retrained, fell from 0.131 to 0.093 in seven weeks. Its lists held 0.1% new products or less; the retrained model reached 10.2%.
  • Compare models on the same customers and week. The gap (0.050 to 0.069 from week 4) is steadier than one model’s weekly value.
  • A release check compares the retrained and the current model by segment, with coverage, short lists and speed. Keep the old files to roll back.
  • One segment in one week is noisy: do not act on a single week. The fallback needs a retrain schedule too.

The recommender is now served, monitored and retrained. But every list is built from what customers did, and that is personal data. Next, you handle a customer who asks to be forgotten, a customer who opts out, and a family who shares one inbox.

Next: Privacy and sensitive suggestions