Batch or real time
- Lesson 27
- 40 min
- Module 9: Run it in production
Where we are
In Module 8, we chose a list with a simulated online test, and saw how a home page that learns only from itself gets stuck. Now the list must reach real pages, fast, every day. In this lesson, we compare suggestions computed each night with suggestions computed at each request, and we run a small service that falls back to popularity.
Lesson 27 of 29 · Recommendation and Personalisation: Show Each Person What MattersBatch or real time
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Decide when Larkfield computes its suggestions: once a night for every customer, or at the moment a page asks. Then run a small recommendation service that always answers, even when the model is missing.
NotebookDo this lesson in a notebook
The notebook computes a simple people-also-bought list for every customer at once, as a nightly job does. Then it serves the lists as a lookup, with a fallback for a customer it has never seen. Last, it times scoring one customer at request time. Its setup cells download about 8 MB. About 15 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The trending list and why it is hard to beat, from The baselines to beat.
- The two-stage system and its business rules, from Why two stages and Business rules and re-ranking.
- Amira’s speed guardrail of 100 milliseconds, from Online tests and guardrails.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject at the end of Module 8, with its virtual environment active.recs checkprintsSetup check passed. - The practice files in
../nextia-learning-labs, beside the project folder. You copy five files and a newrequirements.txtfrom its snapshotend-of-m09. curl, or a web browser, to send requests to the service.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.
A bakery that bakes at night
Section titled “A bakery that bakes at night”Think of a bakery. The bakers come in at four in the morning and bake the day’s bread. When the shop opens, a customer asks for a loaf, and gets it in two seconds. The work was done before anyone asked.
Some customers want something special: a birthday cake with a name on it. The bakery cannot bake that in advance, because it does not know the name. So it bakes the cake when the order comes. The customer waits, but the cake is exactly right.
And when the oven breaks? The bakery does not close. It sells yesterday’s bread and the cakes in the freezer. It is not the best the shop can do, but nobody leaves with nothing.
A recommender has the same three choices. It can compute lists before anyone asks. It can compute a list when the page asks. And it needs something to show when its model is not there.
The problem: a page that waits, or a page that is empty
Section titled “The problem: a page that waits, or a page that is empty”Until now, every method in this course ran in a notebook or in recs evaluate. The command trained a model, scored 2,118 test customers, and printed a number. Nobody was waiting.
A real home page is different. A customer opens it, and the page asks the recommender for 10 products. Two things can go wrong.
- The page waits. Training the two-stage system takes 12 seconds. If the page trained the model at each visit, the customer would leave. Even scoring one customer with a ready model takes time, and a busy page answers thousands of customers.
- The page is empty. The model file is missing after a failed night job. Or the customer is new, and no model knows them. Without a plan, the page shows an error or a blank box where the suggestions should be.
The idea: batch, real time, and a fallback
Section titled “The idea: batch, real time, and a fallback”Serving means answering requests from real pages. A request is one question from a page, for example “10 products for customer C100123”. The latency is the time from the request to the answer. It is measured in milliseconds (ms): one thousandth of a second.
There are two ways to serve suggestions:
- Batch serving computes the lists for every customer at a fixed time, usually once a night. It saves them in a file. A request only looks up the saved list. It is the bread baked at four in the morning. The name comes from a batch of bread: many loaves made in one go.
- Real-time serving keeps the model in memory and computes a list when the request comes. It can use what the customer did one minute ago. It is the birthday cake.
And both need a fallback: a simple list that the service shows when it cannot show a personal one. It is yesterday’s bread when the oven breaks.
Text description of the diagram
The picture shows two paths. On the left, batch serving, once a night. The job trains on every event so far and scores all 40,000 customers at once. The two-stage system takes 111 seconds. Then the job saves each customer’s top 50 products (17.6 MB). Later, a request only looks up the saved list, in 0.72 ms. On the right, real-time serving, at each request. The model waits in memory. A request reads the customer’s events up to this second, scores this one customer (10.3 ms for the two-stage system), and answers.
Two ways to describe speed: p50 and p95
Section titled “Two ways to describe speed: p50 and p95”Requests do not all take the same time. So we describe speed with two numbers. Imagine 100 requests, sorted from the fastest to the slowest.
- p50, the median, is the time of the 50th request. Half of the requests are faster.
- p95 is the time of the 95th request. Only 5 requests in 100 are slower. It tells you how bad a slow request is.
A page feels slow when its slow requests are slow, so a speed budget usually limits p95, not only p50.
A tiny example with every value visible
Section titled “A tiny example with every value visible”Here is a constructed example with four garden products. The model knows how close the products are, as in Item-item collaborative filtering:
| Close to… | Gloves | Compost | Seed tray |
|---|---|---|---|
| Trowel | 0.6 | 0.3 | 0.2 |
| Seed tray | 0.1 | 0.8 | (the same product) |
Ana has viewed the trowel. At night, the batch job adds the closeness to each product she viewed. Gloves get 0.6, compost 0.3 and the seed tray 0.2. Her saved list is gloves, compost, seed tray.
At eleven in the morning, Ana views the seed tray. Now she probably wants to sow seeds.
- Batch: the page looks up the saved list. She still sees gloves first. The list learns about the seed tray only tomorrow night.
- Real time: the service adds both rows now. Gloves get 0.6 + 0.1 = 0.7, compost 0.3 + 0.8 = 1.1. The seed tray is already viewed, so it is not shown. She sees compost first.
Real time is fresher by half a day. It also did more work during the request: it read her latest events and added two rows. With 3,000 products and a ranker, that work is what the 10.3 ms measures.
Larkfield’s numbers
Section titled “Larkfield’s numbers”The course measured both ways on its laptop (Apple M4, one thread) with the project’s commands. These numbers are measured:
| What | Time |
|---|---|
| Nightly batch, item-cf, 40,000 customers | 1.5 s |
| Nightly batch, two-stage, 40,000 customers | 111 s (about 2 minutes) |
| Size of the saved lists (top 50 per customer, and product neighbours) | 17.6 MB |
| Service: look up a saved list | p50 0.72 ms, p95 0.79 ms |
| Service: fallback only (no model loaded) | p50 0.70 ms, p95 0.78 ms |
| Real time: ALS, model in memory | p50 0.06 ms |
| Real time: two-stage, model in memory | p50 10.3 ms, p95 11.1 ms |
Text description of the chart
A bar chart of the median time per request, in milliseconds, on the course laptop. Two-stage scored at request time: 10.3 ms. Looking up a saved list in the service: 0.72 ms. The service with no model, answering with the fallback: 0.70 ms. ALS scored at request time: 0.06 ms. Amira’s budget is 100 ms per request, far above every bar.
Three things stand out.
First, every way is fast enough for Amira’s budget of 100 ms. Even the two-stage system at request time uses about a tenth of it.
Second, the lookup (0.72 ms) is almost all the service’s own work: reading the request, applying the rules, writing the answer. The fallback alone takes 0.70 ms. Finding a saved list adds almost nothing.
Third, ALS at request time is the fastest of all: 0.06 ms. Its score is one multiplication of a customer vector by the product vectors, as in Hidden tastes. The two-stage system is slower because it collects about 200 candidates from five sources, computes their features, and runs the ranker.
What each way costs
Section titled “What each way costs”Speed is not the only cost. Under the course’s assumption that a server with 4 CPUs costs US$0.17 an hour:
- Batch: 111 seconds a night is 111 ÷ 3,600 × 0.17 ≈ half a cent a night of computing. The server that answers requests only reads a 17.6 MB file, so it can be small. The cost is freshness: a list can be up to a day old. Ana’s seed tray reaches her list tomorrow.
- Real time: at 10.3 ms per request, one CPU can score about 1,000 ÷ 10.3 ≈ 97 requests a second. This is rough arithmetic: a real server also does other work. The service must keep the model, the features and the customer’s latest events in memory. All of them must be ready at every request. The gain is a list that knows what the customer did a minute ago.
Prices change: check your provider’s current page before you plan with them.
The fallback: what to show when you cannot be personal
Section titled “The fallback: what to show when you cannot be personal”A fallback answers three cases. The batch file is missing or broken. The customer has no saved list, because they are new. Or the customer must not get a personal list. You meet the last case in Privacy and sensitive suggestions.
Larkfield’s fallback is the trending list: the most ordered products of the last 28 days. It is not a poor answer. In The baselines to beat, trending scored recall@10 0.156 on the test period. That beat every single personal method. The service also applies the business rules to the fallback, so it never shows an out-of-stock or age-restricted product.
Text description of the diagram
The picture shows how the service chooses its answer. A request for customer C100123 reaches a first question: did the batch files load? If not, because they are missing or broken, the service uses the fallback, the trending list. If they loaded, it asks a second question: does this customer have a saved list? A new customer has none, so the service uses the fallback again. Otherwise it takes the customer’s saved top 50. Both paths go through the same step: the rules remove blocked products, and a short list is filled from the fallback. The answer always has 10 products, with its source and the reason.
Guided practice: serve the suggestions
Section titled “Guided practice: serve the suggestions”Your project at the end of Module 8 has no service yet. You copy five files and a new requirements.txt from the end-of-module-9 snapshot:
serving.pycomputes and loads the saved lists, and chooses the answer as in the picture.service.pyis the web service. It uses FastAPI, a Python library for small web services, and uvicorn, the program that runs it.commands/batch.py,commands/serve.pyandcommands/bench.pyaddrecs batch,recs serveandrecs bench.
-
In your
larkfield-recsfolder, with the virtual environment active, copy the files:Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m09cp $S/larkfield_recs/serving.py $S/larkfield_recs/service.py larkfield_recs/cp $S/larkfield_recs/commands/batch.py $S/larkfield_recs/commands/serve.py $S/larkfield_recs/commands/bench.py larkfield_recs/commands/cp $S/requirements.txt .Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m09"Copy-Item "$S\larkfield_recs\serving.py", "$S\larkfield_recs\service.py" larkfield_recs\Copy-Item "$S\larkfield_recs\commands\batch.py", "$S\larkfield_recs\commands\serve.py", "$S\larkfield_recs\commands\bench.py" larkfield_recs\commands\Copy-Item "$S\requirements.txt" . -
Install the three new packages (FastAPI, uvicorn and httpx). The other lines of the file are already installed:
Terminal window pip install -r requirements.txt -
Run the nightly job with the fast item-cf model:
Terminal window recs batch item-cfBatch: item-cf, trained on the events before 2026-07-06; lists for 40,000 customers; 2 sWritten to outputs/batch/ (lists.npz, fallback.json, meta.json)The job trained on the whole log, up to its last day.
lists.npzholds every customer’s top 50 and every product’s 20 neighbours.fallback.jsonholds the trending list. -
Start the service. A port is the number that a program on your computer listens on. Here it is 8037:
Terminal window recs serve --port 8037Larkfield recommendations on http://127.0.0.1:8037/ (try /health, /recommend/C100123, /similar/P10020). Stop with Ctrl+C.The command keeps running. Leave this terminal open.
-
Open a second terminal. Ask the service whether it is healthy:
Terminal window curl -s http://127.0.0.1:8037/healthTerminal window curl.exe -s http://127.0.0.1:8037/health{"status":"ok","model":"item-cf","customers_with_lists":40000,"fallback_only":false}You can also open the same address in your browser.
-
Ask for three suggestions for a known customer and for a customer the shop has never seen. On Windows, write
curl.exe:Terminal window curl -s "http://127.0.0.1:8037/recommend/C100123?k=3"curl -s "http://127.0.0.1:8037/recommend/C999999?k=3"{"customer_id":"C100123","surface":"home","source":"model","reason":null,"products":[{"product_id":"P10065","title":"Silver Lamb's Ear Perennial","price":20.99},{"product_id":"P12645","title":"Garden Fern in Natural Pot","price":15.99},{"product_id":"P11784","title":"Blueberry Shrub in Oak Pot","price":14.99}],"ms":0.116}{"customer_id":"C999999","surface":"home","source":"fallback","reason":"no history for this customer","products":[{"product_id":"P11118","title":"Lumi Green Nordic Coffee Table","price":174.99},{"product_id":"P10614","title":"Bramble Terracotta Mosaic Side Table","price":241.99},{"product_id":"P10219","title":"Navy Nitrile-Grip Garden Gloves","price":7.99}],"ms":0.182}C100123 gets plants from the model. C999999 gets the fallback, and the answer says why. The
msvalue is the service’s own time; yours will differ. -
Ask for the product page list of the Driftwood armchair:
Terminal window curl -s "http://127.0.0.1:8037/similar/P10020?k=3"The first product is P12928, the Driftwood Terracotta Rattan Armchair. You met it in People also bought.
-
Go back to the first terminal and press Ctrl+C. The service stops.
-
Now run the nightly job with the two-stage system. It takes about two minutes:
Terminal window recs batch two-stageBatch: two-stage, trained on the events before 2026-07-06; lists for 40,000 customers; 113 sWritten to outputs/batch/ (lists.npz, fallback.json, meta.json)It replaces the item-cf files. A service that you start now serves the two-stage lists.
-
Time the service and real-time scoring. The command trains ALS and two-stage once, so it takes about half a minute:
Terminal window recs bench --requests 200The course team’s run printed:
service, batch lookup (two-stage), 200 requests: p50 0.82 ms, p95 1.29 msals at request time (model in memory, 100 customers): p50 0.06 ms, p95 0.06 ms; training took 3.4 stwo-stage at request time (model in memory, 100 customers): p50 9.94 ms, p95 13.37 ms; training took 12.0 sYour numbers will differ: they depend on your computer and on what else it is doing. The order should be the same. ALS is the fastest, the lookup comes next, and the two-stage system is the slowest. Before these lines, you may see a
StarletteDeprecationWarningabouthttpx2. It is harmless. -
Run the tests with
python -m pytest -q. You see28 passed. The tests for this module’s files come in the last lesson of the module.
If recs batch stops with No module named 'fastapi', or recs serve with address already in use, read the common mistakes below.
Your turn: break the model on purpose
Section titled “Your turn: break the model on purpose”A fallback that nobody has tested is a hope, not a plan. Make the model unavailable, and check that the service still answers usefully.
- Move
outputs/batch/lists.npzout of thebatchfolder, for example to the project folder. - Start the service again, and ask for
/healthand for/recommend/C100123?k=3. - Predict first: what do
fallback_only,sourceandreasonsay? Then compare. - Move the file back. Restart the service and check that C100123 gets the model’s list again.
Show the answer
The course team’s run, with lists.npz moved away:
{"status":"ok","model":null,"customers_with_lists":0,"fallback_only":true}{"customer_id":"C100123","surface":"home","source":"fallback","reason":"no model loaded","products":[{"product_id":"P11118","title":"Lumi Green Nordic Coffee Table","price":174.99},{"product_id":"P10614","title":"Bramble Terracotta Mosaic Side Table","price":241.99},{"product_id":"P10219","title":"Navy Nitrile-Grip Garden Gloves","price":7.99}],"ms":2.37}The service starts, says that it runs on the fallback only, and gives C100123 the trending list with the reason no model loaded. /similar/P10020 also answers with the fallback. The page shows 10 good products instead of an error.
A broken file is the same case. The project treats a lists.npz that is cut short, for example by a full disk, as a missing file. A test in the last lesson of this module checks it.
When to use which, and what it costs
Section titled “When to use which, and what it costs”| Batch | Real time | |
|---|---|---|
| Freshness | Up to a day old | Up to the last click |
| Time per request | A lookup: under 1 ms here | The model’s work: 0.06 ms (ALS) to 10.3 ms (two-stage) here |
| Computing | Every customer, every night, even if they never visit | Only customers who visit |
| What must be ready at a request | One file | The model, its features and the latest events |
| If the model fails | Yesterday’s lists, or the fallback | The fallback |
Use batch when a day-old list is good enough. Weekly emails and most home pages fit: Larkfield’s customers rarely change their tastes overnight. Use real time when the latest action matters a lot. A product page and the next item in a visit are examples. In Sessions, sequences and LLMs, briefly, the current product predicted the next view far better than trending did.
Many shops mix them. The nightly batch computes the personal lists. A small real-time step changes them at the request. For example, it removes a product that went out of stock an hour ago. Or it adds neighbours of the product on the screen.
Every way needs a fallback, and a fallback needs a test. The MLOps course goes further on running models as services, for example in Provider and dependency outages.
Common mistakes
Section titled “Common mistakes”Common mistake
ModuleNotFoundError: No module named 'fastapi'
Symptom: after copying the files, every recs command, even recs batch, ends with:
File ".../larkfield_recs/service.py", line 9, in <module> from fastapi import FastAPI, HTTPExceptionModuleNotFoundError: No module named 'fastapi'Cause: the project loads every file in larkfield_recs/commands/ at the start. bench.py imports the service, and the service needs FastAPI. You get this error after copying the files and before step 2.
Fix: with the virtual environment active, run pip install -r requirements.txt with the new file from step 1.
Common mistake
address already in use
Symptom: recs serve --port 8037 prints its first line, then:
ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8037): [errno 48] address already in useCause: another program already listens on port 8037. Usually it is your own service, still running in another terminal. You get this message on macOS with a first service still running. The text on Windows is different (not tested).
Fix: stop the first service with Ctrl+C in its terminal. Or start this one on another port, for example recs serve --port 8038, and use that port in your requests.
Knowledge check
Section titled “Knowledge check”Check your understanding
Larkfield's weekly email goes out on Monday morning. Should its lists come from a batch or from real-time scoring? Give one reason and one cost.
Show the answer
A batch. The email is built for all customers at a known time, and nobody waits for it, so the lists can be computed the night before. The cost is freshness: what a customer did on Monday morning is not in the list. For a weekly email, that matters little. Real-time scoring would add work at the moment of sending, and gain only a few hours of freshness.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The service answers a lookup in 0.72 ms and the fallback alone in 0.70 ms. What does this tell you about where the service spends its time?
Show the answer
Almost all the time goes to the service’s own work. It reads the request, applies the rules, describes the products and writes the answer. Finding the customer’s saved list adds only about 0.02 ms. So a faster model would not make a lookup service faster. Real-time scoring is different: there, the model’s work is most of the time (10.3 ms for the two-stage system).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In your project, move outputs/batch/lists.npz away and start the service. What do /health and /recommend/C100123 return, and why is that a good result?
Show the answer
/health returns "fallback_only":true and "customers_with_lists":0. /recommend/C100123 returns "source":"fallback" with the reason no model loaded, and the trending list. It is a good result. The page still shows 10 products that sell well. And the answer says why, so Amira’s monitoring can see that the model is missing.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Batch serving computes every customer’s list once a night; a request is a lookup (0.72 ms here). Real-time serving scores at the request (two-stage 10.3 ms, ALS 0.06 ms) and uses the latest events.
- p50 is the median time; p95 is the time that only 5 requests in 100 exceed.
- Larkfield’s two-stage batch takes 111 seconds a night, about half a cent under the course’s server price. Its cost is freshness: up to a day old.
- A fallback (here the trending list, with the rules) answers when the files are missing or broken, or when the customer has no list. Test it by breaking the model on purpose.
- Use batch when a day-old list is good enough (emails, most home pages). Use real time when the latest action matters (product pages, the next item in a visit).
The service now answers every request. But the shop does not stand still: new products arrive, seasons change, and a model trained in May slowly goes wrong. Next, you watch the service week by week, and decide when a new model may replace the old one.
Next: Monitor and retrain