Skip to content
Français

Batch or real time

  • Lesson 27
  • 40 min
  • Module 9: Run it in production

Where we are

In Module 8, we chose a list with a simulated online test, and saw how a home page that learns only from itself gets stuck. Now the list must reach real pages, fast, every day. In this lesson, we compare suggestions computed each night with suggestions computed at each request, and we run a small service that falls back to popularity.

What you will be able to do

Decide when Larkfield computes its suggestions: once a night for every customer, or at the moment a page asks. Then run a small recommendation service that always answers, even when the model is missing.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes a simple people-also-bought list for every customer at once, as a nightly job does. Then it serves the lists as a lookup, with a fallback for a customer it has never seen. Last, it times scoring one customer at request time. Its setup cells download about 8 MB. About 15 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project at the end of Module 8, with its virtual environment active. recs check prints Setup check passed.
  • The practice files in ../nextia-learning-labs, beside the project folder. You copy five files and a new requirements.txt from its snapshot end-of-m09.
  • curl, or a web browser, to send requests to the service.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

Think of a bakery. The bakers come in at four in the morning and bake the day’s bread. When the shop opens, a customer asks for a loaf, and gets it in two seconds. The work was done before anyone asked.

Some customers want something special: a birthday cake with a name on it. The bakery cannot bake that in advance, because it does not know the name. So it bakes the cake when the order comes. The customer waits, but the cake is exactly right.

And when the oven breaks? The bakery does not close. It sells yesterday’s bread and the cakes in the freezer. It is not the best the shop can do, but nobody leaves with nothing.

A recommender has the same three choices. It can compute lists before anyone asks. It can compute a list when the page asks. And it needs something to show when its model is not there.

The problem: a page that waits, or a page that is empty

Section titled “The problem: a page that waits, or a page that is empty”

Until now, every method in this course ran in a notebook or in recs evaluate. The command trained a model, scored 2,118 test customers, and printed a number. Nobody was waiting.

A real home page is different. A customer opens it, and the page asks the recommender for 10 products. Two things can go wrong.

  • The page waits. Training the two-stage system takes 12 seconds. If the page trained the model at each visit, the customer would leave. Even scoring one customer with a ready model takes time, and a busy page answers thousands of customers.
  • The page is empty. The model file is missing after a failed night job. Or the customer is new, and no model knows them. Without a plan, the page shows an error or a blank box where the suggestions should be.

The idea: batch, real time, and a fallback

Section titled “The idea: batch, real time, and a fallback”

Serving means answering requests from real pages. A request is one question from a page, for example “10 products for customer C100123”. The latency is the time from the request to the answer. It is measured in milliseconds (ms): one thousandth of a second.

There are two ways to serve suggestions:

  • Batch serving computes the lists for every customer at a fixed time, usually once a night. It saves them in a file. A request only looks up the saved list. It is the bread baked at four in the morning. The name comes from a batch of bread: many loaves made in one go.
  • Real-time serving keeps the model in memory and computes a list when the request comes. It can use what the customer did one minute ago. It is the birthday cake.

And both need a fallback: a simple list that the service shows when it cannot show a personal one. It is yesterday’s bread when the oven breaks.

Two paths side by side. Batch, once a night: train on every event so far, score all 40,000 customers at once (two-stage: 111 seconds), save each customer's top 50 (17.6 MB), then a request looks up the saved list in 0.72 ms. Real time, at each request: the model waits in memory, a request reads the customer's events up to this second, scores this customer now (two-stage: 10.3 ms), and answers.
Batch and real-time serving of Larkfield's two-stage system, with the times measured on the course laptop.
Text description of the diagram

The picture shows two paths. On the left, batch serving, once a night. The job trains on every event so far and scores all 40,000 customers at once. The two-stage system takes 111 seconds. Then the job saves each customer’s top 50 products (17.6 MB). Later, a request only looks up the saved list, in 0.72 ms. On the right, real-time serving, at each request. The model waits in memory. A request reads the customer’s events up to this second, scores this one customer (10.3 ms for the two-stage system), and answers.

Requests do not all take the same time. So we describe speed with two numbers. Imagine 100 requests, sorted from the fastest to the slowest.

  • p50, the median, is the time of the 50th request. Half of the requests are faster.
  • p95 is the time of the 95th request. Only 5 requests in 100 are slower. It tells you how bad a slow request is.

A page feels slow when its slow requests are slow, so a speed budget usually limits p95, not only p50.

Here is a constructed example with four garden products. The model knows how close the products are, as in Item-item collaborative filtering:

Close to… Gloves Compost Seed tray
Trowel 0.6 0.3 0.2
Seed tray 0.1 0.8 (the same product)

Ana has viewed the trowel. At night, the batch job adds the closeness to each product she viewed. Gloves get 0.6, compost 0.3 and the seed tray 0.2. Her saved list is gloves, compost, seed tray.

At eleven in the morning, Ana views the seed tray. Now she probably wants to sow seeds.

  • Batch: the page looks up the saved list. She still sees gloves first. The list learns about the seed tray only tomorrow night.
  • Real time: the service adds both rows now. Gloves get 0.6 + 0.1 = 0.7, compost 0.3 + 0.8 = 1.1. The seed tray is already viewed, so it is not shown. She sees compost first.

Real time is fresher by half a day. It also did more work during the request: it read her latest events and added two rows. With 3,000 products and a ranker, that work is what the 10.3 ms measures.

The course measured both ways on its laptop (Apple M4, one thread) with the project’s commands. These numbers are measured:

What Time
Nightly batch, item-cf, 40,000 customers 1.5 s
Nightly batch, two-stage, 40,000 customers 111 s (about 2 minutes)
Size of the saved lists (top 50 per customer, and product neighbours) 17.6 MB
Service: look up a saved list p50 0.72 ms, p95 0.79 ms
Service: fallback only (no model loaded) p50 0.70 ms, p95 0.78 ms
Real time: ALS, model in memory p50 0.06 ms
Real time: two-stage, model in memory p50 10.3 ms, p95 11.1 ms
Median time per request on the course laptop ('now': scored at request time). All four are far below Amira's budget of 100 ms, which would be eight times higher than the top of this chart.
Text description of the chart

A bar chart of the median time per request, in milliseconds, on the course laptop. Two-stage scored at request time: 10.3 ms. Looking up a saved list in the service: 0.72 ms. The service with no model, answering with the fallback: 0.70 ms. ALS scored at request time: 0.06 ms. Amira’s budget is 100 ms per request, far above every bar.

Three things stand out.

First, every way is fast enough for Amira’s budget of 100 ms. Even the two-stage system at request time uses about a tenth of it.

Second, the lookup (0.72 ms) is almost all the service’s own work: reading the request, applying the rules, writing the answer. The fallback alone takes 0.70 ms. Finding a saved list adds almost nothing.

Third, ALS at request time is the fastest of all: 0.06 ms. Its score is one multiplication of a customer vector by the product vectors, as in Hidden tastes. The two-stage system is slower because it collects about 200 candidates from five sources, computes their features, and runs the ranker.

Speed is not the only cost. Under the course’s assumption that a server with 4 CPUs costs US$0.17 an hour:

  • Batch: 111 seconds a night is 111 ÷ 3,600 × 0.17 ≈ half a cent a night of computing. The server that answers requests only reads a 17.6 MB file, so it can be small. The cost is freshness: a list can be up to a day old. Ana’s seed tray reaches her list tomorrow.
  • Real time: at 10.3 ms per request, one CPU can score about 1,000 ÷ 10.3 ≈ 97 requests a second. This is rough arithmetic: a real server also does other work. The service must keep the model, the features and the customer’s latest events in memory. All of them must be ready at every request. The gain is a list that knows what the customer did a minute ago.

Prices change: check your provider’s current page before you plan with them.

The fallback: what to show when you cannot be personal

Section titled “The fallback: what to show when you cannot be personal”

A fallback answers three cases. The batch file is missing or broken. The customer has no saved list, because they are new. Or the customer must not get a personal list. You meet the last case in Privacy and sensitive suggestions.

Larkfield’s fallback is the trending list: the most ordered products of the last 28 days. It is not a poor answer. In The baselines to beat, trending scored recall@10 0.156 on the test period. That beat every single personal method. The service also applies the business rules to the fallback, so it never shows an out-of-stock or age-restricted product.

A request for customer C100123 reaches a first question: did the batch files load? If no, because they are missing or broken, the answer is the fallback, the trending list. If yes, a second question: does this customer have a saved list? If no, a new customer, the fallback again. If yes, the customer's saved top 50. Both paths go through the same step: remove blocked products with the rules, and fill a short list from the fallback. The answer has 10 products, with a source and a reason.
How Larkfield's service chooses its answer. Every path ends with 10 products.
Text description of the diagram

The picture shows how the service chooses its answer. A request for customer C100123 reaches a first question: did the batch files load? If not, because they are missing or broken, the service uses the fallback, the trending list. If they loaded, it asks a second question: does this customer have a saved list? A new customer has none, so the service uses the fallback again. Otherwise it takes the customer’s saved top 50. Both paths go through the same step: the rules remove blocked products, and a short list is filled from the fallback. The answer always has 10 products, with its source and the reason.

Your project at the end of Module 8 has no service yet. You copy five files and a new requirements.txt from the end-of-module-9 snapshot:

  • serving.py computes and loads the saved lists, and chooses the answer as in the picture.
  • service.py is the web service. It uses FastAPI, a Python library for small web services, and uvicorn, the program that runs it.
  • commands/batch.py, commands/serve.py and commands/bench.py add recs batch, recs serve and recs bench.
  1. In your larkfield-recs folder, with the virtual environment active, copy the files:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m09
    cp $S/larkfield_recs/serving.py $S/larkfield_recs/service.py larkfield_recs/
    cp $S/larkfield_recs/commands/batch.py $S/larkfield_recs/commands/serve.py $S/larkfield_recs/commands/bench.py larkfield_recs/commands/
    cp $S/requirements.txt .
  2. Install the three new packages (FastAPI, uvicorn and httpx). The other lines of the file are already installed:

    Terminal window
    pip install -r requirements.txt
  3. Run the nightly job with the fast item-cf model:

    Terminal window
    recs batch item-cf
    Batch: item-cf, trained on the events before 2026-07-06; lists for 40,000 customers; 2 s
    Written to outputs/batch/ (lists.npz, fallback.json, meta.json)

    The job trained on the whole log, up to its last day. lists.npz holds every customer’s top 50 and every product’s 20 neighbours. fallback.json holds the trending list.

  4. Start the service. A port is the number that a program on your computer listens on. Here it is 8037:

    Terminal window
    recs serve --port 8037
    Larkfield recommendations on http://127.0.0.1:8037/ (try /health, /recommend/C100123, /similar/P10020). Stop with Ctrl+C.

    The command keeps running. Leave this terminal open.

  5. Open a second terminal. Ask the service whether it is healthy:

    Terminal window
    curl -s http://127.0.0.1:8037/health
    {"status":"ok","model":"item-cf","customers_with_lists":40000,"fallback_only":false}

    You can also open the same address in your browser.

  6. Ask for three suggestions for a known customer and for a customer the shop has never seen. On Windows, write curl.exe:

    Terminal window
    curl -s "http://127.0.0.1:8037/recommend/C100123?k=3"
    curl -s "http://127.0.0.1:8037/recommend/C999999?k=3"
    {"customer_id":"C100123","surface":"home","source":"model","reason":null,"products":[{"product_id":"P10065","title":"Silver Lamb's Ear Perennial","price":20.99},{"product_id":"P12645","title":"Garden Fern in Natural Pot","price":15.99},{"product_id":"P11784","title":"Blueberry Shrub in Oak Pot","price":14.99}],"ms":0.116}
    {"customer_id":"C999999","surface":"home","source":"fallback","reason":"no history for this customer","products":[{"product_id":"P11118","title":"Lumi Green Nordic Coffee Table","price":174.99},{"product_id":"P10614","title":"Bramble Terracotta Mosaic Side Table","price":241.99},{"product_id":"P10219","title":"Navy Nitrile-Grip Garden Gloves","price":7.99}],"ms":0.182}

    C100123 gets plants from the model. C999999 gets the fallback, and the answer says why. The ms value is the service’s own time; yours will differ.

  7. Ask for the product page list of the Driftwood armchair:

    Terminal window
    curl -s "http://127.0.0.1:8037/similar/P10020?k=3"

    The first product is P12928, the Driftwood Terracotta Rattan Armchair. You met it in People also bought.

  8. Go back to the first terminal and press Ctrl+C. The service stops.

  9. Now run the nightly job with the two-stage system. It takes about two minutes:

    Terminal window
    recs batch two-stage
    Batch: two-stage, trained on the events before 2026-07-06; lists for 40,000 customers; 113 s
    Written to outputs/batch/ (lists.npz, fallback.json, meta.json)

    It replaces the item-cf files. A service that you start now serves the two-stage lists.

  10. Time the service and real-time scoring. The command trains ALS and two-stage once, so it takes about half a minute:

    Terminal window
    recs bench --requests 200

    The course team’s run printed:

    service, batch lookup (two-stage), 200 requests: p50 0.82 ms, p95 1.29 ms
    als at request time (model in memory, 100 customers): p50 0.06 ms, p95 0.06 ms; training took 3.4 s
    two-stage at request time (model in memory, 100 customers): p50 9.94 ms, p95 13.37 ms; training took 12.0 s

    Your numbers will differ: they depend on your computer and on what else it is doing. The order should be the same. ALS is the fastest, the lookup comes next, and the two-stage system is the slowest. Before these lines, you may see a StarletteDeprecationWarning about httpx2. It is harmless.

  11. Run the tests with python -m pytest -q. You see 28 passed. The tests for this module’s files come in the last lesson of the module.

If recs batch stops with No module named 'fastapi', or recs serve with address already in use, read the common mistakes below.

A fallback that nobody has tested is a hope, not a plan. Make the model unavailable, and check that the service still answers usefully.

  1. Move outputs/batch/lists.npz out of the batch folder, for example to the project folder.
  2. Start the service again, and ask for /health and for /recommend/C100123?k=3.
  3. Predict first: what do fallback_only, source and reason say? Then compare.
  4. Move the file back. Restart the service and check that C100123 gets the model’s list again.
Show the answer

The course team’s run, with lists.npz moved away:

{"status":"ok","model":null,"customers_with_lists":0,"fallback_only":true}
{"customer_id":"C100123","surface":"home","source":"fallback","reason":"no model loaded","products":[{"product_id":"P11118","title":"Lumi Green Nordic Coffee Table","price":174.99},{"product_id":"P10614","title":"Bramble Terracotta Mosaic Side Table","price":241.99},{"product_id":"P10219","title":"Navy Nitrile-Grip Garden Gloves","price":7.99}],"ms":2.37}

The service starts, says that it runs on the fallback only, and gives C100123 the trending list with the reason no model loaded. /similar/P10020 also answers with the fallback. The page shows 10 good products instead of an error.

A broken file is the same case. The project treats a lists.npz that is cut short, for example by a full disk, as a missing file. A test in the last lesson of this module checks it.

Batch Real time
Freshness Up to a day old Up to the last click
Time per request A lookup: under 1 ms here The model’s work: 0.06 ms (ALS) to 10.3 ms (two-stage) here
Computing Every customer, every night, even if they never visit Only customers who visit
What must be ready at a request One file The model, its features and the latest events
If the model fails Yesterday’s lists, or the fallback The fallback

Use batch when a day-old list is good enough. Weekly emails and most home pages fit: Larkfield’s customers rarely change their tastes overnight. Use real time when the latest action matters a lot. A product page and the next item in a visit are examples. In Sessions, sequences and LLMs, briefly, the current product predicted the next view far better than trending did.

Many shops mix them. The nightly batch computes the personal lists. A small real-time step changes them at the request. For example, it removes a product that went out of stock an hour ago. Or it adds neighbours of the product on the screen.

Every way needs a fallback, and a fallback needs a test. The MLOps course goes further on running models as services, for example in Provider and dependency outages.

Common mistake

ModuleNotFoundError: No module named 'fastapi'

Symptom: after copying the files, every recs command, even recs batch, ends with:

File ".../larkfield_recs/service.py", line 9, in <module>
from fastapi import FastAPI, HTTPException
ModuleNotFoundError: No module named 'fastapi'

Cause: the project loads every file in larkfield_recs/commands/ at the start. bench.py imports the service, and the service needs FastAPI. You get this error after copying the files and before step 2.

Fix: with the virtual environment active, run pip install -r requirements.txt with the new file from step 1.

Common mistake

address already in use

Symptom: recs serve --port 8037 prints its first line, then:

ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8037): [errno 48] address already in use

Cause: another program already listens on port 8037. Usually it is your own service, still running in another terminal. You get this message on macOS with a first service still running. The text on Windows is different (not tested).

Fix: stop the first service with Ctrl+C in its terminal. Or start this one on another port, for example recs serve --port 8038, and use that port in your requests.

Check your understanding

Larkfield's weekly email goes out on Monday morning. Should its lists come from a batch or from real-time scoring? Give one reason and one cost.

Show the answer

A batch. The email is built for all customers at a known time, and nobody waits for it, so the lists can be computed the night before. The cost is freshness: what a customer did on Monday morning is not in the list. For a weekly email, that matters little. Real-time scoring would add work at the moment of sending, and gain only a few hours of freshness.

Check your understanding

The service answers a lookup in 0.72 ms and the fallback alone in 0.70 ms. What does this tell you about where the service spends its time?

Show the answer

Almost all the time goes to the service’s own work. It reads the request, applies the rules, describes the products and writes the answer. Finding the customer’s saved list adds only about 0.02 ms. So a faster model would not make a lookup service faster. Real-time scoring is different: there, the model’s work is most of the time (10.3 ms for the two-stage system).

Check your understanding

In your project, move outputs/batch/lists.npz away and start the service. What do /health and /recommend/C100123 return, and why is that a good result?

Show the answer

/health returns "fallback_only":true and "customers_with_lists":0. /recommend/C100123 returns "source":"fallback" with the reason no model loaded, and the trending list. It is a good result. The page still shows 10 products that sell well. And the answer says why, so Amira’s monitoring can see that the model is missing.

Recap

  • Batch serving computes every customer’s list once a night; a request is a lookup (0.72 ms here). Real-time serving scores at the request (two-stage 10.3 ms, ALS 0.06 ms) and uses the latest events.
  • p50 is the median time; p95 is the time that only 5 requests in 100 exceed.
  • Larkfield’s two-stage batch takes 111 seconds a night, about half a cent under the course’s server price. Its cost is freshness: up to a day old.
  • A fallback (here the trending list, with the rules) answers when the files are missing or broken, or when the customer has no list. Test it by breaking the model on purpose.
  • Use batch when a day-old list is good enough (emails, most home pages). Use real time when the latest action matters (product pages, the next item in a visit).

The service now answers every request. But the shop does not stand still: new products arrive, seasons change, and a model trained in May slowly goes wrong. Next, you watch the service week by week, and decide when a new model may replace the old one.

Next: Monitor and retrain