Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

Each module added one part of a recommender and measured it: the split, the harness, the methods, the ranker, the rules and the service. In this assignment, you use them all for Larkfield's new category and weekly email.

What you will be able to do

Use every module of the course on one new situation, and deliver a recommendation pack that Omar, Hana, Rafael and Amira can act on. The new category has no history, many customers are new, and nobody has received the email yet. So you measure each choice again, on this data, instead of copying it from the lessons.

Before you startWhat to know and to have readyAbout 180 minutes
  • You completed Modules 1 to 9. Each part below names the lessons that it uses.
  • You do not need the case studies of Module 10 for this assignment. The certificate asks for at least one of them: Recommend for an online retailer, Next item from browsing sessions or Books for readers. They are good practice for this assignment.
  • Your folder larkfield-recs works at the end of Module 9: python -m pytest -q prints 33 passed, 1 warning, as in Privacy and sensitive suggestions. The course data is in larkfield-recs/data, and the folder nextia-learning-labs is beside larkfield-recs, as in Meet the shop log.
  • About 3 to 4 hours. Most of it is reading results and writing decisions. The slowest step is the nightly batch with the two-stage ranker. On the course team’s busy laptop, it took about 4 minutes when the ranker served only loyal customers. It took about 21 minutes when the ranker served every customer.
  • Tested on macOS with Python 3.12.13 and the project’s pinned packages. The Windows commands are written, but they were not run.

The situation: Home office and a weekly email

Section titled “The situation: Home office and a weekly email”

Larkfield does two new things at the same time:

  • A new category. On 20 July 2026, Larkfield launches Home office: 150 products in 6 subcategories. On their first day, they have no views and no orders. This is the cold start of Cold start for new products and new customers, for a whole category.
  • A weekly email. Every week, each customer gets an email with 10 products. Nobody has received it yet, so there is no email log.

Many customers are new too. 5,000 people opened an account from 6 July to 16 August. Most of them have one visit or none.

The team has questions, as in the course:

  • Omar, the product manager, asks what the email is worth. His main measure is added revenue per email: money from suggested products that the customer would not have bought anyway. His limit from Module 6 still holds: a cold-start change may lower known customers’ recall@10 by at most 2% of its value.
  • Hana, the merchandising lead, wants the new category to be seen from the first week.
  • Rafael, the email and customer-relationship manager, watches unsubscribes. He reminds everybody that some customers share one email address with their family, and that some customers opted out of personal suggestions.
  • Amira, the developer, owns the service. She needs a fallback, a batch that runs in the night, and a tested deletion request.
  • Tomás, the AI engineer, has an answer ready: “Two-stage won the course’s table. Ship two-stage for everything.”

Priya asks you to answer all of them with evidence.

Be honest about the data. The shop log, the new products and the new customers are synthetic: the course’s code made them with a model of customer behaviour. The titles were written by a language model from our attributes. The data has practice problems we made on purpose. The course team knows them, and you do not. The email has never been sent, so any number about it is a plan or a simulation, never a measurement on people. Quote your numbers “on Larkfield’s synthetic log”, with their intervals.

The data is in the labs repository, in C37/final/. The course data, C37/data/, stays as it was.

File Rows What one row is
products_new.csv 150 A Home office product, with the same 17 columns as the course’s products.csv. All were added on 2026-07-20.
customers_new.csv 5,000 A new customer, with the same 5 columns as customers.csv. first_seen is empty when the customer has no event in the files yet.
events_final.csv.gz 37,140 An event (view, basket add or order) from 2026-07-20 to 2026-08-02, of old and new customers.
embeddings/products_new.npy 150 × 384 The new products’ vectors, made with the course’s model, in file order.
make_data.py Makes one data folder that the recs command can read.
export_lists.py Writes the lists that you hand in, from recs batch.

The folder also has embeddings/model.json, README.md, dataset-card.md and SHA256SUMS. It is about 0.6 MB.

Two periods are not in the files. The picture shows where they are:

A time line from left to right. The course log, 1 February 2025 to 5 July 2026, is in data; its last two weeks, 22 June to 5 July, are the default validation period. Then a grey dashed box: 6 to 19 July, missing, the course team keeps it. Then 20 to 26 July, the first week of Home office. Then 27 July to 2 August, the default test week, used once. Then a red box: 3 to 16 August, the graded weeks, not in the files.
What you have and what you do not have. You make your own time split from the days you have; the course team grades on the two red weeks.
Text description of the diagram

Five boxes from left to right. The course log, from 1 February 2025 to 5 July 2026, is in your data folder. Its last two weeks, 22 June to 5 July, are the default validation period. Next, a grey box with a dashed border: 6 to 19 July, missing, because the course team keeps it. Next, 20 to 26 July: the first week of Home office. Next, 27 July to 2 August: the default test week, which you use once. Last, a red box: 3 to 16 August, the graded weeks, which are not in the files.

  • The gap, 6 to 19 July. Your data has no events there. Just after the gap, “the last 28 days” of trending holds only two weeks of events. The ranker’s two training weeks hold one week of events, or none. Keep it in mind when you choose your periods.
  • The graded weeks, 3 to 16 August. The course team scores the lists that you hand in on the orders of these two weeks. You cannot tune on them, and you do not need to.

Check the files, make the data folder, and give it to the project. Run this in your larkfield-recs folder, with .venv active:

Terminal window
git -C ../nextia-learning-labs pull
F=../nextia-learning-labs/C37/final
(cd "$F" && shasum -a 256 -c SHA256SUMS | grep -vc ': OK$')
python "$F/make_data.py"
export LARKFIELD_DATA=final-data LARKFIELD_OUTPUTS=outputs-final
recs check
recs info

The number that shasum prints is the number of wrong files. It must be 0. Then you see:

Wrote final-data/: 3,144 products (150 new in Home office), 45,000 customers (5,000 new, 2,088 of them with events), 842,456 events to 2026-08-02
Periods: launch 2026-07-20 to 2026-08-02, validation 2026-06-22 to 2026-07-05, test 2026-07-27 to 2026-08-02
No events from 2026-07-06 to 2026-07-19; the files stop on 2026-08-02.
Data: 8 files, all right (final-data)
popular on the validation period (quick): recall@10 0.106 (expected 0.106)
Setup check passed.
Products: 3,144 in 21 categories and 107 subcategories (synthetic)
Customers: 45,000; 24,902 ordered at least once, 54% of them only once
Events: 842,456 (598,678 views, 156,615 basket adds, 87,163 order lines in 60,035 orders), 2025-02-01 to 2026-08-02
Periods: validation 2026-06-22 to 2026-07-05, test 2026-07-27 to 2026-08-02
Customer-product pairs with any event: 510,773 of 141,480,000 (0.36% filled)

What happened:

  • make_data.py wrote a new folder, final-data/. Your data/ folder did not change. The new folder has the course log, the new products, the new customers and the new events, with new checksums.
  • It set three periods in final-data/splits.json, with the names that the recs commands know. launch is the first two weeks of Home office, trained on the events before the launch: the same replay as the summer range in Module 6. validation is the course’s old test period, so you tune on two full weeks with no gap before them. test is the last week that you have: Home office has one week of history before it.
  • LARKFIELD_DATA gives the folder to every recs command. LARKFIELD_OUTPUTS keeps the new models and runs apart from the course’s. Close the terminal, or unset both, to go back to the course data.

Your own split. The default periods are a choice, not a rule. The validation period has no Home office product. You can tune on the first week of Home office instead:

Terminal window
python "$F/make_data.py" --validation 2026-07-20 2026-07-26 --test 2026-07-27 2026-08-02

Read The ranker has nothing to learn from first: just after the gap, some methods learn from almost nothing.

A period must end by 2 August, and it cannot touch the gap. recs check knows the expected value only for the default periods. With the weeks above, it checks the files and then prints:

popular on the validation period (quick): recall@10 0.088 (no expected value for this validation period)
Data check passed. The result check needs the default periods: use `recs info` to see yours.

Write your split and your reason in your report.

Make a folder recs-pack/ in your larkfield-recs folder. Put every file of the assignment in it. What to submit lists the files.

The pack follows the course’s completion rules: one part for each required item.

A box on top: recs-pack with REPORT.md. Below it, five boxes: 1 problem statements, 2 comparison, 3 chosen system, 4 cold start, 5 service and email test, each with its files.
The five parts of the recommendation pack and the files that they hand in.
Text description of the diagram

A box on top: the folder recs-pack, with REPORT.md and the commands that make every number. Below it, five boxes. 1 Problem statements: PROBLEM.md, for the product page and the email, with the business measure, the guardrails and the cost of a bad suggestion. 2 Comparison: comparison.csv, with popular, trending and at least four method families, by segment. 3 Chosen system: rules.csv with the effect of each rule, and OMAR.md, which says why this design and what it costs. 4 Cold start: COLD-START.md, with evidence that new products are shown and that new customers are served. 5 Service and email test: email.csv and similar.csv, service.md, and EMAIL-TEST.md, a plan decided in advance.

Part 1: a problem statement for each surface

Section titled “Part 1: a problem statement for each surface”

Uses A good shop assistant and What success means.

Write recs-pack/PROBLEM.md, one half page for each surface: the product page of a Home office product, and the weekly email. For each surface, write:

  1. Who sees it, and when. For example: the product page is seen by people who are already looking at one product. The email reaches every customer, even those who have not visited for months.
  2. The business measure. For the email, Omar’s added revenue per email. For the product page, choose one and say why.
  3. The offline stand-in. This is a measure that you can compute from the log, for example recall@10 on the orders of a later week. Say why it is only a stand-in.
  4. The guardrails. At least two per surface. For example: returns, unsubscribes, new products shown, products that a customer already bought, sensitive products in a shared inbox, and Amira’s 100 ms.
  5. The cost of a bad suggestion, in dollars, with the course’s assumptions in final-data/costs.json. A return costs 12 dollars to handle. A subscriber is worth 18 dollars a year. Work one example, as in The cost of two mistakes. Say “under these assumptions”.

Uses Split by time, The baselines to beat, Ranking metrics, Beyond accuracy, Evaluate by segment and the lessons of Modules 4 to 7. The case study Recommend for an online retailer makes the same kind of comparison on real transactions.

  1. Fix your split first, and do not change it after you see results. Write in REPORT.md which week tunes and which week reports, and why no event of a later day is in the training data.

  2. The baselines. Run popular and trending. Every other design is compared with them.

    Terminal window
    recs evaluate popular --period validation --by segment
    recs evaluate trending --period validation --by segment
  3. At least four method families. A family is one idea: neighbours (cooc, lift, item-cf, user-user), factorisation (als, bpr), content (content), hybrids (hybrid-blend, hybrid-switch, als+new) and two stages (two-stage). Two settings of one family count as one family.

    Terminal window
    recs evaluate item-cf --period validation --by segment
    recs evaluate als --period validation --by segment --set factors=32 --set regularization=0.1 --set alpha=1
    recs evaluate content --period validation --by segment
    recs evaluate two-stage --period validation --by segment

    The ranker learns on the 14 days just before the period. With the default periods, those days have orders. Before your test week, they hold one week of events and one week of the gap.

  4. Tune on validation only. If you tune a method, as in Tune without fooling yourself, choose its settings on the validation week. Then run each design that you report once on the test week, with no change.

  5. Report the course’s columns for each design in recs-pack/comparison.csv: recall@10 with its 95% interval, NDCG@10, MAP@10, coverage, novelty, diversity, the share of new products in the lists, recall@10 by segment (new, occasional, loyal) with the size of each segment, and the fit time. The output of recs evaluate has all of them, and the runs are saved in outputs-final/runs/.

  6. Compare on the same customers. Use recs compare for every difference that you claim, as in Compare on the same customers:

    Terminal window
    recs compare trending two-stage --period test --by segment
  7. Look at the products as well as the customers. Count how often Home office products appear in each design’s lists. The line new products in the lists of recs evaluate gives it. A design that learned only from history shows almost none in the first week.

Part 3: cold start for the new category and the new customers

Section titled “Part 3: cold start for the new category and the new customers”

Uses Content-based recommendations, Cold start for new products and new customers, Hybrids and People also bought.

  1. Replay the launch. The period launch trains on the events before 20 July, so Home office has no history at all. Measure how many new products each design shows:

    Terminal window
    recs evaluate als --period launch --by segment
    recs evaluate content --period launch --by segment
    recs evaluate als+new --period launch --by segment --set slots=1

    Report new products in the lists and recall@10 on them. Then check Omar’s limit: known customers’ recall@10 may fall by at most 2% of its value. Use recs compare on the launch period.

  2. New customers. Find how your design serves a customer with no order and a customer with no event at all. The segment new of recs evaluate is customers with no order before the period. A customer with no event gets the fallback of the service. Say what the fallback is, and whether it is good for them. Next item from browsing sessions shows how far a visit alone can go when a visitor has no history.

  3. The product page of a new product. A Home office product has two weeks of history when the graded weeks start. Look at its neighbours with each method:

    Terminal window
    recs similar P20001 --method item-cf
    recs similar P20001 --method lift
    recs similar P20001 --method content

    recs similar trains on the events before your test week. Ask which neighbours are alternatives (another product of the same kind) and which are complements (a product that people buy next). Module 4 showed that both appear, for different customers. Decide what the page should show, and measure what you can.

  4. Write recs-pack/COLD-START.md: the plan for the first week, the plan for later weeks, and the evidence. “New products are shown” needs numbers. Give the share of lists (or of list places) with a Home office product. Also give the number of Home office products shown at least once.

Part 4: the chosen system, its rules and its service

Section titled “Part 4: the chosen system, its rules and its service”

Uses Why two stages, A learned ranker, Business rules and re-ranking, Batch or real time, Monitor and retrain and Privacy and sensitive suggestions.

  1. Choose one system for the email on your validation week: a single method, a hybrid, or two stages. Your design may not be one of the project’s methods. Then add it as a file in larkfield_recs/methods/, as the course’s methods are. recs evaluate and recs batch can then use it.

  2. Measure the effect of each rule on your design, as in What each rule costs:

    Terminal window
    recs rules two-stage --period validation

    Write one row per rule in recs-pack/rules.csv: recall@10 before and after, and the share of lists that the rule changed. Then say which rules you keep, and why. A rule can be right even when it lowers recall.

  3. Write recs-pack/OMAR.md, one page and no code. Say what your design does better, for which segments, and which segment gets worse, with your comment. Answer Tomás: is two-stage the right choice here, for each surface?

  4. Measure the serving cost. Build the batch for every customer, time it, and start the service:

    Terminal window
    recs batch two-stage
    recs bench --requests 200

    Give the batch time and the p50 and p95 of a lookup. (Half of the lookups are faster than p50, and 95% are faster than p95.) Give the cost of the nightly batch at the course’s server price: US$0.17 an hour for 4 vCPUs, an assumption in costs.json. Check Amira’s 100 ms budget.

  5. Export the lists that you hand in:

    Terminal window
    python "$F/export_lists.py" --out recs-pack
    Wrote recs-pack/email.csv: 45,000 customers, 40,786 personal lists and 4,214 fallback lists (two-stage, trained on the events before 2026-08-03)
    Wrote recs-pack/similar.csv: 3,144 product pages (batch)

    email.csv has 10 products for every customer: the batch list with the rules of your rules.toml, and no sensitive product for a shared family inbox. A customer with no batch list, an opted-out customer or a deleted customer gets the fallback. similar.csv has 10 products for every product page. --similar lift (or cooc, item-cf, content) computes the pages with another method. If your product page is a design of your own, write similar.csv yourself, with the same three columns.

  6. Test the fallback and a deletion request in a separate outputs folder, so that your batch does not change. Choose a customer who is in your batch:

    Terminal window
    LARKFIELD_OUTPUTS=outputs-forget recs forget C100123

    Then start the service on that folder and ask for the customer’s list, as in Apply a deletion request. Ask for a customer with no event too, and for an email list of a shared family inbox. Stop the service with Ctrl+C. Paste the answers in recs-pack/service.md, and say what each one shows.

Uses Offline wins are not online wins, Online tests and guardrails and Feedback loops and popularity bias.

Your offline test says which lists find more of next week’s orders. It does not say what the email earns. Write the plan for the online test in recs-pack/EMAIL-TEST.md, before any test starts:

  1. The question and the two versions. For example: your email lists against an email with the trending list. Customers are split at random.
  2. The main measure that decides, for example added revenue per email.
  3. Guardrails, at least two: for example unsubscribes, returns, the share of shared inboxes with a sensitive product (it must be 0), and new products shown.
  4. The size, decided in advance. Use the course’s simulated A/B test to size it: recs online, run with the course data (the two variables unset). Then use the rough rule of Your turn: decide which list to release: an interval shrinks with the square root of the number of customers. Turn the size into weeks of emails.
  5. The stopping rule. Look once, at the planned size. Stop early only if a guardrail breaks, for example if unsubscribes are clearly higher. Do not stop because your version “looks good” after a few days.
  6. The decision rule and the rollback: what you do for each result, and how the email goes back to trending in one step.

This course does not teach test design in depth; that belongs to a planned course on experiments.

The folder recs-pack/ in your larkfield-recs folder:

recs-pack/
PROBLEM.md Part 1: the two surfaces, measures, guardrails, cost of a bad suggestion
comparison.csv Part 2: one row per design and period, the course's columns, by segment
COLD-START.md Part 3: new products and new customers, with the evidence
rules.csv Part 4: one row per rule, recall before and after, lists changed
OMAR.md Part 4: one page for Omar, no code
email.csv Part 4: the email lists, from export_lists.py
similar.csv Part 4: the product pages
lists.json Part 4: what made the lists (written by export_lists.py)
service.md Part 4: the fallback, the deletion request, the shared inbox, the timings
EMAIL-TEST.md Part 5: the online test plan
methods/ any method file that you added, and any other script
REPORT.md Parts 1 to 5, with the commands that make every number
requirements.txt exact versions (pip freeze)

REPORT.md has these sections:

# Recommendation pack: Home office and the weekly email
## Data checks (files, checksums, the gap, what you see in the new category)
## Split (your periods, and why there is no leak)
## Problem statements (Part 1: a summary of PROBLEM.md)
## Comparison (Part 2: the table, the segments, the differences with intervals)
## Cold start (Part 3: the launch replay, new customers, the product page)
## The choice (Part 4: rules, cost, the segment that got worse, the test-week run)
## Service (Part 4: fallback, deletion, latency)
## Email test plan (Part 5: a summary of EMAIL-TEST.md)
## Limits (what your offline test cannot see)
## How to reproduce (commands, versions, run times, a fresh environment)

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. The course is complete only when all five items are demonstrated. You can improve an item and send it again.

Required item Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Problem statements (Module 1) For both surfaces: who sees it, the business measure, the offline stand-in, at least two guardrails, and the cost of a bad suggestion worked in dollars with the assumptions named. One surface only, or a cost with no assumption. “Maximise clicks” as the business measure, with no guardrail.
2. Comparison (Modules 2 to 7) A time split with no later event in training, written before the results. popular, trending and at least four families on the same customers. recall@10 with intervals, NDCG, coverage and novelty, by segment with sizes, and new products in the lists. Differences claimed with recs compare. No intervals, or no segments. Or settings tuned on the test week, then reported on the same week. A random split. Or no popularity baseline. Or fewer than four families.
3. Chosen system (Module 7) A design chosen on validation, run once on the test week. The effect of each rule measured, and each kept rule explained. The segment that gets worse named, with a comment. Rules applied but not measured one by one. The choice made on the average alone, while a segment gets worse with no comment.
4. Cold start (Module 6) The launch replay with the share of new products shown and Omar’s 2% limit checked. A plan for new customers, with what the fallback shows. A product page for a new product, with what it shows and why. New products shown, but the cost for known customers not measured. No evidence that new products are shown.
5. Service and email test (Modules 8 and 9) The batch built and timed, the service answering, the fallback shown, a deletion request tested on the lists, and a shared inbox with no sensitive product. An email test plan with a main measure, at least two guardrails, a size decided in advance with its source, a stopping rule and a rollback. No deletion test, or a plan with no size. “Send it and see.” Or no stopping rule.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • A leaky split: a random split, or training data that contains events of the period that you score. (Item 2.)
  • The test week used for tuning: settings, weights or rules changed after the test-week run, or the test week run again until it looked good. (Items 2 and 3.)
  • No popularity baseline in the comparison. (Item 2.)
  • A design chosen on the average alone. A segment that the course taught gets worse (new, occasional or loyal customers, or new products), and the pack says nothing about it. (Item 3.)
  • Lists that break the shop’s own rules: a sensitive product in a shared family inbox, or a personal list for a customer who opted out. (Item 5.)

A good pack is honest before it is impressive. It has three things:

  • Numbers with their sizes. Each recall has its interval. It also says how many customers are behind it, and which period it comes from.
  • A simple baseline in each comparison. For the email, the baseline is trending. For the product page, it is the course’s neighbours. If your design does not beat the baseline, the pack says so.
  • A decision with its cost. For example: “On the test week, this design finds more of next week’s orders for loyal customers. It shows the new category to this share of customers. It costs this much a night. It is worse for occasional customers, and we accept that because …”

The course team solved the assignment once, from the public files only, with the course’s code. Then it scored those lists on the graded weeks and sent them in the simulated email. The results below come from that run. Your design and your split can be different. Compare the method and the size of each result, not the exact number.

The case studies of Module 10 met many of the same results on real data. Read them next to this list: Recommend for an online retailer, Next item from browsing sessions and Books for readers.

  • The test week is small. 1,234 customers ordered in it. The intervals of recall@10 were about ±0.02. A segment of about 300 customers proves little.
  • Trending was hard to beat again. On the test week, trending had a recall@10 of about 0.16, and about 0.14 with the rules. Every single method was below trending. Books for readers found the same on real data.
  • The two-stage ranker won offline. With the rules, it was better than trending by about 0.04 on the test week. The difference was clear for every segment.

A validation week can choose the wrong design

Section titled “A validation week can choose the wrong design”

The course team tuned on 20 to 26 July, the first week after the gap. In that week, the ranker could learn only from old weeks. So trending beat the ranker for new and occasional customers.

The course team therefore chose a switch: trending for customers with fewer than 3 orders, and the ranker for the others. On the test week, the ranker alone was better.

The course team kept the switch. Changing the design after the test run would make the test week a second validation period. This is why the default validation period is now two full weeks before the gap.

  • The course team’s switch was a little better than trending: about 0.15 against about 0.14. All of the gain came from loyal customers.
  • The ranker alone was better still: about 0.17. Its gain came from occasional and loyal customers. New customers were about the same as with trending.
  • Showing new products is not the same as guessing them. One kept place showed almost every Home office product to about two thirds of the customers who ordered. It found about 1% of their Home office orders.
  • It had a cost. It lowered known customers’ recall by about 4%. That is more than Omar’s 2% limit. A pack must say this, and decide.
  • The rules matter more for the email. Without rules, about 8% of the places in the course team’s lists were sensitive health products. Removing them cost about 0.016 of recall. All the rules together changed 95% of the lists.
  • The batch neighbours were alternatives. On the pages of new furniture, about 8 of the 10 neighbours were products of the same kind. A small table of “bought next”, by subcategory, added complements to many pages. Next item from browsing sessions shows a related idea: products seen in the same visit.

About 4,700 customers received two simulated emails. Half of them received trending, and half received the course team’s lists. Trending earned about 1.2 dollars of added revenue per email.

  • The course team’s lists earned less. In 18 of 20 simulated tests, the difference was not clear. In 2, the course team’s lists were clearly worse.
  • The offline winner earned the least. The ranker alone had the best score on the graded weeks. In the simulation, it earned about 0.9 dollars per email, and it was clearly worse than trending in 3 of 20 tests.

An offline gain does not promise an online gain. This is why Part 5 asks for a test plan before anything is sent.

On the course team’s laptop, other heavy jobs ran at the same time, so these times are slow:

  • The nightly batch took about 4 minutes for the switch, and about 21 minutes for the ranker for every customer (42,088 customers).
  • A lookup in the service had a p50 of about 1.5 ms. Its p95 was about 93 ms: just under Amira’s 100 ms.
  • The ranker at request time took about 145 ms for half of the customers. That is over the budget, so the email and the page come from the batch.

Common mistake

The ranker has nothing to learn from

Symptom: you chose the validation week 20 to 26 July. recs evaluate two-stage --period validation stops with a long error from LightGBM. Its last line is:

ValueError: Input data must be 2 dimensional and non empty.

Cause: the ranker learns on the 14 days before the period. Before 20 July, those days are the gap: no orders, so no labels.

Fix: choose a validation period with two weeks of events before it, as the default one. Or add --set train_days=21, so that the ranker reaches back to the course log. Then the ranker learns from old weeks only, and it is weak: in the course team’s run, it lost to trending on that week. It won on the test week, where it had one week of Home office. A validation period that does not look like the period you serve can choose the wrong design. Say in your report which window the ranker learned on.

Consequence in AI work: a model that trains on an empty window fails quietly in production too. Check what each training window contains.

Common mistake

A better average that hides a worse segment

Symptom: your chosen design has the best recall@10 overall. A colleague finds that occasional customers got worse, or that the new category almost never appears.

Cause: the average mixes the segments, and new products are a small share of the orders. A large gain for loyal customers can hide a loss elsewhere.

Fix: give every segment with its size, use recs compare --by segment, and report new products separately. Name the group that loses, and say what you do about it.

Consequence in AI work: a change that “won” is rolled back after complaints from one group of customers.

  • The graded weeks guessed from the course log. Tuning a design to “what sold in August last year” is not a time split. Use only the days before your period.
  • Trending read without the gap. Just after the gap, “the last 28 days” holds only two weeks of events. Say so when you read a trending list.
  • Tomás’s answer accepted. Two-stage led the course’s offline table, and it won offline here too. In the course team’s simulated email, it earned the least. The new category is also a different problem. Measure each surface here.
  • A product page that shows only alternatives. Ten other desks on a desk’s page can be right for a customer who is still choosing. It is wrong for a customer who bought one. Say whom your page serves.
  • A deletion tested on the wrong batch. recs forget rebuilds the batch. Run it in a separate outputs folder, or after you export your lists.
  • A test stopped when it “looks good”. Decide the size first, and look once.
Show the public check values

Open this only after you have your own numbers. These values come from the public files, so you can check them. They are not the answers to the assignment.

  • Data: 150 new products in 6 subcategories, all added on 2026-07-20; 5,000 new customers, 2,088 with an event in the files; 37,140 events from 2026-07-20 to 2026-08-02. Every file in SHA256SUMS is OK.
  • The data folder: make_data.py writes 3,144 products, 45,000 customers and 842,456 events. recs check with the default periods gives 0.106 for popular (quick).
  • The default periods: 2,118 customers ordered in the validation period (the course’s test period) and 1,234 in the test week. trending on the validation period: recall@10 0.156, as in the course.
  • Speed on the course team’s busy laptop (Apple M4, one thread): the nightly batch for 42,088 customers took about 4 minutes with the switch. It took about 21 minutes with the ranker for every customer.

Recap

You can now take a recommendation problem from a new category to a decision. You state what each surface is for and what a bad suggestion costs. You make a time split with no leak, and compare popularity with other families of methods by segment, with intervals. You show new products from their first week, and you serve customers with no history. You choose a system with its rules, its cost and the group that loses in plain view. You run it as a service with a fallback and a deletion request. And you plan the email test with its size and its stopping rule decided in advance. You know that an offline winner may not be clearly better online.