Business rules and re-ranking
- Lesson 22
- 40 min
- Module 7: Two stages: candidates and ranking
Where we are
Our ranker orders candidates by predicted interest. The shop also has rules: stock, age limits, margins and variety. In this lesson, we add business rules and diversity, and we measure the cost of each rule.
Lesson 22 of 29 · Recommendation and Personalisation: Show Each Person What MattersBusiness rules and re-ranking
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Make the ranker’s lists safe and sensible to show. Remove the products that the shop must not suggest, keep the list varied, and fill the empty places. Measure what each rule costs in recall, and who pays for it.
NotebookDo this lesson in a notebook
The notebook looks at the all-time popular list and finds the products that a shop should not show. It applies the rules for stock, alcohol and sensitive health products to a list for every test customer. You write the rule that removes products a customer already bought. Then it measures what each rule costs in recall. Its setup cells download about 7 MB of practice data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The two-stage recommender, from Why two stages and A learned ranker.
- Coverage and diversity, from Beyond accuracy.
What you need on your computer (only for the project steps)
- Your
larkfield-recsproject with the files of the last lesson:recs evaluate two-stageworks.
Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.
The assistant who knows the shop’s rules
Section titled “The assistant who knows the shop’s rules”A good shop assistant knows more than taste. She knows the shop’s rules, and she uses common sense:
- She does not suggest a product that is out of stock.
- She does not offer wine to someone who may be too young.
- She does not suggest a second sofa to someone who bought one last month.
- She never holds up a pregnancy test in front of the customer’s family.
- She does not fill the counter with five nearly identical lamps.
The ranker of the last lesson knows none of this. It learned what customers order. It did not learn what the shop may show.
The problem: the best-scored list is not a list we can show
Section titled “The problem: the best-scored list is not a list we can show”We looked at the two-stage lists for the 2,118 test customers, before any rule:
Text description of the chart
A dot chart with four rows. 35.2% of the lists contain a product the customer already bought (a product people do not buy again). 5.7% of all places in the lists show a product that is out of stock. 5.7% of the places show a sensitive health product. 0.8% of the places show alcohol.
Each problem has a cause:
- Already bought: 35% of the lists. The ranker’s top feature is “days since the customer touched it”. A product the customer bought last month is very recent.
- Out of stock: 5.7% of the places. Most of them show the Oak Timber Swing Seat. It is #4 in trending, and its stock flag says out of stock since 15 June. This is a practice problem we made on purpose.
- Sensitive: 5.7% of the places. Larkfield sells mobility aids, continence care, and pregnancy and fertility products. People buy continence care again and again, so it is popular: #4 of the all-time popular list.
- Alcohol: 0.8% of the places. Larkfield sells wine and spirits, and it does not know the customer’s age.
Here is one real list, for customer C100142, a loyal customer:
| Place | Product | Subcategory | Problem |
|---|---|---|---|
| 1 | Green Waterproof Chair Protector | Continence care | sensitive |
| 2 | Lumi Green Nordic Coffee Table | Garden tables | |
| 3 | Bramble Terracotta Mosaic Side Table | Garden tables | |
| 4 | Red Wooden Pull-Along Train | Toys and games | |
| 5 | Bramble White Rustic Single Wardrobe | Wardrobes | |
| 6 | Modern White MDF 3-Drawer Chest | Chests of drawers | |
| 7 | Blush Pink Hopscotch Mat | Outdoor play | out of stock, already bought |
| 8 | Blush Pink Felt Finger Puppets | Toys and games | out of stock |
| 9 | Black Rustic Wooden Planter Box | Pots and planters | |
| 10 | Terracotta Linen Roman-Style Curtains | Curtains |
The first suggestion is a continence product. On a home page that a family shares, or in a weekly email, that is a privacy problem. Places 7 and 8 cannot be ordered.
The idea: rules after the ranker
Section titled “The idea: rules after the ranker”We keep the ranker as it is, and we add a last step. It has two kinds of rules.
Filters: remove what the shop must not show
Section titled “Filters: remove what the shop must not show”A filter removes a product from a list. Some filters are the same for every customer:
- out of stock today;
- age-restricted (wine and spirits);
- sensitive health products;
- a margin below Hana’s floor (the share of the price that the shop keeps).
One filter depends on the customer: already bought. It removes a product that the customer ordered before, unless people buy it again. The product file says which products people buy again: compost, pet food, continence care. A sofa is bought once.
Re-ranking: keep the list varied
Section titled “Re-ranking: keep the list varied”A list with five nearly identical lamps is a poor list, even if each lamp has a high score. Re-ranking changes the order to make the list more varied. That is diversity, which you measured in Beyond accuracy.
A well-known method is maximal marginal relevance (MMR). It picks the list one place at a time. Each time, it takes the product with the best value of:
value = 0.5 × score − 0.5 × (highest similarity to a product already in the list)
The 0.5 and 0.5 set the balance between relevance and variety.
A worked example: lamps and a rug
Section titled “A worked example: lamps and a rug”Four candidates (a tiny example we made). Similarity between two lamps is 0.9; between a lamp and the rug, 0.1.
| Candidate | Score | Place 1: 0.5 × score | Place 2: 0.5 × score − 0.5 × similarity to lamp A |
|---|---|---|---|
| Lamp A | 0.90 | 0.450 | (chosen) |
| Lamp B | 0.85 | 0.425 | 0.425 − 0.450 = −0.025 |
| Lamp C | 0.80 | 0.400 | 0.400 − 0.450 = −0.050 |
| Rug | 0.70 | 0.350 | 0.350 − 0.050 = 0.300 |
Lamp A takes place 1. For place 2, the rug wins, although its score is the lowest. Lamps B and C are too similar to lamp A.
The project uses a simpler rule with the same aim: at most 3 products of one subcategory in a list. It is easy to explain to Hana, and it needs no similarity.
The order of the steps
Section titled “The order of the steps”Text description of the diagram
Step 1: take the ranker’s first 50 products, not 10. Step 2: remove the products that no list may show: out of stock, alcohol, sensitive health products, and a margin below the floor. Step 3: remove the products that this customer already bought, unless people buy them again. Step 4: keep at most 3 products of one subcategory. Step 5: keep the first 10. If fewer than 10 are left, fill the empty places from the trending list, with the same rules.
The rules start from 50 products for a reason. If we cut to 10 first and then remove three, the list has seven products. With 50, the next good products move up.
For C100142, the rules remove the continence product (sensitive), the hopscotch mat (out of stock, and already bought) and the finger puppets (out of stock). Seven products move up. Three come from places 11 and below of the ranker’s list: Dockside Oak Metal Wall Shelf, Hollow Farm Black Barn-Door Bedside Cabinet, Black Classic Wooden Kids' Wardrobe.
Larkfield: what each rule costs
Section titled “Larkfield: what each rule costs”recs rules measures each rule alone, then all rules together, on the same lists:
Text description of the chart
A dot chart with a line at the recall without rules, 0.234. remove out of stock: 0.236; at most 3 per subcategory: 0.231; margin at least 12%: 0.230; remove alcohol: 0.229; remove sensitive health: 0.229; remove already bought: 0.224; all rules (no margin floor): 0.215. The margin floor is Hana’s suggestion and is not part of “all rules”.
All rules together cost 0.019 of recall@10: from 0.234 to 0.215 (95% interval 0.199 to 0.231). They change 90% of the lists. Coverage falls from 29.5% to 26.1%.
One rule raises recall
Section titled “One rule raises recall”Removing out-of-stock products raises recall, from 0.234 to 0.236. The stock flag says that the swing seat is out of stock from 15 June. Our synthetic log still has a few orders of it in the test period. Think of them as back-orders, which the shop cannot send soon. Its place goes to a product that more customers can buy now. So almost every list that showed it was wasting a place.
Who pays for the rules
Section titled “Who pays for the rules”The rules do not cost every segment the same:
| Segment | No rules | All rules | Lists with an already-bought product (no rules) |
|---|---|---|---|
| New | 0.198 | 0.195 | 0.0% |
| Occasional | 0.268 | 0.248 | 42.8% |
| Loyal | 0.258 | 0.217 | 83.3% |
Loyal customers pay the most: −0.041. 83% of their lists had a product they already bought. The already-bought rule alone takes their recall from 0.258 to 0.230. New customers have bought nothing, so this rule does not touch them.
Why does that rule cost recall at all? Because some customers do order the same product again. Customer C106777 ordered the same duvet cover twice. The offline measure counts such an order as a hit. But a list full of things you already own also annoys people, and recall cannot see that. Hana and Rafael decided that the rule is worth its cost. The simulated online test in Online tests and guardrails looks at the other side: returns and unsubscribes.
A rule can help one customer
Section titled “A rule can help one customer”On average, the rules cost recall. For one customer, they can help. In the last lesson, C106777’s two ordered products were at places 11 and 15, behind two wines, a storage basket and a duvet cover. The rules remove the wines (alcohol), the basket (out of stock and already bought) and the duvet cover (already bought). The side table and the gloves move into the list. This customer’s recall goes from 0.000 to 0.667.
Note the age rule here. This customer has ordered wine twice. A shop that checks age at checkout could allow wine for customers it knows are adults. A blunt rule is safe, but it has a price.
When to use rules, when not, and what they cost
Section titled “When to use rules, when not, and what they cost”Use rules for what the model cannot know or must not decide: stock today, the law (age), privacy (sensitive products), and business choices (margin, variety). Keep them in one file that Hana and Rafael can read: the project’s rules.toml.
Do not use a rule to fix a problem that the data or the model can fix. The winter coats in July are an example: the ranker’s season feature already pushes them down. One rule per season would grow into a list nobody understands. And do not judge a privacy rule by recall alone.
Cost: 0.234 → 0.215 recall@10 on Larkfield’s test period, and coverage 29.5% → 26.1%. The rules themselves take almost no time.
Some shops add fairness rules: for example, a minimum share of places for new products or small suppliers. The one place for the new range in Cold start is a rule of this kind. Measure such rules as we measured these: alone, then together, and by segment.
Guided practice: add the rules
Section titled “Guided practice: add the rules”-
Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.
-
Run section 1. It finds the out-of-stock, sensitive and age-restricted products in the all-time popular top 10.
-
In section 2, write the already-bought rule. The check prints
Check passed. -
Run section 3. It measures recall@10 with each rule.
-
In a terminal, in your
larkfield-recsfolder with the virtual environment active, copy the rules files. The last two replace files you have: they add--rulestorecs evaluate.Terminal window S=../nextia-learning-labs/C37/snapshots/end-of-m07cp $S/rules.toml .cp $S/larkfield_recs/rules.py larkfield_recs/cp $S/larkfield_recs/commands/rules.py larkfield_recs/commands/cp $S/larkfield_recs/harness.py larkfield_recs/cp $S/larkfield_recs/commands/evaluate.py larkfield_recs/commands/Terminal window $S = "..\nextia-learning-labs\C37\snapshots\end-of-m07"Copy-Item "$S\rules.toml" .Copy-Item "$S\larkfield_recs\rules.py" larkfield_recs\Copy-Item "$S\larkfield_recs\commands\rules.py" larkfield_recs\commands\Copy-Item "$S\larkfield_recs\harness.py" larkfield_recs\Copy-Item "$S\larkfield_recs\commands\evaluate.py" larkfield_recs\commands\ -
Open
rules.toml. Each rule is one line that Hana can change:[rules]remove_out_of_stock = true # no stock today (the stock file is from 2026-06-15; not applied before that day)remove_age_restricted = true # wine and spirits: we do not know the customer's ageremove_already_bought = true # bought before, unless it is a product people buy again (compost, pet food ...)remove_sensitive = true # health products that a shared family email could revealmin_margin = 0.0 # Hana: products with a lower margin are not suggested (0.12 = 12%)max_per_subcategory = 3 # diversity: at most this many products of one subcategory in a list -
Measure each rule. This takes about 20 seconds; add
--quickfor a sample of 1,000 customers.Terminal window recs rules two-stagetwo-stage, test period, 2,118 customerslists recall@10 NDCG@10 coverage diversity changed out of stock sensitive alcoholno rules 0.234 0.156 29.5% 0.93 0% 5.7% 5.7% 0.8%only remove_out_of_stock = True 0.236 0.158 28.7% 0.93 54% 0.0% 5.7% 0.8%only remove_age_restricted = True 0.229 0.152 28.9% 0.93 5% 5.7% 5.7% 0.0%only remove_already_bought = True 0.224 0.151 28.3% 0.93 35% 5.8% 6.1% 0.9%only remove_sensitive = True 0.229 0.152 29.2% 0.92 54% 5.7% 0.0% 0.8%only max_per_subcategory = 3 0.231 0.155 28.9% 0.94 9% 5.7% 5.8% 0.8%all rules 0.215 0.143 26.1% 0.93 90% 0.0% 0.0% 0.0% -
Score the two-stage recommender with all the rules:
Terminal window recs evaluate two-stage --rules --by segmentMethod two-stage (candidates from 5 sources, then a LightGBM LambdaMART ranker), test period, rules: max_per_subcategory, min_margin, remove_age_restricted, remove_already_bought, remove_out_of_stock, remove_sensitiveCustomers scored: 2,118; trained on events before 2026-06-22recall@10 0.215 (95% interval 0.199 to 0.231), NDCG@10 0.143, MAP@10 0.109, precision@10 0.033, hit rate 28.6%coverage 26.1% of the products on sale, novelty 5.86 bits, diversity 0.93, new products in the lists 28.2%time: fit 11.6 s, suggestions 4.4 ssegment customers recall@10 NDCG@10 95% interval (recall)new 952 0.195 0.132 0.171 to 0.218occasional 556 0.248 0.162 0.217 to 0.285loyal 610 0.217 0.144 0.189 to 0.247new products: 647 orders of new products in the period; recall@10 on them 0.334 -
Compare the lists with and without rules on the same customers:
Terminal window recs compare two-stage two-stage+rules --by segmentrecall@10: two-stage 0.234, two-stage+rules 0.215difference -0.018 (95% interval -0.028 to -0.010); better for 67 customers, worse for 112, the same for 1,939The difference is clear: the interval does not include 0.coverage 29.5% -> 26.1%; novelty 5.68 -> 5.86new 952 customers -0.003 (-0.013 to +0.006)occasional 556 customers -0.020 (-0.036 to -0.005)loyal 610 customers -0.041 (-0.063 to -0.019)The exact difference is −0.018. The 0.019 above is 0.234 − 0.215, from rounded numbers.
Your turn: Hana’s margin floor
Section titled “Your turn: Hana’s margin floor”Hana wants no suggestion with a margin below 12%. Before you run it, predict: how many lists will change, and how much recall will it cost?
recs rules two-stage --set min_margin=0.12Show the answer
The table gets two changed lines:
only min_margin = 0.12 0.230 0.155 27.7% 0.93 63% 5.8% 5.9% 0.8%all rules 0.214 0.143 24.9% 0.93 95% 0.0% 0.0% 0.0%The margin floor changes 63% of the lists, but costs only 0.004 of recall@10 (0.234 → 0.230). Low-margin products are common in the lists, and there are good products to replace them. With all rules, it costs 0.001 more, and coverage falls to 24.9%.
Recall cannot say whether the floor is worth it. The shop keeps more money on each sale, but it shows fewer kinds of product. That is a question for Omar’s online measure, added revenue per visit, in Module 8.
Common mistakes
Section titled “Common mistakes”Common mistake
Rules applied after the list is cut to 10
Symptom: after the rules, some customers’ lists have seven products, and a few have four.
Cause: the code took the ranker’s top 10, then removed products. Nothing replaced them.
Diagnose: count the products in each list after the rules. Empty places show up as missing IDs.
Fix: apply the rules to a longer list (the project uses 50), then keep the first 10. Fill any remaining places from a fallback list that passes the same rules.
Common mistake
A privacy rule judged by recall
Symptom: Tomás wants to drop the sensitive-products rule. “It costs 0.005 of recall, and customers do buy continence care.”
Cause: recall measures orders, not harm. A continence product on a shared family computer, or in an email that the whole family reads, tells others about someone’s health. Rafael has customers with a shared email address.
Diagnose: ask what a wrong suggestion costs, not only what a right one earns.
Fix: keep the rule. Sensitive products can still be found by search, and shown on their own product pages. Privacy and sensitive suggestions adds the email rule for shared inboxes.
Knowledge check
Section titled “Knowledge check”Check your understanding
Why does the project apply the rules to the ranker's first 50 products, not to its first 10?
Show the answer
The rules remove products. If they started from 10, every removed product would leave an empty place. With 50, the next products of the ranker move up, so the list stays full of products the ranker likes. Only if fewer than 10 survive does the trending fallback fill the rest.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Removing out-of-stock products raised recall@10 from 0.234 to 0.236. Removing alcohol lowered it to 0.229. Explain the difference.
Show the answer
The stock flag says the swing seat is out of stock, so it is rarely ordered in the test period. Its few orders there are back-orders. Its place in a list is almost never a hit, so giving that place to another product helps. Alcohol is on sale, and some customers did order it in the test period. Removing it from their lists removes some hits.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With at most 2 products per subcategory instead of 3, recall@10 falls from 0.231 to 0.224, and diversity rises from 0.94 to 0.96. Who should decide which cap to use?
Show the answer
The team measures; the business decides. Priya shows the cost (0.007 of recall) and the gain (more varied lists). Hana, who owns the rules, and Omar, who owns the business measure, choose. An online test can show whether customers click and buy more from more varied lists.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The ranker learns what customers order, not what the shop may show. Without rules, 35% of Larkfield’s lists have an already-bought product, and 5.7% of the places show a sensitive health product.
- Filters remove products: out of stock, alcohol, sensitive, low margin, already bought (unless people buy it again). Re-ranking keeps the list varied: maximal marginal relevance, or the project’s cap of 3 per subcategory.
- Apply the rules to a long list (50), then cut to 10, then fill from a fallback with the same rules.
- All rules cost 0.019 of recall@10 (0.234 → 0.215) and change 90% of the lists. Loyal customers pay the most. The out-of-stock rule alone raises recall.
- Keep rules in one file that the business owns, and do not judge a privacy rule by recall.
Our system uses a customer’s whole history. Sometimes we only know the current visit. Next, we try simple session suggestions, and we see what an LLM adds for re-ranking and explanations, and what it costs.