Skip to content
Français

Business rules and re-ranking

  • Lesson 22
  • 40 min
  • Module 7: Two stages: candidates and ranking

Where we are

Our ranker orders candidates by predicted interest. The shop also has rules: stock, age limits, margins and variety. In this lesson, we add business rules and diversity, and we measure the cost of each rule.

What you will be able to do

Make the ranker’s lists safe and sensible to show. Remove the products that the shop must not suggest, keep the list varied, and fill the empty places. Measure what each rule costs in recall, and who pays for it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook looks at the all-time popular list and finds the products that a shop should not show. It applies the rules for stock, alcohol and sensitive health products to a list for every test customer. You write the rule that removes products a customer already bought. Then it measures what each rule costs in recall. Its setup cells download about 7 MB of practice data. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the files of the last lesson: recs evaluate two-stage works.

Tested on macOS with Python 3.12 and LightGBM 4.7.0. The Windows (PowerShell) and Linux commands were not run.

The assistant who knows the shop’s rules

Section titled “The assistant who knows the shop’s rules”

A good shop assistant knows more than taste. She knows the shop’s rules, and she uses common sense:

  • She does not suggest a product that is out of stock.
  • She does not offer wine to someone who may be too young.
  • She does not suggest a second sofa to someone who bought one last month.
  • She never holds up a pregnancy test in front of the customer’s family.
  • She does not fill the counter with five nearly identical lamps.

The ranker of the last lesson knows none of this. It learned what customers order. It did not learn what the shop may show.

The problem: the best-scored list is not a list we can show

Section titled “The problem: the best-scored list is not a list we can show”

We looked at the two-stage lists for the 2,118 test customers, before any rule:

What the two-stage ranker's lists contain before any rule, on Larkfield's test period. A third of the lists suggest a product the customer already bought.
Text description of the chart

A dot chart with four rows. 35.2% of the lists contain a product the customer already bought (a product people do not buy again). 5.7% of all places in the lists show a product that is out of stock. 5.7% of the places show a sensitive health product. 0.8% of the places show alcohol.

Each problem has a cause:

  • Already bought: 35% of the lists. The ranker’s top feature is “days since the customer touched it”. A product the customer bought last month is very recent.
  • Out of stock: 5.7% of the places. Most of them show the Oak Timber Swing Seat. It is #4 in trending, and its stock flag says out of stock since 15 June. This is a practice problem we made on purpose.
  • Sensitive: 5.7% of the places. Larkfield sells mobility aids, continence care, and pregnancy and fertility products. People buy continence care again and again, so it is popular: #4 of the all-time popular list.
  • Alcohol: 0.8% of the places. Larkfield sells wine and spirits, and it does not know the customer’s age.

Here is one real list, for customer C100142, a loyal customer:

Place Product Subcategory Problem
1 Green Waterproof Chair Protector Continence care sensitive
2 Lumi Green Nordic Coffee Table Garden tables
3 Bramble Terracotta Mosaic Side Table Garden tables
4 Red Wooden Pull-Along Train Toys and games
5 Bramble White Rustic Single Wardrobe Wardrobes
6 Modern White MDF 3-Drawer Chest Chests of drawers
7 Blush Pink Hopscotch Mat Outdoor play out of stock, already bought
8 Blush Pink Felt Finger Puppets Toys and games out of stock
9 Black Rustic Wooden Planter Box Pots and planters
10 Terracotta Linen Roman-Style Curtains Curtains

The first suggestion is a continence product. On a home page that a family shares, or in a weekly email, that is a privacy problem. Places 7 and 8 cannot be ordered.

We keep the ranker as it is, and we add a last step. It has two kinds of rules.

Filters: remove what the shop must not show

Section titled “Filters: remove what the shop must not show”

A filter removes a product from a list. Some filters are the same for every customer:

  • out of stock today;
  • age-restricted (wine and spirits);
  • sensitive health products;
  • a margin below Hana’s floor (the share of the price that the shop keeps).

One filter depends on the customer: already bought. It removes a product that the customer ordered before, unless people buy it again. The product file says which products people buy again: compost, pet food, continence care. A sofa is bought once.

A list with five nearly identical lamps is a poor list, even if each lamp has a high score. Re-ranking changes the order to make the list more varied. That is diversity, which you measured in Beyond accuracy.

A well-known method is maximal marginal relevance (MMR). It picks the list one place at a time. Each time, it takes the product with the best value of:

value = 0.5 × score − 0.5 × (highest similarity to a product already in the list)

The 0.5 and 0.5 set the balance between relevance and variety.

Four candidates (a tiny example we made). Similarity between two lamps is 0.9; between a lamp and the rug, 0.1.

Candidate Score Place 1: 0.5 × score Place 2: 0.5 × score − 0.5 × similarity to lamp A
Lamp A 0.90 0.450 (chosen)
Lamp B 0.85 0.425 0.425 − 0.450 = −0.025
Lamp C 0.80 0.400 0.400 − 0.450 = −0.050
Rug 0.70 0.350 0.350 − 0.050 = 0.300

Lamp A takes place 1. For place 2, the rug wins, although its score is the lowest. Lamps B and C are too similar to lamp A.

The project uses a simpler rule with the same aim: at most 3 products of one subcategory in a list. It is easy to explain to Hana, and it needs no similarity.

Five steps from left to right. The ranker's first 50 products. Remove products no list may show: out of stock, alcohol, sensitive health, margin below the floor. Remove products this customer already bought, unless people buy them again. Keep at most 3 of one subcategory. Keep the first 10. If places are empty, fill them from trending with the same rules.
How the project applies the rules to each customer's list. It starts from 50 products, so that the removed ones can be replaced.
Text description of the diagram

Step 1: take the ranker’s first 50 products, not 10. Step 2: remove the products that no list may show: out of stock, alcohol, sensitive health products, and a margin below the floor. Step 3: remove the products that this customer already bought, unless people buy them again. Step 4: keep at most 3 products of one subcategory. Step 5: keep the first 10. If fewer than 10 are left, fill the empty places from the trending list, with the same rules.

The rules start from 50 products for a reason. If we cut to 10 first and then remove three, the list has seven products. With 50, the next good products move up.

For C100142, the rules remove the continence product (sensitive), the hopscotch mat (out of stock, and already bought) and the finger puppets (out of stock). Seven products move up. Three come from places 11 and below of the ranker’s list: Dockside Oak Metal Wall Shelf, Hollow Farm Black Barn-Door Bedside Cabinet, Black Classic Wooden Kids' Wardrobe.

recs rules measures each rule alone, then all rules together, on the same lists:

Recall@10 of the two-stage lists with each rule alone and with all rules, on Larkfield's test period (2,118 customers). One rule raises recall; all rules together cost 0.019.
Text description of the chart

A dot chart with a line at the recall without rules, 0.234. remove out of stock: 0.236; at most 3 per subcategory: 0.231; margin at least 12%: 0.230; remove alcohol: 0.229; remove sensitive health: 0.229; remove already bought: 0.224; all rules (no margin floor): 0.215. The margin floor is Hana’s suggestion and is not part of “all rules”.

All rules together cost 0.019 of recall@10: from 0.234 to 0.215 (95% interval 0.199 to 0.231). They change 90% of the lists. Coverage falls from 29.5% to 26.1%.

Removing out-of-stock products raises recall, from 0.234 to 0.236. The stock flag says that the swing seat is out of stock from 15 June. Our synthetic log still has a few orders of it in the test period. Think of them as back-orders, which the shop cannot send soon. Its place goes to a product that more customers can buy now. So almost every list that showed it was wasting a place.

The rules do not cost every segment the same:

Segment No rules All rules Lists with an already-bought product (no rules)
New 0.198 0.195 0.0%
Occasional 0.268 0.248 42.8%
Loyal 0.258 0.217 83.3%

Loyal customers pay the most: −0.041. 83% of their lists had a product they already bought. The already-bought rule alone takes their recall from 0.258 to 0.230. New customers have bought nothing, so this rule does not touch them.

Why does that rule cost recall at all? Because some customers do order the same product again. Customer C106777 ordered the same duvet cover twice. The offline measure counts such an order as a hit. But a list full of things you already own also annoys people, and recall cannot see that. Hana and Rafael decided that the rule is worth its cost. The simulated online test in Online tests and guardrails looks at the other side: returns and unsubscribes.

On average, the rules cost recall. For one customer, they can help. In the last lesson, C106777’s two ordered products were at places 11 and 15, behind two wines, a storage basket and a duvet cover. The rules remove the wines (alcohol), the basket (out of stock and already bought) and the duvet cover (already bought). The side table and the gloves move into the list. This customer’s recall goes from 0.000 to 0.667.

Note the age rule here. This customer has ordered wine twice. A shop that checks age at checkout could allow wine for customers it knows are adults. A blunt rule is safe, but it has a price.

When to use rules, when not, and what they cost

Section titled “When to use rules, when not, and what they cost”

Use rules for what the model cannot know or must not decide: stock today, the law (age), privacy (sensitive products), and business choices (margin, variety). Keep them in one file that Hana and Rafael can read: the project’s rules.toml.

Do not use a rule to fix a problem that the data or the model can fix. The winter coats in July are an example: the ranker’s season feature already pushes them down. One rule per season would grow into a list nobody understands. And do not judge a privacy rule by recall alone.

Cost: 0.234 → 0.215 recall@10 on Larkfield’s test period, and coverage 29.5% → 26.1%. The rules themselves take almost no time.

Some shops add fairness rules: for example, a minimum share of places for new products or small suppliers. The one place for the new range in Cold start is a rule of this kind. Measure such rules as we measured these: alone, then together, and by segment.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page. Run the setup cells.

  2. Run section 1. It finds the out-of-stock, sensitive and age-restricted products in the all-time popular top 10.

  3. In section 2, write the already-bought rule. The check prints Check passed.

  4. Run section 3. It measures recall@10 with each rule.

  5. In a terminal, in your larkfield-recs folder with the virtual environment active, copy the rules files. The last two replace files you have: they add --rules to recs evaluate.

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m07
    cp $S/rules.toml .
    cp $S/larkfield_recs/rules.py larkfield_recs/
    cp $S/larkfield_recs/commands/rules.py larkfield_recs/commands/
    cp $S/larkfield_recs/harness.py larkfield_recs/
    cp $S/larkfield_recs/commands/evaluate.py larkfield_recs/commands/
  6. Open rules.toml. Each rule is one line that Hana can change:

    [rules]
    remove_out_of_stock = true # no stock today (the stock file is from 2026-06-15; not applied before that day)
    remove_age_restricted = true # wine and spirits: we do not know the customer's age
    remove_already_bought = true # bought before, unless it is a product people buy again (compost, pet food ...)
    remove_sensitive = true # health products that a shared family email could reveal
    min_margin = 0.0 # Hana: products with a lower margin are not suggested (0.12 = 12%)
    max_per_subcategory = 3 # diversity: at most this many products of one subcategory in a list
  7. Measure each rule. This takes about 20 seconds; add --quick for a sample of 1,000 customers.

    Terminal window
    recs rules two-stage
    two-stage, test period, 2,118 customers
    lists recall@10 NDCG@10 coverage diversity changed out of stock sensitive alcohol
    no rules 0.234 0.156 29.5% 0.93 0% 5.7% 5.7% 0.8%
    only remove_out_of_stock = True 0.236 0.158 28.7% 0.93 54% 0.0% 5.7% 0.8%
    only remove_age_restricted = True 0.229 0.152 28.9% 0.93 5% 5.7% 5.7% 0.0%
    only remove_already_bought = True 0.224 0.151 28.3% 0.93 35% 5.8% 6.1% 0.9%
    only remove_sensitive = True 0.229 0.152 29.2% 0.92 54% 5.7% 0.0% 0.8%
    only max_per_subcategory = 3 0.231 0.155 28.9% 0.94 9% 5.7% 5.8% 0.8%
    all rules 0.215 0.143 26.1% 0.93 90% 0.0% 0.0% 0.0%
  8. Score the two-stage recommender with all the rules:

    Terminal window
    recs evaluate two-stage --rules --by segment
    Method two-stage (candidates from 5 sources, then a LightGBM LambdaMART ranker), test period, rules: max_per_subcategory, min_margin, remove_age_restricted, remove_already_bought, remove_out_of_stock, remove_sensitive
    Customers scored: 2,118; trained on events before 2026-06-22
    recall@10 0.215 (95% interval 0.199 to 0.231), NDCG@10 0.143, MAP@10 0.109, precision@10 0.033, hit rate 28.6%
    coverage 26.1% of the products on sale, novelty 5.86 bits, diversity 0.93, new products in the lists 28.2%
    time: fit 11.6 s, suggestions 4.4 s
    segment customers recall@10 NDCG@10 95% interval (recall)
    new 952 0.195 0.132 0.171 to 0.218
    occasional 556 0.248 0.162 0.217 to 0.285
    loyal 610 0.217 0.144 0.189 to 0.247
    new products: 647 orders of new products in the period; recall@10 on them 0.334
  9. Compare the lists with and without rules on the same customers:

    Terminal window
    recs compare two-stage two-stage+rules --by segment
    recall@10: two-stage 0.234, two-stage+rules 0.215
    difference -0.018 (95% interval -0.028 to -0.010); better for 67 customers, worse for 112, the same for 1,939
    The difference is clear: the interval does not include 0.
    coverage 29.5% -> 26.1%; novelty 5.68 -> 5.86
    new 952 customers -0.003 (-0.013 to +0.006)
    occasional 556 customers -0.020 (-0.036 to -0.005)
    loyal 610 customers -0.041 (-0.063 to -0.019)

    The exact difference is −0.018. The 0.019 above is 0.234 − 0.215, from rounded numbers.

Hana wants no suggestion with a margin below 12%. Before you run it, predict: how many lists will change, and how much recall will it cost?

Terminal window
recs rules two-stage --set min_margin=0.12
Show the answer

The table gets two changed lines:

only min_margin = 0.12 0.230 0.155 27.7% 0.93 63% 5.8% 5.9% 0.8%
all rules 0.214 0.143 24.9% 0.93 95% 0.0% 0.0% 0.0%

The margin floor changes 63% of the lists, but costs only 0.004 of recall@10 (0.234 → 0.230). Low-margin products are common in the lists, and there are good products to replace them. With all rules, it costs 0.001 more, and coverage falls to 24.9%.

Recall cannot say whether the floor is worth it. The shop keeps more money on each sale, but it shows fewer kinds of product. That is a question for Omar’s online measure, added revenue per visit, in Module 8.

Common mistake

Rules applied after the list is cut to 10

Symptom: after the rules, some customers’ lists have seven products, and a few have four.

Cause: the code took the ranker’s top 10, then removed products. Nothing replaced them.

Diagnose: count the products in each list after the rules. Empty places show up as missing IDs.

Fix: apply the rules to a longer list (the project uses 50), then keep the first 10. Fill any remaining places from a fallback list that passes the same rules.

Common mistake

A privacy rule judged by recall

Symptom: Tomás wants to drop the sensitive-products rule. “It costs 0.005 of recall, and customers do buy continence care.”

Cause: recall measures orders, not harm. A continence product on a shared family computer, or in an email that the whole family reads, tells others about someone’s health. Rafael has customers with a shared email address.

Diagnose: ask what a wrong suggestion costs, not only what a right one earns.

Fix: keep the rule. Sensitive products can still be found by search, and shown on their own product pages. Privacy and sensitive suggestions adds the email rule for shared inboxes.

Check your understanding

Why does the project apply the rules to the ranker's first 50 products, not to its first 10?

Show the answer

The rules remove products. If they started from 10, every removed product would leave an empty place. With 50, the next products of the ranker move up, so the list stays full of products the ranker likes. Only if fewer than 10 survive does the trending fallback fill the rest.

Check your understanding

Removing out-of-stock products raised recall@10 from 0.234 to 0.236. Removing alcohol lowered it to 0.229. Explain the difference.

Show the answer

The stock flag says the swing seat is out of stock, so it is rarely ordered in the test period. Its few orders there are back-orders. Its place in a list is almost never a hit, so giving that place to another product helps. Alcohol is on sale, and some customers did order it in the test period. Removing it from their lists removes some hits.

Check your understanding

With at most 2 products per subcategory instead of 3, recall@10 falls from 0.231 to 0.224, and diversity rises from 0.94 to 0.96. Who should decide which cap to use?

Show the answer

The team measures; the business decides. Priya shows the cost (0.007 of recall) and the gain (more varied lists). Hana, who owns the rules, and Omar, who owns the business measure, choose. An online test can show whether customers click and buy more from more varied lists.

Recap

  • The ranker learns what customers order, not what the shop may show. Without rules, 35% of Larkfield’s lists have an already-bought product, and 5.7% of the places show a sensitive health product.
  • Filters remove products: out of stock, alcohol, sensitive, low margin, already bought (unless people buy it again). Re-ranking keeps the list varied: maximal marginal relevance, or the project’s cap of 3 per subcategory.
  • Apply the rules to a long list (50), then cut to 10, then fill from a fallback with the same rules.
  • All rules cost 0.019 of recall@10 (0.234 → 0.215) and change 90% of the lists. Loyal customers pay the most. The out-of-stock rule alone raises recall.
  • Keep rules in one file that the business owns, and do not judge a privacy rule by recall.

Our system uses a customer’s whole history. Sometimes we only know the current visit. Next, we try simple session suggestions, and we see what an LLM adds for re-ranking and explanations, and what it costs.

Next: Sessions, sequences and LLMs, briefly