Skip to content
Français

Feedback loops and popularity bias

  • Lesson 26
  • 35 min
  • Module 8: From offline scores to the real shop

Where we are

We chose a list with an online test. But after release, a recommender keeps learning, and it learns from the clicks on its own suggestions. In this lesson, we see a feedback loop, add a small share of exploration, and watch coverage over time.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

See how a recommender that learns from its own suggestions can lock itself into the same few products. Measure it with coverage over time, and see what a small share of exploration buys and costs in Larkfield’s simulation.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook runs a toy home page of 5 places that learns only from its own clicks, with and without one random place. You write coverage over time, then plot Larkfield’s simulated 12 weeks. Its setup cells download less than 1 MB. About 15 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the project steps)

  • Your larkfield-recs project with the online command from Online tests and guardrails, and its virtual environment active.
  • The practice files in ../nextia-learning-labs, beside the project folder. You copy two files from its snapshot end-of-m08.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not run.

Think of a small radio station. Every week, it plays the ten songs that listeners requested most last week. That sounds fair: the listeners choose.

But listeners mostly request songs they heard on the station. A new song that was never played gets no requests, so it is never played. After a few weeks, the same ten songs fill every week. The station did not decide to ignore new music. Its own rule did it.

The problem: a home page that only shows what it showed

Section titled “The problem: a home page that only shows what it showed”

A recommender in production has the same problem. It shows a list. Customers click some of the products in it. The clicks go into the log. Next week, the model learns from the log, and it shows what got clicks. But only shown products can get clicks.

This circle is a feedback loop: the system’s output becomes its next input.

A circle of four steps. This week's home page shows the top 10 products. Customers see only these 10 and click some. The log records clicks on the 10 products shown. The learner counts the clicks of the last two weeks and chooses next week's top 10, which closes the circle. A dotted arrow from outside the circle: exploration, 1 place of 10 shows a random product on sale.
The feedback loop of a home page that learns only from its own clicks. Exploration is the only way in for a product that is not already shown.
Text description of the diagram

The picture is a circle of four steps. This week’s home page shows the top 10 products. Customers see only these 10 and click some of them. The log records the clicks on the 10 products shown. The learner counts the clicks of the last two weeks and chooses next week’s top 10, and the circle starts again. From outside the circle, a dotted arrow shows exploration: 1 place of 10 shows a random product on sale. It is the only way in for a product that is not already shown.

Here is a constructed example. A shop has four products, and the home page has two places. Each week, the page shows the two products with the most clicks last week. Product D is the one that customers like most, but the old page never showed it.

Week Page shows Clicks: A B C D Next week’s page
1 A, B 30 20 0 0 A, B
2 A, B 31 19 0 0 A, B
3 A, B 29 21 0 0 A, B

C and D get 0 clicks because nobody sees them, not because nobody wants them. The loop can never learn that D is the best product. Now keep place 1 for the product with the most clicks last week, and show a random product in place 2:

Week Page shows Clicks: A B C D Place 1 next week
1 A, C (random) 30 0 12 0 A
2 A, D (random) 30 0 0 45 D
3 D, B (random) 0 15 0 60 D

In week 2, the random place happens to show D. It gets more clicks than A, so from week 3 it leads the page. One random place opened the loop.

The course ran a loop like this on its behaviour model. It is simulated, not a real home page. It ran for 12 weeks from 6 April 2026, with 6,000 customers. Each customer visited in a given week with a chance of 40%. In week 1, the page showed the 10 most ordered products of March 2026. After that, it showed what its learner chose from its own log of the last two weeks.

There were two learners:

  • clicks: the 10 products with the most clicks;
  • click rate (ctr in the output, for click-through rate): the 10 products with the best clicks per time shown. It counts 100 extra “average” showings for each product, so that one lucky click does not win.

Exploration means giving some places to products that the learner would not choose. Here, at 10%, one of the 10 places on each visit shows a random product on sale, at a random place. At 20%, two places do.

Coverage over time is the number of different products shown at least once, counted week by week. It is the measure of a loop.

  • clicks, exploration 10%
  • click rate, exploration 20%
  • clicks, no exploration
Simulated: products shown at least once, week by week. Without exploration, the line stays at 10. With exploration, every one of the 2,994 products is shown by week 11.
Text description of the chart

A line chart over 12 weeks. The vertical axis is the number of products shown at least once so far, from 0 to 3,000. Clicks, no exploration: a flat line at 10 for all 12 weeks. Clicks, exploration 10%: 1,581 products in week 1, 2,219 in week 2 and 2,509 in week 3. Then 2,841 in week 5, 2,970 in week 7, and 2,994, every product, in weeks 11 and 12. Click rate, exploration 20%: 2,266 in week 1, 2,656 in week 2, and 2,994 from week 9. The click-rate learner without exploration is the same as the clicks learner without exploration: 10 products.

Without exploration, both learners show the same 10 products for all 12 weeks: 10 of 2,994. Every one of them is a product from week 1. The loop never let another product in.

With 10% exploration, one random place on each of about 2,400 visits a week reaches many products fast. In week 1 alone, 1,581 different products were shown. By week 11, all 2,994 had been shown at least once.

Coverage is not the business measure. Here is revenue per visit, week by week. It is the money from orders of suggested products, divided by visits, in this simulation.

  • clicks, no exploration
  • clicks, exploration 10%
  • click rate, exploration 10%
Simulated: revenue per visit, week by week. The page without exploration earns the most. Exploration lowers revenue in this simulation, and the click-rate learner with exploration earns the least.
Text description of the chart

A line chart over 12 weeks. The vertical axis is revenue per visit in dollars, from 0 to 1.8. The values move up and down from week to week. Clicks, no exploration: between 0.75 and 1.48 dollars, with a mean of 0.88 in weeks 1 to 4 and 1.31 in weeks 9 to 12. Clicks, exploration 10%: between 0.35 and 0.82, with a mean of 0.62 in weeks 1 to 4 and 0.47 in weeks 9 to 12. Click rate, exploration 10%: 0.63 in week 1, then between 0.10 and 0.46, with a mean of 0.14 in weeks 9 to 12.

This is the honest result of this simulation: exploration costs revenue here. In weeks 9 to 12, the page without exploration earns 1.31 dollars per visit. With 10% exploration, the clicks learner earns 0.47, and the click-rate learner 0.14.

Why? The simulation does not record a reason, but the behaviour model suggests one. The week-1 products are March’s bestsellers, and the model has a long tail: most products appeal to few customers. So a random place usually goes to a product that rarely sells, and it pushes the bestsellers down one place. The weekly values are also noisy: each week has only about 40 to 60 orders from suggestions. Read the four-week means, not single weeks.

In the toy shop of the notebook, the result is the other way round. Its week-1 products were five products picked without any data, so the random place found better ones, and clicks per visit rose. Whether exploration pays depends on how good the current list already is.

The click-rate learner shows a second danger. With exploration, it replaced 8 of the 10 week-1 products after one week, and its revenue fell further. Exploration changes the data that the learner reads. So you must check the learner as well as the exploration.

Exploration buys two things that the 12 weeks do not price. The first is information about products that the page never showed. The second is coverage: a chance for every product to be seen. Treat it as a cost that you choose and measure, not as a free gain.

A feedback loop makes popularity bias worse: popular products are shown, so they get clicks, so they stay popular. Exploration is one remedy. It adds random products. You already know a second remedy from Beyond accuracy: a popularity penalty, which pushes popular products down a little in every personal list.

The explorer below is the same measured result as in that lesson: tuned ALS on the test period, with nine penalties. This time, look at coverage and at the share of places that go to the 1% most popular products.

Explore

A popularity penalty against popularity bias

Penalty
0.0
Recall@10
0.149
NDCG@10
0.097
Coverage
16.5%
Novelty (bits)
5.70
Top 1% share
38.8%
New products
10.0%

No penalty: tuned ALS as it is. Recall@10 is 0.149. 16.5% of the products on sale appear in at least one list, and the 1% most popular products take 38.8% of all places.

  • Chosen penalty
  • Level with no penalty (dashed)
0%15%30%0.00.10.20.30.40.50.60.70.80.91.0Coverage of the products on salelevel with no penaltyCoverage of the products on salePopularity penalty 0.1: 17.4%Popularity penalty 0.2: 18.4%Popularity penalty 0.3: 20.2%Popularity penalty 0.4: 22.6%Popularity penalty 0.5: 24.7%Popularity penalty 0.6: 25.0%Popularity penalty 0.8: 20.9%Popularity penalty 1.0: 14.9%Popularity penalty 0.0: 16.5%16.5%0%25%50%0.00.10.20.30.40.50.60.70.80.91.0Share of the 1% most popular productsShare of the 1% most popular productsPopularity penalty 0.1: 37.4%Popularity penalty 0.2: 36.0%Popularity penalty 0.3: 34.5%Popularity penalty 0.4: 33.0%Popularity penalty 0.5: 31.6%Popularity penalty 0.6: 30.1%Popularity penalty 0.8: 25.9%Popularity penalty 1.0: 21.6%Popularity penalty 0.0: 38.8%38.8%0.000.100.200.00.10.20.30.40.50.60.70.80.91.0Popularity penaltyRecall@10Recall@10Popularity penalty 0.1: 0.150Popularity penalty 0.2: 0.151Popularity penalty 0.3: 0.151Popularity penalty 0.4: 0.148Popularity penalty 0.5: 0.143Popularity penalty 0.6: 0.137Popularity penalty 0.8: 0.113Popularity penalty 1.0: 0.068Popularity penalty 0.0: 0.1490.149
0%15%30%0.00.20.40.60.81.0Coverage of the products on saleCoverage of the products on salePopularity penalty 0.1: 17.4%Popularity penalty 0.2: 18.4%Popularity penalty 0.3: 20.2%Popularity penalty 0.4: 22.6%Popularity penalty 0.5: 24.7%Popularity penalty 0.6: 25.0%Popularity penalty 0.8: 20.9%Popularity penalty 1.0: 14.9%Popularity penalty 0.0: 16.5%16.5%0%25%50%0.00.20.40.60.81.0Share of the 1% most popular productsShare of the 1% most popular productsPopularity penalty 0.1: 37.4%Popularity penalty 0.2: 36.0%Popularity penalty 0.3: 34.5%Popularity penalty 0.4: 33.0%Popularity penalty 0.5: 31.6%Popularity penalty 0.6: 30.1%Popularity penalty 0.8: 25.9%Popularity penalty 1.0: 21.6%Popularity penalty 0.0: 38.8%38.8%0.000.100.200.00.20.40.60.81.0Popularity penaltyRecall@10Recall@10Popularity penalty 0.1: 0.150Popularity penalty 0.2: 0.151Popularity penalty 0.3: 0.151Popularity penalty 0.4: 0.148Popularity penalty 0.5: 0.143Popularity penalty 0.6: 0.137Popularity penalty 0.8: 0.113Popularity penalty 1.0: 0.068Popularity penalty 0.0: 0.1490.149
Tuned ALS on the test period (2,118 customers), with nine measured popularity penalties. Watch coverage rise and the share of the 1% most popular products fall, and check what happens to recall@10.
Text description of the explorer

The explorer shows measured results of tuned ALS with a popularity penalty, on the 2,118 test customers. A slider chooses one of nine penalties: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.8 and 1. Three panels show coverage, the share of places that go to the 1% most popular products, and recall@10, with the chosen penalty marked.

With no penalty: coverage 16.5%, top 1% share 38.8%, recall@10 0.149. Penalty 0.3: coverage 20.2%, top 1% share 34.5%, recall 0.151. Penalty 0.6: coverage 25.0%, the highest, top 1% share 30.1%, recall 0.137. Penalty 1: coverage 14.9%, top 1% share 21.6%, recall 0.068. The top 1% share falls at every step; coverage rises up to 0.6 and then falls.

The top 1% share falls steadily, from 38.8% with no penalty to 21.6% at 1. Coverage is highest at 0.6, 25.0%, where recall@10 has fallen from 0.149 to 0.137. A small penalty, 0.3, gives part of the coverage for no loss of recall.

The two remedies differ. A penalty is measured offline and costs no live traffic. But it only reorders products that the model already scores, so it cannot add information. Exploration adds information about products that nobody has seen, and it spends live visits to get it.

You copy the loop command and the tests of this module from the end-of-module-8 snapshot. The simulated weeks, data/online/feedback_loop.csv, are already in your data folder.

  1. In your larkfield-recs folder, with the virtual environment active, copy the command and the tests:

    Terminal window
    S=../nextia-learning-labs/C37/snapshots/end-of-m08
    cp $S/larkfield_recs/commands/loop.py larkfield_recs/commands/
    cp $S/tests/test_m08.py tests/
  2. Read the summary of the simulated loop:

    Terminal window
    recs loop
    SIMULATED on the course's behaviour model: 12 weeks, the home page learns only from its own log
    learner exploration products shown in 12 weeks week-1 products still in the top 10 revenue/visit weeks 1-4 weeks 9-12
    clicks 0% 10 10 0.88 1.31
    clicks 10% 2,994 9 0.62 0.47
    ctr 0% 10 10 0.88 1.36
    ctr 10% 2,994 3 0.33 0.14
    ctr 20% 2,994 2 0.55 0.23
  3. Show every week too:

    Terminal window
    recs loop --weeks

    After the summary, it prints one row per week and variant, 60 rows. The first rows of the clicks learner with 10% exploration:

    week week_start learner exploration visits click_rate orders_per_1000_visits revenue_per_visit products_shown_this_week products_shown_so_far top10_same_as_week1
    1 2026-04-06 clicks 0.1 2396 0.080134 20.450751 0.634186 1581 1581 10
    2 2026-04-13 clicks 0.1 2351 0.082093 19.991493 0.710561 1585 2219 9
    3 2026-04-20 clicks 0.1 2353 0.076923 17.849554 0.510234 1588 2509 9
  4. Run the tests with python -m pytest -q. You see 28 passed.

Your project now matches the snapshot end-of-m08, except its README.md. To check on macOS and Linux: diff -rq larkfield_recs ../nextia-learning-labs/C37/snapshots/end-of-m08/larkfield_recs prints only lines about __pycache__ folders.

The loop is quiet. Nothing crashes, and every week’s list looks reasonable on its own. That is why a team must watch it. Amira adds three numbers to Larkfield’s weekly report:

  • products shown this week, and the share of the products on sale;
  • products shown so far since the last release;
  • the share of places that go to the 1% most popular products.

A sudden fall in products shown is a warning. Without exploration, the simulated page showed 10 products in week 1 and never one more. In Monitor and retrain, the project’s weekly report prints coverage week by week.

Look at the output of step 2. Before you check, answer two questions. Which learner kept the most week-1 products with exploration? Which variant would you choose if Hana asks for every product to be seen, at the lowest cost in revenue?

Show the answer

The clicks learner with 10% exploration kept 9 of the 10 week-1 products. The click-rate learner kept only 3 (10%) or 2 (20%). For Hana’s goal, choose the clicks learner with 10% exploration. It shows all 2,994 products and earns 0.47 dollars per visit in weeks 9 to 12. That is the most of the three variants with exploration. It is still much less than 1.31 without exploration. So the honest answer to Hana is a price. In this simulation, showing every product costs about 0.84 dollars per visit.

Explore when the list depends on its own log. Examples are a home page that learns from clicks, or a “trending” list built from suggested orders. Keep the share small, fixed and measured, and give the random products places that are not the top place.

Do not explore with sensitive or age-restricted products, or on surfaces where one bad suggestion is costly, such as a family’s shared email. The business rules of Module 7 still apply to random products.

The costs, in Larkfield’s simulated case: with 10% exploration, revenue per visit in weeks 9 to 12 fell from 1.31 to 0.47 dollars. In return, every product was shown, and the log now has data about all of them.

Common mistake

'The list is stable, so the model is good'

Symptom: Tomás sees that the home page has shown the same 10 products for 12 weeks. He says that the model has found the best products.

Cause: the model only learns from what it showed. In the simulation, the same 10 products stayed because no other product could get a click, not because they were the best.

Fix: watch coverage over time beside revenue. Ten products of 2,994 for 12 weeks is a loop, not a result.

Common mistake

FAILED tests/test_m08.py::test_loop

Symptom: after copying only the tests, python -m pytest -q ends with:

FAILED tests/test_m08.py::test_loop - SystemExit: 2
1 failed, 27 passed in 0.75s

Above it, the captured output says recs: error: argument COMMAND: invalid choice: 'loop'. You see this in a project with online.py and the tests, but without loop.py.

Cause: tests/test_m08.py tests both commands of this module, online and loop. Here, loop.py is missing.

Fix: copy larkfield_recs/commands/loop.py as in step 1, then run the tests again.

Check your understanding

In the simulation, the home page without exploration showed the same 10 products for 12 weeks. Explain why with the feedback loop.

Show the answer

The page learned only from clicks on what it showed. In week 1, it showed March’s 10 bestsellers. Only those 10 could get clicks, so they were again the products with the most clicks, and they were shown again. No other product could ever get a click, so none could enter. The loop is closed: 10 of 2,994 products for 12 weeks.

Check your understanding

With 10% exploration, all 2,994 products were shown, but revenue per visit in weeks 9 to 12 fell from 1.31 to 0.47 dollars. Is exploration a mistake? Give one reason for it and one against it.

Show the answer

Not necessarily: it is a trade-off with a measured price. For it: every product gets a chance to be seen. The log gets data about products that the page never showed, so the learner can find good ones that the loop hid. Against it: in this simulation, it cost about 0.84 dollars per visit in weeks 9 to 12. The random places mostly went to products that rarely sell. A team chooses the share of exploration by what the information and the coverage are worth to them.

Check your understanding

In your project, run recs loop. Which row shows that the click-rate learner changed the home page much more than the clicks learner, and what is the number?

Show the answer

The row ctr 10%: the column “week-1 products still in the top 10” is 3, against 9 for clicks 10%. With the same 10% exploration, the click-rate learner replaced 7 of the week-1 products by week 12 (and 8 after one week). Its revenue per visit in weeks 9 to 12 was the lowest, 0.14 dollars.

Recap

  • A feedback loop: the recommender’s suggestions decide its clicks, and its clicks decide its next suggestions. Only shown products can get clicks.
  • In Larkfield’s simulation, a page that learns only from its own clicks showed the same 10 products for 12 weeks.
  • Exploration (10%: 1 random place of 10) showed all 2,994 products. But revenue per visit in weeks 9 to 12 fell from 1.31 to 0.47 dollars (0.14 for the click-rate learner). It buys information and coverage at a measured price.
  • A popularity penalty is a second remedy, measured offline. At 0.3, tuned ALS has coverage 20.2% against 16.5%, with no loss of recall.
  • Monitor coverage over time: products shown each week, products shown so far, and the share of the 1% most popular products.

You can now judge a recommender offline, online and over time. Module 9 takes it into production. First, you choose between lists computed every night and lists computed at each request, and you see what each one costs.

Next: Batch or real time