Skip to content
Français

Novelty and learning effects

  • Lesson 15
  • 30 min
  • Module 4: Traps in real tests

Where we are

We can now control false winners over time and over metrics. Some effects are real but do not last. In this lesson, we look at a new widget that wins for two weeks and then loses, and we measure the effect by week and by first exposure.

What you will be able to do

Tell a lasting effect from one that fades. A change can win because it is new, then lose when customers get used to it. Look at the effect by week since each customer first saw the change, and you see where it is going.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate a 6-week test of a widget that wins at first and then loses. You compute the effect by calendar week and by week since the first visit, and compare both with the truth. Its setup cells install the packages, so it needs no local setup. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the project steps)

  • The project larkfield-experiments at the snapshot end-of-m04, from Peeking, with .venv active.

Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.

A café puts up a big, colourful menu board. In the first week, customers stop and read it. Many try a new drink. Sales go up. In the third week, the regular customers no longer look at it. They order their usual coffee, and the long queue in front of the board slows everyone down. Sales are now a little lower than before.

If the owner had judged the board after one week, he would keep it. After two months, he would remove it. The board did not change. The customers changed: it was new, and then it was not.

A novelty effect is a reaction to something because it is new. It is strong at first and fades as people get used to it. The café’s board is a novelty effect.

A learning effect goes the other way. People need time to learn a new thing, so the effect starts low, or even negative, and grows. A new checkout flow can annoy customers in the first week (they look for the old button), and help them a month later. The first weeks’ dislike of any change is also called change aversion.

Both have the same problem: the effect in the test’s first weeks is not the effect after the launch. And the tools to see them are the same.

The problem: a widget that wins, then loses

Section titled “The problem: a widget that wins, then loses”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a new “bought together” widget on the product page for 6 weeks. The simulator has this test as a scenario. Its novelty effect is a practice problem we made on purpose. The widget raises the chance to order in a customer’s first days with it. Later, it lowers that chance.

  1. Run the widget test.

    Terminal window
    sim run --scenario novelty
    Scenario novelty (seed 40): A new 'bought together' widget, 6 weeks: it wins at first, then loses (planted novelty effect).
    Wrote runs/novelty-s40: customers (75,291 rows), visits (326,517 rows). Synthetic data. 8.2 s
  2. Analyse the whole test, and compare it with the truth.

    Terminal window
    xp analyse
    xp check
    Run novelty-s40 (synthetic): A new 'bought together' widget, 6 weeks: it wins at first, then loses (planted novelty effect).
    Metric: revenue. A: 37,425 customers, mean 34.3121. B: 37,866 customers, mean 34.6506.
    difference +0.3384 (+1.0%), 95% interval -0.9879 to +1.6648, p = 0.617 (Welch t-test)
    Saved to runs/novelty-s40/analysis.json. Compare with the true effect: xp check
    Run novelty-s40: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.2754 dollars per customer.
    Your estimate: +0.3384, 95% interval -0.9879 to +1.6648: the interval contains the true effect.

Over 6 weeks, the widget adds +0.34 dollars per customer, and the truth is +0.28. The estimate is honest: over 100 seeds, its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contained the truth in 94% of the runs. But this is the effect of these 6 weeks. It mixes the good first weeks with the bad later ones. It does not say what the widget does in month 3.

xp weekly splits the test by week, in two ways. The first is the calendar week of the test: week 1 is 8 to 14 September. The second is the week since the first visit: week 1 is each customer’s own first week with the widget, whenever it started.

Terminal window
xp weekly
Revenue per visit, B − A, by calendar week:
1: +11.7% (+0.359 to +1.486) +#######################
2: +10.9% (+0.272 to +1.380) +#####################
3: +2.3% (-0.369 to +0.731) +####
4: -3.3% (-0.787 to +0.277) -######
5: -9.1% (-1.295 to -0.227) -##################
6: -2.4% (-0.720 to +0.339) -####
Revenue per visit, B − A, by week since first visit:
1: +8.4% (+0.253 to +0.982) +################
2: +3.9% (-0.268 to +0.878) +#######
3: +0.5% (-0.534 to +0.619) +#
4: -7.7% (-1.256 to -0.053) -###############
5: -7.1% (-1.249 to +0.024) -##############
6: +0.6% (-0.772 to +0.871) +#

The numbers in brackets are 95% intervals of the difference in dollars per visit. A team that stopped after two weeks would report “+11%”. By weeks 4 to 6, the widget loses money.

In calendar week 1, every customer is new to the widget: nobody saw it before the test started. In week 2, some customers are in their first week and some in their second. By week 6, most visits come from customers who know the widget well.

A grid with calendar weeks 1 to 6 as columns and four rows of customers: those who first saw the widget in week 1, 2, 3 and 4. Each cell says which week it is for those customers: 1st, 2nd, 3rd and so on. The 1st-week cells are dark blue and lie on a diagonal. Below the grid, the share of visits that are in the customer's first week: 100% in calendar week 1, then 56%, 27%, 17%, 12% and 9%.
Calendar week against week since the first visit. Calendar week 1 contains only first-week visits. Later calendar weeks mix customers at different stages, so the calendar curve shows a changing crowd as much as a changing effect.

So the calendar curve mixes two things: how the effect changes for a customer, and who is in the shop that week. The curve by week since the first visit compares customers at the same stage. It shows the effect’s path for one customer, and its end shows where the effect is going.

The simulator knows the true effect by week since the first visit. The chart puts the two estimates and the truth together.

  • Estimate by calendar week
  • Estimate by week since the first visit
  • Truth by week since the first visit (simulator)
The widget's effect on revenue per visit by week. The true effect (orange) is +6.7% in a customer's first week with the widget, then negative and settling near −2.5%. The calendar-week estimate starts higher, because calendar weeks 1 and 2 are full of first-week customers.
Text description of the chart

A line chart over weeks 1 to 6, from −12% to +15%. The truth by week since the first visit: +6.7%, −0.7%, −2.1%, −3.2%, −2.6%, −2.5%. The estimate by week since the first visit: +8.4%, +3.9%, +0.5%, −7.7%, −7.1%, +0.6%. The estimate by calendar week: +11.7%, +10.9%, +2.3%, −3.3%, −9.1%, −2.4%. Both estimates are noisy around the truth; their 95% intervals are 10 to 20 points wide.

Compare your estimates with the truth:

Terminal window
xp check
Run novelty-s40: the last analysis was `xp weekly`.
Week since first visit: your estimate against the true effect (revenue per visit):
1: yours +8.4%, true +6.7%
2: yours +3.9%, true -0.7%
3: yours +0.5%, true -2.1%
4: yours -7.7%, true -3.2%
5: yours -7.1%, true -2.6%
6: yours +0.6%, true -2.5%

Each weekly estimate is noisy: its interval is 10 to 20 points wide. Read the direction over several weeks, not one week’s number. Here the direction is clear: positive in week 1, negative from week 3. The long-run effect is about −2.5% of revenue per visit.

Hana, the merchandising lead, asks: “So how long must a test run?” There is no single answer, but there are four habits.

  1. Run whole weeks, and more than you need for power. Two weeks is too short to see a novelty fade. For a visible change, plan three to six weeks.
  2. Look by week since the first exposure. xp weekly gives this table. If the effect falls week after week, the last weeks are the best guess of the future.
  3. Look at new customers. They never knew the old page, so they show no change aversion and less surprise at the new one. Their effect is closer to the long-run effect for future customers. But they are only a part of the shop.
  4. Keep a holdout after the launch. A holdout is a small group, for example 5% of customers, who do not get the change for some months. Comparing them with everyone else shows the long-run effect.

Each habit costs something. Longer tests delay every decision. A holdout keeps a few customers on the old version, and someone must watch it. Use them for changes that customers see and use every visit, such as a widget or a new layout. A change that customers do not notice, such as a faster page, rarely has a novelty effect.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell.
  3. In section 1, predict when the effect becomes negative. Then print the curve. It crosses zero at about day 7.
  4. In section 2, simulate the 6-week test. You see a small positive estimate over the 6 weeks.
  5. In section 3, compute the effect by calendar week. Week 1 has only first-week visits.
  6. In section 4, compute the table by week since the first visit, and the mean of the truth for weeks 3 to 6.
  7. Run the check cell. It prints Check passed.
Show the answer to step 6
table = by("week_since_first")
long_run = round(float(table.loc[3:6, "truth"].mean()), 3)
print(table)
print(long_run)

The truth is positive only in week 1 since the first visit. The mean for weeks 3 to 6 is the long-run effect: −0.026, about −2.6%.

TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. reads the xp weekly output above. He asks: “In which week since the first visit does the estimate first turn negative? Does the truth turn in the same week?” Answer from the output, without a new run.

Show the answer

The estimate by week since the first visit is first negative in week 4 (−7.7%). The truth is negative already in week 2 (−0.7%), and clearly in week 3 (−2.1%). The estimates of weeks 2 and 3 (+3.9% and +0.5%) are above the truth, by noise: their intervals contain the truth. This is why one week’s number is not enough. The trend over weeks 1 to 4 shows the fall early: +8.4%, +3.9%, +0.5%, −7.7%.

Common mistake

“It won by 11% in two weeks”

Symptom: the team runs the widget test for two weeks and sees +11.7% and +10.9% on revenue per visit. They ship the widget. Three months later, revenue per visit is about 2.5% lower than before the launch.

Cause: a novelty effect. In the first two calendar weeks, most visits come from customers who are new to the widget, and new things get clicks. The effect fades and turns negative once customers get used to it. A two-week test measures the novelty, not the widget.

Fix: run the test for several weeks, and read the effect by week since the first visit (xp weekly). If it falls week after week, trust the last weeks. After a launch, keep a small holdout to check the long-run effect.

Check your understanding

Why is every visit in calendar week 1 of a test also in its customer's first week with the change? What does this mean for the calendar-week curve?

Show the answer

Before the test started, nobody had seen the change. So in calendar week 1, every customer is in their first week with it. Later calendar weeks mix new customers and customers who know the change. The calendar-week curve therefore shows both how the effect changes for a customer and how the crowd changes. The curve by week since the first visit compares customers at the same stage.

Check your understanding

A new checkout flow shows −3% on orders in a customer's first week with it, −1% in the second week, +1% in the third and +2% in the fourth. What kind of effect is this, and what would you recommend?

Show the answer

This is a learning effect, with change aversion at the start: customers need time to learn the new flow, and then it helps. A short test would reject a good change. Recommend a longer test: run it until the effect by week since the first exposure stops changing. Also check new customers, who have no old habit to unlearn. If the later weeks stay positive, the flow is a good change.

Check your understanding

The 6-week widget test gives +0.34 dollars per customer, and its interval contains the true 6-week effect (+0.28). Hana says: “The interval is right, so the widget is good.” What is missing?

Show the answer

The interval is honest for the 6-week average, but that average mixes the strong first weeks with the weaker later ones. After the launch, most customers will have seen the widget for a long time, so the long-run effect matters. By week since the first visit, the true effect settles near −2.5% of revenue per visit. The widget wins at first and then loses money.

Recap

  • A novelty effect is strong at first and fades. A learning effect starts low and grows; change aversion is the first weeks’ dislike of any change.
  • The widget test (a practice problem we made on purpose) showed +11.7% and +10.9% in its first two calendar weeks. The true effect by week since the first visit was +6.7% in week 1, then near −2.5%.
  • The 6-week average (+0.34 dollars against a true +0.28) is honest for those 6 weeks. It is not the long-run effect.
  • Calendar weeks mislead: week 1 contains only first-week customers. Read the effect by week since the first exposure with xp weekly.
  • Run tests for several weeks, look at new customers, and keep a holdout after the launch.

So far, one customer’s version did not change another customer’s result. Next, a test where it does: shared stock.

Next: When units affect each other