Novelty and learning effects
- Lesson 15
- 30 min
- Module 4: Traps in real tests
Where we are
We can now control false winners over time and over metrics. Some effects are real but do not last. In this lesson, we look at a new widget that wins for two weeks and then loses, and we measure the effect by week and by first exposure.
Lesson 15 of 30 · Experiments and Causal Impact: Did Our Change Work?Novelty and learning effects
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Tell a lasting effect from one that fades. A change can win because it is new, then lose when customers get used to it. Look at the effect by week since each customer first saw the change, and you see where it is going.
NotebookDo this lesson in a notebook
In the notebook, you simulate a 6-week test of a widget that wins at first and then loses. You compute the effect by calendar week and by week since the first visit, and compare both with the truth. Its setup cells install the packages, so it needs no local setup. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Peeking, from Peeking.
- The effect in segments, from Segments and different effects.
- Why a test has a planned length, from Sample size and power.
What you need on your computer (only for the project steps)
- The project
larkfield-experimentsat the snapshotend-of-m04, from Peeking, with.venvactive.
Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.
A new menu board at the café
Section titled “A new menu board at the café”A café puts up a big, colourful menu board. In the first week, customers stop and read it. Many try a new drink. Sales go up. In the third week, the regular customers no longer look at it. They order their usual coffee, and the long queue in front of the board slows everyone down. Sales are now a little lower than before.
If the owner had judged the board after one week, he would keep it. After two months, he would remove it. The board did not change. The customers changed: it was new, and then it was not.
Two effects that change with time
Section titled “Two effects that change with time”A novelty effect is a reaction to something because it is new. It is strong at first and fades as people get used to it. The café’s board is a novelty effect.
A learning effect goes the other way. People need time to learn a new thing, so the effect starts low, or even negative, and grows. A new checkout flow can annoy customers in the first week (they look for the old button), and help them a month later. The first weeks’ dislike of any change is also called change aversion.
Both have the same problem: the effect in the test’s first weeks is not the effect after the launch. And the tools to see them are the same.
The problem: a widget that wins, then loses
Section titled “The problem: a widget that wins, then loses”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a new “bought together” widget on the product page for 6 weeks. The simulator has this test as a scenario. Its novelty effect is a practice problem we made on purpose. The widget raises the chance to order in a customer’s first days with it. Later, it lowers that chance.
-
Run the widget test.
Terminal window sim run --scenario noveltyScenario novelty (seed 40): A new 'bought together' widget, 6 weeks: it wins at first, then loses (planted novelty effect).Wrote runs/novelty-s40: customers (75,291 rows), visits (326,517 rows). Synthetic data. 8.2 s -
Analyse the whole test, and compare it with the truth.
Terminal window xp analysexp checkRun novelty-s40 (synthetic): A new 'bought together' widget, 6 weeks: it wins at first, then loses (planted novelty effect).Metric: revenue. A: 37,425 customers, mean 34.3121. B: 37,866 customers, mean 34.6506.difference +0.3384 (+1.0%), 95% interval -0.9879 to +1.6648, p = 0.617 (Welch t-test)Saved to runs/novelty-s40/analysis.json. Compare with the true effect: xp checkRun novelty-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.2754 dollars per customer.Your estimate: +0.3384, 95% interval -0.9879 to +1.6648: the interval contains the true effect.
Over 6 weeks, the widget adds +0.34 dollars per customer, and the truth is +0.28. The estimate is honest: over 100 seeds, its TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contained the truth in 94% of the runs. But this is the effect of these 6 weeks. It mixes the good first weeks with the bad later ones. It does not say what the widget does in month 3.
The effect week by week
Section titled “The effect week by week”xp weekly splits the test by week, in two ways. The first is the calendar week of the test: week 1 is 8 to 14 September. The second is the week since the first visit: week 1 is each customer’s own first week with the widget, whenever it started.
xp weeklyRevenue per visit, B − A, by calendar week: 1: +11.7% (+0.359 to +1.486) +####################### 2: +10.9% (+0.272 to +1.380) +##################### 3: +2.3% (-0.369 to +0.731) +#### 4: -3.3% (-0.787 to +0.277) -###### 5: -9.1% (-1.295 to -0.227) -################## 6: -2.4% (-0.720 to +0.339) -####Revenue per visit, B − A, by week since first visit: 1: +8.4% (+0.253 to +0.982) +################ 2: +3.9% (-0.268 to +0.878) +####### 3: +0.5% (-0.534 to +0.619) +# 4: -7.7% (-1.256 to -0.053) -############### 5: -7.1% (-1.249 to +0.024) -############## 6: +0.6% (-0.772 to +0.871) +#The numbers in brackets are 95% intervals of the difference in dollars per visit. A team that stopped after two weeks would report “+11%”. By weeks 4 to 6, the widget loses money.
Why calendar weeks mislead
Section titled “Why calendar weeks mislead”In calendar week 1, every customer is new to the widget: nobody saw it before the test started. In week 2, some customers are in their first week and some in their second. By week 6, most visits come from customers who know the widget well.
So the calendar curve mixes two things: how the effect changes for a customer, and who is in the shop that week. The curve by week since the first visit compares customers at the same stage. It shows the effect’s path for one customer, and its end shows where the effect is going.
The simulator knows the true effect by week since the first visit. The chart puts the two estimates and the truth together.
- Estimate by calendar week
- Estimate by week since the first visit
- Truth by week since the first visit (simulator)
Text description of the chart
A line chart over weeks 1 to 6, from −12% to +15%. The truth by week since the first visit: +6.7%, −0.7%, −2.1%, −3.2%, −2.6%, −2.5%. The estimate by week since the first visit: +8.4%, +3.9%, +0.5%, −7.7%, −7.1%, +0.6%. The estimate by calendar week: +11.7%, +10.9%, +2.3%, −3.3%, −9.1%, −2.4%. Both estimates are noisy around the truth; their 95% intervals are 10 to 20 points wide.
Compare your estimates with the truth:
xp checkRun novelty-s40: the last analysis was `xp weekly`.Week since first visit: your estimate against the true effect (revenue per visit): 1: yours +8.4%, true +6.7% 2: yours +3.9%, true -0.7% 3: yours +0.5%, true -2.1% 4: yours -7.7%, true -3.2% 5: yours -7.1%, true -2.6% 6: yours +0.6%, true -2.5%Each weekly estimate is noisy: its interval is 10 to 20 points wide. Read the direction over several weeks, not one week’s number. Here the direction is clear: positive in week 1, negative from week 3. The long-run effect is about −2.5% of revenue per visit.
What to do about it
Section titled “What to do about it”Hana, the merchandising lead, asks: “So how long must a test run?” There is no single answer, but there are four habits.
- Run whole weeks, and more than you need for power. Two weeks is too short to see a novelty fade. For a visible change, plan three to six weeks.
- Look by week since the first exposure.
xp weeklygives this table. If the effect falls week after week, the last weeks are the best guess of the future. - Look at new customers. They never knew the old page, so they show no change aversion and less surprise at the new one. Their effect is closer to the long-run effect for future customers. But they are only a part of the shop.
- Keep a holdout after the launch. A holdout is a small group, for example 5% of customers, who do not get the change for some months. Comparing them with everyone else shows the long-run effect.
Each habit costs something. Longer tests delay every decision. A holdout keeps a few customers on the old version, and someone must watch it. Use them for changes that customers see and use every visit, such as a widget or a new layout. A change that customers do not notice, such as a faster page, rarely has a novelty effect.
Guided practice: see the novelty curve
Section titled “Guided practice: see the novelty curve”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell.
- In section 1, predict when the effect becomes negative. Then print the curve. It crosses zero at about day 7.
- In section 2, simulate the 6-week test. You see a small positive estimate over the 6 weeks.
- In section 3, compute the effect by calendar week. Week 1 has only first-week visits.
- In section 4, compute the table by week since the first visit, and the mean of the truth for weeks 3 to 6.
- Run the check cell. It prints
Check passed.
Show the answer to step 6
table = by("week_since_first")long_run = round(float(table.loc[3:6, "truth"].mean()), 3)print(table)print(long_run)The truth is positive only in week 1 since the first visit. The mean for weeks 3 to 6 is the long-run effect: −0.026, about −2.6%.
Your turn: when does it turn?
Section titled “Your turn: when does it turn?”TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. reads the xp weekly output above. He asks: “In which week since the first visit does the estimate first turn negative? Does the truth turn in the same week?” Answer from the output, without a new run.
Show the answer
The estimate by week since the first visit is first negative in week 4 (−7.7%). The truth is negative already in week 2 (−0.7%), and clearly in week 3 (−2.1%). The estimates of weeks 2 and 3 (+3.9% and +0.5%) are above the truth, by noise: their intervals contain the truth. This is why one week’s number is not enough. The trend over weeks 1 to 4 shows the fall early: +8.4%, +3.9%, +0.5%, −7.7%.
Common mistake
“It won by 11% in two weeks”
Symptom: the team runs the widget test for two weeks and sees +11.7% and +10.9% on revenue per visit. They ship the widget. Three months later, revenue per visit is about 2.5% lower than before the launch.
Cause: a novelty effect. In the first two calendar weeks, most visits come from customers who are new to the widget, and new things get clicks. The effect fades and turns negative once customers get used to it. A two-week test measures the novelty, not the widget.
Fix: run the test for several weeks, and read the effect by week since the first visit (xp weekly). If it falls week after week, trust the last weeks. After a launch, keep a small holdout to check the long-run effect.
Check your understanding
Why is every visit in calendar week 1 of a test also in its customer's first week with the change? What does this mean for the calendar-week curve?
Show the answer
Before the test started, nobody had seen the change. So in calendar week 1, every customer is in their first week with it. Later calendar weeks mix new customers and customers who know the change. The calendar-week curve therefore shows both how the effect changes for a customer and how the crowd changes. The curve by week since the first visit compares customers at the same stage.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A new checkout flow shows −3% on orders in a customer's first week with it, −1% in the second week, +1% in the third and +2% in the fourth. What kind of effect is this, and what would you recommend?
Show the answer
This is a learning effect, with change aversion at the start: customers need time to learn the new flow, and then it helps. A short test would reject a good change. Recommend a longer test: run it until the effect by week since the first exposure stops changing. Also check new customers, who have no old habit to unlearn. If the later weeks stay positive, the flow is a good change.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The 6-week widget test gives +0.34 dollars per customer, and its interval contains the true 6-week effect (+0.28). Hana says: “The interval is right, so the widget is good.” What is missing?
Show the answer
The interval is honest for the 6-week average, but that average mixes the strong first weeks with the weaker later ones. After the launch, most customers will have seen the widget for a long time, so the long-run effect matters. By week since the first visit, the true effect settles near −2.5% of revenue per visit. The widget wins at first and then loses money.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A novelty effect is strong at first and fades. A learning effect starts low and grows; change aversion is the first weeks’ dislike of any change.
- The widget test (a practice problem we made on purpose) showed +11.7% and +10.9% in its first two calendar weeks. The true effect by week since the first visit was +6.7% in week 1, then near −2.5%.
- The 6-week average (+0.34 dollars against a true +0.28) is honest for those 6 weeks. It is not the long-run effect.
- Calendar weeks mislead: week 1 contains only first-week customers. Read the effect by week since the first exposure with
xp weekly. - Run tests for several weeks, look at new customers, and keep a holdout after the launch.
So far, one customer’s version did not change another customer’s result. Next, a test where it does: shared stock.