Skip to content
Français

The feature that "raised sales"

  • Lesson 1
  • 25 min
  • Module 1: Why "it went up" is not an answer

Where we are

In the mathematics course, we learned to compute a confidence interval for a difference. Larkfield now has a harder question. Omar says that the new recommendation block raised sales by 8%. In this lesson, we look at that number, at what else happened in the same weeks, and at how far the number is from the true effect. First, think of a shop with a new window display.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Learn to distrust a number that went up after a change. A rise after a launch mixes the change with everything else that happened in the same weeks, and the dashboard cannot tell them apart.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

What you need on your computer

Nothing. Keep a pen and paper ready for the practice.

Think of a small gift shop. On Monday, the owner puts a new display in the window. That week, 20% more customers come in. “The display works,” she says.

But that week was also the week before a holiday. People were buying presents. And the café next door had a street festival on Saturday. Any of these could bring more customers.

The owner wants to know one thing: how many more customers came because of the display? To answer, she needs the same week with the old display. It must have the same holiday and the same festival. That week does not exist. She lived only one version of it.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield has the same problem, with more money at stake.

Larkfield is an online shop for home and garden products. It sells in Canada, so prices are in Canadian dollars. Everything about Larkfield in this course is synthetic: our simulator made the shop, its customers and their orders. There are no real people.

In this course, Larkfield wants to know if four changes worked. They are a lower threshold for free delivery, a new recommendation block on the product pages, a reminder email, and a new prompt for the support assistant. You meet the people who own these changes:

  • TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses., a data scientist, is your guide. She designs and analyses the tests.
  • TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., a product manager, owns the recommendation block and its decision.
  • TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses., a developer, owns the code that assigns customers and logs what they see.
  • TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., an AI engineer, asks the questions that many of us ask at first.
  • Hana owns the free-delivery threshold, Rafael owns the reminder email, and Mateo keeps the list of past tests.

The recommendation block shows “customers also bought” products under each product. It went live for every customer on Monday 6 October 2025. Four weeks later, Omar opens the sales dashboard and compares two numbers:

Weeks Dates Sales
4 weeks before the launch 8 September to 5 October 2025 3,877,532 dollars
4 weeks after the launch 6 October to 2 November 2025 4,199,656 dollars

4,199,656 ÷ 3,877,532 = 1.083. Sales rose by 8.3%. Omar writes to the team: “The block raised sales by 8%. Let us add blocks to the cart page too.”

This comparison has a name: a before-and-after comparison. You take a number before a change and the same number after it, and you call the difference the effect of the change. It is fast, and every dashboard can do it. In this course, we call it a naive answer: an answer that looks at the data and ignores everything else that could explain it.

This number is also a practice problem we made on purpose. We set the size of the sale in the simulator so that the before-and-after rise is close to Omar’s 8%. Everything else in the scenario follows from the simulator’s rules.

Priya opens the calendar before she opens a spreadsheet. The four weeks after the launch were not ordinary weeks:

  • Thanksgiving was on Monday 13 October. In the simulated shop, the five shopping days around it get 12% more orders.
  • The autumn sale ran from 9 to 19 October: 15% off garden furniture and decor. During the sale, 11% more visits end in an order, and baskets are 4% smaller.
  • Slow growth. Larkfield gets a few more customers every month, so any later period tends to sell a little more.
  • Chance. Every day is a little higher or lower than expected, for no reason anybody can name.
Five boxes on the left point to one box on the right. The boxes are: the recommendation block, live from 6 October, with an arrow labelled +2.0%; Thanksgiving on 13 October, with more orders on 5 shopping days; the autumn sale from 9 to 19 October, with more orders and smaller baskets; slow growth, a few more customers every month; and chance, with a dotted arrow, because each day is a little higher or lower. The box on the right is the sales in the 4 weeks after the launch: +8.3% against the 4 weeks before.
One number on the dashboard, five causes behind it. Only one of them is the block.
Text description of the diagram

The dashboard shows one number, +8.3%. Five things pushed it: the block, Thanksgiving, the autumn sale, slow growth and chance. The arrow from the block carries +2.0%, the true effect that the simulator knows. The other arrows carry the rest. The dashboard adds them all up and shows only the total.

So the 8.3% is the sum of several causes. The block is one of them. The dashboard has no way to split the sum.

The problem: the answer depends on the window

Section titled “The problem: the answer depends on the window”

If the 8.3% were the block’s effect, it should not depend much on how many weeks we compare. Priya tries other windows, with the same number of weeks on each side of the launch:

Weeks before and after Before-and-after change
1 week +14.6%
2 weeks +12.6%
3 weeks +9.1%
4 weeks +8.3%

The answer falls from +14.6% to +8.3%. Why? The first week after the launch holds Thanksgiving and the first days of the sale. A short window after the launch is mostly holiday and sale, so it shows a big rise. A longer window adds normal weeks, so the rise shrinks.

This is the first warning sign of a naive answer: change a choice that should not matter, and the answer moves. The block did not get weaker over four weeks. The comparison changed.

Tomás has an idea: “Why not use a longer window? Eight weeks before and four after.” That gives +8.7%. It is still far from the truth, as you will see. A longer window does not remove the holiday or the sale. It only adds more weeks with other things in them.

The idea: compare with the world without the change

Section titled “The idea: compare with the world without the change”

The question “did the block raise sales?” really asks this: how much would Larkfield have sold in the same four weeks without the block? The effect is the difference between what happened and that other world.

A real shop never sees the other world, like the gift shop owner. But our Larkfield lives in a simulator: a program that makes the shop’s data from known rules. The simulator ran the same four weeks twice: once with the block, and once without it. Both runs had the same customers, the same holiday, the same sale and the same chance. Only the block was different.

So we know the answer that real teams never know:

Sales Dollars
4 weeks before the launch 3,877,532
4 weeks after, without the block (simulated) 4,117,094
4 weeks after, with the block (what happened) 4,199,656

Read the table from the top:

  1. Without the block, sales would still have risen: 4,117,094 ÷ 3,877,532 = 1.062, so +6.2%. That is the holiday, the sale, the growth and chance.
  2. With the block, sales were 4,199,656. Against the world without the block: 4,199,656 ÷ 4,117,094 = 1.020, so +2.0%. That is the block: 82,562 dollars, or about 1,000 extra orders in four weeks.
  3. Omar’s +8.3% combines both rises: 1.062 × 1.020 = 1.083.
The block is the small step between the last two bars. Most of the rise would have happened anyway. Synthetic data from the simulator.
Text description of the chart

Three bars, in million dollars, on an axis that starts at 3.6 so the differences show. The 4 weeks before the launch: 3.88. The 4 weeks after the launch without the block, which only the simulator knows: 4.12. The 4 weeks after with the block, what really happened: 4.20. The step from the first bar to the second, +6.2%, is the holiday, the sale, growth and chance. The step from the second bar to the third, +2.0%, is the block.

The axis of this chart starts at 3.6 million, not at zero. This makes the steps visible, but it also makes them look larger than they are. Read the numbers, not only the heights.

The block’s true effect is about a quarter of the rise on the dashboard. Omar’s number is about four times too large. The block is useful: 82,562 dollars in four weeks is real money. But the plan to add blocks everywhere was built on 8.3%, not on 2.0%.

The explorer below shows the daily sales around the launch. The shaded parts are the two windows: before the launch and after it. The black lines show the average sales per day in each window. The naive change compares those two averages. Short bars above the chart mark the holiday days and the sale days.

Explore

The naive answer moves, the true effect stays

Average per day before
138,483
Average per day after
149,988
Naive change (after against before)
+8.3%
True effect (simulator)
+2.0%
Naive minus truth
+6.3 points

Average sales per day: 138,483 dollars before, 149,988 dollars after. The naive change is +8.3%. Without the block, the same 4 weeks after the change would have sold 4,117,094 dollars. The block added +2.0% (+82,562 dollars). The naive answer is 6.3 points too high.

  • Sales (with the block)
  • Sales without the block (simulator)
  • holiday
  • sale

Shaded: the two windows; the black lines are their averages

4 weeks (8 Sep to 5 Oct 2025)4 weeks (6 Oct to 2 Nov 2025)100120140160180200Sales per day (thousand dollars)11 Aug25 Aug8 Sep22 Sep6 Oct20 Octholiday: 1 Sep 2025holiday: 9 Oct 2025holiday: 10 Oct 2025holiday: 11 Oct 2025holiday: 12 Oct 2025holiday: 13 Oct 2025sale: 9 Oct 2025sale: 10 Oct 2025sale: 11 Oct 2025sale: 12 Oct 2025sale: 13 Oct 2025sale: 14 Oct 2025sale: 15 Oct 2025sale: 16 Oct 2025sale: 17 Oct 2025sale: 18 Oct 2025sale: 19 Oct 2025launch 6 OctSales without the block (simulator)Sales (with the block)Average per period before: 138,483Average per period after: 149,988
4 weeks (8 Sep to 5 Oct 2025)4 weeks (6 Oct to 2 Nov 2025)100120140160180200Thousand $ a day11 Aug8 Sep6 Octholiday: 1 Sep 2025holiday: 9 Oct 2025holiday: 10 Oct 2025holiday: 11 Oct 2025holiday: 12 Oct 2025holiday: 13 Oct 2025sale: 9 Oct 2025sale: 10 Oct 2025sale: 11 Oct 2025sale: 12 Oct 2025sale: 13 Oct 2025sale: 14 Oct 2025sale: 15 Oct 2025sale: 16 Oct 2025sale: 17 Oct 2025sale: 18 Oct 2025sale: 19 Oct 2025launch 6 OctSales without the block (simulator)Sales (with the block)Average per period before: 138,483Average per period after: 149,988
Daily sales of the simulated Larkfield around the launch of the recommendation block. Synthetic data, seed 40.
Text description of the explorer

The chart shows daily sales from 11 August to 2 November 2025, in thousand dollars per day. A vertical line marks the launch on 6 October. The window before the launch and the window after it are shaded, and a black line shows the average of each window. With 4 weeks on each side, the average is 138,483 dollars a day before and 149,988 after: a naive change of +8.3%. The days of Thanksgiving (9 to 13 October) and of the autumn sale (9 to 19 October) are marked above the chart. They all fall in the window after the launch. One more holiday mark, Labour Day on 1 September, enters the window before the launch only when it is 5 weeks or longer. The button shows a dashed line: the sales of the same days without the block, from the simulator. It runs just under the real sales, about 2% lower. The true effect is +2.0%, 82,562 dollars in the 4 weeks, so the naive answer is 6.3 points too high. With 1 week on each side, the naive change is +14.6%; with 2 weeks, +12.6%; with 3 weeks, +9.1%. The true effect stays at +2.0% for every window after the launch.

Try these three things:

  1. Set both windows to 1 week. The naive change jumps to +14.6%, because the week after the launch holds Thanksgiving and the start of the sale.
  2. Show the world without the block. The dashed line is almost on top of the real sales. The gap between them is the block’s effect.
  3. Move the windows again with the dashed line on. The naive change moves a lot. The true effect stays at +2.0%.

The true effect does not depend on your windows, because it compares the same days in two worlds. The naive answer compares different days, and different days differ for many reasons.

A real data team never sees the dashed line. That is exactly why it is hard to learn this subject from real data: you can compute an answer, but you cannot check it.

Our simulator removes that problem. For every scenario in this course, it knows the true effect, because it ran the world both ways. So every method that you learn gets the same treatment: you compute an estimate, and then you compare it with the truth. Some methods land close. Some, like the before-and-after comparison, land far away. A few look careful and still miss, and we show those too.

Remember what the simulator gives you and what it does not. It gives you a truth to check against. It does not make Larkfield real: real customers are messier, and real data has no answer key. The methods are the same; only the checking is easier.

Omar asks, “So is the dashboard always wrong?” Not always.

A before-and-after comparison works when nothing else changed in the two periods. The change must also be large next to the normal ups and downs. Suppose the checkout page was broken for a day and orders fell to almost zero. After the fix, orders go back to normal. Nobody needs a test to say that the fix worked.

It fails when the effect is small next to other causes. Holidays, sales, prices, weather, marketing emails and slow growth all move sales by more than 2%. Most product changes have small effects. So most before-and-after numbers for product changes mix the change with something larger.

It is cheap to compute. It is expensive when it is wrong. A team that believes +8.3% may build the next three features on the same idea. Another team may stop a change that really works, because a rainy month hid it.

First, look at Priya’s notes for Omar’s claim. They have three parts:

  1. The naive comparison: sales in the 4 weeks after the launch against the 4 weeks before: +8.3%.
  2. What else changed in those weeks: Thanksgiving, the autumn sale, slow growth.
  3. The question to ask: what would sales have been in the same weeks without the block?

Now do the same for Hana’s claim. Free delivery from 50 dollars instead of 75 started in Quebec on 2 June 2025. Hana says: “Quebec’s sales rose 7% in the 26 weeks after we started, against the 26 weeks before.”

Write the three parts on paper. Then compare with one possible answer:

  1. The naive comparison: Quebec’s sales in the 26 weeks after 2 June against the 26 weeks before: +7%.
  2. What else changed: the 26 weeks before are December to May, and the 26 weeks after are June to November. The season is different: summer and autumn are a garden shop’s busy months. Slow growth adds to the rise too.
  3. The question to ask: what would Quebec have sold from June to November without free delivery? Other provinces, which kept the old threshold, may help to answer it. Module 5 does exactly that.

Common mistake

The window decides the answer

Symptom: you report “+8.3%”. A colleague computes the same before-and-after change with 1-week windows and gets “+14.6%”. You both used the same data.

Cause: the windows hold different days. The first week after the launch is full of holiday and sale days, so a short window shows a bigger rise. Neither number is the block’s effect.

Diagnose: put the calendar next to the chart. Mark every holiday, sale, price change and email in both windows. If the after window has an event that the before window does not have, the comparison mixes that event with the change.

Fix: do not search for the “right” window. No window removes the holiday. You need a comparison with the world without the change: a test, or a method that builds that world from other data. The next lessons start there.

Check your understanding

In your own words: why is the +8.3% rise in sales after the launch not the effect of the recommendation block?

Show the answer

The +8.3% compares four weeks after the launch with four different weeks before it. The weeks after the launch also had Thanksgiving, the autumn sale, a little growth and chance. All of these raised sales too. The dashboard shows the sum of all the causes and cannot split it. The simulator, which ran the same weeks without the block, shows that the block caused +2.0% (82,562 dollars). The rest, +6.2%, would have happened anyway.

Check your understanding

Predict and explain: Priya computes the before-and-after change with 1 week on each side of the launch. Will the result be larger or smaller than +8.3%, and why? What happens to the true effect?

Show the answer

Larger: +14.6%. The first week after the launch holds Thanksgiving and the first days of the autumn sale, so it sells much more than a normal week. With four weeks, normal weeks dilute the holiday and the sale. The true effect stays at +2.0%. It compares the same days with and without the block, so the holiday and the sale are in both worlds.

Check your understanding

Practical check: use the table of this lesson. Compute the rise that would have happened without the block, and the block's own effect, both in percent. Show the divisions.

Show the answer

Without the block: 4,117,094 ÷ 3,877,532 = 1.062, a rise of +6.2%. The block’s effect: 4,199,656 ÷ 4,117,094 = 1.020, so +2.0%, or 4,199,656 − 4,117,094 = 82,562 dollars. Together they give the dashboard’s 4,199,656 ÷ 3,877,532 = 1.083, so +8.3%. The two percentages multiply (1.062 × 1.020 = 1.083); they do not add exactly.

Recap

  • A before-and-after comparison takes a number before a change and after it, and calls the difference the effect. It is a naive answer: it ignores everything else that happened.
  • After the recommendation block launched, sales rose +8.3% (3,877,532 to 4,199,656 dollars, four weeks each side). This is a practice problem we made on purpose.
  • The same weeks had Thanksgiving, an autumn sale, slow growth and chance. The simulator, which ran the weeks without the block, shows a true effect of +2.0% (82,562 dollars). Without the block, sales would still have risen +6.2%.
  • A warning sign of a naive answer: it moves when you change a choice that should not matter. The window is one: +14.6% with 1 week, +8.3% with 4.
  • The real question is always: what would have happened in the same time without the change?

You can now see why a rise on a dashboard is not proof. But “the world without the change” still sounds vague, and the block is not the only case. Customers who used the block also spent a third more than the others. Next, you give the world without the change its name, and you meet the confounders that make two groups look different for other reasons.

Next: Causes and comparisons