Skip to content
Français

Causes and comparisons

  • Lesson 2
  • 30 min
  • Module 1: Why "it went up" is not an answer

Where we are

We have seen a before-and-after number that looked like proof and was not. To do better, we need to say exactly what we compare with. In this lesson, we meet the counterfactual, the world without the change, and confounders, which make two groups look different for other reasons. First, think of a headache and a pill.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Say exactly what a change should be compared with: the same customers, at the same time, without the change. Then you can see why a comparison of the people who chose a feature with the people who did not is not an answer.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. Keep a pen and paper ready for the practice.

You have a headache. You take a pill. Two hours later, the headache is gone. Did the pill work?

You cannot be sure. Many headaches go away by themselves in two hours. To know what the pill did, you would need a second version of the same afternoon: the same headache, the same day, but no pill. If the headache stayed in that version, the pill helped. If it went away too, the pill did nothing.

You can never live both versions. You took the pill, so the afternoon without the pill never happened. This is not a lack of data. It is the nature of the question.

The version that did not happen has a name: the counterfactual. The word means “against the facts”. It is the world that would have happened if one thing had been different.

Every question about an effect is a question about a counterfactual:

  • Did the pill cure the headache? Compare with the same afternoon without the pill.
  • Did the block raise sales? Compare with the same four weeks without the block.
  • Did the reminder email bring orders? Compare with the same customers, in the same two weeks, without the email.

The effect of a change is the difference between the two worlds. It is the outcome with the change minus the outcome without it, for the same units at the same time. A unit is the thing that gets the change: a customer, a ticket, a day or a province. The outcome is what we measure: an order, revenue, a resolved ticket.

In the last lesson, the simulator ran the four weeks after the launch in both worlds. The diagram shows them as two lines from the same start.

A start box on the left: Monday 6 October, the same shop, the same customers, the same holiday and sale. A solid arrow goes to World 1, the block goes live: sales in the next 4 weeks are 4,199,656 dollars, which is what happened and what we see. A dotted arrow goes to World 2, no block: sales in the same 4 weeks are 4,117,094 dollars; this never happened and only the simulator knows it. Both point to a box on the right: the effect of the block is World 1 minus World 2, which is 82,562 dollars, or +2.0%.
The counterfactual: two worlds from the same start. We live in one of them. The effect is the gap between them.
Text description of the diagram

Both worlds start on Monday 6 October with the same shop, the same customers, the same holiday and the same sale. In World 1, the block goes live, and the shop sells 4,199,656 dollars in four weeks. This is the world we see. In World 2, there is no block, and the shop sells 4,117,094 dollars. This world never happened; only the simulator can run it. The block’s effect is the gap: 82,562 dollars, or +2.0%.

Here is a made-up example with four customers. Imagine that we could see both worlds for each of them. The outcome is 1 if the customer ordered in two weeks, and 0 if not.

Customer Active before? Orders without the email Orders with the email Effect
Ana yes 0 1 +1
Ben yes 1 1 0
Chloé no 0 0 0
Dev no 0 0 0

The email changed only Ana’s choice. Ben orders anyway, and Chloé and Dev do not order either way. So the average effect over the four customers is (1 + 0 + 0 + 0) ÷ 4 = 0.25. The email adds a quarter of an order per customer.

Now the real situation. Nobody sees both columns. Each customer either got the email or did not, so we see only one of the two outcomes:

Customer Got the email? Orders without the email Orders with the email
Ana yes ? 1
Ben yes ? 1
Chloé no 0 ?
Dev no 0 ?

Suppose Rafael sent the email only to the active customers, Ana and Ben. The customers with the email ordered 1 time on average: (1 + 1) ÷ 2. The others ordered 0 times. The difference is 1.0, four times the true average effect of 0.25.

What went wrong? Ana and Ben were active customers. Ben would have ordered without any email. The comparison mixes the email’s effect with the difference between active and inactive customers. The question marks hide exactly the numbers we need.

This is the central problem of the whole course. Every method in it is a way to fill in the question marks honestly.

Confounders: people with umbrellas get wet

Section titled “Confounders: people with umbrellas get wet”

Here is a fact that is true in many cities: people who carry an umbrella get wet more often than people who do not. Does the umbrella make them wet?

Of course not. Rain causes both. On rainy days, people take umbrellas, and people get wet. On dry days, they do neither. If you compare umbrella carriers with the others, you compare rainy days with dry days.

Rain is a confounder: a cause of both who gets the “treatment” (the umbrella) and the outcome (getting wet). A confounder makes two things move together even when one does not cause the other. The name comes from the Latin for “to mix up”: it mixes its own effect into the comparison.

Two small graphs side by side. Left, titled People with umbrellas get wet: Rain, the confounder, has an arrow to Carries an umbrella and an arrow to Gets wet. A dotted line with no arrowhead joins umbrella and wet, labelled no arrow: an umbrella does not make you wet. Right, titled Users of the block spend more: An active customer, who spent more before, visits more and has a garden, has an arrow to Uses the block and an arrow to Spends more. A small arrow from Uses the block to Spends more is labelled small: +2.73 dollars.
A confounder causes both the treatment and the outcome. Left, rain; right, the kind of customer.
Text description of the diagram

On the left, rain causes people to carry umbrellas and also causes them to get wet. There is no arrow from the umbrella to getting wet. Umbrella carriers get wet more often only because of the rain. On the right, the same shape at Larkfield. An active customer, who spent more before, visits more and may have a garden, is more likely to use the block and also spends more. The block also raises spending, but only a little: +2.73 dollars per user.

The treatment is the change whose effect we want: the umbrella, the email, the block. The word comes from medicine, but in this course a treatment can be any change.

Back at TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield. TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. has a second number. The block went live for everyone, and in the 28 days after the launch, some customers clicked a product in it and others did not. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses.’s logs show:

Group Customers Mean revenue in 28 days
Used the block 13,338 39.19 dollars
Did not use the block 26,662 29.48 dollars

The users spent 9.70 dollars more, which is +32.9%. Omar: “Customers who use the block spend a third more!”

This is the second naive answer of the course: users against non-users. Like the first, it is a practice problem we made on purpose. We set the simulator so that the users spend about a third more.

Nobody chose who used the block. Customers chose for themselves. So who are the users? TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. looks at the 8 weeks before the launch, when the block did not exist yet. In those weeks, the future users had spent 51.77 dollars on average, and the future non-users 41.28 dollars. The users were bigger spenders before they ever saw the block. They also visit more often, more of them use a computer instead of a phone, and they have been customers for longer.

And one cause is not in the logs at all. In the simulator, customers with a garden are more likely to use the block, and they also spend more. Larkfield does not record who has a garden. Module 6 comes back to this hidden cause.

So the users would have spent more even without the block. The simulator knows how much more, because it ran the 28 days both ways:

Of the 9.70-dollar gap, 2.73 dollars is the block. The rest is who the users are. Synthetic data from the simulator.
Text description of the chart

Three bars. Non-users spent 29.48 dollars on average. The users, in the world where they did not use the block, would have spent 36.46 dollars: only the simulator knows this. The users really spent 39.19 dollars. The step from the first bar to the second, 6.98 dollars, is the difference between the kinds of customers. The step from the second bar to the third, 2.73 dollars, is the block’s effect on its users.

Read the bars with the counterfactual in mind:

  1. The users spent 39.19 dollars.
  2. Without the block, the same users would have spent 36.46 dollars (39.19 − 2.73).
  3. So the block’s effect on its users is +2.73 dollars per user. The other 6.98 dollars of the gap come from who the users are.

The naive gap of 9.70 dollars is about 3.6 times the true effect.

The last number needs one more word. +2.73 dollars is the effect on the customers who used the block. The simulator can also say what the block would do if every customer used it: +2.37 dollars per customer. The two numbers differ because the users are different people: the block helps them a little more.

Both are honest answers to different questions. “What did the block do for the people who used it?” is the first. “What would it do for everyone?” is the second. A good report says which one it gives. Later modules name them and estimate each.

Is Omar’s +32.9% useless, then? No. It answers a real question, just not his.

“Do users of the block spend more than non-users?” Yes, they do. If Larkfield wants to find its best customers, block use is a good sign. This is a correlation: the two things move together. It is useful for prediction.

“Does the block make customers spend more?” That is a question about the counterfactual. A correlation cannot answer it when a confounder causes both.

TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. asks, “What if we compare users and non-users who spent the same before the launch?” That is a good idea. Module 6 does it with regression and matching. But it works only if you know and measure every confounder. The garden is not in the logs, so even a careful comparison of this kind stays a little biased.

A comparison answers a causal question when the two groups are alike before the change, so the change is the only difference. There are three ways to get such groups, and this course teaches all three:

  1. Flip a coin. Let chance decide who gets the change. Then active and inactive customers, gardeners and the rest, end up in both groups in about the same numbers. This is an A/B test, from Module 2.
  2. Find a natural experiment. Sometimes the world makes the groups for you: a rollout in one province first, or a rule with a threshold. Module 5 uses these.
  3. Adjust for the confounders you measured. Compare customers who were alike before. Module 6 shows how, and what goes wrong when a confounder is hidden.

The coin flip is the strongest, because chance also balances the confounders that nobody measured, like the garden. The other two need an assumption that you cannot fully check.

Worry about confounders every time a group chose the treatment, or someone chose for them:

  • customers chose to use a feature, to open an email, or to join a programme;
  • a manager chose the regions, the stores or the customers that got the change first;
  • the change started at a time when something else changed too, like a season or a sale.

Comparing users with non-users costs nothing, and the logs already have the data. A wrong answer costs much more. If Larkfield believed the +32.9%, it might pay to show the block to everyone everywhere, and expect a third more revenue that never comes.

First, read Priya’s notes on Omar’s claim. They have three parts:

  1. The naive comparison: customers who used the block against customers who did not: 39.19 against 29.48 dollars, +32.9%.
  2. One confounder: spending before the launch. Big spenders use the block more, and they spend more anyway (51.77 against 41.28 dollars in the 8 weeks before).
  3. The comparison that would answer: flip a coin for each customer to show the block or not. Then compare the two groups, users and non-users together.

Now write the same three parts for each of these claims:

  • Rafael: “Customers who opened the reminder email ordered much more often than customers who did not open it.”
  • Hana: “Since free delivery from 50 dollars started in Quebec in June, Quebec’s sales are up.”
  • Support team: “Since the new support prompt went live, fewer tickets go to a person.”
One possible answer for each claim
  • Rafael. Naive comparison: openers against non-openers. Confounder: interest in Larkfield. Customers who open emails often are the ones who buy often, email or not. Comparison that would answer: a coin flip decides who is sent the email. Then you compare everyone who was sent it with everyone who was not, openers or not.
  • Hana. Naive comparison: Quebec after June against Quebec before June. Confounder: the season. June to November is a busy time for a garden shop, and other changes happen too. Comparison that would answer: Quebec against provinces that kept the old threshold, in the same months (Module 5).
  • Support team. Naive comparison: tickets before the launch against tickets after it. Confounder: the kind of questions. After a sale, for example, more tickets are about returns, which need a person more often. Comparison that would answer: on the same days, a coin flip decides which prompt answers each customer’s tickets (Module 8).

Common mistake

Comparing by what customers did after the change

Symptom: Rafael’s email was sent to a random half of the customers, a fair coin flip. But his report compares the customers who opened it with everyone else. The openers ordered much more often, and the report calls it the email’s effect.

Cause: opening the email is a choice that customers made after the coin flip. Customers who like Larkfield open more emails and order more anyway. The comparison by opening brings back the confounder that the coin flip had removed.

Diagnose: ask of each group, “Who decided that this customer is in this group?” If the answer is the customer, or anything that happened after the change started, the groups are not comparable.

Fix: compare the groups that the coin made: everyone who was sent the email against everyone who was not, whatever they did next. This is the effect of sending the email, which is also the decision Rafael has to make.

Check your understanding

Module check: Omar says, “Customers who used the block spent 30% more, so the block made them spend 30% more.” Explain why this does not show that the block caused it.

Show the answer

The users chose to use the block, so they are different customers from the non-users. They spent more before the block existed: 51.77 against 41.28 dollars in the 8 weeks before. They also visit more often, and more of them are gardeners. These differences are confounders: they cause both block use and spending. So the users would have spent more even without the block. The comparison mixes the block’s effect with who the users are. In the simulator, the block’s true effect on its users is +2.73 dollars, not +9.70. To answer the question, the two groups must be alike before the change, for example by a coin flip.

Check your understanding

Predict: Priya compares the users and the non-users of the block in the 8 weeks before the launch, when the block did not exist. What do you expect to see, and what does it tell you?

Show the answer

The users already spent more before the block existed: 51.77 dollars against 41.28. The block cannot cause something that happened before it existed, so this gap must come from who the users are. It is a direct sign of a confounder: the two groups were not alike before the change. A gap in the time before a change is one of the simplest warnings that a comparison is biased.

Check your understanding

Practical check: from the numbers of this lesson, compute how much of the 9.70-dollar gap between users and non-users comes from the block, and how much from the kind of customer. Then give the naive gap as a multiple of the true effect.

Show the answer

The block: 2.73 dollars, the true effect on the users. The kind of customer: 9.70 − 2.73 = 6.97 dollars (the bars show 6.98, because the means are rounded). Without the block, the users would have spent 39.19 − 2.73 = 36.46 dollars, against 29.48 for the non-users. The naive gap is 9.70 ÷ 2.73 = 3.6 times the true effect.

Recap

  • The counterfactual is the world that did not happen: the same units, at the same time, without the change. The effect of a change is the outcome with the change minus the outcome without it.
  • We see only one of the two worlds for each unit. Every method in this course fills in the missing one, with an assumption.
  • A confounder causes both who gets the treatment and the outcome, like rain for umbrellas and wet clothes. It makes a comparison show an effect that the treatment did not cause.
  • At Larkfield, users of the block spent +9.70 dollars (+32.9%) more than non-users. This is a practice problem we made on purpose. The block’s true effect on its users is +2.73 dollars. The users were bigger spenders before the block existed.
  • A comparison answers a causal question when the groups are alike before the change. A coin flip, a natural experiment, or adjustment for measured confounders can make them alike.

You can now name the counterfactual and spot a confounder. But Larkfield’s logs have many columns: pages viewed, reviews, visits, past spending. Which of them are confounders, and which ones would make the answer worse if you used them? Next, you draw the causes as simple graphs and let the graph decide.

Next: Draw the causes