Skip to content
Français

Draw the causes

  • Lesson 3
  • 35 min
  • Module 1: Why "it went up" is not an answer

Where we are

We know that a confounder can make a change look good or bad. But which variables are confounders, and which ones must we leave alone? In this lesson, we draw simple causal graphs for Larkfield’s four changes and use them to decide what to adjust for. First, we go back to the gift shop and its new window display.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Decide which variables to use in a comparison, and which ones to leave out, by drawing the causes first. “Use every column” can make an estimate worse, and even turn it to the wrong sign.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing. This lesson has no code. Keep a pen and paper ready: you will draw graphs.

The owner of the gift shop wants a clearer answer about her new window display. She takes a sheet of paper and writes down what causes what. She draws an arrow from each cause to its result.

  • The holiday brings more visitors. It also made her choose that week for the new display. So the holiday has two arrows: one to “new display” and one to “visitors”.
  • The new display makes people stop at the window, and people who stop come in. So: display → stop at the window → visitors.
  • A local newspaper writes about shops that are busy or that have a beautiful window. So the newspaper story has two arrows coming in: one from the display and one from the visitors.

Now she can see three different kinds of variables. The holiday is a confounder. Stopping at the window is a step on the way from the display to the visitors. The newspaper story is a result of both. As you will see, each kind needs different treatment.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s logs have a column for every customer fact that TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. records. For the recommendation block, they include:

Column What it is
pre_spend, pre_orders spending and orders in the 8 weeks before the launch
visits visits in the 28 days
tenure_days days since the customer’s first order
new_customer, device a new customer or not; phone or computer
pages_viewed pages seen in the 28 days
left_review wrote a product review in the 28 days
used_block clicked a product in the block (the treatment)
revenue spending in the 28 days (the outcome)

In the last lesson, TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. suggested comparing customers who were alike before the launch. A regression does this for many columns at once. It estimates the block’s effect for customers who are alike in every column you give it. Module 6 teaches it. Here, only one question matters: which columns should it get?

Tomás has a simple answer: “All of them. More control is always better.”

The course measured what happens. The simulator also knows the truth: if everyone used the block, revenue would rise by 2.37 dollars per customer.

The same logs, four answers. Adding the wrong column moves the estimate to the wrong sign. Synthetic data from the simulator; regression adjustment as in Module 6.
Text description of the chart

Four estimates of the block’s effect in dollars per customer, with their 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI where they exist. A line at 0 marks no effect, and a line at +2.37 marks the true effect if everyone used the block. Users against non-users: +9.70 (7.56 to 11.84). Adjusted for the 6 confounders (spending and orders before, visits, tenure, new customer, device): +3.38 (1.74 to 5.03). The same and the mediator, pages viewed: +2.79 (0.81 to 4.76). The 6 confounders and the collider, left a review, instead of the mediator: −3.16, below zero.

Read the four rows from the top:

  1. Users against non-users: +9.70. No adjustment. The confounders are all in it.
  2. Adjusted for 6 confounders: +3.38. Much closer to the truth. It is still too high, because one confounder, the garden, is not in the logs.
  3. And the mediator pages_viewed: +2.79. Closer still, but for a bad reason. Two errors cancel by luck: the hidden garden pushes the estimate up, and the mediator pulls it down. You cannot count on that luck.
  4. And the collider left_review instead: −3.16. The block now seems to lose money. This is not a small error. The sign is wrong.

So “all of them” is not a rule. One column fixed the answer, and another one broke it. We need a way to tell them apart before we look at the results. Looking at the results first and keeping the answer you like is not analysis.

A causal graph is a drawing of what causes what. Each variable is a box. An arrow from A to B means “A is one of the causes of B”. The graph does not say how big each effect is. It only says which causes exist, and which do not.

You draw the graph from what you know about the business, before you look at the results. TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. draws it with Amira, who knows the logs, and with TermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop., who knows the customers.

The technical name is a DAG, a directed acyclic graph. “Directed” means each arrow has a direction. “Acyclic” means you can never follow the arrows back to where you started: nothing causes itself.

Every variable in a graph can play a role relative to the treatment and the outcome. Three roles matter most.

A graph with five boxes. Confounder, past spending, has arrows to Treatment, used the block, and to Outcome, revenue in 28 days. The treatment has an arrow to Mediator, pages viewed, which has an arrow to the outcome. The treatment also has a direct arrow to the outcome. The treatment and the outcome both have arrows into Collider, left a review.
Three roles: a confounder causes both, a mediator is on the path, and a collider is caused by both.
Text description of the diagram

Past spending is a confounder: it causes block use and it causes revenue. Pages viewed is a mediator: the block adds pages, and pages lead to purchases, so part of the block’s effect travels through it. Left a review is a collider: both using the block and spending cause reviews, so two arrows meet in it.

A confounder causes both the treatment and the outcome. It opens a false path between them. Adjust for it: compare like with like. Past spending and the holiday are confounders.

A mediator is on the path from the treatment to the outcome: treatment → mediator → outcome. It is part of how the treatment works. If you adjust for it, you compare customers who saw the same number of pages. That removes the part of the effect that goes through the pages. Leave it alone when you want the whole effect. In the gift shop, “stop at the window” is a mediator. Compare only days with the same number of people stopping, and the display seems to do nothing.

A collider is caused by both the treatment and the outcome: two arrows collide in it. On its own, it does no harm. But if you adjust for it, or look only at one value of it, you create a link that is not there. Leave it alone.

The collider is the strangest of the three, so here is a made-up example with every value visible. The gift shop has four kinds of days. In this example, the display has no effect on how busy the shop is. The newspaper writes about the shop when it has a new display or when it is busy.

Day New display Busy Newspaper story
1 yes yes yes
2 yes no yes
3 no yes yes
4 no no no

Look at all four days. With the display, the shop was busy on 1 day of 2. Without it, also 1 day of 2. No link, which is right.

Now look only at the days with a newspaper story (days 1, 2 and 3). With the display, the shop was busy on 1 day of 2. Without the display, it was busy on 1 day of 1: every time. Among these days, the display seems to make the shop less busy.

Why? A story without a display needs a busy day: that is the only other reason the newspaper writes. Choosing the story days filled the “no display” group with busy days. The collider created the link. Larkfield’s left_review does the same: among customers who wrote a review, the ones who did not use the block are mostly big spenders. That is how the estimate turned to −3.16.

For the whole effect of a treatment, a simple rule covers most cases:

  • Adjust for the causes of both the treatment and the outcome, measured before the treatment.
  • Do not adjust for anything that the treatment can change: mediators, colliders, and other results of the treatment.

So the time order is a good first test. A variable recorded before the change cannot be a mediator or a collider of that change. A variable recorded after it needs a careful look at the graph.

Priya draws one graph for each of the four changes. Two kinds of graph appear. When a coin flip assigns the change, the treatment has only one arrow coming in: from the coin. When customers or managers choose, other arrows come in, and those are the confounders.

A graph from top to bottom. Coin flip, A or B, points to Free delivery from 50 dollars. Free delivery points to Orders and basket size, which points to Revenue. Free delivery also points to Returns. Past spending and season point to Revenue.
Hana's test: the coin flip is the only arrow into the treatment.
Text description of the diagram

A coin flip puts each customer in A or B. Group B gets free delivery from 50 dollars. Free delivery changes orders and basket size, which change revenue, and it also changes returns. Past spending and the season also cause revenue, but no arrow goes from them to free delivery.

Past spending and the season cause revenue, but they do not choose who gets free delivery: the coin does. So there is no confounder, and the simple difference between A and B estimates the effect. Orders and basket size are mediators: do not adjust for them. Returns are a second outcome: Hana must check that they do not rise too much. Module 2 calls such an outcome a guardrail.

A graph from top to bottom. Past spending, visits, tenure and device point to Used the block and to Revenue. Has a garden, not in the log, points to Used the block and to Revenue. Used the block points to Pages viewed, which points to Revenue, and it points directly to Revenue. Used the block and Revenue both point to Left a review.
Omar's block: customers chose, so confounders point into the treatment. One of them is not in the log.
Text description of the diagram

Past spending, visits, tenure and device cause both block use and revenue: they are confounders in the log. Having a garden is a confounder that the log does not record. Pages viewed is a mediator between the block and revenue. Left a review is a collider, caused by both the block and revenue.

This graph explains the whole interval chart above. The six confounders in the log belong in the comparison. The mediator and the collider do not. And the garden cannot go in, because Larkfield never recorded it. So even the best comparison of these logs stays a little too high. Module 6 measures how much, and how to test what a hidden confounder could do.

A graph from top to bottom. Coin flip, email or not, points to Reminder email. The email points to Ordered in 14 days and to Unsubscribed. Kind of customer, new, heavy buyer, days since last order, points to Ordered in 14 days.
Rafael's test: a coin flip again. The kind of customer changes the outcome, not who gets the email.
Text description of the diagram

A coin flip decides who gets the reminder email. The email causes orders and also unsubscribes. The kind of customer also causes orders: a new customer, a heavy buyer, or the days since the last order. But it does not decide who gets the email.

The kind of customer is not a confounder here: it does not choose who gets the email. But it matters in another way. The email may help some kinds of customers and annoy others. Module 7 is about exactly that.

A graph from top to bottom. Coin flip, prompt A or B, points to Support prompt. The prompt points to The answer, correct and short, which points to Ticket resolved. The prompt points to Hand-off to a person, which points to Cost per ticket and to Ticket resolved. Kind of question points to The answer and to Hand-off.
The support assistant's test: the answer and the hand-off are mediators.
Text description of the diagram

A coin flip decides which prompt answers a customer’s tickets. The prompt changes the answer, which changes whether the ticket is resolved. The prompt also changes how often the assistant hands the ticket to a person. The hand-off changes the cost per ticket and whether the ticket is resolved. The kind of question changes both the answer and the hand-off.

The answer and the hand-off are both mediators. They are how the prompt works. To measure the prompt’s whole effect on resolved tickets and on cost, compare A and B without adjusting for them. You can still look at them, to understand why one prompt wins. Looking is fine; adjusting is the mistake.

Draw a graph every time you estimate an effect without a coin flip. Draw one even with a coin flip: it shows the mediators and the second outcomes, such as returns, unsubscribes or cost.

A graph costs little: a sheet of paper and ten minutes with the people who know the business. Disagreement is useful. If Amira and Omar draw different arrows, the team has found an assumption that it must check.

A graph has limits too. It shows assumptions, not facts. The data cannot prove that an arrow is missing. A graph also cannot add a variable that nobody recorded, like the garden. And for a large graph, the rule above needs more care. Libraries such as DoWhy can read a graph and find what to adjust for. The course project uses DoWhy in Module 6.

First, follow Priya on one column. Is visits a confounder, a mediator or a collider?

  1. When is it measured? In the 28 days after the launch, so the time test alone does not decide.
  2. Can the block change it? In Larkfield’s simulator, no: customers visit for their own reasons, and the block shows only during a visit.
  3. Does it cause the treatment and the outcome? Yes. More visits give more chances to see the block and more chances to order.

So visits is a confounder, and it belongs in the comparison. If the block made people come back more often, visits would also be a mediator, and the answer would change. The graph forces you to say which you believe.

Now your turn. Suppose Rafael had not used a coin flip. He sent the reminder email to the customers he thought most likely to buy. Draw the graph with these variables: the email, ordered in 14 days, days since the last order, opened the email, unsubscribed. Then mark each variable as a confounder, a mediator, a collider or a second outcome.

One possible answer
  • Days since the last order is a confounder. Rafael used it to choose who got the email, and recent buyers also order more anyway. Adjust for it.
  • Opened the email is a mediator: the email must be opened before it can bring an order. It is also caused by the customer’s interest in Larkfield, which causes orders too. So it is a mediator and it opens a false path. Do not adjust for it, and do not compare openers with non-openers.
  • Unsubscribed is a second outcome, caused by the email. Check it, but do not adjust for it.
  • With the coin flip back in place, the arrow from “days since the last order” to the email disappears, and no adjustment is needed.

Common mistake

Every column in the regression

Symptom: you add one more column to the comparison, and the estimate changes a lot. At Larkfield, adding left_review turns the block’s effect from +3.38 to −3.16 dollars.

Cause: the column is a result of the treatment. Here, it is a collider: both the block and spending cause reviews. Adjusting for it links the block and spending in a way that the block did not cause.

Diagnose: for each column, ask two questions. Was it measured before the treatment started? Could the treatment change it? A column that the treatment can change is a mediator, a collider or another outcome.

Fix: draw the graph first, and adjust only for the confounders. Remove every column that the treatment can change. Decide the list before you see any estimate, and write it down.

Check your understanding

In your own words: what is a confounder, a mediator and a collider? For each one, say whether you adjust for it when you want the whole effect of a change.

Show the answer

A confounder causes both the treatment and the outcome, like past spending for the block and revenue. Adjust for it, so you compare like with like. A mediator is on the path from the treatment to the outcome, like pages viewed. Do not adjust for it: that removes the part of the effect that travels through it. A collider is caused by both the treatment and the outcome, like leaving a review. Do not adjust for it: that creates a link between the treatment and the outcome that is not there.

Check your understanding

Predict and explain: an analyst studies the block using only the customers who wrote a review. What do you expect to happen to the estimated effect, and why?

Show the answer

The estimate will be biased, probably toward a negative effect. Writing a review is a collider: both the block and high spending cause it. Among reviewers, a customer who did not use the block probably wrote the review because of high spending. So the non-users in this group are mostly big spenders, and the block looks worse than it is. On the full logs, adjusting for left_review gave −3.16 dollars, against a true effect of +2.37.

Check your understanding

Practical check: classify each column of the block's logs as a confounder, a mediator, a collider, the treatment or the outcome: pre_spend, tenure_days, device, pages_viewed, left_review, used_block, revenue. Which important cause is missing from the logs?

Show the answer

Confounders: pre_spend, tenure_days and device (also pre_orders, visits and new_customer). Mediator: pages_viewed. Collider: left_review. Treatment: used_block. Outcome: revenue. The missing cause is whether the customer has a garden. It makes customers more likely to use the block and to spend more, and Larkfield does not record it. So even the right comparison of these logs stays a little too high (+3.38 against +2.37).

Recap

  • A causal graph (a DAG) draws each variable as a box and each cause as an arrow. You draw it from knowledge of the business, before you look at the results.
  • A confounder causes both the treatment and the outcome: adjust for it. A mediator is on the path: leave it alone for the whole effect. A collider is caused by both: leave it alone.
  • A first test: adjust for causes measured before the treatment, never for anything the treatment can change.
  • On Larkfield’s logs, the 6 confounders give +3.38 dollars, still too high because the garden is hidden. Adding the mediator gives +2.79 only because two errors cancel. Adding the collider gives −3.16, the wrong sign. The truth for everyone is +2.37.
  • With a coin flip, the only arrow into the treatment comes from the coin, so no confounder remains. The free-delivery test, the reminder email and the prompt test use one.

You can now draw the causes of each change and say what to compare. But so far, every number came from us. Next, you set up the course project, run the simulator yourself, and check a naive answer and a test’s answer against the truth.

Next: Meet the simulator