Skip to content
Français

Experiments and Causal Impact: Did Our Change Work?

Find out whether a change really worked. Design and analyse A/B tests, avoid the traps of peeking and many metrics, estimate effects when you cannot run a test, find who benefits with uplift models, and check every answer against a simulator that knows the truth.

Read every lesson as a guest, or sign in to enroll and track your progress.

Is this the right course for you?What this course covers and does not cover, and 3 courses to take for other needs.

This course teaches how to get a trustworthy answer to a causal question in a business: randomised A/B tests from the plan to the report, the traps in real tests, natural experiments and adjustment for confounders when you cannot randomise, uplift models that decide who to treat, bandits, and online tests of AI features. It does not teach statistics from the start, how to build an experiment platform, or offline evaluation of AI systems.

Take another course if you want to:

What you'll learn

By the end of this course, you will be able to:

  • Turn a business question into a causal question with a unit, a treatment, an outcome and a decision, and explain why correlation in logs does not answer it.
  • Design an A/B test: randomisation unit, primary and guardrail metrics, minimum detectable effect, sample size and duration, and a stopping rule.
  • Analyse a test correctly: confidence intervals, checks for sample ratio mismatch, variance reduction with CUPED, multiple metrics, peeking and sequential tests, and novelty effects.
  • Estimate an effect when you cannot randomise, with difference-in-differences, synthetic control, regression discontinuity, interrupted time series or propensity-score methods, and state the assumption each one needs.
  • Find who benefits from a treatment with uplift models, and choose who to treat under a budget.
  • Test AI features (models, prompts, recommendations) online, choose between an A/B test and a bandit, and report a result that a decision-maker can trust.

About this course

After Larkfield launched a new recommendation block on its product pages, sales rose 8%. Omar, the product manager, wants to say that the block raised sales by 8%. But the same week had a holiday and a price cut. How much of the 8% did the block cause, if any?

In this course, you follow Priya, Larkfield’s data scientist, as she answers questions like this one for four changes: a lower free-delivery threshold, the recommendation block, a reminder email and a new prompt for the support assistant. You design A/B tests, analyse them, and learn the traps that make a test lie: stopping when it wins, checking twenty metrics, a split that is not really 50/50, and an effect that fades after two weeks.

Sometimes a test is not possible. Free delivery started in Quebec first, and the loyalty tier starts at 500 dollars of spending. For these, you use the methods that economists and data teams use: difference-in-differences, synthetic control, regression discontinuity and interrupted time series, and adjustment for confounders in logs. Then you find who benefits: the reminder email helps some customers and annoys others, and you choose whom to send it to under a budget.

You do all this with a small course project. Its simulator creates Larkfield’s traffic and knows the true effect of every change, so you can check each of your answers against the truth and see how wrong the naive answer was. All Larkfield data is synthetic: we made it for this course, and it has no real people. The case studies use real open data.

Before you start

You completed Essential Mathematics and Statistics for AI, or you have the same skills. You can compute a difference between two groups, a confidence interval and a p-value, and you know what they mean. You can read and change Python and pandas code and run a notebook. The machine learning course helps in the uplift module but is not required. You need Python 3.12 on your computer, or a free Google or Kaggle account for the notebooks. You need no key and no money.

Courses to take first: Essential Mathematics and Statistics for AI.

Who this is for

Data scientists, AI engineers, analysts and product people who must say whether a change, such as a new feature, a model, a price, an email or a prompt, caused a result.

How we made this course

Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses

Inside the course

Final module · choose at least oneCase studies

3 case studies

Three end-to-end case studies on real open data. A randomised mailing and its uplift, real online experiments at a fashion shop, and a minimum-wage rise studied without a test. Complete at least one for the certificate; each one is a portfolio piece.

Complete at least one case study for the certificate. The others are optional. What each one covers

  1. Did the mailing work, and for whom?150 min
  2. Real tests at an online shop150 min
  3. A minimum wage without a test150 min

After the case studiesFinal assignment

Use every module in one practice project. You collect evidence, and check it against the course outcomes.

  1. Final assignment180 min

Case studies

Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.

  1. Did the mailing work, and for whom?2 hr 30 min

    A real randomised mailing to 344,000 people with four letters and a control group (CC0): analyse the test with intervals, then build an uplift model and choose whom to mail under a budget.

  2. Real tests at an online shop2 hr 30 min

    Daily results of 78 real online experiments of a fashion shop (CC BY 4.0): intervals, sample ratio checks, peeking against a sequential test, and a decision for each test.

  3. A minimum wage without a test2 hr 30 min

    Restaurant jobs in New Jersey and Pennsylvania counties around New Jersey's 1992 minimum-wage rise, from official public-domain statistics: difference-in-differences, parallel trends and placebo tests.

Reviews

Loading the reviews…

    Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.

    Share this course