Experiments and Causal Impact: Did Our Change Work?
Find out whether a change really worked. Design and analyse A/B tests, avoid the traps of peeking and many metrics, estimate effects when you cannot run a test, find who benefits with uplift models, and check every answer against a simulator that knows the truth.
Read every lesson as a guest, or sign in to enroll and track your progress.
Enroll to save the lessons that you complete. It is free.
You are enrolled. From now on, the lessons that you complete are saved.
To get your certificate, mark every knowledge check as matching. These lessons have checks to finish:
Is this the right course for you?What this course covers and does not cover, and 3 courses to take for other needs.
This course teaches how to get a trustworthy answer to a causal question in a business: randomised A/B tests from the plan to the report, the traps in real tests, natural experiments and adjustment for confounders when you cannot randomise, uplift models that decide who to treat, bandits, and online tests of AI features. It does not teach statistics from the start, how to build an experiment platform, or offline evaluation of AI systems.
Turn a business question into a causal question with a unit, a treatment, an outcome and a decision, and explain why correlation in logs does not answer it.
Design an A/B test: randomisation unit, primary and guardrail metrics, minimum detectable effect, sample size and duration, and a stopping rule.
Analyse a test correctly: confidence intervals, checks for sample ratio mismatch, variance reduction with CUPED, multiple metrics, peeking and sequential tests, and novelty effects.
Estimate an effect when you cannot randomise, with difference-in-differences, synthetic control, regression discontinuity, interrupted time series or propensity-score methods, and state the assumption each one needs.
Find who benefits from a treatment with uplift models, and choose who to treat under a budget.
Test AI features (models, prompts, recommendations) online, choose between an A/B test and a bandit, and report a result that a decision-maker can trust.
About this course
After Larkfield launched a new recommendation block on its product pages, sales rose 8%. Omar, the product manager, wants to say that the block raised sales by 8%. But the same week had a holiday and a price cut. How much of the 8% did the block cause, if any?
In this course, you follow Priya, Larkfield’s data scientist, as she answers questions like this one for four changes: a lower free-delivery threshold, the recommendation block, a reminder email and a new prompt for the support assistant. You design A/B tests, analyse them, and learn the traps that make a test lie: stopping when it wins, checking twenty metrics, a split that is not really 50/50, and an effect that fades after two weeks.
Sometimes a test is not possible. Free delivery started in Quebec first, and the loyalty tier starts at 500 dollars of spending. For these, you use the methods that economists and data teams use: difference-in-differences, synthetic control, regression discontinuity and interrupted time series, and adjustment for confounders in logs. Then you find who benefits: the reminder email helps some customers and annoys others, and you choose whom to send it to under a budget.
You do all this with a small course project. Its simulator creates Larkfield’s traffic and knows the true effect of every change, so you can check each of your answers against the truth and see how wrong the naive answer was. All Larkfield data is synthetic: we made it for this course, and it has no real people. The case studies use real open data.
Before you start
You completed Essential Mathematics and Statistics for AI, or you have the same skills. You can compute a difference between two groups, a confidence interval and a p-value, and you know what they mean. You can read and change Python and pandas code and run a notebook. The machine learning course helps in the uplift module but is not required. You need Python 3.12 on your computer, or a free Google or Kaggle account for the notebooks. You need no key and no money.
Data scientists, AI engineers, analysts and product people who must say whether a change, such as a new feature, a model, a price, an email or a prompt, caused a result.
How we made this course
Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses
Inside the course
Module 1Why "it went up" is not an answer
4 lessons
Why "it went up" is not an answer. A launch that "raised sales 8%" in a week with a holiday and a price cut, counterfactuals and confounders, simple causal graphs, and a first run of the simulator that knows the true effect.
Design an A/B test. Randomise with the right unit, choose a primary metric and guardrails, compute the sample size and the power, and write the test plan before the test starts.
Analyse the test. The effect and its confidence interval, the bootstrap, checks for a sample ratio mismatch and A/A tests, more precision with CUPED, and segments planned in advance.
Traps in real tests. Peeking and sequential tests, many metrics and false discoveries, novelty effects that fade, and customers who affect each other through shared stock.
Module 5When you cannot randomise: natural experiments
4 lessons
When you cannot randomise. Difference-in-differences and synthetic control for a rollout in Quebec first, regression discontinuity at a loyalty threshold, and interrupted time series.
Module 6Adjust for confounders in observational data
3 lessons
Adjust for confounders in observational data. Regression adjustment, propensity scores, matching and weighting, and refutation tests that try to break your own answer.
Uplift. Find the customers an email persuades and the ones it annoys, train T- and X-learners, and choose whom to email under a budget with uplift and Qini curves.
Module 8Experiments for AI features and in production
4 lessons
Experiments for AI features and in production. An online test of a support-assistant prompt, bandits, a result report people trust, and experiments as a team habit.
Three end-to-end case studies on real open data. A randomised mailing and its uplift, real online experiments at a fashion shop, and a minimum-wage rise studied without a test. Complete at least one for the certificate; each one is a portfolio piece.
Complete at least one case study for the certificate. The others are optional. What each one covers
Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.
A real randomised mailing to 344,000 people with four letters and a control group (CC0): analyse the test with intervals, then build an uplift model and choose whom to mail under a budget.
Daily results of 78 real online experiments of a fashion shop (CC BY 4.0): intervals, sample ratio checks, peeking against a sequential test, and a decision for each test.
Restaurant jobs in New Jersey and Pennsylvania counties around New Jersey's 1992 minimum-wage rise, from official public-domain statistics: difference-in-differences, parallel trends and placebo tests.
Reviews
Loading the reviews…
Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.