Meet the simulator
- Lesson 4
- 35 min
- Module 1: Why "it went up" is not an answer
Where we are
We have graphs and counterfactuals on paper. We have not run anything yet. In this lesson, we set up the course project, run the simulator, which knows the true effect of every change, and compare the naive answer and the test’s answer with the truth. First, think of a driving school’s practice track.
Lesson 4 of 30 · Experiments and Causal Impact: Did Our Change Work?Meet the simulator
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Set up the course project and check your first answers against the truth. The simulator knows the true effect of every change, so from now on you can test each method before you trust it.
NotebookDo the analysis part in a notebook
The notebook reads the same email test and the same block logs as the project. It computes the difference and its 95% TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI step by step with pandas, and checks them against the true effect. Then it asks you to compute the users’ +32.9%. Its first cell installs pandas, SciPy, statsmodels and matplotlib, and the next one downloads the data, so it needs no local setup. About 20 minutes. The project steps of this lesson still need your computer.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The naive answers of this module, from The feature that “raised sales” and Causes and comparisons.
- A difference between two groups, its 95% confidence interval and a p-value, from Compare alternatives in Essential Mathematics and Statistics for AI.
- How to use a terminal: change folders, run a command, activate a virtual environment.
What you need on your computer
- Python 3.12 and a terminal. VS Code helps but is not required.
- About 300 MB of free disk space for the packages. An internet connection for the setup.
- No account, no key and no money.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are given but were not run.
A practice track
Section titled “A practice track”A driving school has a practice track. It looks like a real road: curves, crossings, a hill, a parked car that suddenly opens its door. But the instructor built it, so the instructor knows exactly where every danger is. When a learner brakes late, the instructor can say how late, because the track has an answer key.
A real road has no answer key. That is why people learn on the track first.
The course’s simulator is TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s practice track. It makes the shop’s data from rules that we wrote, so it knows the true effect of every change. You can try a method, see how far it lands from the truth, and only then use it where nobody knows the truth.
How the simulator knows the truth
Section titled “How the simulator knows the truth”In the first lesson, the simulator ran the weeks after the block’s launch twice: with the block and without it. It does the same for every customer of every scenario.
Take the reminder email. For each customer, the simulator draws the random numbers that decide how this customer behaves: how often they visit, how likely they are to order. Then it uses the same random numbers twice: once in the world without the email, and once in the world with it. Only the email is different. So any difference between the two worlds is the email’s effect, for this customer.
Text description of the diagram
Each customer is simulated twice, with the same random numbers: once without the email and once with it. The average of the differences over all customers is the true effect, saved in truth.json. A coin flip puts each customer in group A or B. The log keeps only the world of that group, as a real log would. xp analyse reads only the log. It estimates the effect with a 95% interval. xp check then compares that estimate with truth.json.
The true effect of a run is the average of (with − without) over its customers. It is exact for that run, with those customers. The log is what a real team would have. For each customer, it keeps only the world of the group the coin chose. So the analysis sees exactly what a real analysis sees, and nothing more. Only xp check opens the answer key.
The project: two commands
Section titled “The project: two commands”The course project is called larkfield-experiments. It has two commands:
simis the simulator.sim run --scenario emailmakes the data of one scenario and writes it in a folder underruns/.xpis the analysis tool.xp analyseestimates an effect, andxp naivecomputes a naive answer.xp checkcompares your last answer with the truth, andxp resetdeletes all runs.
The project grows in each module. Module 2 adds sample sizes, Module 3 adds checks of a test, and so on. At the start, it has four scenarios: email, delivery, launch and logs. All its data is synthetic.
Set up the project
Section titled “Set up the project”You copy the snapshot start from the course’s labs repository, install its packages and run its tests. A snapshot is the project as it is at one point of the course. If your copy ever breaks, you can start again from the snapshot of the module you are on.
-
Get the files of the course. Open a terminal. If you have Git, clone the labs repository once, in a folder for your projects, such as
~/projects. If you have the clone already, rungit -C nextia-learning-labs pullinstead.Terminal window cd ~/projectsgit clone https://github.com/Nextia-Intelligence/nextia-learning-labs.gitWithout Git, download the ZIP file of the repository and unzip it in
~/projects. Its folder is thennextia-learning-labs-main. Use that name in the next step. -
Copy the snapshot
startto a new folder,larkfield-experiments, beside the labs folder. Then go into it.Terminal window cp -R nextia-learning-labs/C40/snapshots/start larkfield-experimentscd larkfield-experimentsTerminal window Copy-Item -Recurse nextia-learning-labs\C40\snapshots\start larkfield-experimentscd larkfield-experiments -
Make a TermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering and activate it. If PowerShell refuses to run
Activate.ps1, runSet-ExecutionPolicy -Scope CurrentUser RemoteSignedonce, then try again.Terminal window python3.12 -m venv .venvsource .venv/bin/activateTerminal window py -3.12 -m venv .venv.venv\Scripts\Activate.ps1 -
Install the pinned packages, then the project itself. The first command takes about a minute.
Terminal window python -m pip install -r requirements.txtpython -m pip install -e . --no-depsThe project uses NumPy 2.4.6, pandas 3.1.1, SciPy 1.15.3, statsmodels 0.15.0 and pytest 9.2.0. The second command installs the
simandxpcommands into your environment. -
Run the tests.
Terminal window python -m pytest -q......... [100%]11 passed in 1.38sYour time will differ. Eleven tests pass: the project works on your computer.
-
List the scenarios.
Terminal window sim listdelivery Free delivery from 50 dollars instead of 75: an A/B test by customer, 28 daysemail A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 dayslaunch Daily orders and sales around the recommendation block's launch (6 October 2025): no test, one serieslogs 28 days of logs after the recommendation block's launch: who used it and what they spent (no test)
Two of the scenarios are randomised tests: email and delivery. The other two have no test. launch is the daily sales of the first lesson, and logs is the users and non-users of the second lesson. sim about email prints what a scenario simulates.
Run a test and check it
Section titled “Run a test and check it”Rafael’s reminder email went to customers who had not ordered for 30 days. A coin flip chose who got it: group B got the email, group A did not. The outcome is an order in the 14 days after the email.
Predict first. About 9% of these customers order in two weeks without an email. How many more percentage points do you expect with it? Write a number down.
-
Run the scenario.
Terminal window sim run --scenario emailScenario email (seed 40): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Wrote runs/email-s40: customers (60,000 rows), targets (50,000 rows), truth_targets (50,000 rows). Synthetic data. 1.0 sThe simulator wrote the folder
runs/email-s40. s40 is the seed: the number that starts the random numbers. The same seed always gives the same data, so your numbers match this page. -
Analyse it.
Terminal window xp analyseRun email-s40 (synthetic): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Metric: ordered. A: 30,208 customers, mean 0.0896. B: 29,792 customers, mean 0.1043.difference +0.0147 (+16.4%), 95% interval +0.0100 to +0.0194, p = 1.14e-09 (Welch t-test)Saved to runs/email-s40/analysis.json. Compare with the true effect: xp check -
Check it against the truth.
Terminal window xp checkRun email-s40: the last analysis was `xp analyse`.True effect (from the simulator): +0.0119 share of customers with an order.Your estimate: +0.0147, 95% interval +0.0100 to +0.0194: the interval contains the true effect.
Read the result
Section titled “Read the result”Read the analysis line by line:
- A: 30,208 customers, mean 0.0896. Without the email, 8.96% of the customers ordered in two weeks.
- B: 29,792 customers, mean 0.1043. With the email, 10.43% ordered.
- difference +0.0147 (+16.4%). The email raised the share by 1.47 percentage points: from 8.96% to 10.43%. In relative terms, 16.4% more customers ordered.
- 95% interval +0.0100 to +0.0194. The email added between 1.00 and 1.94 points. The interval comes from the standard error, as in the mathematics course. Welch’s t-test is the test that gives it; it allows the two groups to have different spreads.
- p = 1.14e-09. If the email did nothing, a difference this large would almost never appear by chance.
Now the check. The true effect is +0.0119: 1.19 points. Your estimate, 1.47, is not the truth. It is one draw of chance, from one set of customers. But the interval, 1.00 to 1.94, contains the truth. That is what a 95% interval promises: not the exact answer, but a range that contains the truth in about 95 tests out of 100.
Text description of the chart
One estimate on an axis in percentage points. The email test at seed 40 estimates +1.47 points. Its 95% interval goes from +1.00 to +1.94. A line at 0 marks no effect, and a line at +1.19 marks the true effect from the simulator. The interval is far from 0 and contains +1.19.
Compare with the naive answers of the last two lessons. The before-and-after number was four times too large, and the users’ number 3.6 times. The coin flip’s answer misses by 0.28 points, and its interval says how far it could miss. This is the difference between a test and a naive comparison.
Check the naive answers too
Section titled “Check the naive answers too”The same project computes the naive answers of this module, so you can check them the same way.
-
The block’s launch, before and after:
Terminal window sim run --scenario launchxp naivexp checkScenario launch (seed 40): Daily orders and sales around the recommendation block's launch (6 October 2025): no test, one series.Wrote runs/launch-s40: daily (427 rows), counterfactual (427 rows). Synthetic data. 0.0 sSales in the 4 weeks before the launch (2025-09-08 to 2025-10-05): 3,877,532 dollarsSales in the 4 weeks after the launch (2025-10-06 to 2025-11-02): 4,199,656 dollarsChange: +8.3%. The same weeks had: Thanksgiving, an autumn sale.Saved to runs/launch-s40/analysis.json. Compare with the true effect: xp checkRun launch-s40: the last analysis was `xp naive`.Naive answer: sales +8.3% after the launch.True effect of the block (from the simulator): +2.0% of sales (82,562 dollars in the 4 weeks after the launch).The naive answer is 6.3 percentage points too high: the holiday, the sale and the trend are in it too. -
The users and non-users of the block:
Terminal window sim run --scenario logsxp naivexp checkScenario logs (seed 40): 28 days of logs after the recommendation block's launch: who used it and what they spent (no test).Wrote runs/logs-s40: customers (40,000 rows). Synthetic data. 0.2 sCustomers who used the block: 13,338, mean revenue 39.19 dollarsCustomers who did not: 26,662, mean revenue 29.48 dollarsDifference: +9.70 dollars (+32.9%).Saved to runs/logs-s40/analysis.json. Compare with the true effect: xp checkRun logs-s40: the last analysis was `xp naive`.Naive answer: users spent +9.70 dollars more (+32.9%).True effect of the block on its users (from the simulator): +2.73 dollars per customer.The naive answer is 3.6 times the true effect: the users were different customers before they used the block.
These are the numbers of the first two lessons, now from your own computer. Both naive numbers are practice problems that we made on purpose. xp naive --weeks 2 changes the windows of the launch, as the explorer of the first lesson did: it prints +12.6%.
Your turn: another seed
Section titled “Your turn: another seed”Change one thing: the seed. A new seed is a new copy of the same test with other customers, as if Rafael ran it again next month.
Predict first. Will the new estimate be the same as before? Will the true effect be the same?
sim run --scenario email --seed 41xp analysexp checkScenario email (seed 41): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Wrote runs/email-s41: customers (60,000 rows), targets (50,000 rows), truth_targets (50,000 rows). Synthetic data. 1.0 sRun email-s41 (synthetic): A reminder email to customers who have not ordered for 30 days: an A/B test by customer, 14 days.Metric: ordered. A: 30,006 customers, mean 0.0896. B: 29,994 customers, mean 0.1048. difference +0.0152 (+17.0%), 95% interval +0.0105 to +0.0199, p = 3.22e-10 (Welch t-test)Saved to runs/email-s41/analysis.json. Compare with the true effect: xp checkRun email-s41: the last analysis was `xp analyse`.True effect (from the simulator): +0.0145 share of customers with an order.Your estimate: +0.0152, 95% interval +0.0105 to +0.0199: the interval contains the true effect.Both changed. The estimate is now +1.52 points, and the true effect is +1.45 points. The truth is the average over the customers of this run, and these are other customers. The interval contains the truth again.
The course ran the email test at 20 seeds, 40 to 59. The chart shows the first 10. Each row is one interval, moved so that its own true effect sits at zero. An interval that crosses the zero line contains its truth.
Text description of the chart
Ten rows, seeds 40 to 49. Each row shows the estimate minus that seed’s true effect, in percentage points, with its 95% interval moved the same way. A line at 0 marks the true effect. The estimates scatter between about −0.4 and +0.3 points. Every interval crosses the zero line, so every one contains its own true effect. Seed 40 is the run of this lesson: +0.28 points from its truth.
All 20 intervals contained their truth. Over 200 seeds, the course measured 97%. A 95% interval should be right about 95 times in 100, so 97% is about what we expect. With 200 seeds, a few points either way is normal.
This is the check that real teams can never run. Here, you can run it for every method in the course.
Start again: reset
Section titled “Start again: reset”xp reset deletes the runs/ folder with all its runs. Your code does not change. It asks first:
xp resetDelete runs with 4 runs? [y/N] yDeleted runs (4 runs). Start again with: sim run --scenario emailType y and press Enter. Any other answer deletes nothing. xp reset --yes deletes without asking. You can always make the runs again: the seeds give the same data every time.
When to use the simulator
Section titled “When to use the simulator”Use it to learn and to test a method. It shows how far an answer lands from the truth, and how often an interval contains it. Use it before you trust a method on real data, and to show a colleague why a naive number misleads.
Do not use it to learn about Larkfield’s customers. The simulator’s customers follow our rules, and we chose the effects. A method that works here can still fail on real data that breaks its assumptions. The simulator shows you how a method behaves when you know its assumptions; real data does not tell you which assumptions hold.
The cost is a few minutes of setup and a little discipline. Write “simulated” next to every number from it, as sim does in its own output.
Common mistake
command not found: sim
Symptom: you type sim run --scenario email, and the terminal answers zsh: command not found: sim (or a similar message in bash or PowerShell).
Cause: the virtual environment is not active. This happens in every new terminal window. The sim and xp commands live inside .venv.
Fix: go to the project folder and activate the environment again: source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows. If the command is still missing, run python -m pip install -e . --no-deps again.
Common mistake
No run yet, or the wrong kind of run
Symptom: xp analyse stops with one of these messages:
No run yet. Create one with: sim run --scenario emailThe run launch-s40 has no table 'customers'. It has: daily, counterfactual.Cause: in the first case, the folder runs/ has no run. You have not run a scenario yet, you reset, or your terminal is in another folder. In the second case, the last run is launch, which has no test: it has daily sales, not two groups of customers.
Diagnose: run ls runs to see your runs. The file runs/LATEST names the last one.
Fix: go to the larkfield-experiments folder and run a scenario first. Use xp analyse on the tests (email, delivery) and xp naive on the runs without a test (launch, logs). You can also name a run: xp analyse email.
Knowledge check
Section titled “Knowledge check”Check your understanding
In your own words: how does the simulator know the true effect of the email, when the log shows each customer in only one group?
Show the answer
The simulator runs each customer twice, with the same random numbers: once in the world without the email and once in the world with it. Only the email differs, so the difference between the two worlds is the email’s effect on that customer. The true effect is the average of these differences over all customers, saved in truth.json. The log keeps only the world of the group that the coin chose, like a real log. So xp analyse sees what a real analysis sees, and only xp check uses the truth.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: you run the email scenario with seed 42. Will the estimate be +0.0147 again? Will the true effect be +0.0119? What do you expect from the interval?
Show the answer
No to both. A new seed gives other customers, so the estimate changes. The true effect changes too, because it is the average over this run’s customers. (At seed 41, the estimate was +0.0152 and the truth +0.0145.) The interval will most likely contain the new truth: over 200 seeds, 97% of the intervals did. But not always: about 1 interval in 20 can miss.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: show that your project works. What do the tests, xp analyse and xp check print for the email scenario at its default seed?
Show the answer
python -m pytest -q prints 11 passed. After sim run --scenario email, xp analyse prints difference +0.0147 (+16.4%), 95% interval +0.0100 to +0.0194, p = 1.14e-09 (Welch t-test), with 30,208 customers in A and 29,792 in B. xp check prints the true effect +0.0119 and says that the interval contains the true effect.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The course project
larkfield-experimentshas two commands:simmakes a scenario’s data, andxpanalyses it.xp checkcompares your last answer with the truth, andxp resetdeletes all runs. - The simulator runs every customer twice with the same random numbers, with and without the change. The average difference is the true effect in truth.json. The log keeps one world per customer, like a real log.
- The email test estimates +1.47 points (95% interval +1.00 to +1.94). The true effect is +1.19: the estimate misses, the interval contains it. Over 200 seeds, 97% of the intervals contain their truth.
- The naive answers land far away: +8.3% against +2.0% for the launch, +9.70 against +2.73 dollars for the users.
- A new seed is a new copy of the test: a new estimate and a new truth.
You can now run a scenario and check any answer against the truth. The email test was close because a coin chose who got the email. But why does a coin flip remove the confounders that broke the naive answers? And what must the coin decide: a customer, a visit, or an order? Next, you randomise.
Next: Randomise