Final assignment
- After the case studies
- about 3 hr
Where we are
Each module added one tool and tried it on its own: the plan, the analysis, the checks, natural experiments, adjustment, uplift and the report. In this assignment, you use them together. Larkfield wants to know whether the new "complete the look" block should stay, and for whom.
What you will be able to do
Use every module of the course on one new change, and deliver an evidence pack that Omar can act on. You write the plan before you open the result. You check the test, measure the effect with its interval, and look for the traps of Module 4. You estimate the effect again from logs without a coin flip, and you say what that estimate can and cannot tell. Then you choose who should see the block next month, and you write the report.
Before you startWhat to know and to have ready
- You completed Modules 1 to 8. Each part below names the lessons that it uses.
- You do not need the case studies of Module 9 for this assignment. The certificate asks for at least one of them: Did the mailing work, and for whom?, Real tests at an online shop or A minimum wage without a test. They are good practice for this assignment.
- Your folder
larkfield-experimentsis at the snapshotend-of-m08, from Test a model or a prompt online, with.venvactive.python -m pytest -qpasses. The foldernextia-learning-labsis beside it, as in Meet the simulator. - About 3 hours. Most of it is reading results and writing decisions. Every command runs in under a minute on a laptop.
- Tested on macOS with Python 3.12.13 and the project’s pinned packages. The Windows commands are written, but they were not run.
The situation: a new block on the product pages
Section titled “The situation: a new block on the product pages”Larkfield adds a “complete the look” block to its product pages. This assignment is a separate scenario: it replays October and November 2025 with this block in place of the “customers also bought” block of Module 1. Under a lamp, for example, the block shows a table, a shade and a bulb that go with it. The team must decide two things: should the block stay, and for whom?
Three sources of evidence exist, one after the other:
- A soft launch. For four weeks in October, customers could turn on a “new look” with the block. About one customer in five turned it on. Nobody chose at random.
- A randomised test. For four weeks in November, a coin flip put each customer in A (no block) or B (the block).
- Next month’s customers. The block’s product suggestions come from a paid service. Next month, its contract pays for at most 15,000 customers of the 50,000.
Text description of the diagram
Three boxes from left to right. The soft launch, from 6 October to 2 November 2025: customers chose a toggle, 40,000 customers, no coin flip. The randomised test, from 3 to 30 November 2025: a coin flip for each customer, A without the block and B with it, 93,463 customers. Next month: 50,000 customers, and you can show the block to at most 15,000. A fourth box, your plan, points with a dashed arrow to the test: you write it before you open any result.
The team has questions, as in the course:
- Omar, the product manager, read the soft-launch dashboard. “The customers with the new look spent much more. Why do we need the test at all?”
- Amira, the developer, owns the block’s code and its logs. She says the test split customers 50/50, by customer, on the first visit. She expected about 3,300 new customers a day in the test.
- Hana, the merchandising lead, asks whether the block helps every kind of customer. Mobile screens are small, and the block takes space.
- Mateo, the growth analyst, keeps the test log. He wants a plan with a date before anybody reads the result.
- Tomás, the AI engineer, has an answer ready: “The dashboard showed a winner after a few days. We can ship it to everyone, and train an uplift model for next month.”
Priya asks you to answer all of them with evidence.
Be honest about the data. All of it is synthetic: the course simulator made it, and there are no real people. It has practice problems we made on purpose. The course team knows the true effects. You do not, and xp check cannot help you here. Your reviewer compares your answers with the truth. Quote your numbers “on Larkfield’s simulated test”, with their intervals.
The data
Section titled “The data”The data is in the labs repository, in C40/final-assignment/. Run every command of this assignment in your larkfield-experiments folder. Update your copy of the labs first:
git -C ../nextia-learning-labs pull| File | Rows | What one row is |
|---|---|---|
data/scenario.json |
What the files are: the test’s dates and split, and the budget. | |
data/soft_launch_customers.csv |
40,000 | A customer of the soft launch: province, device, new or returning, tenure, orders and spending in the 8 weeks before (pre_orders, pre_spend), visits, pages viewed, turned_on_new_look (0 or 1), and revenue in the four weeks. |
data/test_customers.csv.gz |
93,463 | A customer of the test: the variant, the first visit, and the visits, orders, revenue and returns in the 28 days. Also the same columns from before the test. |
data/test_visits.csv.gz |
337,741 | A visit of the test: the date, the customer, the variant, and whether it had an order, its revenue and a return. |
data/next_month_customers.csv |
50,000 | A customer of next month, with only the columns from before a test. |
data/SHA256SUMS |
The checksum of each file. | |
make_runs.py |
Checks the files and turns them into two runs that xp can read. |
The folder also has a README.md. It is about 5.4 MB.
Do not open the test files yet. You may read scenario.json and the soft launch now. The soft launch happened before the test, so a plan may use it. The test’s files wait until your plan is committed, in Part 1.
Make a folder evidence-pack/ in larkfield-experiments. Put every file of the assignment in it. What to submit lists the files.
The five parts
Section titled “The five parts”The pack follows the course’s completion rules: one part for each required item.
Text description of the diagram
A box on top: the folder evidence-pack, with REPORT.md, one page for Omar, and the commands that make every number. Below it, five boxes. 1 Plan: PLAN.md, committed before any result. 2 The test: the checks, the effect and its interval, CUPED, the effect by week, and the segments. 3 The soft launch: two methods, their assumptions, and a comparison with the test. 4 Targeting: an uplift model, a simple rule, and targets.csv. 5 The report: the decision, for whom, and its limits.
Part 1: the plan, before any result
Section titled “Part 1: the plan, before any result”Uses Choose the metrics, Sample size and power, Write the plan first and Segments and different effects.
The test has already run, but nobody has read its result. You write the plan now, as if the test had not started. Then the plan decides what the result means, and not the other way round.
-
Start the plan with the project’s template:
Terminal window xp plan new complete-the-lookIt writes
plans/complete-the-look.mdwith eleven fields, allTODO. The project’s.gitignoreleaves the folderplans/out of Git, so move the file into your pack:mv plans/complete-the-look.md evidence-pack/PLAN.md(in PowerShell,Move-Item plans\complete-the-look.md evidence-pack\PLAN.md). Fill inevidence-pack/PLAN.md. -
Choose the unit and the metrics. The unit is the customer: that is how Amira split the test. Choose one primary metric and at least one guardrail, as in Choose the metrics. Write which metrics only help you understand the result (diagnostics), for example orders per customer.
-
Name the segments now. Hana’s question needs segments. Write them in the plan, and say how you will treat several segments at once. Many metrics and many tests showed why. The runs give a column
segmentwith four values: device (desktop or mobile) and new or returning. You may plan other segments, but only from columns that exist before the test. -
Size the test. Take the baseline and the spread of your primary metric from the soft launch’s customers who did not turn the toggle on. For revenue:
Terminal window python -c "import pandas as pd; s = pd.read_csv('../nextia-learning-labs/C40/final-assignment/data/soft_launch_customers.csv'); a = s[s.turned_on_new_look == 0].revenue; print(len(a), round(a.mean(), 2), round(a.std(), 2))"The same command works in PowerShell: Python accepts
/in paths on Windows too. It prints the number of customers, the mean and the standard deviation. Then usexp power --baseline … --sd … --mde … --per-day 3300, as in Sample size and power. Say why these customers are only a stand-in for group A. -
Find the smallest effect that pays. Use two assumptions of the course. Larkfield keeps about 30% of revenue as margin: 26 dollars per order of about 86.75 dollars. The block’s service costs 0.05 dollars per customer a month. Both are assumptions, not measured costs. Work out the break-even gain in revenue per customer. Then compare it with the effect that 28 days can find. If the test cannot see the break-even, say so in the plan.
-
Write the stopping rule and the three decisions: if B is better, if B is worse, if there is no clear effect. Add a fourth rule for segments: what you do if one segment is clearly worse.
-
Check and commit the plan.
Terminal window xp plan check evidence-pack/PLAN.mdgit add evidence-pack/PLAN.mdgit commit -m "Plan for the complete-the-look test"xp plan checkprintsReady: all 11 fields are filled in.If your folder is not a Git repository yet, rungit initonce first. The commit’s date shows that the plan came first. -
Now make the runs. Give your plan’s primary metric and guardrails:
Terminal window python ../nextia-learning-labs/C40/final-assignment/make_runs.py --primary revenue --guardrails return_rateTerminal window python ..\nextia-learning-labs\C40\final-assignment\make_runs.py --primary revenue --guardrails return_rateFiles: all right (…/nextia-learning-labs/C40/final-assignment/data).Wrote runs/final-test: customers (93,463 rows), visits (337,741 rows), targets (50,000 rows). Primary metric: revenue; guardrails: return_rate.Wrote runs/final-soft: customers (40,000 rows), treatment used_block (= turned_on_new_look).Synthetic data. There is no true effect in these runs: xp check cannot help here.The first line says that every file matches
SHA256SUMS. If your plan has other metrics, give them instead.xp reportuses them in Part 5.
Part 2: the test
Section titled “Part 2: the test”Uses Check the test before the result, The result and its uncertainty, Get more precise with CUPED, Segments and different effects, Peeking, Many metrics and many tests and Novelty and learning effects. Name the run in every command: final-test.
-
Check the split before the result. Run
xp srm final-test, thenxp srm final-test --by segment. Decide with the rule of your plan. A mismatch would stop the analysis here. -
The primary metric, with and without CUPED:
Terminal window xp analyse final-testxp analyse final-test --cupedReport the difference, its 95% interval, the share of the variance that CUPED removed, and what the interval rules out. Compare the interval with your MDE and your break-even.
-
The guardrails and the diagnostics. For example
xp analyse final-test --metric return_rate, and--metric ordersor--metric converted. A guardrail must not be clearly worse. A diagnostic explains the primary result: more orders, or larger orders? -
The effect over time. Run
xp weekly final-test. Read the table by week since the first visit, not only by calendar week. Is the effect steady? If it falls, which weeks are the best guess of next month? Write what that means for a decision about next month. -
The segments of your plan. For example:
Terminal window xp analyse final-test --cuped --by segmentxp analyse final-test --metric orders --by segmentUse the correction that your plan named. A segment is “clearly worse” only by your plan’s rule. “Not clearly better” is not the same as “worse”, and not the same as “no effect”.
-
What a dashboard would have said. Run
xp peek final-test. On which day did the fixed test first show p < 0.05? When did the sequential test decide? Answer Tomás: would stopping on that day have been right, and why is it still not allowed by your plan? -
Write
evidence-pack/TEST.md: every command with its output, and one sentence after each that says what it means.
Part 3: the soft launch
Section titled “Part 3: the soft launch”Uses Causes and comparisons, Draw the causes, Regression adjustment, Propensity scores, matching and weighting and Test your own answer.
Omar’s dashboard compares the customers who turned on the new look with the others. Your job is to say what those logs can tell, and what they cannot.
-
Draw the causes first, before any estimate, as in Draw the causes. Who turns a toggle on? Which columns cause both the choice and the revenue (confounders)? Which column can the new look itself change (a mediator)? Put the graph in
evidence-pack/SOFT-LAUNCH.md. A photo of a drawing on paper is fine. -
The naive answer and two families of methods.
xp adjust final-softprints the naive difference and three estimates. The regression models the outcome. Matching and weighting model the choice, with propensity scores. So the regression and a propensity method are two different families. Choose the covariates from your graph with--covariates, and say why. -
Check the assumptions that you can check. Read the overlap line and the balance line. Is every kind of customer present in both groups? Is the balance below 0.1 after weighting?
-
Make the mediator mistake once, on purpose. Run
xp adjust final-soft --add pages_viewed. Explain what happens to the overlap and to the weighting estimate, and why. -
Try to break your answer. Run
xp refute final-soft. Passing these tests does not prove the estimate. Read the grid for a hidden confounder: how strong must a missing cause be to move the estimate to 0? -
Compare with the test. Put the naive difference, your two estimates and the test’s estimate in one table, each with its interval. Say which effect each one estimates: the effect on the customers who chose the toggle (ATT), or on everyone (ATE). Also say what was different in October: other customers, a short time with the block, and no coin flip.
-
Write what the soft launch can and cannot say, in a few sentences for Omar. List every assumption that your estimates need, for example no hidden confounder, overlap, and the right covariates.
Part 4: who should see the block next month
Section titled “Part 4: who should see the block next month”Uses The average hides the people, Uplift models and Evaluate and use uplift.
The service pays for at most 15,000 of next month’s 50,000 customers. Choose them, and show that your choice is better than simpler ones.
xp uplift works only on the email run, so you write a short script. The project gives the parts: EconML’s learners, the same base model, and the uplift curve and Qini coefficient of Module 7. This starting point trains an X-learner on revenue, on 70% of the test, and evaluates it on the other 30%:
import numpy as npimport pandas as pdfrom econml.metalearners import XLearnerfrom sklearn.linear_model import LogisticRegressionfrom threadpoolctl import threadpool_limits
from larkfield_experiments.uplift import _regressor, qini_coefficient, uplift_curve
test = pd.read_csv("runs/final-test/customers.csv.gz")nextm = pd.read_csv("runs/final-test/targets.csv")cols = ["new_customer", "tenure_days", "pre_orders", "pre_spend"] # your choice: columns known before the test
def X(df): return df[cols].assign(mobile=(df.device == "mobile").astype(int)).to_numpy(float)
hold = np.random.default_rng(0).random(len(test)) < 0.3train, ev = test[~hold], test[hold]model = XLearner(models=_regressor(), propensity_model=LogisticRegression(max_iter=1000))with threadpool_limits(1): model.fit(train.revenue.to_numpy(float), (train.variant == "B").to_numpy(int), X=X(train)) score_ev, score_next = model.effect(X(ev)), model.effect(X(nextm))
shares, gains = uplift_curve(score_ev, ev.revenue.to_numpy(float), (ev.variant == "B").to_numpy(int))print("extra revenue in the evaluation group, top 10% ... 100%:", np.round(gains[2::2]).tolist())print("Qini coefficient:", round(qini_coefficient(shares, gains), 1))Save it as evidence-pack/targeting.py and run it with python evidence-pack/targeting.py. It takes a few seconds. The numbers on the curve are dollars of revenue in the evaluation group, not orders.
-
Choose the features. Use only columns that exist before the test and in
targets.csv.visits,ordersandfirst_visithappen during the test: a model that reads them cannot score next month’s customers, and it learns from the outcome. -
Compare at least three lists on the same evaluation group. The first is your uplift model: a T-learner, an X-learner, or both. The second is a simple rule built from your planned segments. The third is a “who will buy” list, for example the highest
pre_spend. A simple rule is a score too: give each customer the effect of their segment, and draw its uplift curve. -
Train on another split. Change the seed of the split, as in Your turn: train on another split. If the curve and the list change a lot, the model is not reliable here.
-
Price the list. Use the assumptions of Part 1. The profit of one customer is 0.30 × their revenue gain − 0.05 dollars. Keep a customer only if your estimate pays. Choose at most 15,000.
-
Think about time. The test’s 28 days include each customer’s first weeks with the block. Next month, many customers will not be new to it. Use what Part 2 found by week, and say whether your expected gain is too high.
-
Write the list as
evidence-pack/targets.csv, with one column,customer_id, and at most 15,000 rows. Writeevidence-pack/TARGETING.md: the rule in one sentence, the curves, the seeds, the price, and which customers you leave out and why.
Part 5: the report
Section titled “Part 5: the report”Uses Report a result people trust and Experiments as a team habit.
-
Write the draft. Run
xp report final-test --cuped --plan evidence-pack/PLAN.md --out evidence-pack/report-draft.md(leave out--cupedif CUPED is not in your plan). Read its decision line, and ask what the draft cannot see. -
Write
evidence-pack/REPORT.md: one page for Omar, with no code. It answers his two questions: should the block stay, and for whom? It has:- the decision, in one sentence, from your plan’s rules;
- the effect on the primary metric with its interval, and what the interval rules out;
- the checks: the split, the duration, the guardrails, and peeking;
- the effect by week, and what it means for next month;
- the segments, and every segment that is worse, with what you do about it;
- the soft launch in two sentences: what it can and cannot say;
- the target list: its size, its price and what it leaves out;
- the limits, and what you would measure next (for example, a small holdout after the launch).
-
Answer each person in one line: Omar’s dashboard, Hana’s mobile question, Mateo’s plan, and Tomás’s “winner after a few days”.
-
Optional: add the test to the test log, as in Experiments as a team habit.
xp log addlogs the last analysis of the run, so run your primary analysis again first:xp analyse final-test --cuped, thenxp log add final-test --decision … --plan evidence-pack/PLAN.md.
What to submit
Section titled “What to submit”The folder evidence-pack/ in your larkfield-experiments folder:
evidence-pack/ PLAN.md Part 1: the plan, committed before the runs were made TEST.md Part 2: checks, effect, CUPED, guardrails, weeks, segments, peeking, with outputs SOFT-LAUNCH.md Part 3: the graph, two families, assumptions, the mediator, refutation, the comparison targeting.py Part 4: your script TARGETING.md Part 4: the lists compared, the seeds, the price, the rule targets.csv Part 4: customer_id, at most 15,000 rows report-draft.md Part 5: the draft from xp report REPORT.md Part 5: one page for Omar requirements.txt exact versions (python -m pip freeze)Add the output of git log --format="%ad %s" -- evidence-pack/PLAN.md at the end of REPORT.md. It shows when the plan was written, and every later change.
Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. The course is complete only when all five items are demonstrated. You can improve an item and send it again.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. The plan (Module 2) | Committed before the runs were made, with the unit, the primary metric, guardrails, the segments and their correction, the MDE and the sample size with their source, the break-even with its assumptions, a stopping rule and the decisions. Any later change has a dated note. | A plan with no segments, or a sample size with no source. | No plan, or a plan written after the result. |
| 2. The test (Modules 3 and 4) | The SRM check first. The primary effect with its 95% interval, with and without CUPED, and what the interval rules out. Guardrails. The effect by week since the first visit, read correctly. The planned segments with the planned correction, and every segment that is clearly worse named. The peeking question answered. | The effect without CUPED or without the weekly table. Or segments read without a correction. | No interval. Or no SRM check. Or a segment that is clearly worse and not named. |
| 3. The soft launch (Modules 1, 5 and 6) | A causal graph. Two methods from different families, each with its interval, its target (ATT or ATE) and its assumptions. Overlap and balance checked. The mediator explained. The refutation run and read. A comparison with the test, and what the soft launch can and cannot say. | One family only. Or assumptions listed but not checked where they can be. | The naive difference reported as the effect. Or an estimate with no assumptions. |
| 4. Targeting (Module 7) | A model trained only on the randomised test, with features known before the test. Evaluated on held-out customers with an uplift curve. Compared with a simple rule and a “who will buy” list, on more than one split. A list of at most 15,000 with its price under named assumptions, and the effect of time considered. | One list only, or no held-out evaluation. | A list ranked by “who will buy” alone. Or more than 15,000 customers. Or features from the test’s outcomes. |
| 5. The report (Module 8) | One page with the decision from the plan’s rules, the effect and its interval, the checks, the weeks, the segments, the soft launch, the list and the limits. Each person’s question answered. | A report that follows the draft of xp report without the segments or the weeks. |
“Ship B” with one number. Or a decision that the plan’s rules do not support, with no reason. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- The plan written or changed after the result without a dated note. For example, the primary metric changed to the one that “won”. (Item 1.)
- A result used from a test with a sample ratio mismatch, or no SRM check at all. (Item 2.)
- A test stopped, or a winner claimed, at a peek. “It was significant on day 5” is not a result. (Item 2.)
- A segment that is clearly worse and not reported, or a decision for everyone that hides it. (Items 2 and 5.)
- The soft launch’s naive difference reported as the effect of the block. (Item 3.)
- An observational estimate reported without its assumptions. (Item 3.)
- A target list that breaks the budget, or that was chosen with the test’s outcomes as features. (Item 4.)
What a good pack looks like
Section titled “What a good pack looks like”A good pack is honest before it is impressive. It has three things:
- Numbers with their intervals and their sizes. Each estimate has its 95% interval, the number of customers behind it, and the run it comes from.
- The plan’s rules, applied. The decision comes from rules written before the result. If the result does not fit any rule, the pack says so.
- A decision with its cost and its losers. For example: “Keep the block for these customers. It earns about this much, with this range, under these assumptions. It is worse for this group, so we do not show it to them. We could not measure this, so we will check it with a holdout.”
The author’s results
Section titled “The author’s results”The author solved the assignment once, from the public files only, with the course’s commands and the script of Part 4. Then the course team scored the author’s target lists against the simulator’s truth. Open the results only after you have your own numbers. Your choices can differ. Compare the method and the size of each result, not the exact number.
Show the author’s results
The plan. The soft launch’s customers without the toggle gave a spread of about 66 dollars per customer. With about 3,300 customers a day, 28 days can find an effect of about 1.2 dollars per customer with 80% power. The break-even under the assumptions is far smaller: about 0.17 dollars. So the plan said: “a gain can pay and still be unclear.”
The test.
- The split was fine: no mismatch overall or in any segment.
- Revenue went up, with a wide interval. The interval was above 0, but its lower end was close to 0. CUPED removed about a fifth of the variance, and the interval became a little narrower. The return rate was not clearly worse.
- A dashboard would have shown a winner within the first week. The sequential test decided a few days later. The plan’s rule was to look once, on day 28.
- The effect fell week by week. By week since the first visit, it went from about +10% in the first week to about 0 in weeks 3 and 4. The intervals of the last weeks were wide. So the long-run effect is smaller than the 28-day effect, and nobody can say from this test how much smaller.
- One planned segment was worse. On orders per customer, it was clearly below 0, even after a correction for four segments. On revenue, its interval still reached above 0. A pack that looks at revenue alone misses it. Another segment gained most, with an interval clearly above 0.
The soft launch.
- The naive difference was many times larger than any adjusted estimate.
- The regression and the weighting gave about +3 dollars per customer, each with an interval about 2 dollars wide on each side. Matching, which estimates the effect on the customers who chose the toggle, gave more. Balance after weighting was good, and the overlap was fine.
- With
pages_viewed, the propensity scores reached 1.000, and the weighting estimate became large and negative. - The three refutation tests passed. That did not make the estimate right: a hidden cause of moderate size would move it to 0.
- The test’s estimate was inside the intervals of the regression and the weighting.
Targeting.
- The author’s X-learner list reached about half to three fifths of the best possible list, in the course team’s score. The T-learner did a little worse. The Qini coefficients changed a lot from one split to the next.
- A simple rule from the planned segments did better than both models: about 70% of the best possible. It left out the segment that the test showed worse.
- The “who will buy” list did worst, below half of the best. It contained more than 2,000 customers who lose because of the block.
- The models’ own expected gain was 2 to 7 times too high. The 28-day test includes the first weeks, and the curves are noisy.
- Showing the block to all 50,000 would cost more than it earns for many of them, under the assumptions. The best list of 15,000 earns more than showing it to everyone.
Common mistakes
Section titled “Common mistakes”Common mistake
The soft launch "proves" the block
Symptom: the report says that the block raised revenue per customer by more than 25 dollars, from the soft launch.
Cause: the customers who turned on the new look were already better customers. They visited more and spent more before. The difference mixes the block with who chose it. This is the problem of The feature that “raised sales”, again.
Fix: use the randomised test for the effect. Use the soft launch only with adjustment, its assumptions and its intervals, and compare it with the test.
Consequence in AI work: an opt-in feature, such as an assistant that users turn on, always looks good in its users’ numbers. Test it with a coin flip before you report its value.
Common mistake
A good average that hides a worse segment
Symptom: the overall interval is above 0, and the report says “ship to everyone”. A month later, one group of customers orders less.
Cause: the average mixes the segments. A large gain for one group can hide a loss in another. The loss can also hide in a noisy metric such as revenue, and show clearly in orders.
Fix: plan the segments, read them with the planned correction, and look at the diagnostics as well as the primary metric. Name every group that is clearly worse, and decide what to do for it.
Consequence in AI work: a model or a prompt that “won” on average is rolled back after complaints from one group of users.
Common mistake
A list built from the first weeks
Symptom: the target list promises a large gain next month. The real gain is much smaller.
Cause: the test’s 28 days include each customer’s first weeks with the block, when the effect is largest. The uplift scores learned that early effect. Noise in the curves adds to the error.
Fix: read the effect by week since the first visit. Say how much smaller the long-run effect may be, and price your list with that in mind. Keep a small random holdout next month to measure the real gain.
Consequence in AI work: a new feature’s first weeks are not its future. Measure again after the novelty fades.
- The plan sized on a guess. Take the baseline and the spread from data that existed before the test, and say where they came from.
- “Not clearly better” read as “worse”. A segment whose interval contains 0 has no clear effect either way. Only your plan’s rule makes a segment “clearly worse”.
- The weekly table read by calendar week only. In the first calendar weeks, most visits are first visits. Read the table by week since the first visit.
pages_viewedused as a covariate. The new look adds pages. Adjusting for it removes part of the block’s effect, and it breaks the overlap.- Refuters read as proof. The placebo and the other two tests passed in the course’s own example of a biased estimate too. Say what they cannot see.
- The best “who will buy” model used for targeting. It finds customers who spend anyway. Some of them lose because of the block.
- The draft report sent as it is.
xp reportdoes not see segments or weeks. Write the real report yourself.
Show the public check values
Open this only after you have your own numbers. These values come from the public files, so you can check them. They are not the answers to the assignment.
- Files:
make_runs.pyprintsFiles: all right. The test has 93,463 customers and 337,741 visits from 3 to 30 November 2025. The soft launch has 40,000 customers, and 8,251 of them turned the toggle on. Next month has 50,000 customers. - The plan’s baseline: 31,749 soft-launch customers did not turn the toggle on. In the four weeks, they spent 21.61 dollars on average, with a standard deviation of 65.65 dollars.
xp power --baseline 21.61 --sd 65.65 --mde 1.2 --per-day 3300gives 46,985 customers per group. - The split: A 46,826 customers, B 46,637.
- The starting script of Part 4, unchanged, prints a Qini coefficient of
16217.2.
Recap
You can now take a change from a plan to a decision with evidence. You write the plan before the result, with its size, its segments and its stopping rule. You check the split, estimate the effect with its interval and CUPED, and read it by week and by segment. You estimate the effect again from self-selected logs with two families of methods. You say what that estimate needs and what it cannot see. You choose whom to treat within a budget with an uplift model, and you compare it with a simple rule. And you write a report that says for whom the change works, for whom it does not, and what is still unknown.