Skip to content
Français

Peeking

  • Lesson 13
  • 40 min
  • Module 4: Traps in real tests

Where we are

We can analyse a finished test well. Real tests are watched while they run, and Tomás wants to stop as soon as the change wins. In this lesson, we simulate what daily peeking does to false winners, and we meet sequential tests, which allow us to look every day.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Decide when a running test may stop. Some teams look at a test every day and stop at the first “significant” result. They find false winners five times as often as they think. A sequential test lets you look every day and keep the promise of the 5%.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

In the notebook, you simulate 2,000 A/A tests, look at each one every day, and count the false winners. Then you run a sequential test on the same tests. Its setup cells install the packages, so it needs no local setup. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the project steps)

Tested on macOS with Python 3.12 and the end-of-m04 snapshot of the project. Windows was not tested.

A window display that “worked” on Wednesday

Section titled “A window display that “worked” on Wednesday”

A shop owner puts a new display in the window on Monday. Every evening, he compares the day’s customers with the same day last month. Monday: a little more. Tuesday: a little less. Wednesday: many more! He calls his sister: “The display works.”

On Thursday the numbers are normal again. By the end of the month, they are just like last month. The display did nothing. On Wednesday, he saw a lucky day, and he stopped looking at the right moment.

Every day has ups and downs. If you look every day and stop on a good day, you will often stop on a lucky one. The more often you look, the more lucky days you meet.

The problem: a checkout button that wins on day 3

Section titled “The problem: a checkout button that wins on day 3”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield tested a new green checkout button. TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails., the AI engineer, watched the dashboard every day. On day 3, B was ahead with p = 0.02. He said: “It is significant. Let’s stop and ship it.” The simulator has this test as a scenario. Its early stop is a practice problem we made on purpose: the button has no real effect.

First, update the project to the snapshot end-of-m04. It adds every tool of this module: xp peek, xp metrics, xp weekly, the sequential test and six new scenarios. You copy its files over your project folder. Your own runs and plans stay.

  1. Open a terminal in the folder that holds larkfield-experiments and nextia-learning-labs. If you cloned the labs repository with Git, update it first.

    Terminal window
    git -C nextia-learning-labs pull
  2. Copy the snapshot over your project.

    Terminal window
    cp -R nextia-learning-labs/C40/snapshots/end-of-m04/. larkfield-experiments/
    cd larkfield-experiments
    source .venv/bin/activate

    The packages are the same as in Module 3, so you install nothing new.

  3. Run the tests.

    Terminal window
    python -m pytest -q
    ................... [100%]
    21 passed in 3.44s

    Twenty-one tests pass. Your time will differ.

  4. Run the checkout-button test.

    Terminal window
    sim run --scenario early-stop
    Scenario early-stop (seed 1197): A new checkout button with no real effect, stopped on day 3 when it looked like a win (planted).
    Wrote runs/early-stop-s1197: customers (33,090 rows), visits (107,059 rows). Synthetic data. 3.0 s
  5. Look at it day by day, as the dashboard did.

    Terminal window
    xp peek
    Run early-stop-s1197, metric revenue: one look a day. 'fixed p' is the usual test at that day; the sequential columns are valid at every look (tau 1.24).
    day date customers difference fixed p sequential interval seq. p
    1 2025-09-08 3,500 +1.1842 0.223 -2.0675 to +4.4359 1.000
    2 2025-09-09 6,571 +1.1951 0.131 -1.3291 to +3.7194 0.828
    3 2025-09-10 9,244 +1.7557 0.020 -0.6315 to +3.7194 0.267 <
    4 2025-09-11 11,587 +1.4692 0.034 -0.6315 to +3.6371 0.267 <
    5 2025-09-12 13,647 +0.8414 0.207 -0.6315 to +2.9137 0.267
    6 2025-09-13 15,624 +0.6416 0.319 -0.6315 to +2.6316 0.267
    7 2025-09-14 17,312 +0.7028 0.275 -0.6315 to +2.6316 0.267
    8 2025-09-15 18,637 +0.9617 0.137 -0.6315 to +2.6316 0.267
    9 2025-09-16 19,880 +1.4123 0.032 -0.6284 to +2.6316 0.267 <
    10 2025-09-17 20,993 +1.4680 0.028 -0.6074 to +2.6316 0.267 <
    11 2025-09-18 22,126 +1.5251 0.022 -0.5342 to +2.6316 0.267 <
    12 2025-09-19 23,203 +1.5892 0.020 -0.5342 to +2.6316 0.262 <
    13 2025-09-20 24,186 +1.2393 0.072 -0.5342 to +2.6316 0.262
    14 2025-09-21 25,137 +1.5858 0.026 -0.5342 to +2.6316 0.262 <
    15 2025-09-22 25,857 +1.2731 0.073 -0.5342 to +2.6316 0.262
    16 2025-09-23 26,597 +1.3112 0.067 -0.5342 to +2.6316 0.262
    17 2025-09-24 27,278 +1.4985 0.037 -0.5342 to +2.6316 0.262 <
    18 2025-09-25 27,948 +1.4660 0.047 -0.5342 to +2.6316 0.262 <
    19 2025-09-26 28,571 +1.5365 0.042 -0.5342 to +2.6316 0.262 <
    20 2025-09-27 29,190 +1.3275 0.082 -0.5342 to +2.6316 0.262
    21 2025-09-28 29,788 +0.9141 0.237 -0.5342 to +2.6316 0.262
    22 2025-09-29 30,246 +0.8297 0.285 -0.5342 to +2.6316 0.262
    23 2025-09-30 30,778 +0.8105 0.301 -0.5342 to +2.6316 0.262
    24 2025-10-01 31,294 +0.6923 0.384 -0.5342 to +2.6316 0.262
    25 2025-10-02 31,760 +0.8130 0.311 -0.5342 to +2.6316 0.262
    26 2025-10-03 32,228 +0.8993 0.268 -0.5342 to +2.6316 0.262
    27 2025-10-04 32,689 +0.8118 0.325 -0.5342 to +2.6316 0.262
    28 2025-10-05 33,090 +0.8510 0.307 -0.5342 to +2.6316 0.262
    First day with p < 0.05 (peeking): 3. First day the sequential test decides: none.
  6. Analyse the whole test, as the plan said, and compare it with the truth.

    Terminal window
    xp analyse
    xp check
    Run early-stop-s1197 (synthetic): A new checkout button with no real effect, stopped on day 3 when it looked like a win (planted).
    Metric: revenue. A: 16,471 customers, mean 24.8770. B: 16,619 customers, mean 25.7280.
    difference +0.8510 (+3.4%), 95% interval -0.7827 to +2.4847, p = 0.307 (Welch t-test)
    Saved to runs/early-stop-s1197/analysis.json. Compare with the true effect: xp check
    Run early-stop-s1197: the last analysis was `xp analyse`.
    True effect (from the simulator): +0.0000 dollars per customer.
    Your estimate: +0.8510, 95% interval -0.7827 to +2.4847: the interval contains the true effect.

On day 3, B was 1.76 dollars ahead, with p = 0.020. On day 28, B is 0.85 dollars ahead, with p = 0.31. The TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI contains zero. The < marks show the days with p < 0.05. There are 10 of them, spread over the month. A team that looks every day has many chances to see a “win”.

  • Difference B − A
  • Usual 95% interval
  • Sequential interval
The checkout-button test day by day. The true effect is zero. The usual interval leaves zero on day 3 and on several later days. The sequential interval always contains zero.
Text description of the chart

A line chart over the 28 days of the test, with the difference B − A in dollars per customer from −2.5 to +4.5. The difference starts at +1.18 on day 1, peaks at +1.76 on day 3, and ends at +0.85 on day 28. Two dashed grey lines show the usual interval. It is −0.72 to +3.09 on day 1 and −0.78 to +2.48 on day 28. Its low end is above zero on day 3 (+0.28), day 4 (+0.11) and on days 9 to 12, 14 and 17 to 19. The sequential interval, two green lines, is wider: −0.63 to +3.72 on day 3, and −0.53 to +2.63 from day 11. Its low end never rises above zero.

A p-value below 0.05 has a meaning only for one look. “5% of tests with no effect cross the line” is true when you look once, at the end. Each extra look is another chance for noise to cross the line. You stop at the first crossing, so the chances add up.

Suppose each look were a fresh, independent test. Then the chance that at least one of k looks crosses the line is 1 − 0.95k, as for many segments in Segments and different effects.

Looks Chance of at least one false “win”
1 1 − 0.95 = 5%
4 (once a week) 1 − 0.954 = 1 − 0.815 = 18.5%
14 1 − 0.9514 = 1 − 0.488 = 51%
28 (every day) 1 − 0.9528 = 1 − 0.238 = 76%

Real looks are not independent. Day 4’s data contains day 3’s data, so two days next to each other give almost the same answer. The real rate is lower than this table, but much higher than 5%. A simulation tells us how much higher.

The explorer shows 40 simulated A/A tests: both groups get the same shop, so every “winner” is false. Each test has 2,000 new customers per group a day, and 22% of them order. Each row is one test, and each square is one day. Choose how you look, and move the day.

Explore

Peeking at 40 A/A tests

Looks so far
28
False winners in these 40 tests
10 of 40 (25%)
Share of 10,000 tests by day 28
27.6% every day, 1.8% sequential

By day 28, with a look every day, 10 of the 40 A/A tests have stopped with a false winner. None of these tests has a real effect.

  • A look: p ≥ 0.05
  • A look: p < 0.05, stop (a false winner)
  • No look that day
  • After the stop
  • Day not reached yet
40 A/A tests, one row eachTest 1, day 1: p = 0.428 (a look)Test 1, day 2: p = 0.611 (a look)Test 1, day 3: p = 0.584 (a look)Test 1, day 4: p = 0.350 (a look)Test 1, day 5: p = 0.297 (a look)Test 1, day 6: p = 0.156 (a look)Test 1, day 7: p = 0.362 (a look)Test 1, day 8: p = 0.234 (a look)Test 1, day 9: p = 0.321 (a look)Test 1, day 10: p = 0.231 (a look)Test 1, day 11: p = 0.222 (a look)Test 1, day 12: p = 0.190 (a look)Test 1, day 13: p = 0.249 (a look)Test 1, day 14: p = 0.169 (a look)Test 1, day 15: p = 0.340 (a look)Test 1, day 16: p = 0.524 (a look)Test 1, day 17: p = 0.605 (a look)Test 1, day 18: p = 0.441 (a look)Test 1, day 19: p = 0.369 (a look)Test 1, day 20: p = 0.345 (a look)Test 1, day 21: p = 0.512 (a look)Test 1, day 22: p = 0.777 (a look)Test 1, day 23: p = 0.956 (a look)Test 1, day 24: p = 0.858 (a look)Test 1, day 25: p = 0.692 (a look)Test 1, day 26: p = 0.806 (a look)Test 1, day 27: p = 0.798 (a look)Test 1, day 28: p = 0.880 (a look)Test 2, day 1: p = 0.099 (a look)Test 2, day 2: p = 0.606 (a look)Test 2, day 3: p = 0.860 (a look)Test 2, day 4: p = 0.744 (a look)Test 2, day 5: p = 0.945 (a look)Test 2, day 6: p = 0.629 (a look)Test 2, day 7: p = 0.341 (a look)Test 2, day 8: p = 0.264 (a look)Test 2, day 9: p = 0.204 (a look)Test 2, day 10: p = 0.106 (a look)Test 2, day 11: p = 0.076 (a look)Test 2, day 12: p = 0.039 (a look: p < 0.05, the test stops with a false winner)Test 2, day 13: p = 0.042 (after the stop)Test 2, day 14: p = 0.011 (after the stop)Test 2, day 15: p = 0.008 (after the stop)Test 2, day 16: p = 0.018 (after the stop)Test 2, day 17: p = 0.011 (after the stop)Test 2, day 18: p = 0.046 (after the stop)Test 2, day 19: p = 0.058 (after the stop)Test 2, day 20: p = 0.048 (after the stop)Test 2, day 21: p = 0.027 (after the stop)Test 2, day 22: p = 0.065 (after the stop)Test 2, day 23: p = 0.066 (after the stop)Test 2, day 24: p = 0.022 (after the stop)Test 2, day 25: p = 0.040 (after the stop)Test 2, day 26: p = 0.042 (after the stop)Test 2, day 27: p = 0.030 (after the stop)Test 2, day 28: p = 0.036 (after the stop)stop, day 12Test 3, day 1: p = 0.970 (a look)Test 3, day 2: p = 0.936 (a look)Test 3, day 3: p = 0.777 (a look)Test 3, day 4: p = 0.533 (a look)Test 3, day 5: p = 0.499 (a look)Test 3, day 6: p = 0.537 (a look)Test 3, day 7: p = 0.390 (a look)Test 3, day 8: p = 0.271 (a look)Test 3, day 9: p = 0.164 (a look)Test 3, day 10: p = 0.186 (a look)Test 3, day 11: p = 0.151 (a look)Test 3, day 12: p = 0.213 (a look)Test 3, day 13: p = 0.086 (a look)Test 3, day 14: p = 0.096 (a look)Test 3, day 15: p = 0.156 (a look)Test 3, day 16: p = 0.184 (a look)Test 3, day 17: p = 0.122 (a look)Test 3, day 18: p = 0.105 (a look)Test 3, day 19: p = 0.110 (a look)Test 3, day 20: p = 0.208 (a look)Test 3, day 21: p = 0.199 (a look)Test 3, day 22: p = 0.140 (a look)Test 3, day 23: p = 0.118 (a look)Test 3, day 24: p = 0.155 (a look)Test 3, day 25: p = 0.148 (a look)Test 3, day 26: p = 0.224 (a look)Test 3, day 27: p = 0.173 (a look)Test 3, day 28: p = 0.184 (a look)Test 4, day 1: p = 0.232 (a look)Test 4, day 2: p = 0.455 (a look)Test 4, day 3: p = 0.601 (a look)Test 4, day 4: p = 0.806 (a look)Test 4, day 5: p = 0.567 (a look)Test 4, day 6: p = 0.963 (a look)Test 4, day 7: p = 0.977 (a look)Test 4, day 8: p = 0.852 (a look)Test 4, day 9: p = 0.668 (a look)Test 4, day 10: p = 0.565 (a look)Test 4, day 11: p = 0.523 (a look)Test 4, day 12: p = 0.710 (a look)Test 4, day 13: p = 0.867 (a look)Test 4, day 14: p = 0.871 (a look)Test 4, day 15: p = 0.544 (a look)Test 4, day 16: p = 0.525 (a look)Test 4, day 17: p = 0.832 (a look)Test 4, day 18: p = 0.971 (a look)Test 4, day 19: p = 0.930 (a look)Test 4, day 20: p = 0.925 (a look)Test 4, day 21: p = 0.803 (a look)Test 4, day 22: p = 0.968 (a look)Test 4, day 23: p = 0.781 (a look)Test 4, day 24: p = 0.762 (a look)Test 4, day 25: p = 0.767 (a look)Test 4, day 26: p = 0.911 (a look)Test 4, day 27: p = 0.971 (a look)Test 4, day 28: p = 0.903 (a look)Test 5, day 1: p = 0.646 (a look)Test 5, day 2: p = 0.089 (a look)Test 5, day 3: p = 0.035 (a look: p < 0.05, the test stops with a false winner)Test 5, day 4: p = 0.229 (after the stop)Test 5, day 5: p = 0.444 (after the stop)Test 5, day 6: p = 0.113 (after the stop)Test 5, day 7: p = 0.159 (after the stop)Test 5, day 8: p = 0.276 (after the stop)Test 5, day 9: p = 0.224 (after the stop)Test 5, day 10: p = 0.194 (after the stop)Test 5, day 11: p = 0.243 (after the stop)Test 5, day 12: p = 0.357 (after the stop)Test 5, day 13: p = 0.500 (after the stop)Test 5, day 14: p = 0.434 (after the stop)Test 5, day 15: p = 0.651 (after the stop)Test 5, day 16: p = 0.621 (after the stop)Test 5, day 17: p = 0.733 (after the stop)Test 5, day 18: p = 0.979 (after the stop)Test 5, day 19: p = 0.958 (after the stop)Test 5, day 20: p = 0.872 (after the stop)Test 5, day 21: p = 0.636 (after the stop)Test 5, day 22: p = 0.500 (after the stop)Test 5, day 23: p = 0.628 (after the stop)Test 5, day 24: p = 0.816 (after the stop)Test 5, day 25: p = 0.642 (after the stop)Test 5, day 26: p = 0.648 (after the stop)Test 5, day 27: p = 0.665 (after the stop)Test 5, day 28: p = 0.277 (after the stop)stop, day 3Test 6, day 1: p = 0.386 (a look)Test 6, day 2: p = 0.667 (a look)Test 6, day 3: p = 0.692 (a look)Test 6, day 4: p = 0.529 (a look)Test 6, day 5: p = 0.194 (a look)Test 6, day 6: p = 0.367 (a look)Test 6, day 7: p = 0.489 (a look)Test 6, day 8: p = 0.434 (a look)Test 6, day 9: p = 0.526 (a look)Test 6, day 10: p = 0.847 (a look)Test 6, day 11: p = 1.000 (a look)Test 6, day 12: p = 0.939 (a look)Test 6, day 13: p = 0.800 (a look)Test 6, day 14: p = 0.654 (a look)Test 6, day 15: p = 0.568 (a look)Test 6, day 16: p = 0.266 (a look)Test 6, day 17: p = 0.252 (a look)Test 6, day 18: p = 0.609 (a look)Test 6, day 19: p = 0.513 (a look)Test 6, day 20: p = 0.580 (a look)Test 6, day 21: p = 0.666 (a look)Test 6, day 22: p = 0.650 (a look)Test 6, day 23: p = 0.868 (a look)Test 6, day 24: p = 0.852 (a look)Test 6, day 25: p = 0.790 (a look)Test 6, day 26: p = 0.800 (a look)Test 6, day 27: p = 0.676 (a look)Test 6, day 28: p = 0.605 (a look)Test 7, day 1: p = 0.135 (a look)Test 7, day 2: p = 0.765 (a look)Test 7, day 3: p = 0.164 (a look)Test 7, day 4: p = 0.168 (a look)Test 7, day 5: p = 0.123 (a look)Test 7, day 6: p = 0.090 (a look)Test 7, day 7: p = 0.369 (a look)Test 7, day 8: p = 0.491 (a look)Test 7, day 9: p = 0.485 (a look)Test 7, day 10: p = 0.673 (a look)Test 7, day 11: p = 0.513 (a look)Test 7, day 12: p = 0.553 (a look)Test 7, day 13: p = 0.612 (a look)Test 7, day 14: p = 0.569 (a look)Test 7, day 15: p = 0.523 (a look)Test 7, day 16: p = 0.241 (a look)Test 7, day 17: p = 0.448 (a look)Test 7, day 18: p = 0.388 (a look)Test 7, day 19: p = 0.186 (a look)Test 7, day 20: p = 0.362 (a look)Test 7, day 21: p = 0.356 (a look)Test 7, day 22: p = 0.354 (a look)Test 7, day 23: p = 0.441 (a look)Test 7, day 24: p = 0.581 (a look)Test 7, day 25: p = 0.642 (a look)Test 7, day 26: p = 0.899 (a look)Test 7, day 27: p = 0.769 (a look)Test 7, day 28: p = 0.880 (a look)Test 8, day 1: p = 0.729 (a look)Test 8, day 2: p = 0.606 (a look)Test 8, day 3: p = 0.537 (a look)Test 8, day 4: p = 0.985 (a look)Test 8, day 5: p = 0.707 (a look)Test 8, day 6: p = 0.988 (a look)Test 8, day 7: p = 0.665 (a look)Test 8, day 8: p = 0.829 (a look)Test 8, day 9: p = 0.629 (a look)Test 8, day 10: p = 0.981 (a look)Test 8, day 11: p = 0.704 (a look)Test 8, day 12: p = 0.708 (a look)Test 8, day 13: p = 0.775 (a look)Test 8, day 14: p = 0.691 (a look)Test 8, day 15: p = 0.898 (a look)Test 8, day 16: p = 0.621 (a look)Test 8, day 17: p = 0.740 (a look)Test 8, day 18: p = 0.686 (a look)Test 8, day 19: p = 0.800 (a look)Test 8, day 20: p = 0.824 (a look)Test 8, day 21: p = 0.751 (a look)Test 8, day 22: p = 0.714 (a look)Test 8, day 23: p = 0.685 (a look)Test 8, day 24: p = 0.497 (a look)Test 8, day 25: p = 0.477 (a look)Test 8, day 26: p = 0.595 (a look)Test 8, day 27: p = 0.436 (a look)Test 8, day 28: p = 0.493 (a look)Test 9, day 1: p = 0.512 (a look)Test 9, day 2: p = 0.034 (a look: p < 0.05, the test stops with a false winner)Test 9, day 3: p = 0.066 (after the stop)Test 9, day 4: p = 0.243 (after the stop)Test 9, day 5: p = 0.119 (after the stop)Test 9, day 6: p = 0.234 (after the stop)Test 9, day 7: p = 0.444 (after the stop)Test 9, day 8: p = 0.256 (after the stop)Test 9, day 9: p = 0.400 (after the stop)Test 9, day 10: p = 0.482 (after the stop)Test 9, day 11: p = 0.738 (after the stop)Test 9, day 12: p = 0.581 (after the stop)Test 9, day 13: p = 0.581 (after the stop)Test 9, day 14: p = 0.475 (after the stop)Test 9, day 15: p = 0.454 (after the stop)Test 9, day 16: p = 0.600 (after the stop)Test 9, day 17: p = 0.719 (after the stop)Test 9, day 18: p = 0.565 (after the stop)Test 9, day 19: p = 0.535 (after the stop)Test 9, day 20: p = 0.469 (after the stop)Test 9, day 21: p = 0.474 (after the stop)Test 9, day 22: p = 0.426 (after the stop)Test 9, day 23: p = 0.356 (after the stop)Test 9, day 24: p = 0.283 (after the stop)Test 9, day 25: p = 0.208 (after the stop)Test 9, day 26: p = 0.226 (after the stop)Test 9, day 27: p = 0.256 (after the stop)Test 9, day 28: p = 0.317 (after the stop)stop, day 2Test 10, day 1: p = 0.057 (a look)Test 10, day 2: p = 0.667 (a look)Test 10, day 3: p = 0.315 (a look)Test 10, day 4: p = 0.865 (a look)Test 10, day 5: p = 0.574 (a look)Test 10, day 6: p = 0.698 (a look)Test 10, day 7: p = 0.508 (a look)Test 10, day 8: p = 0.686 (a look)Test 10, day 9: p = 0.780 (a look)Test 10, day 10: p = 0.664 (a look)Test 10, day 11: p = 0.791 (a look)Test 10, day 12: p = 0.930 (a look)Test 10, day 13: p = 0.899 (a look)Test 10, day 14: p = 0.775 (a look)Test 10, day 15: p = 0.836 (a look)Test 10, day 16: p = 0.939 (a look)Test 10, day 17: p = 0.919 (a look)Test 10, day 18: p = 0.978 (a look)Test 10, day 19: p = 0.779 (a look)Test 10, day 20: p = 0.714 (a look)Test 10, day 21: p = 0.714 (a look)Test 10, day 22: p = 0.832 (a look)Test 10, day 23: p = 0.673 (a look)Test 10, day 24: p = 0.559 (a look)Test 10, day 25: p = 0.567 (a look)Test 10, day 26: p = 0.554 (a look)Test 10, day 27: p = 0.537 (a look)Test 10, day 28: p = 0.603 (a look)Test 11, day 1: p = 0.191 (a look)Test 11, day 2: p = 0.166 (a look)Test 11, day 3: p = 0.241 (a look)Test 11, day 4: p = 0.251 (a look)Test 11, day 5: p = 0.019 (a look: p < 0.05, the test stops with a false winner)Test 11, day 6: p = 0.113 (after the stop)Test 11, day 7: p = 0.162 (after the stop)Test 11, day 8: p = 0.201 (after the stop)Test 11, day 9: p = 0.109 (after the stop)Test 11, day 10: p = 0.109 (after the stop)Test 11, day 11: p = 0.051 (after the stop)Test 11, day 12: p = 0.108 (after the stop)Test 11, day 13: p = 0.072 (after the stop)Test 11, day 14: p = 0.089 (after the stop)Test 11, day 15: p = 0.083 (after the stop)Test 11, day 16: p = 0.223 (after the stop)Test 11, day 17: p = 0.161 (after the stop)Test 11, day 18: p = 0.156 (after the stop)Test 11, day 19: p = 0.072 (after the stop)Test 11, day 20: p = 0.112 (after the stop)Test 11, day 21: p = 0.186 (after the stop)Test 11, day 22: p = 0.163 (after the stop)Test 11, day 23: p = 0.093 (after the stop)Test 11, day 24: p = 0.155 (after the stop)Test 11, day 25: p = 0.174 (after the stop)Test 11, day 26: p = 0.375 (after the stop)Test 11, day 27: p = 0.469 (after the stop)Test 11, day 28: p = 0.698 (after the stop)stop, day 5Test 12, day 1: p = 0.645 (a look)Test 12, day 2: p = 0.236 (a look)Test 12, day 3: p = 0.212 (a look)Test 12, day 4: p = 0.732 (a look)Test 12, day 5: p = 0.798 (a look)Test 12, day 6: p = 0.139 (a look)Test 12, day 7: p = 0.011 (a look: p < 0.05, the test stops with a false winner)Test 12, day 8: p = 0.050 (after the stop)Test 12, day 9: p = 0.124 (after the stop)Test 12, day 10: p = 0.045 (after the stop)Test 12, day 11: p = 0.107 (after the stop)Test 12, day 12: p = 0.058 (after the stop)Test 12, day 13: p = 0.112 (after the stop)Test 12, day 14: p = 0.171 (after the stop)Test 12, day 15: p = 0.126 (after the stop)Test 12, day 16: p = 0.106 (after the stop)Test 12, day 17: p = 0.088 (after the stop)Test 12, day 18: p = 0.150 (after the stop)Test 12, day 19: p = 0.305 (after the stop)Test 12, day 20: p = 0.522 (after the stop)Test 12, day 21: p = 0.623 (after the stop)Test 12, day 22: p = 0.801 (after the stop)Test 12, day 23: p = 0.709 (after the stop)Test 12, day 24: p = 0.518 (after the stop)Test 12, day 25: p = 0.578 (after the stop)Test 12, day 26: p = 0.535 (after the stop)Test 12, day 27: p = 0.558 (after the stop)Test 12, day 28: p = 0.625 (after the stop)stop, day 7Test 13, day 1: p = 0.148 (a look)Test 13, day 2: p = 0.549 (a look)Test 13, day 3: p = 0.250 (a look)Test 13, day 4: p = 0.894 (a look)Test 13, day 5: p = 0.595 (a look)Test 13, day 6: p = 0.532 (a look)Test 13, day 7: p = 0.362 (a look)Test 13, day 8: p = 0.474 (a look)Test 13, day 9: p = 0.459 (a look)Test 13, day 10: p = 0.370 (a look)Test 13, day 11: p = 0.677 (a look)Test 13, day 12: p = 0.912 (a look)Test 13, day 13: p = 0.686 (a look)Test 13, day 14: p = 0.743 (a look)Test 13, day 15: p = 0.560 (a look)Test 13, day 16: p = 0.612 (a look)Test 13, day 17: p = 0.690 (a look)Test 13, day 18: p = 0.766 (a look)Test 13, day 19: p = 0.909 (a look)Test 13, day 20: p = 0.871 (a look)Test 13, day 21: p = 0.980 (a look)Test 13, day 22: p = 0.708 (a look)Test 13, day 23: p = 0.836 (a look)Test 13, day 24: p = 0.803 (a look)Test 13, day 25: p = 0.818 (a look)Test 13, day 26: p = 0.816 (a look)Test 13, day 27: p = 0.831 (a look)Test 13, day 28: p = 0.778 (a look)Test 14, day 1: p = 0.848 (a look)Test 14, day 2: p = 0.244 (a look)Test 14, day 3: p = 0.199 (a look)Test 14, day 4: p = 0.673 (a look)Test 14, day 5: p = 0.878 (a look)Test 14, day 6: p = 0.640 (a look)Test 14, day 7: p = 0.863 (a look)Test 14, day 8: p = 0.627 (a look)Test 14, day 9: p = 0.656 (a look)Test 14, day 10: p = 0.476 (a look)Test 14, day 11: p = 0.407 (a look)Test 14, day 12: p = 0.238 (a look)Test 14, day 13: p = 0.172 (a look)Test 14, day 14: p = 0.169 (a look)Test 14, day 15: p = 0.106 (a look)Test 14, day 16: p = 0.057 (a look)Test 14, day 17: p = 0.072 (a look)Test 14, day 18: p = 0.037 (a look: p < 0.05, the test stops with a false winner)Test 14, day 19: p = 0.026 (after the stop)Test 14, day 20: p = 0.025 (after the stop)Test 14, day 21: p = 0.035 (after the stop)Test 14, day 22: p = 0.025 (after the stop)Test 14, day 23: p = 0.067 (after the stop)Test 14, day 24: p = 0.046 (after the stop)Test 14, day 25: p = 0.034 (after the stop)Test 14, day 26: p = 0.044 (after the stop)Test 14, day 27: p = 0.021 (after the stop)Test 14, day 28: p = 0.027 (after the stop)stop, day 18Test 15, day 1: p = 0.909 (a look)Test 15, day 2: p = 0.538 (a look)Test 15, day 3: p = 0.879 (a look)Test 15, day 4: p = 0.598 (a look)Test 15, day 5: p = 0.438 (a look)Test 15, day 6: p = 0.458 (a look)Test 15, day 7: p = 0.616 (a look)Test 15, day 8: p = 0.851 (a look)Test 15, day 9: p = 0.723 (a look)Test 15, day 10: p = 0.639 (a look)Test 15, day 11: p = 0.740 (a look)Test 15, day 12: p = 0.887 (a look)Test 15, day 13: p = 1.000 (a look)Test 15, day 14: p = 0.816 (a look)Test 15, day 15: p = 0.739 (a look)Test 15, day 16: p = 0.409 (a look)Test 15, day 17: p = 0.495 (a look)Test 15, day 18: p = 0.311 (a look)Test 15, day 19: p = 0.272 (a look)Test 15, day 20: p = 0.234 (a look)Test 15, day 21: p = 0.344 (a look)Test 15, day 22: p = 0.385 (a look)Test 15, day 23: p = 0.260 (a look)Test 15, day 24: p = 0.189 (a look)Test 15, day 25: p = 0.215 (a look)Test 15, day 26: p = 0.282 (a look)Test 15, day 27: p = 0.379 (a look)Test 15, day 28: p = 0.300 (a look)Test 16, day 1: p = 0.595 (a look)Test 16, day 2: p = 0.628 (a look)Test 16, day 3: p = 0.709 (a look)Test 16, day 4: p = 0.985 (a look)Test 16, day 5: p = 0.610 (a look)Test 16, day 6: p = 0.238 (a look)Test 16, day 7: p = 0.195 (a look)Test 16, day 8: p = 0.300 (a look)Test 16, day 9: p = 0.274 (a look)Test 16, day 10: p = 0.745 (a look)Test 16, day 11: p = 0.629 (a look)Test 16, day 12: p = 0.775 (a look)Test 16, day 13: p = 0.719 (a look)Test 16, day 14: p = 0.943 (a look)Test 16, day 15: p = 0.813 (a look)Test 16, day 16: p = 0.614 (a look)Test 16, day 17: p = 0.684 (a look)Test 16, day 18: p = 0.719 (a look)Test 16, day 19: p = 0.416 (a look)Test 16, day 20: p = 0.670 (a look)Test 16, day 21: p = 0.683 (a look)Test 16, day 22: p = 0.764 (a look)Test 16, day 23: p = 0.562 (a look)Test 16, day 24: p = 0.533 (a look)Test 16, day 25: p = 0.512 (a look)Test 16, day 26: p = 0.464 (a look)Test 16, day 27: p = 0.394 (a look)Test 16, day 28: p = 0.407 (a look)Test 17, day 1: p = 0.438 (a look)Test 17, day 2: p = 0.589 (a look)Test 17, day 3: p = 0.186 (a look)Test 17, day 4: p = 0.594 (a look)Test 17, day 5: p = 0.517 (a look)Test 17, day 6: p = 0.708 (a look)Test 17, day 7: p = 0.954 (a look)Test 17, day 8: p = 0.882 (a look)Test 17, day 9: p = 0.683 (a look)Test 17, day 10: p = 0.942 (a look)Test 17, day 11: p = 0.854 (a look)Test 17, day 12: p = 0.974 (a look)Test 17, day 13: p = 0.857 (a look)Test 17, day 14: p = 0.854 (a look)Test 17, day 15: p = 0.737 (a look)Test 17, day 16: p = 0.833 (a look)Test 17, day 17: p = 0.904 (a look)Test 17, day 18: p = 0.943 (a look)Test 17, day 19: p = 0.965 (a look)Test 17, day 20: p = 0.824 (a look)Test 17, day 21: p = 0.815 (a look)Test 17, day 22: p = 0.890 (a look)Test 17, day 23: p = 0.848 (a look)Test 17, day 24: p = 0.656 (a look)Test 17, day 25: p = 0.635 (a look)Test 17, day 26: p = 0.574 (a look)Test 17, day 27: p = 0.551 (a look)Test 17, day 28: p = 0.452 (a look)Test 18, day 1: p = 0.970 (a look)Test 18, day 2: p = 0.935 (a look)Test 18, day 3: p = 0.691 (a look)Test 18, day 4: p = 0.788 (a look)Test 18, day 5: p = 0.430 (a look)Test 18, day 6: p = 0.381 (a look)Test 18, day 7: p = 0.623 (a look)Test 18, day 8: p = 0.498 (a look)Test 18, day 9: p = 0.413 (a look)Test 18, day 10: p = 0.497 (a look)Test 18, day 11: p = 0.262 (a look)Test 18, day 12: p = 0.245 (a look)Test 18, day 13: p = 0.655 (a look)Test 18, day 14: p = 0.615 (a look)Test 18, day 15: p = 0.441 (a look)Test 18, day 16: p = 0.400 (a look)Test 18, day 17: p = 0.308 (a look)Test 18, day 18: p = 0.368 (a look)Test 18, day 19: p = 0.139 (a look)Test 18, day 20: p = 0.142 (a look)Test 18, day 21: p = 0.230 (a look)Test 18, day 22: p = 0.193 (a look)Test 18, day 23: p = 0.161 (a look)Test 18, day 24: p = 0.119 (a look)Test 18, day 25: p = 0.231 (a look)Test 18, day 26: p = 0.199 (a look)Test 18, day 27: p = 0.207 (a look)Test 18, day 28: p = 0.229 (a look)Test 19, day 1: p = 0.727 (a look)Test 19, day 2: p = 0.702 (a look)Test 19, day 3: p = 1.000 (a look)Test 19, day 4: p = 0.728 (a look)Test 19, day 5: p = 0.209 (a look)Test 19, day 6: p = 0.461 (a look)Test 19, day 7: p = 0.485 (a look)Test 19, day 8: p = 0.430 (a look)Test 19, day 9: p = 0.565 (a look)Test 19, day 10: p = 0.467 (a look)Test 19, day 11: p = 0.533 (a look)Test 19, day 12: p = 0.799 (a look)Test 19, day 13: p = 0.718 (a look)Test 19, day 14: p = 0.767 (a look)Test 19, day 15: p = 0.745 (a look)Test 19, day 16: p = 0.579 (a look)Test 19, day 17: p = 0.697 (a look)Test 19, day 18: p = 0.822 (a look)Test 19, day 19: p = 0.888 (a look)Test 19, day 20: p = 0.959 (a look)Test 19, day 21: p = 0.980 (a look)Test 19, day 22: p = 0.896 (a look)Test 19, day 23: p = 0.962 (a look)Test 19, day 24: p = 0.833 (a look)Test 19, day 25: p = 0.848 (a look)Test 19, day 26: p = 0.674 (a look)Test 19, day 27: p = 0.718 (a look)Test 19, day 28: p = 0.723 (a look)Test 20, day 1: p = 0.273 (a look)Test 20, day 2: p = 0.957 (a look)Test 20, day 3: p = 0.648 (a look)Test 20, day 4: p = 0.822 (a look)Test 20, day 5: p = 0.775 (a look)Test 20, day 6: p = 0.988 (a look)Test 20, day 7: p = 0.977 (a look)Test 20, day 8: p = 0.883 (a look)Test 20, day 9: p = 0.650 (a look)Test 20, day 10: p = 0.711 (a look)Test 20, day 11: p = 0.648 (a look)Test 20, day 12: p = 0.563 (a look)Test 20, day 13: p = 0.543 (a look)Test 20, day 14: p = 0.816 (a look)Test 20, day 15: p = 0.423 (a look)Test 20, day 16: p = 0.461 (a look)Test 20, day 17: p = 0.514 (a look)Test 20, day 18: p = 0.387 (a look)Test 20, day 19: p = 0.244 (a look)Test 20, day 20: p = 0.189 (a look)Test 20, day 21: p = 0.206 (a look)Test 20, day 22: p = 0.210 (a look)Test 20, day 23: p = 0.148 (a look)Test 20, day 24: p = 0.168 (a look)Test 20, day 25: p = 0.113 (a look)Test 20, day 26: p = 0.136 (a look)Test 20, day 27: p = 0.095 (a look)Test 20, day 28: p = 0.111 (a look)Test 21, day 1: p = 0.970 (a look)Test 21, day 2: p = 0.872 (a look)Test 21, day 3: p = 0.725 (a look)Test 21, day 4: p = 0.324 (a look)Test 21, day 5: p = 0.433 (a look)Test 21, day 6: p = 0.359 (a look)Test 21, day 7: p = 0.818 (a look)Test 21, day 8: p = 0.935 (a look)Test 21, day 9: p = 0.789 (a look)Test 21, day 10: p = 0.781 (a look)Test 21, day 11: p = 0.662 (a look)Test 21, day 12: p = 0.921 (a look)Test 21, day 13: p = 0.656 (a look)Test 21, day 14: p = 0.610 (a look)Test 21, day 15: p = 0.508 (a look)Test 21, day 16: p = 0.613 (a look)Test 21, day 17: p = 0.650 (a look)Test 21, day 18: p = 0.760 (a look)Test 21, day 19: p = 0.979 (a look)Test 21, day 20: p = 0.858 (a look)Test 21, day 21: p = 0.521 (a look)Test 21, day 22: p = 0.473 (a look)Test 21, day 23: p = 0.170 (a look)Test 21, day 24: p = 0.215 (a look)Test 21, day 25: p = 0.218 (a look)Test 21, day 26: p = 0.271 (a look)Test 21, day 27: p = 0.370 (a look)Test 21, day 28: p = 0.391 (a look)Test 22, day 1: p = 0.103 (a look)Test 22, day 2: p = 0.257 (a look)Test 22, day 3: p = 0.167 (a look)Test 22, day 4: p = 0.158 (a look)Test 22, day 5: p = 0.505 (a look)Test 22, day 6: p = 0.697 (a look)Test 22, day 7: p = 0.697 (a look)Test 22, day 8: p = 0.205 (a look)Test 22, day 9: p = 0.191 (a look)Test 22, day 10: p = 0.419 (a look)Test 22, day 11: p = 0.696 (a look)Test 22, day 12: p = 0.783 (a look)Test 22, day 13: p = 0.672 (a look)Test 22, day 14: p = 0.676 (a look)Test 22, day 15: p = 0.568 (a look)Test 22, day 16: p = 0.724 (a look)Test 22, day 17: p = 0.711 (a look)Test 22, day 18: p = 0.455 (a look)Test 22, day 19: p = 0.425 (a look)Test 22, day 20: p = 0.233 (a look)Test 22, day 21: p = 0.224 (a look)Test 22, day 22: p = 0.241 (a look)Test 22, day 23: p = 0.126 (a look)Test 22, day 24: p = 0.149 (a look)Test 22, day 25: p = 0.114 (a look)Test 22, day 26: p = 0.107 (a look)Test 22, day 27: p = 0.172 (a look)Test 22, day 28: p = 0.298 (a look)Test 23, day 1: p = 0.276 (a look)Test 23, day 2: p = 0.200 (a look)Test 23, day 3: p = 0.597 (a look)Test 23, day 4: p = 0.985 (a look)Test 23, day 5: p = 0.824 (a look)Test 23, day 6: p = 0.876 (a look)Test 23, day 7: p = 0.896 (a look)Test 23, day 8: p = 0.829 (a look)Test 23, day 9: p = 0.711 (a look)Test 23, day 10: p = 0.654 (a look)Test 23, day 11: p = 0.489 (a look)Test 23, day 12: p = 0.567 (a look)Test 23, day 13: p = 0.397 (a look)Test 23, day 14: p = 0.397 (a look)Test 23, day 15: p = 0.396 (a look)Test 23, day 16: p = 0.560 (a look)Test 23, day 17: p = 0.559 (a look)Test 23, day 18: p = 0.640 (a look)Test 23, day 19: p = 0.827 (a look)Test 23, day 20: p = 0.707 (a look)Test 23, day 21: p = 0.790 (a look)Test 23, day 22: p = 0.877 (a look)Test 23, day 23: p = 0.918 (a look)Test 23, day 24: p = 0.640 (a look)Test 23, day 25: p = 0.731 (a look)Test 23, day 26: p = 0.544 (a look)Test 23, day 27: p = 0.445 (a look)Test 23, day 28: p = 0.493 (a look)Test 24, day 1: p = 0.877 (a look)Test 24, day 2: p = 0.514 (a look)Test 24, day 3: p = 0.723 (a look)Test 24, day 4: p = 0.502 (a look)Test 24, day 5: p = 0.441 (a look)Test 24, day 6: p = 0.134 (a look)Test 24, day 7: p = 0.122 (a look)Test 24, day 8: p = 0.199 (a look)Test 24, day 9: p = 0.220 (a look)Test 24, day 10: p = 0.304 (a look)Test 24, day 11: p = 0.209 (a look)Test 24, day 12: p = 0.326 (a look)Test 24, day 13: p = 0.231 (a look)Test 24, day 14: p = 0.308 (a look)Test 24, day 15: p = 0.355 (a look)Test 24, day 16: p = 0.452 (a look)Test 24, day 17: p = 0.868 (a look)Test 24, day 18: p = 0.907 (a look)Test 24, day 19: p = 0.902 (a look)Test 24, day 20: p = 0.765 (a look)Test 24, day 21: p = 0.532 (a look)Test 24, day 22: p = 0.580 (a look)Test 24, day 23: p = 0.616 (a look)Test 24, day 24: p = 0.528 (a look)Test 24, day 25: p = 0.393 (a look)Test 24, day 26: p = 0.346 (a look)Test 24, day 27: p = 0.304 (a look)Test 24, day 28: p = 0.445 (a look)Test 25, day 1: p = 0.033 (a look: p < 0.05, the test stops with a false winner)Test 25, day 2: p = 0.155 (after the stop)Test 25, day 3: p = 0.759 (after the stop)Test 25, day 4: p = 0.864 (after the stop)Test 25, day 5: p = 0.851 (after the stop)Test 25, day 6: p = 0.401 (after the stop)Test 25, day 7: p = 0.413 (after the stop)Test 25, day 8: p = 0.667 (after the stop)Test 25, day 9: p = 0.761 (after the stop)Test 25, day 10: p = 0.782 (after the stop)Test 25, day 11: p = 0.706 (after the stop)Test 25, day 12: p = 0.709 (after the stop)Test 25, day 13: p = 0.751 (after the stop)Test 25, day 14: p = 0.691 (after the stop)Test 25, day 15: p = 0.760 (after the stop)Test 25, day 16: p = 0.682 (after the stop)Test 25, day 17: p = 0.985 (after the stop)Test 25, day 18: p = 0.740 (after the stop)Test 25, day 19: p = 0.681 (after the stop)Test 25, day 20: p = 0.939 (after the stop)Test 25, day 21: p = 0.967 (after the stop)Test 25, day 22: p = 0.720 (after the stop)Test 25, day 23: p = 0.842 (after the stop)Test 25, day 24: p = 0.657 (after the stop)Test 25, day 25: p = 0.957 (after the stop)Test 25, day 26: p = 0.893 (after the stop)Test 25, day 27: p = 0.912 (after the stop)Test 25, day 28: p = 0.966 (after the stop)stop, day 1Test 26, day 1: p = 0.593 (a look)Test 26, day 2: p = 0.404 (a look)Test 26, day 3: p = 0.056 (a look)Test 26, day 4: p = 0.022 (a look: p < 0.05, the test stops with a false winner)Test 26, day 5: p = 0.022 (after the stop)Test 26, day 6: p = 0.045 (after the stop)Test 26, day 7: p = 0.068 (after the stop)Test 26, day 8: p = 0.040 (after the stop)Test 26, day 9: p = 0.028 (after the stop)Test 26, day 10: p = 0.015 (after the stop)Test 26, day 11: p = 0.017 (after the stop)Test 26, day 12: p = 0.034 (after the stop)Test 26, day 13: p = 0.035 (after the stop)Test 26, day 14: p = 0.040 (after the stop)Test 26, day 15: p = 0.051 (after the stop)Test 26, day 16: p = 0.050 (after the stop)Test 26, day 17: p = 0.055 (after the stop)Test 26, day 18: p = 0.123 (after the stop)Test 26, day 19: p = 0.084 (after the stop)Test 26, day 20: p = 0.089 (after the stop)Test 26, day 21: p = 0.074 (after the stop)Test 26, day 22: p = 0.098 (after the stop)Test 26, day 23: p = 0.180 (after the stop)Test 26, day 24: p = 0.258 (after the stop)Test 26, day 25: p = 0.171 (after the stop)Test 26, day 26: p = 0.233 (after the stop)Test 26, day 27: p = 0.331 (after the stop)Test 26, day 28: p = 0.282 (after the stop)stop, day 4Test 27, day 1: p = 0.273 (a look)Test 27, day 2: p = 0.606 (a look)Test 27, day 3: p = 0.451 (a look)Test 27, day 4: p = 0.702 (a look)Test 27, day 5: p = 0.865 (a look)Test 27, day 6: p = 0.284 (a look)Test 27, day 7: p = 0.565 (a look)Test 27, day 8: p = 0.381 (a look)Test 27, day 9: p = 0.760 (a look)Test 27, day 10: p = 0.745 (a look)Test 27, day 11: p = 0.612 (a look)Test 27, day 12: p = 0.337 (a look)Test 27, day 13: p = 0.356 (a look)Test 27, day 14: p = 0.443 (a look)Test 27, day 15: p = 0.348 (a look)Test 27, day 16: p = 0.339 (a look)Test 27, day 17: p = 0.231 (a look)Test 27, day 18: p = 0.137 (a look)Test 27, day 19: p = 0.118 (a look)Test 27, day 20: p = 0.163 (a look)Test 27, day 21: p = 0.260 (a look)Test 27, day 22: p = 0.230 (a look)Test 27, day 23: p = 0.274 (a look)Test 27, day 24: p = 0.310 (a look)Test 27, day 25: p = 0.453 (a look)Test 27, day 26: p = 0.674 (a look)Test 27, day 27: p = 0.453 (a look)Test 27, day 28: p = 0.426 (a look)Test 28, day 1: p = 0.623 (a look)Test 28, day 2: p = 0.248 (a look)Test 28, day 3: p = 0.930 (a look)Test 28, day 4: p = 0.970 (a look)Test 28, day 5: p = 0.798 (a look)Test 28, day 6: p = 0.876 (a look)Test 28, day 7: p = 0.829 (a look)Test 28, day 8: p = 0.850 (a look)Test 28, day 9: p = 0.723 (a look)Test 28, day 10: p = 0.273 (a look)Test 28, day 11: p = 0.142 (a look)Test 28, day 12: p = 0.156 (a look)Test 28, day 13: p = 0.163 (a look)Test 28, day 14: p = 0.308 (a look)Test 28, day 15: p = 0.334 (a look)Test 28, day 16: p = 0.214 (a look)Test 28, day 17: p = 0.182 (a look)Test 28, day 18: p = 0.265 (a look)Test 28, day 19: p = 0.192 (a look)Test 28, day 20: p = 0.121 (a look)Test 28, day 21: p = 0.234 (a look)Test 28, day 22: p = 0.201 (a look)Test 28, day 23: p = 0.242 (a look)Test 28, day 24: p = 0.204 (a look)Test 28, day 25: p = 0.252 (a look)Test 28, day 26: p = 0.242 (a look)Test 28, day 27: p = 0.132 (a look)Test 28, day 28: p = 0.122 (a look)Test 29, day 1: p = 0.591 (a look)Test 29, day 2: p = 0.957 (a look)Test 29, day 3: p = 0.724 (a look)Test 29, day 4: p = 0.301 (a look)Test 29, day 5: p = 0.584 (a look)Test 29, day 6: p = 0.791 (a look)Test 29, day 7: p = 0.796 (a look)Test 29, day 8: p = 0.656 (a look)Test 29, day 9: p = 0.960 (a look)Test 29, day 10: p = 0.631 (a look)Test 29, day 11: p = 0.599 (a look)Test 29, day 12: p = 0.599 (a look)Test 29, day 13: p = 0.643 (a look)Test 29, day 14: p = 0.471 (a look)Test 29, day 15: p = 0.474 (a look)Test 29, day 16: p = 0.582 (a look)Test 29, day 17: p = 0.352 (a look)Test 29, day 18: p = 0.220 (a look)Test 29, day 19: p = 0.307 (a look)Test 29, day 20: p = 0.287 (a look)Test 29, day 21: p = 0.319 (a look)Test 29, day 22: p = 0.311 (a look)Test 29, day 23: p = 0.243 (a look)Test 29, day 24: p = 0.187 (a look)Test 29, day 25: p = 0.373 (a look)Test 29, day 26: p = 0.670 (a look)Test 29, day 27: p = 0.720 (a look)Test 29, day 28: p = 0.615 (a look)Test 30, day 1: p = 0.970 (a look)Test 30, day 2: p = 0.662 (a look)Test 30, day 3: p = 0.436 (a look)Test 30, day 4: p = 0.673 (a look)Test 30, day 5: p = 0.471 (a look)Test 30, day 6: p = 0.310 (a look)Test 30, day 7: p = 0.136 (a look)Test 30, day 8: p = 0.336 (a look)Test 30, day 9: p = 0.674 (a look)Test 30, day 10: p = 0.595 (a look)Test 30, day 11: p = 0.612 (a look)Test 30, day 12: p = 0.529 (a look)Test 30, day 13: p = 0.582 (a look)Test 30, day 14: p = 0.450 (a look)Test 30, day 15: p = 0.636 (a look)Test 30, day 16: p = 0.468 (a look)Test 30, day 17: p = 0.321 (a look)Test 30, day 18: p = 0.439 (a look)Test 30, day 19: p = 0.318 (a look)Test 30, day 20: p = 0.169 (a look)Test 30, day 21: p = 0.278 (a look)Test 30, day 22: p = 0.286 (a look)Test 30, day 23: p = 0.194 (a look)Test 30, day 24: p = 0.409 (a look)Test 30, day 25: p = 0.497 (a look)Test 30, day 26: p = 0.589 (a look)Test 30, day 27: p = 0.566 (a look)Test 30, day 28: p = 0.440 (a look)Test 31, day 1: p = 0.970 (a look)Test 31, day 2: p = 0.270 (a look)Test 31, day 3: p = 0.895 (a look)Test 31, day 4: p = 0.879 (a look)Test 31, day 5: p = 0.747 (a look)Test 31, day 6: p = 0.664 (a look)Test 31, day 7: p = 0.275 (a look)Test 31, day 8: p = 0.747 (a look)Test 31, day 9: p = 0.909 (a look)Test 31, day 10: p = 0.875 (a look)Test 31, day 11: p = 0.670 (a look)Test 31, day 12: p = 0.783 (a look)Test 31, day 13: p = 0.874 (a look)Test 31, day 14: p = 0.862 (a look)Test 31, day 15: p = 0.976 (a look)Test 31, day 16: p = 0.724 (a look)Test 31, day 17: p = 0.465 (a look)Test 31, day 18: p = 0.565 (a look)Test 31, day 19: p = 0.581 (a look)Test 31, day 20: p = 0.714 (a look)Test 31, day 21: p = 0.565 (a look)Test 31, day 22: p = 0.690 (a look)Test 31, day 23: p = 0.917 (a look)Test 31, day 24: p = 0.994 (a look)Test 31, day 25: p = 0.976 (a look)Test 31, day 26: p = 0.982 (a look)Test 31, day 27: p = 0.854 (a look)Test 31, day 28: p = 0.723 (a look)Test 32, day 1: p = 0.078 (a look)Test 32, day 2: p = 0.013 (a look: p < 0.05, the test stops with a false winner)Test 32, day 3: p = 0.019 (after the stop)Test 32, day 4: p = 0.006 (after the stop)Test 32, day 5: p = <0.001 (after the stop)Test 32, day 6: p = 0.002 (after the stop)Test 32, day 7: p = <0.001 (after the stop)Test 32, day 8: p = 0.004 (after the stop)Test 32, day 9: p = 0.009 (after the stop)Test 32, day 10: p = 0.012 (after the stop)Test 32, day 11: p = 0.021 (after the stop)Test 32, day 12: p = 0.065 (after the stop)Test 32, day 13: p = 0.195 (after the stop)Test 32, day 14: p = 0.263 (after the stop)Test 32, day 15: p = 0.504 (after the stop)Test 32, day 16: p = 0.620 (after the stop)Test 32, day 17: p = 0.618 (after the stop)Test 32, day 18: p = 0.634 (after the stop)Test 32, day 19: p = 0.707 (after the stop)Test 32, day 20: p = 0.766 (after the stop)Test 32, day 21: p = 0.709 (after the stop)Test 32, day 22: p = 0.620 (after the stop)Test 32, day 23: p = 0.793 (after the stop)Test 32, day 24: p = 0.669 (after the stop)Test 32, day 25: p = 0.698 (after the stop)Test 32, day 26: p = 0.952 (after the stop)Test 32, day 27: p = 0.971 (after the stop)Test 32, day 28: p = 0.943 (after the stop)stop, day 2Test 33, day 1: p = 0.339 (a look)Test 33, day 2: p = 0.914 (a look)Test 33, day 3: p = 0.947 (a look)Test 33, day 4: p = 0.954 (a look)Test 33, day 5: p = 0.473 (a look)Test 33, day 6: p = 0.684 (a look)Test 33, day 7: p = 0.707 (a look)Test 33, day 8: p = 0.626 (a look)Test 33, day 9: p = 0.838 (a look)Test 33, day 10: p = 0.846 (a look)Test 33, day 11: p = 0.853 (a look)Test 33, day 12: p = 0.938 (a look)Test 33, day 13: p = 0.734 (a look)Test 33, day 14: p = 0.659 (a look)Test 33, day 15: p = 0.766 (a look)Test 33, day 16: p = 0.992 (a look)Test 33, day 17: p = 0.724 (a look)Test 33, day 18: p = 0.821 (a look)Test 33, day 19: p = 0.888 (a look)Test 33, day 20: p = 0.764 (a look)Test 33, day 21: p = 0.713 (a look)Test 33, day 22: p = 0.719 (a look)Test 33, day 23: p = 0.892 (a look)Test 33, day 24: p = 0.633 (a look)Test 33, day 25: p = 0.908 (a look)Test 33, day 26: p = 0.816 (a look)Test 33, day 27: p = 0.819 (a look)Test 33, day 28: p = 0.578 (a look)Test 34, day 1: p = 0.757 (a look)Test 34, day 2: p = 0.724 (a look)Test 34, day 3: p = 0.519 (a look)Test 34, day 4: p = 0.408 (a look)Test 34, day 5: p = 0.295 (a look)Test 34, day 6: p = 0.552 (a look)Test 34, day 7: p = 0.562 (a look)Test 34, day 8: p = 0.552 (a look)Test 34, day 9: p = 0.750 (a look)Test 34, day 10: p = 0.673 (a look)Test 34, day 11: p = 0.721 (a look)Test 34, day 12: p = 0.895 (a look)Test 34, day 13: p = 0.849 (a look)Test 34, day 14: p = 0.831 (a look)Test 34, day 15: p = 0.760 (a look)Test 34, day 16: p = 0.683 (a look)Test 34, day 17: p = 0.904 (a look)Test 34, day 18: p = 0.823 (a look)Test 34, day 19: p = 0.787 (a look)Test 34, day 20: p = 0.792 (a look)Test 34, day 21: p = 0.561 (a look)Test 34, day 22: p = 0.335 (a look)Test 34, day 23: p = 0.634 (a look)Test 34, day 24: p = 0.681 (a look)Test 34, day 25: p = 0.533 (a look)Test 34, day 26: p = 0.314 (a look)Test 34, day 27: p = 0.341 (a look)Test 34, day 28: p = 0.317 (a look)Test 35, day 1: p = 0.259 (a look)Test 35, day 2: p = 0.011 (a look: p < 0.05, the test stops with a false winner)Test 35, day 3: p = 0.052 (after the stop)Test 35, day 4: p = 0.009 (after the stop)Test 35, day 5: p = 0.009 (after the stop)Test 35, day 6: p = 0.038 (after the stop)Test 35, day 7: p = 0.081 (after the stop)Test 35, day 8: p = 0.135 (after the stop)Test 35, day 9: p = 0.152 (after the stop)Test 35, day 10: p = 0.069 (after the stop)Test 35, day 11: p = 0.165 (after the stop)Test 35, day 12: p = 0.097 (after the stop)Test 35, day 13: p = 0.118 (after the stop)Test 35, day 14: p = 0.138 (after the stop)Test 35, day 15: p = 0.082 (after the stop)Test 35, day 16: p = 0.057 (after the stop)Test 35, day 17: p = 0.076 (after the stop)Test 35, day 18: p = 0.176 (after the stop)Test 35, day 19: p = 0.119 (after the stop)Test 35, day 20: p = 0.200 (after the stop)Test 35, day 21: p = 0.260 (after the stop)Test 35, day 22: p = 0.484 (after the stop)Test 35, day 23: p = 0.696 (after the stop)Test 35, day 24: p = 0.749 (after the stop)Test 35, day 25: p = 0.765 (after the stop)Test 35, day 26: p = 0.869 (after the stop)Test 35, day 27: p = 0.959 (after the stop)Test 35, day 28: p = 0.745 (after the stop)stop, day 2Test 36, day 1: p = 0.672 (a look)Test 36, day 2: p = 0.935 (a look)Test 36, day 3: p = 0.824 (a look)Test 36, day 4: p = 0.878 (a look)Test 36, day 5: p = 0.732 (a look)Test 36, day 6: p = 0.628 (a look)Test 36, day 7: p = 0.624 (a look)Test 36, day 8: p = 0.426 (a look)Test 36, day 9: p = 0.268 (a look)Test 36, day 10: p = 0.252 (a look)Test 36, day 11: p = 0.528 (a look)Test 36, day 12: p = 0.379 (a look)Test 36, day 13: p = 0.472 (a look)Test 36, day 14: p = 0.541 (a look)Test 36, day 15: p = 0.426 (a look)Test 36, day 16: p = 0.529 (a look)Test 36, day 17: p = 0.657 (a look)Test 36, day 18: p = 0.921 (a look)Test 36, day 19: p = 0.806 (a look)Test 36, day 20: p = 0.952 (a look)Test 36, day 21: p = 0.887 (a look)Test 36, day 22: p = 0.696 (a look)Test 36, day 23: p = 0.524 (a look)Test 36, day 24: p = 0.726 (a look)Test 36, day 25: p = 0.588 (a look)Test 36, day 26: p = 0.394 (a look)Test 36, day 27: p = 0.378 (a look)Test 36, day 28: p = 0.338 (a look)Test 37, day 1: p = 0.276 (a look)Test 37, day 2: p = 0.467 (a look)Test 37, day 3: p = 0.808 (a look)Test 37, day 4: p = 0.581 (a look)Test 37, day 5: p = 0.528 (a look)Test 37, day 6: p = 0.815 (a look)Test 37, day 7: p = 0.565 (a look)Test 37, day 8: p = 0.989 (a look)Test 37, day 9: p = 0.849 (a look)Test 37, day 10: p = 0.828 (a look)Test 37, day 11: p = 0.982 (a look)Test 37, day 12: p = 0.692 (a look)Test 37, day 13: p = 0.992 (a look)Test 37, day 14: p = 0.854 (a look)Test 37, day 15: p = 0.671 (a look)Test 37, day 16: p = 0.450 (a look)Test 37, day 17: p = 0.623 (a look)Test 37, day 18: p = 0.466 (a look)Test 37, day 19: p = 0.792 (a look)Test 37, day 20: p = 0.579 (a look)Test 37, day 21: p = 0.537 (a look)Test 37, day 22: p = 0.757 (a look)Test 37, day 23: p = 0.762 (a look)Test 37, day 24: p = 0.685 (a look)Test 37, day 25: p = 0.355 (a look)Test 37, day 26: p = 0.330 (a look)Test 37, day 27: p = 0.346 (a look)Test 37, day 28: p = 0.263 (a look)Test 38, day 1: p = 0.970 (a look)Test 38, day 2: p = 0.979 (a look)Test 38, day 3: p = 0.599 (a look)Test 38, day 4: p = 1.000 (a look)Test 38, day 5: p = 0.932 (a look)Test 38, day 6: p = 0.651 (a look)Test 38, day 7: p = 0.896 (a look)Test 38, day 8: p = 0.903 (a look)Test 38, day 9: p = 0.970 (a look)Test 38, day 10: p = 0.981 (a look)Test 38, day 11: p = 0.572 (a look)Test 38, day 12: p = 0.414 (a look)Test 38, day 13: p = 0.266 (a look)Test 38, day 14: p = 0.136 (a look)Test 38, day 15: p = 0.119 (a look)Test 38, day 16: p = 0.272 (a look)Test 38, day 17: p = 0.295 (a look)Test 38, day 18: p = 0.245 (a look)Test 38, day 19: p = 0.285 (a look)Test 38, day 20: p = 0.278 (a look)Test 38, day 21: p = 0.214 (a look)Test 38, day 22: p = 0.225 (a look)Test 38, day 23: p = 0.327 (a look)Test 38, day 24: p = 0.314 (a look)Test 38, day 25: p = 0.317 (a look)Test 38, day 26: p = 0.485 (a look)Test 38, day 27: p = 0.335 (a look)Test 38, day 28: p = 0.315 (a look)Test 39, day 1: p = 0.848 (a look)Test 39, day 2: p = 0.727 (a look)Test 39, day 3: p = 0.861 (a look)Test 39, day 4: p = 0.529 (a look)Test 39, day 5: p = 0.246 (a look)Test 39, day 6: p = 0.139 (a look)Test 39, day 7: p = 0.364 (a look)Test 39, day 8: p = 0.359 (a look)Test 39, day 9: p = 0.858 (a look)Test 39, day 10: p = 0.735 (a look)Test 39, day 11: p = 1.000 (a look)Test 39, day 12: p = 0.774 (a look)Test 39, day 13: p = 0.687 (a look)Test 39, day 14: p = 0.587 (a look)Test 39, day 15: p = 0.729 (a look)Test 39, day 16: p = 0.553 (a look)Test 39, day 17: p = 0.498 (a look)Test 39, day 18: p = 0.322 (a look)Test 39, day 19: p = 0.197 (a look)Test 39, day 20: p = 0.200 (a look)Test 39, day 21: p = 0.174 (a look)Test 39, day 22: p = 0.290 (a look)Test 39, day 23: p = 0.390 (a look)Test 39, day 24: p = 0.353 (a look)Test 39, day 25: p = 0.359 (a look)Test 39, day 26: p = 0.246 (a look)Test 39, day 27: p = 0.296 (a look)Test 39, day 28: p = 0.406 (a look)Test 40, day 1: p = 0.494 (a look)Test 40, day 2: p = 0.830 (a look)Test 40, day 3: p = 0.843 (a look)Test 40, day 4: p = 0.213 (a look)Test 40, day 5: p = 0.539 (a look)Test 40, day 6: p = 0.342 (a look)Test 40, day 7: p = 0.280 (a look)Test 40, day 8: p = 0.325 (a look)Test 40, day 9: p = 0.182 (a look)Test 40, day 10: p = 0.360 (a look)Test 40, day 11: p = 0.550 (a look)Test 40, day 12: p = 0.692 (a look)Test 40, day 13: p = 0.916 (a look)Test 40, day 14: p = 0.737 (a look)Test 40, day 15: p = 0.805 (a look)Test 40, day 16: p = 0.819 (a look)Test 40, day 17: p = 0.817 (a look)Test 40, day 18: p = 0.935 (a look)Test 40, day 19: p = 0.847 (a look)Test 40, day 20: p = 0.804 (a look)Test 40, day 21: p = 0.940 (a look)Test 40, day 22: p = 0.948 (a look)Test 40, day 23: p = 0.874 (a look)Test 40, day 24: p = 0.925 (a look)Test 40, day 25: p = 0.994 (a look)Test 40, day 26: p = 0.916 (a look)Test 40, day 27: p = 0.965 (a look)Test 40, day 28: p = 0.931 (a look)17142128Day of the test
40 A/A tests, one row eachTest 1, day 1: p = 0.428 (a look)Test 1, day 2: p = 0.611 (a look)Test 1, day 3: p = 0.584 (a look)Test 1, day 4: p = 0.350 (a look)Test 1, day 5: p = 0.297 (a look)Test 1, day 6: p = 0.156 (a look)Test 1, day 7: p = 0.362 (a look)Test 1, day 8: p = 0.234 (a look)Test 1, day 9: p = 0.321 (a look)Test 1, day 10: p = 0.231 (a look)Test 1, day 11: p = 0.222 (a look)Test 1, day 12: p = 0.190 (a look)Test 1, day 13: p = 0.249 (a look)Test 1, day 14: p = 0.169 (a look)Test 1, day 15: p = 0.340 (a look)Test 1, day 16: p = 0.524 (a look)Test 1, day 17: p = 0.605 (a look)Test 1, day 18: p = 0.441 (a look)Test 1, day 19: p = 0.369 (a look)Test 1, day 20: p = 0.345 (a look)Test 1, day 21: p = 0.512 (a look)Test 1, day 22: p = 0.777 (a look)Test 1, day 23: p = 0.956 (a look)Test 1, day 24: p = 0.858 (a look)Test 1, day 25: p = 0.692 (a look)Test 1, day 26: p = 0.806 (a look)Test 1, day 27: p = 0.798 (a look)Test 1, day 28: p = 0.880 (a look)Test 2, day 1: p = 0.099 (a look)Test 2, day 2: p = 0.606 (a look)Test 2, day 3: p = 0.860 (a look)Test 2, day 4: p = 0.744 (a look)Test 2, day 5: p = 0.945 (a look)Test 2, day 6: p = 0.629 (a look)Test 2, day 7: p = 0.341 (a look)Test 2, day 8: p = 0.264 (a look)Test 2, day 9: p = 0.204 (a look)Test 2, day 10: p = 0.106 (a look)Test 2, day 11: p = 0.076 (a look)Test 2, day 12: p = 0.039 (a look: p < 0.05, the test stops with a false winner)Test 2, day 13: p = 0.042 (after the stop)Test 2, day 14: p = 0.011 (after the stop)Test 2, day 15: p = 0.008 (after the stop)Test 2, day 16: p = 0.018 (after the stop)Test 2, day 17: p = 0.011 (after the stop)Test 2, day 18: p = 0.046 (after the stop)Test 2, day 19: p = 0.058 (after the stop)Test 2, day 20: p = 0.048 (after the stop)Test 2, day 21: p = 0.027 (after the stop)Test 2, day 22: p = 0.065 (after the stop)Test 2, day 23: p = 0.066 (after the stop)Test 2, day 24: p = 0.022 (after the stop)Test 2, day 25: p = 0.040 (after the stop)Test 2, day 26: p = 0.042 (after the stop)Test 2, day 27: p = 0.030 (after the stop)Test 2, day 28: p = 0.036 (after the stop)Test 3, day 1: p = 0.970 (a look)Test 3, day 2: p = 0.936 (a look)Test 3, day 3: p = 0.777 (a look)Test 3, day 4: p = 0.533 (a look)Test 3, day 5: p = 0.499 (a look)Test 3, day 6: p = 0.537 (a look)Test 3, day 7: p = 0.390 (a look)Test 3, day 8: p = 0.271 (a look)Test 3, day 9: p = 0.164 (a look)Test 3, day 10: p = 0.186 (a look)Test 3, day 11: p = 0.151 (a look)Test 3, day 12: p = 0.213 (a look)Test 3, day 13: p = 0.086 (a look)Test 3, day 14: p = 0.096 (a look)Test 3, day 15: p = 0.156 (a look)Test 3, day 16: p = 0.184 (a look)Test 3, day 17: p = 0.122 (a look)Test 3, day 18: p = 0.105 (a look)Test 3, day 19: p = 0.110 (a look)Test 3, day 20: p = 0.208 (a look)Test 3, day 21: p = 0.199 (a look)Test 3, day 22: p = 0.140 (a look)Test 3, day 23: p = 0.118 (a look)Test 3, day 24: p = 0.155 (a look)Test 3, day 25: p = 0.148 (a look)Test 3, day 26: p = 0.224 (a look)Test 3, day 27: p = 0.173 (a look)Test 3, day 28: p = 0.184 (a look)Test 4, day 1: p = 0.232 (a look)Test 4, day 2: p = 0.455 (a look)Test 4, day 3: p = 0.601 (a look)Test 4, day 4: p = 0.806 (a look)Test 4, day 5: p = 0.567 (a look)Test 4, day 6: p = 0.963 (a look)Test 4, day 7: p = 0.977 (a look)Test 4, day 8: p = 0.852 (a look)Test 4, day 9: p = 0.668 (a look)Test 4, day 10: p = 0.565 (a look)Test 4, day 11: p = 0.523 (a look)Test 4, day 12: p = 0.710 (a look)Test 4, day 13: p = 0.867 (a look)Test 4, day 14: p = 0.871 (a look)Test 4, day 15: p = 0.544 (a look)Test 4, day 16: p = 0.525 (a look)Test 4, day 17: p = 0.832 (a look)Test 4, day 18: p = 0.971 (a look)Test 4, day 19: p = 0.930 (a look)Test 4, day 20: p = 0.925 (a look)Test 4, day 21: p = 0.803 (a look)Test 4, day 22: p = 0.968 (a look)Test 4, day 23: p = 0.781 (a look)Test 4, day 24: p = 0.762 (a look)Test 4, day 25: p = 0.767 (a look)Test 4, day 26: p = 0.911 (a look)Test 4, day 27: p = 0.971 (a look)Test 4, day 28: p = 0.903 (a look)Test 5, day 1: p = 0.646 (a look)Test 5, day 2: p = 0.089 (a look)Test 5, day 3: p = 0.035 (a look: p < 0.05, the test stops with a false winner)Test 5, day 4: p = 0.229 (after the stop)Test 5, day 5: p = 0.444 (after the stop)Test 5, day 6: p = 0.113 (after the stop)Test 5, day 7: p = 0.159 (after the stop)Test 5, day 8: p = 0.276 (after the stop)Test 5, day 9: p = 0.224 (after the stop)Test 5, day 10: p = 0.194 (after the stop)Test 5, day 11: p = 0.243 (after the stop)Test 5, day 12: p = 0.357 (after the stop)Test 5, day 13: p = 0.500 (after the stop)Test 5, day 14: p = 0.434 (after the stop)Test 5, day 15: p = 0.651 (after the stop)Test 5, day 16: p = 0.621 (after the stop)Test 5, day 17: p = 0.733 (after the stop)Test 5, day 18: p = 0.979 (after the stop)Test 5, day 19: p = 0.958 (after the stop)Test 5, day 20: p = 0.872 (after the stop)Test 5, day 21: p = 0.636 (after the stop)Test 5, day 22: p = 0.500 (after the stop)Test 5, day 23: p = 0.628 (after the stop)Test 5, day 24: p = 0.816 (after the stop)Test 5, day 25: p = 0.642 (after the stop)Test 5, day 26: p = 0.648 (after the stop)Test 5, day 27: p = 0.665 (after the stop)Test 5, day 28: p = 0.277 (after the stop)Test 6, day 1: p = 0.386 (a look)Test 6, day 2: p = 0.667 (a look)Test 6, day 3: p = 0.692 (a look)Test 6, day 4: p = 0.529 (a look)Test 6, day 5: p = 0.194 (a look)Test 6, day 6: p = 0.367 (a look)Test 6, day 7: p = 0.489 (a look)Test 6, day 8: p = 0.434 (a look)Test 6, day 9: p = 0.526 (a look)Test 6, day 10: p = 0.847 (a look)Test 6, day 11: p = 1.000 (a look)Test 6, day 12: p = 0.939 (a look)Test 6, day 13: p = 0.800 (a look)Test 6, day 14: p = 0.654 (a look)Test 6, day 15: p = 0.568 (a look)Test 6, day 16: p = 0.266 (a look)Test 6, day 17: p = 0.252 (a look)Test 6, day 18: p = 0.609 (a look)Test 6, day 19: p = 0.513 (a look)Test 6, day 20: p = 0.580 (a look)Test 6, day 21: p = 0.666 (a look)Test 6, day 22: p = 0.650 (a look)Test 6, day 23: p = 0.868 (a look)Test 6, day 24: p = 0.852 (a look)Test 6, day 25: p = 0.790 (a look)Test 6, day 26: p = 0.800 (a look)Test 6, day 27: p = 0.676 (a look)Test 6, day 28: p = 0.605 (a look)Test 7, day 1: p = 0.135 (a look)Test 7, day 2: p = 0.765 (a look)Test 7, day 3: p = 0.164 (a look)Test 7, day 4: p = 0.168 (a look)Test 7, day 5: p = 0.123 (a look)Test 7, day 6: p = 0.090 (a look)Test 7, day 7: p = 0.369 (a look)Test 7, day 8: p = 0.491 (a look)Test 7, day 9: p = 0.485 (a look)Test 7, day 10: p = 0.673 (a look)Test 7, day 11: p = 0.513 (a look)Test 7, day 12: p = 0.553 (a look)Test 7, day 13: p = 0.612 (a look)Test 7, day 14: p = 0.569 (a look)Test 7, day 15: p = 0.523 (a look)Test 7, day 16: p = 0.241 (a look)Test 7, day 17: p = 0.448 (a look)Test 7, day 18: p = 0.388 (a look)Test 7, day 19: p = 0.186 (a look)Test 7, day 20: p = 0.362 (a look)Test 7, day 21: p = 0.356 (a look)Test 7, day 22: p = 0.354 (a look)Test 7, day 23: p = 0.441 (a look)Test 7, day 24: p = 0.581 (a look)Test 7, day 25: p = 0.642 (a look)Test 7, day 26: p = 0.899 (a look)Test 7, day 27: p = 0.769 (a look)Test 7, day 28: p = 0.880 (a look)Test 8, day 1: p = 0.729 (a look)Test 8, day 2: p = 0.606 (a look)Test 8, day 3: p = 0.537 (a look)Test 8, day 4: p = 0.985 (a look)Test 8, day 5: p = 0.707 (a look)Test 8, day 6: p = 0.988 (a look)Test 8, day 7: p = 0.665 (a look)Test 8, day 8: p = 0.829 (a look)Test 8, day 9: p = 0.629 (a look)Test 8, day 10: p = 0.981 (a look)Test 8, day 11: p = 0.704 (a look)Test 8, day 12: p = 0.708 (a look)Test 8, day 13: p = 0.775 (a look)Test 8, day 14: p = 0.691 (a look)Test 8, day 15: p = 0.898 (a look)Test 8, day 16: p = 0.621 (a look)Test 8, day 17: p = 0.740 (a look)Test 8, day 18: p = 0.686 (a look)Test 8, day 19: p = 0.800 (a look)Test 8, day 20: p = 0.824 (a look)Test 8, day 21: p = 0.751 (a look)Test 8, day 22: p = 0.714 (a look)Test 8, day 23: p = 0.685 (a look)Test 8, day 24: p = 0.497 (a look)Test 8, day 25: p = 0.477 (a look)Test 8, day 26: p = 0.595 (a look)Test 8, day 27: p = 0.436 (a look)Test 8, day 28: p = 0.493 (a look)Test 9, day 1: p = 0.512 (a look)Test 9, day 2: p = 0.034 (a look: p < 0.05, the test stops with a false winner)Test 9, day 3: p = 0.066 (after the stop)Test 9, day 4: p = 0.243 (after the stop)Test 9, day 5: p = 0.119 (after the stop)Test 9, day 6: p = 0.234 (after the stop)Test 9, day 7: p = 0.444 (after the stop)Test 9, day 8: p = 0.256 (after the stop)Test 9, day 9: p = 0.400 (after the stop)Test 9, day 10: p = 0.482 (after the stop)Test 9, day 11: p = 0.738 (after the stop)Test 9, day 12: p = 0.581 (after the stop)Test 9, day 13: p = 0.581 (after the stop)Test 9, day 14: p = 0.475 (after the stop)Test 9, day 15: p = 0.454 (after the stop)Test 9, day 16: p = 0.600 (after the stop)Test 9, day 17: p = 0.719 (after the stop)Test 9, day 18: p = 0.565 (after the stop)Test 9, day 19: p = 0.535 (after the stop)Test 9, day 20: p = 0.469 (after the stop)Test 9, day 21: p = 0.474 (after the stop)Test 9, day 22: p = 0.426 (after the stop)Test 9, day 23: p = 0.356 (after the stop)Test 9, day 24: p = 0.283 (after the stop)Test 9, day 25: p = 0.208 (after the stop)Test 9, day 26: p = 0.226 (after the stop)Test 9, day 27: p = 0.256 (after the stop)Test 9, day 28: p = 0.317 (after the stop)Test 10, day 1: p = 0.057 (a look)Test 10, day 2: p = 0.667 (a look)Test 10, day 3: p = 0.315 (a look)Test 10, day 4: p = 0.865 (a look)Test 10, day 5: p = 0.574 (a look)Test 10, day 6: p = 0.698 (a look)Test 10, day 7: p = 0.508 (a look)Test 10, day 8: p = 0.686 (a look)Test 10, day 9: p = 0.780 (a look)Test 10, day 10: p = 0.664 (a look)Test 10, day 11: p = 0.791 (a look)Test 10, day 12: p = 0.930 (a look)Test 10, day 13: p = 0.899 (a look)Test 10, day 14: p = 0.775 (a look)Test 10, day 15: p = 0.836 (a look)Test 10, day 16: p = 0.939 (a look)Test 10, day 17: p = 0.919 (a look)Test 10, day 18: p = 0.978 (a look)Test 10, day 19: p = 0.779 (a look)Test 10, day 20: p = 0.714 (a look)Test 10, day 21: p = 0.714 (a look)Test 10, day 22: p = 0.832 (a look)Test 10, day 23: p = 0.673 (a look)Test 10, day 24: p = 0.559 (a look)Test 10, day 25: p = 0.567 (a look)Test 10, day 26: p = 0.554 (a look)Test 10, day 27: p = 0.537 (a look)Test 10, day 28: p = 0.603 (a look)Test 11, day 1: p = 0.191 (a look)Test 11, day 2: p = 0.166 (a look)Test 11, day 3: p = 0.241 (a look)Test 11, day 4: p = 0.251 (a look)Test 11, day 5: p = 0.019 (a look: p < 0.05, the test stops with a false winner)Test 11, day 6: p = 0.113 (after the stop)Test 11, day 7: p = 0.162 (after the stop)Test 11, day 8: p = 0.201 (after the stop)Test 11, day 9: p = 0.109 (after the stop)Test 11, day 10: p = 0.109 (after the stop)Test 11, day 11: p = 0.051 (after the stop)Test 11, day 12: p = 0.108 (after the stop)Test 11, day 13: p = 0.072 (after the stop)Test 11, day 14: p = 0.089 (after the stop)Test 11, day 15: p = 0.083 (after the stop)Test 11, day 16: p = 0.223 (after the stop)Test 11, day 17: p = 0.161 (after the stop)Test 11, day 18: p = 0.156 (after the stop)Test 11, day 19: p = 0.072 (after the stop)Test 11, day 20: p = 0.112 (after the stop)Test 11, day 21: p = 0.186 (after the stop)Test 11, day 22: p = 0.163 (after the stop)Test 11, day 23: p = 0.093 (after the stop)Test 11, day 24: p = 0.155 (after the stop)Test 11, day 25: p = 0.174 (after the stop)Test 11, day 26: p = 0.375 (after the stop)Test 11, day 27: p = 0.469 (after the stop)Test 11, day 28: p = 0.698 (after the stop)Test 12, day 1: p = 0.645 (a look)Test 12, day 2: p = 0.236 (a look)Test 12, day 3: p = 0.212 (a look)Test 12, day 4: p = 0.732 (a look)Test 12, day 5: p = 0.798 (a look)Test 12, day 6: p = 0.139 (a look)Test 12, day 7: p = 0.011 (a look: p < 0.05, the test stops with a false winner)Test 12, day 8: p = 0.050 (after the stop)Test 12, day 9: p = 0.124 (after the stop)Test 12, day 10: p = 0.045 (after the stop)Test 12, day 11: p = 0.107 (after the stop)Test 12, day 12: p = 0.058 (after the stop)Test 12, day 13: p = 0.112 (after the stop)Test 12, day 14: p = 0.171 (after the stop)Test 12, day 15: p = 0.126 (after the stop)Test 12, day 16: p = 0.106 (after the stop)Test 12, day 17: p = 0.088 (after the stop)Test 12, day 18: p = 0.150 (after the stop)Test 12, day 19: p = 0.305 (after the stop)Test 12, day 20: p = 0.522 (after the stop)Test 12, day 21: p = 0.623 (after the stop)Test 12, day 22: p = 0.801 (after the stop)Test 12, day 23: p = 0.709 (after the stop)Test 12, day 24: p = 0.518 (after the stop)Test 12, day 25: p = 0.578 (after the stop)Test 12, day 26: p = 0.535 (after the stop)Test 12, day 27: p = 0.558 (after the stop)Test 12, day 28: p = 0.625 (after the stop)Test 13, day 1: p = 0.148 (a look)Test 13, day 2: p = 0.549 (a look)Test 13, day 3: p = 0.250 (a look)Test 13, day 4: p = 0.894 (a look)Test 13, day 5: p = 0.595 (a look)Test 13, day 6: p = 0.532 (a look)Test 13, day 7: p = 0.362 (a look)Test 13, day 8: p = 0.474 (a look)Test 13, day 9: p = 0.459 (a look)Test 13, day 10: p = 0.370 (a look)Test 13, day 11: p = 0.677 (a look)Test 13, day 12: p = 0.912 (a look)Test 13, day 13: p = 0.686 (a look)Test 13, day 14: p = 0.743 (a look)Test 13, day 15: p = 0.560 (a look)Test 13, day 16: p = 0.612 (a look)Test 13, day 17: p = 0.690 (a look)Test 13, day 18: p = 0.766 (a look)Test 13, day 19: p = 0.909 (a look)Test 13, day 20: p = 0.871 (a look)Test 13, day 21: p = 0.980 (a look)Test 13, day 22: p = 0.708 (a look)Test 13, day 23: p = 0.836 (a look)Test 13, day 24: p = 0.803 (a look)Test 13, day 25: p = 0.818 (a look)Test 13, day 26: p = 0.816 (a look)Test 13, day 27: p = 0.831 (a look)Test 13, day 28: p = 0.778 (a look)Test 14, day 1: p = 0.848 (a look)Test 14, day 2: p = 0.244 (a look)Test 14, day 3: p = 0.199 (a look)Test 14, day 4: p = 0.673 (a look)Test 14, day 5: p = 0.878 (a look)Test 14, day 6: p = 0.640 (a look)Test 14, day 7: p = 0.863 (a look)Test 14, day 8: p = 0.627 (a look)Test 14, day 9: p = 0.656 (a look)Test 14, day 10: p = 0.476 (a look)Test 14, day 11: p = 0.407 (a look)Test 14, day 12: p = 0.238 (a look)Test 14, day 13: p = 0.172 (a look)Test 14, day 14: p = 0.169 (a look)Test 14, day 15: p = 0.106 (a look)Test 14, day 16: p = 0.057 (a look)Test 14, day 17: p = 0.072 (a look)Test 14, day 18: p = 0.037 (a look: p < 0.05, the test stops with a false winner)Test 14, day 19: p = 0.026 (after the stop)Test 14, day 20: p = 0.025 (after the stop)Test 14, day 21: p = 0.035 (after the stop)Test 14, day 22: p = 0.025 (after the stop)Test 14, day 23: p = 0.067 (after the stop)Test 14, day 24: p = 0.046 (after the stop)Test 14, day 25: p = 0.034 (after the stop)Test 14, day 26: p = 0.044 (after the stop)Test 14, day 27: p = 0.021 (after the stop)Test 14, day 28: p = 0.027 (after the stop)Test 15, day 1: p = 0.909 (a look)Test 15, day 2: p = 0.538 (a look)Test 15, day 3: p = 0.879 (a look)Test 15, day 4: p = 0.598 (a look)Test 15, day 5: p = 0.438 (a look)Test 15, day 6: p = 0.458 (a look)Test 15, day 7: p = 0.616 (a look)Test 15, day 8: p = 0.851 (a look)Test 15, day 9: p = 0.723 (a look)Test 15, day 10: p = 0.639 (a look)Test 15, day 11: p = 0.740 (a look)Test 15, day 12: p = 0.887 (a look)Test 15, day 13: p = 1.000 (a look)Test 15, day 14: p = 0.816 (a look)Test 15, day 15: p = 0.739 (a look)Test 15, day 16: p = 0.409 (a look)Test 15, day 17: p = 0.495 (a look)Test 15, day 18: p = 0.311 (a look)Test 15, day 19: p = 0.272 (a look)Test 15, day 20: p = 0.234 (a look)Test 15, day 21: p = 0.344 (a look)Test 15, day 22: p = 0.385 (a look)Test 15, day 23: p = 0.260 (a look)Test 15, day 24: p = 0.189 (a look)Test 15, day 25: p = 0.215 (a look)Test 15, day 26: p = 0.282 (a look)Test 15, day 27: p = 0.379 (a look)Test 15, day 28: p = 0.300 (a look)Test 16, day 1: p = 0.595 (a look)Test 16, day 2: p = 0.628 (a look)Test 16, day 3: p = 0.709 (a look)Test 16, day 4: p = 0.985 (a look)Test 16, day 5: p = 0.610 (a look)Test 16, day 6: p = 0.238 (a look)Test 16, day 7: p = 0.195 (a look)Test 16, day 8: p = 0.300 (a look)Test 16, day 9: p = 0.274 (a look)Test 16, day 10: p = 0.745 (a look)Test 16, day 11: p = 0.629 (a look)Test 16, day 12: p = 0.775 (a look)Test 16, day 13: p = 0.719 (a look)Test 16, day 14: p = 0.943 (a look)Test 16, day 15: p = 0.813 (a look)Test 16, day 16: p = 0.614 (a look)Test 16, day 17: p = 0.684 (a look)Test 16, day 18: p = 0.719 (a look)Test 16, day 19: p = 0.416 (a look)Test 16, day 20: p = 0.670 (a look)Test 16, day 21: p = 0.683 (a look)Test 16, day 22: p = 0.764 (a look)Test 16, day 23: p = 0.562 (a look)Test 16, day 24: p = 0.533 (a look)Test 16, day 25: p = 0.512 (a look)Test 16, day 26: p = 0.464 (a look)Test 16, day 27: p = 0.394 (a look)Test 16, day 28: p = 0.407 (a look)Test 17, day 1: p = 0.438 (a look)Test 17, day 2: p = 0.589 (a look)Test 17, day 3: p = 0.186 (a look)Test 17, day 4: p = 0.594 (a look)Test 17, day 5: p = 0.517 (a look)Test 17, day 6: p = 0.708 (a look)Test 17, day 7: p = 0.954 (a look)Test 17, day 8: p = 0.882 (a look)Test 17, day 9: p = 0.683 (a look)Test 17, day 10: p = 0.942 (a look)Test 17, day 11: p = 0.854 (a look)Test 17, day 12: p = 0.974 (a look)Test 17, day 13: p = 0.857 (a look)Test 17, day 14: p = 0.854 (a look)Test 17, day 15: p = 0.737 (a look)Test 17, day 16: p = 0.833 (a look)Test 17, day 17: p = 0.904 (a look)Test 17, day 18: p = 0.943 (a look)Test 17, day 19: p = 0.965 (a look)Test 17, day 20: p = 0.824 (a look)Test 17, day 21: p = 0.815 (a look)Test 17, day 22: p = 0.890 (a look)Test 17, day 23: p = 0.848 (a look)Test 17, day 24: p = 0.656 (a look)Test 17, day 25: p = 0.635 (a look)Test 17, day 26: p = 0.574 (a look)Test 17, day 27: p = 0.551 (a look)Test 17, day 28: p = 0.452 (a look)Test 18, day 1: p = 0.970 (a look)Test 18, day 2: p = 0.935 (a look)Test 18, day 3: p = 0.691 (a look)Test 18, day 4: p = 0.788 (a look)Test 18, day 5: p = 0.430 (a look)Test 18, day 6: p = 0.381 (a look)Test 18, day 7: p = 0.623 (a look)Test 18, day 8: p = 0.498 (a look)Test 18, day 9: p = 0.413 (a look)Test 18, day 10: p = 0.497 (a look)Test 18, day 11: p = 0.262 (a look)Test 18, day 12: p = 0.245 (a look)Test 18, day 13: p = 0.655 (a look)Test 18, day 14: p = 0.615 (a look)Test 18, day 15: p = 0.441 (a look)Test 18, day 16: p = 0.400 (a look)Test 18, day 17: p = 0.308 (a look)Test 18, day 18: p = 0.368 (a look)Test 18, day 19: p = 0.139 (a look)Test 18, day 20: p = 0.142 (a look)Test 18, day 21: p = 0.230 (a look)Test 18, day 22: p = 0.193 (a look)Test 18, day 23: p = 0.161 (a look)Test 18, day 24: p = 0.119 (a look)Test 18, day 25: p = 0.231 (a look)Test 18, day 26: p = 0.199 (a look)Test 18, day 27: p = 0.207 (a look)Test 18, day 28: p = 0.229 (a look)Test 19, day 1: p = 0.727 (a look)Test 19, day 2: p = 0.702 (a look)Test 19, day 3: p = 1.000 (a look)Test 19, day 4: p = 0.728 (a look)Test 19, day 5: p = 0.209 (a look)Test 19, day 6: p = 0.461 (a look)Test 19, day 7: p = 0.485 (a look)Test 19, day 8: p = 0.430 (a look)Test 19, day 9: p = 0.565 (a look)Test 19, day 10: p = 0.467 (a look)Test 19, day 11: p = 0.533 (a look)Test 19, day 12: p = 0.799 (a look)Test 19, day 13: p = 0.718 (a look)Test 19, day 14: p = 0.767 (a look)Test 19, day 15: p = 0.745 (a look)Test 19, day 16: p = 0.579 (a look)Test 19, day 17: p = 0.697 (a look)Test 19, day 18: p = 0.822 (a look)Test 19, day 19: p = 0.888 (a look)Test 19, day 20: p = 0.959 (a look)Test 19, day 21: p = 0.980 (a look)Test 19, day 22: p = 0.896 (a look)Test 19, day 23: p = 0.962 (a look)Test 19, day 24: p = 0.833 (a look)Test 19, day 25: p = 0.848 (a look)Test 19, day 26: p = 0.674 (a look)Test 19, day 27: p = 0.718 (a look)Test 19, day 28: p = 0.723 (a look)Test 20, day 1: p = 0.273 (a look)Test 20, day 2: p = 0.957 (a look)Test 20, day 3: p = 0.648 (a look)Test 20, day 4: p = 0.822 (a look)Test 20, day 5: p = 0.775 (a look)Test 20, day 6: p = 0.988 (a look)Test 20, day 7: p = 0.977 (a look)Test 20, day 8: p = 0.883 (a look)Test 20, day 9: p = 0.650 (a look)Test 20, day 10: p = 0.711 (a look)Test 20, day 11: p = 0.648 (a look)Test 20, day 12: p = 0.563 (a look)Test 20, day 13: p = 0.543 (a look)Test 20, day 14: p = 0.816 (a look)Test 20, day 15: p = 0.423 (a look)Test 20, day 16: p = 0.461 (a look)Test 20, day 17: p = 0.514 (a look)Test 20, day 18: p = 0.387 (a look)Test 20, day 19: p = 0.244 (a look)Test 20, day 20: p = 0.189 (a look)Test 20, day 21: p = 0.206 (a look)Test 20, day 22: p = 0.210 (a look)Test 20, day 23: p = 0.148 (a look)Test 20, day 24: p = 0.168 (a look)Test 20, day 25: p = 0.113 (a look)Test 20, day 26: p = 0.136 (a look)Test 20, day 27: p = 0.095 (a look)Test 20, day 28: p = 0.111 (a look)Test 21, day 1: p = 0.970 (a look)Test 21, day 2: p = 0.872 (a look)Test 21, day 3: p = 0.725 (a look)Test 21, day 4: p = 0.324 (a look)Test 21, day 5: p = 0.433 (a look)Test 21, day 6: p = 0.359 (a look)Test 21, day 7: p = 0.818 (a look)Test 21, day 8: p = 0.935 (a look)Test 21, day 9: p = 0.789 (a look)Test 21, day 10: p = 0.781 (a look)Test 21, day 11: p = 0.662 (a look)Test 21, day 12: p = 0.921 (a look)Test 21, day 13: p = 0.656 (a look)Test 21, day 14: p = 0.610 (a look)Test 21, day 15: p = 0.508 (a look)Test 21, day 16: p = 0.613 (a look)Test 21, day 17: p = 0.650 (a look)Test 21, day 18: p = 0.760 (a look)Test 21, day 19: p = 0.979 (a look)Test 21, day 20: p = 0.858 (a look)Test 21, day 21: p = 0.521 (a look)Test 21, day 22: p = 0.473 (a look)Test 21, day 23: p = 0.170 (a look)Test 21, day 24: p = 0.215 (a look)Test 21, day 25: p = 0.218 (a look)Test 21, day 26: p = 0.271 (a look)Test 21, day 27: p = 0.370 (a look)Test 21, day 28: p = 0.391 (a look)Test 22, day 1: p = 0.103 (a look)Test 22, day 2: p = 0.257 (a look)Test 22, day 3: p = 0.167 (a look)Test 22, day 4: p = 0.158 (a look)Test 22, day 5: p = 0.505 (a look)Test 22, day 6: p = 0.697 (a look)Test 22, day 7: p = 0.697 (a look)Test 22, day 8: p = 0.205 (a look)Test 22, day 9: p = 0.191 (a look)Test 22, day 10: p = 0.419 (a look)Test 22, day 11: p = 0.696 (a look)Test 22, day 12: p = 0.783 (a look)Test 22, day 13: p = 0.672 (a look)Test 22, day 14: p = 0.676 (a look)Test 22, day 15: p = 0.568 (a look)Test 22, day 16: p = 0.724 (a look)Test 22, day 17: p = 0.711 (a look)Test 22, day 18: p = 0.455 (a look)Test 22, day 19: p = 0.425 (a look)Test 22, day 20: p = 0.233 (a look)Test 22, day 21: p = 0.224 (a look)Test 22, day 22: p = 0.241 (a look)Test 22, day 23: p = 0.126 (a look)Test 22, day 24: p = 0.149 (a look)Test 22, day 25: p = 0.114 (a look)Test 22, day 26: p = 0.107 (a look)Test 22, day 27: p = 0.172 (a look)Test 22, day 28: p = 0.298 (a look)Test 23, day 1: p = 0.276 (a look)Test 23, day 2: p = 0.200 (a look)Test 23, day 3: p = 0.597 (a look)Test 23, day 4: p = 0.985 (a look)Test 23, day 5: p = 0.824 (a look)Test 23, day 6: p = 0.876 (a look)Test 23, day 7: p = 0.896 (a look)Test 23, day 8: p = 0.829 (a look)Test 23, day 9: p = 0.711 (a look)Test 23, day 10: p = 0.654 (a look)Test 23, day 11: p = 0.489 (a look)Test 23, day 12: p = 0.567 (a look)Test 23, day 13: p = 0.397 (a look)Test 23, day 14: p = 0.397 (a look)Test 23, day 15: p = 0.396 (a look)Test 23, day 16: p = 0.560 (a look)Test 23, day 17: p = 0.559 (a look)Test 23, day 18: p = 0.640 (a look)Test 23, day 19: p = 0.827 (a look)Test 23, day 20: p = 0.707 (a look)Test 23, day 21: p = 0.790 (a look)Test 23, day 22: p = 0.877 (a look)Test 23, day 23: p = 0.918 (a look)Test 23, day 24: p = 0.640 (a look)Test 23, day 25: p = 0.731 (a look)Test 23, day 26: p = 0.544 (a look)Test 23, day 27: p = 0.445 (a look)Test 23, day 28: p = 0.493 (a look)Test 24, day 1: p = 0.877 (a look)Test 24, day 2: p = 0.514 (a look)Test 24, day 3: p = 0.723 (a look)Test 24, day 4: p = 0.502 (a look)Test 24, day 5: p = 0.441 (a look)Test 24, day 6: p = 0.134 (a look)Test 24, day 7: p = 0.122 (a look)Test 24, day 8: p = 0.199 (a look)Test 24, day 9: p = 0.220 (a look)Test 24, day 10: p = 0.304 (a look)Test 24, day 11: p = 0.209 (a look)Test 24, day 12: p = 0.326 (a look)Test 24, day 13: p = 0.231 (a look)Test 24, day 14: p = 0.308 (a look)Test 24, day 15: p = 0.355 (a look)Test 24, day 16: p = 0.452 (a look)Test 24, day 17: p = 0.868 (a look)Test 24, day 18: p = 0.907 (a look)Test 24, day 19: p = 0.902 (a look)Test 24, day 20: p = 0.765 (a look)Test 24, day 21: p = 0.532 (a look)Test 24, day 22: p = 0.580 (a look)Test 24, day 23: p = 0.616 (a look)Test 24, day 24: p = 0.528 (a look)Test 24, day 25: p = 0.393 (a look)Test 24, day 26: p = 0.346 (a look)Test 24, day 27: p = 0.304 (a look)Test 24, day 28: p = 0.445 (a look)Test 25, day 1: p = 0.033 (a look: p < 0.05, the test stops with a false winner)Test 25, day 2: p = 0.155 (after the stop)Test 25, day 3: p = 0.759 (after the stop)Test 25, day 4: p = 0.864 (after the stop)Test 25, day 5: p = 0.851 (after the stop)Test 25, day 6: p = 0.401 (after the stop)Test 25, day 7: p = 0.413 (after the stop)Test 25, day 8: p = 0.667 (after the stop)Test 25, day 9: p = 0.761 (after the stop)Test 25, day 10: p = 0.782 (after the stop)Test 25, day 11: p = 0.706 (after the stop)Test 25, day 12: p = 0.709 (after the stop)Test 25, day 13: p = 0.751 (after the stop)Test 25, day 14: p = 0.691 (after the stop)Test 25, day 15: p = 0.760 (after the stop)Test 25, day 16: p = 0.682 (after the stop)Test 25, day 17: p = 0.985 (after the stop)Test 25, day 18: p = 0.740 (after the stop)Test 25, day 19: p = 0.681 (after the stop)Test 25, day 20: p = 0.939 (after the stop)Test 25, day 21: p = 0.967 (after the stop)Test 25, day 22: p = 0.720 (after the stop)Test 25, day 23: p = 0.842 (after the stop)Test 25, day 24: p = 0.657 (after the stop)Test 25, day 25: p = 0.957 (after the stop)Test 25, day 26: p = 0.893 (after the stop)Test 25, day 27: p = 0.912 (after the stop)Test 25, day 28: p = 0.966 (after the stop)Test 26, day 1: p = 0.593 (a look)Test 26, day 2: p = 0.404 (a look)Test 26, day 3: p = 0.056 (a look)Test 26, day 4: p = 0.022 (a look: p < 0.05, the test stops with a false winner)Test 26, day 5: p = 0.022 (after the stop)Test 26, day 6: p = 0.045 (after the stop)Test 26, day 7: p = 0.068 (after the stop)Test 26, day 8: p = 0.040 (after the stop)Test 26, day 9: p = 0.028 (after the stop)Test 26, day 10: p = 0.015 (after the stop)Test 26, day 11: p = 0.017 (after the stop)Test 26, day 12: p = 0.034 (after the stop)Test 26, day 13: p = 0.035 (after the stop)Test 26, day 14: p = 0.040 (after the stop)Test 26, day 15: p = 0.051 (after the stop)Test 26, day 16: p = 0.050 (after the stop)Test 26, day 17: p = 0.055 (after the stop)Test 26, day 18: p = 0.123 (after the stop)Test 26, day 19: p = 0.084 (after the stop)Test 26, day 20: p = 0.089 (after the stop)Test 26, day 21: p = 0.074 (after the stop)Test 26, day 22: p = 0.098 (after the stop)Test 26, day 23: p = 0.180 (after the stop)Test 26, day 24: p = 0.258 (after the stop)Test 26, day 25: p = 0.171 (after the stop)Test 26, day 26: p = 0.233 (after the stop)Test 26, day 27: p = 0.331 (after the stop)Test 26, day 28: p = 0.282 (after the stop)Test 27, day 1: p = 0.273 (a look)Test 27, day 2: p = 0.606 (a look)Test 27, day 3: p = 0.451 (a look)Test 27, day 4: p = 0.702 (a look)Test 27, day 5: p = 0.865 (a look)Test 27, day 6: p = 0.284 (a look)Test 27, day 7: p = 0.565 (a look)Test 27, day 8: p = 0.381 (a look)Test 27, day 9: p = 0.760 (a look)Test 27, day 10: p = 0.745 (a look)Test 27, day 11: p = 0.612 (a look)Test 27, day 12: p = 0.337 (a look)Test 27, day 13: p = 0.356 (a look)Test 27, day 14: p = 0.443 (a look)Test 27, day 15: p = 0.348 (a look)Test 27, day 16: p = 0.339 (a look)Test 27, day 17: p = 0.231 (a look)Test 27, day 18: p = 0.137 (a look)Test 27, day 19: p = 0.118 (a look)Test 27, day 20: p = 0.163 (a look)Test 27, day 21: p = 0.260 (a look)Test 27, day 22: p = 0.230 (a look)Test 27, day 23: p = 0.274 (a look)Test 27, day 24: p = 0.310 (a look)Test 27, day 25: p = 0.453 (a look)Test 27, day 26: p = 0.674 (a look)Test 27, day 27: p = 0.453 (a look)Test 27, day 28: p = 0.426 (a look)Test 28, day 1: p = 0.623 (a look)Test 28, day 2: p = 0.248 (a look)Test 28, day 3: p = 0.930 (a look)Test 28, day 4: p = 0.970 (a look)Test 28, day 5: p = 0.798 (a look)Test 28, day 6: p = 0.876 (a look)Test 28, day 7: p = 0.829 (a look)Test 28, day 8: p = 0.850 (a look)Test 28, day 9: p = 0.723 (a look)Test 28, day 10: p = 0.273 (a look)Test 28, day 11: p = 0.142 (a look)Test 28, day 12: p = 0.156 (a look)Test 28, day 13: p = 0.163 (a look)Test 28, day 14: p = 0.308 (a look)Test 28, day 15: p = 0.334 (a look)Test 28, day 16: p = 0.214 (a look)Test 28, day 17: p = 0.182 (a look)Test 28, day 18: p = 0.265 (a look)Test 28, day 19: p = 0.192 (a look)Test 28, day 20: p = 0.121 (a look)Test 28, day 21: p = 0.234 (a look)Test 28, day 22: p = 0.201 (a look)Test 28, day 23: p = 0.242 (a look)Test 28, day 24: p = 0.204 (a look)Test 28, day 25: p = 0.252 (a look)Test 28, day 26: p = 0.242 (a look)Test 28, day 27: p = 0.132 (a look)Test 28, day 28: p = 0.122 (a look)Test 29, day 1: p = 0.591 (a look)Test 29, day 2: p = 0.957 (a look)Test 29, day 3: p = 0.724 (a look)Test 29, day 4: p = 0.301 (a look)Test 29, day 5: p = 0.584 (a look)Test 29, day 6: p = 0.791 (a look)Test 29, day 7: p = 0.796 (a look)Test 29, day 8: p = 0.656 (a look)Test 29, day 9: p = 0.960 (a look)Test 29, day 10: p = 0.631 (a look)Test 29, day 11: p = 0.599 (a look)Test 29, day 12: p = 0.599 (a look)Test 29, day 13: p = 0.643 (a look)Test 29, day 14: p = 0.471 (a look)Test 29, day 15: p = 0.474 (a look)Test 29, day 16: p = 0.582 (a look)Test 29, day 17: p = 0.352 (a look)Test 29, day 18: p = 0.220 (a look)Test 29, day 19: p = 0.307 (a look)Test 29, day 20: p = 0.287 (a look)Test 29, day 21: p = 0.319 (a look)Test 29, day 22: p = 0.311 (a look)Test 29, day 23: p = 0.243 (a look)Test 29, day 24: p = 0.187 (a look)Test 29, day 25: p = 0.373 (a look)Test 29, day 26: p = 0.670 (a look)Test 29, day 27: p = 0.720 (a look)Test 29, day 28: p = 0.615 (a look)Test 30, day 1: p = 0.970 (a look)Test 30, day 2: p = 0.662 (a look)Test 30, day 3: p = 0.436 (a look)Test 30, day 4: p = 0.673 (a look)Test 30, day 5: p = 0.471 (a look)Test 30, day 6: p = 0.310 (a look)Test 30, day 7: p = 0.136 (a look)Test 30, day 8: p = 0.336 (a look)Test 30, day 9: p = 0.674 (a look)Test 30, day 10: p = 0.595 (a look)Test 30, day 11: p = 0.612 (a look)Test 30, day 12: p = 0.529 (a look)Test 30, day 13: p = 0.582 (a look)Test 30, day 14: p = 0.450 (a look)Test 30, day 15: p = 0.636 (a look)Test 30, day 16: p = 0.468 (a look)Test 30, day 17: p = 0.321 (a look)Test 30, day 18: p = 0.439 (a look)Test 30, day 19: p = 0.318 (a look)Test 30, day 20: p = 0.169 (a look)Test 30, day 21: p = 0.278 (a look)Test 30, day 22: p = 0.286 (a look)Test 30, day 23: p = 0.194 (a look)Test 30, day 24: p = 0.409 (a look)Test 30, day 25: p = 0.497 (a look)Test 30, day 26: p = 0.589 (a look)Test 30, day 27: p = 0.566 (a look)Test 30, day 28: p = 0.440 (a look)Test 31, day 1: p = 0.970 (a look)Test 31, day 2: p = 0.270 (a look)Test 31, day 3: p = 0.895 (a look)Test 31, day 4: p = 0.879 (a look)Test 31, day 5: p = 0.747 (a look)Test 31, day 6: p = 0.664 (a look)Test 31, day 7: p = 0.275 (a look)Test 31, day 8: p = 0.747 (a look)Test 31, day 9: p = 0.909 (a look)Test 31, day 10: p = 0.875 (a look)Test 31, day 11: p = 0.670 (a look)Test 31, day 12: p = 0.783 (a look)Test 31, day 13: p = 0.874 (a look)Test 31, day 14: p = 0.862 (a look)Test 31, day 15: p = 0.976 (a look)Test 31, day 16: p = 0.724 (a look)Test 31, day 17: p = 0.465 (a look)Test 31, day 18: p = 0.565 (a look)Test 31, day 19: p = 0.581 (a look)Test 31, day 20: p = 0.714 (a look)Test 31, day 21: p = 0.565 (a look)Test 31, day 22: p = 0.690 (a look)Test 31, day 23: p = 0.917 (a look)Test 31, day 24: p = 0.994 (a look)Test 31, day 25: p = 0.976 (a look)Test 31, day 26: p = 0.982 (a look)Test 31, day 27: p = 0.854 (a look)Test 31, day 28: p = 0.723 (a look)Test 32, day 1: p = 0.078 (a look)Test 32, day 2: p = 0.013 (a look: p < 0.05, the test stops with a false winner)Test 32, day 3: p = 0.019 (after the stop)Test 32, day 4: p = 0.006 (after the stop)Test 32, day 5: p = <0.001 (after the stop)Test 32, day 6: p = 0.002 (after the stop)Test 32, day 7: p = <0.001 (after the stop)Test 32, day 8: p = 0.004 (after the stop)Test 32, day 9: p = 0.009 (after the stop)Test 32, day 10: p = 0.012 (after the stop)Test 32, day 11: p = 0.021 (after the stop)Test 32, day 12: p = 0.065 (after the stop)Test 32, day 13: p = 0.195 (after the stop)Test 32, day 14: p = 0.263 (after the stop)Test 32, day 15: p = 0.504 (after the stop)Test 32, day 16: p = 0.620 (after the stop)Test 32, day 17: p = 0.618 (after the stop)Test 32, day 18: p = 0.634 (after the stop)Test 32, day 19: p = 0.707 (after the stop)Test 32, day 20: p = 0.766 (after the stop)Test 32, day 21: p = 0.709 (after the stop)Test 32, day 22: p = 0.620 (after the stop)Test 32, day 23: p = 0.793 (after the stop)Test 32, day 24: p = 0.669 (after the stop)Test 32, day 25: p = 0.698 (after the stop)Test 32, day 26: p = 0.952 (after the stop)Test 32, day 27: p = 0.971 (after the stop)Test 32, day 28: p = 0.943 (after the stop)Test 33, day 1: p = 0.339 (a look)Test 33, day 2: p = 0.914 (a look)Test 33, day 3: p = 0.947 (a look)Test 33, day 4: p = 0.954 (a look)Test 33, day 5: p = 0.473 (a look)Test 33, day 6: p = 0.684 (a look)Test 33, day 7: p = 0.707 (a look)Test 33, day 8: p = 0.626 (a look)Test 33, day 9: p = 0.838 (a look)Test 33, day 10: p = 0.846 (a look)Test 33, day 11: p = 0.853 (a look)Test 33, day 12: p = 0.938 (a look)Test 33, day 13: p = 0.734 (a look)Test 33, day 14: p = 0.659 (a look)Test 33, day 15: p = 0.766 (a look)Test 33, day 16: p = 0.992 (a look)Test 33, day 17: p = 0.724 (a look)Test 33, day 18: p = 0.821 (a look)Test 33, day 19: p = 0.888 (a look)Test 33, day 20: p = 0.764 (a look)Test 33, day 21: p = 0.713 (a look)Test 33, day 22: p = 0.719 (a look)Test 33, day 23: p = 0.892 (a look)Test 33, day 24: p = 0.633 (a look)Test 33, day 25: p = 0.908 (a look)Test 33, day 26: p = 0.816 (a look)Test 33, day 27: p = 0.819 (a look)Test 33, day 28: p = 0.578 (a look)Test 34, day 1: p = 0.757 (a look)Test 34, day 2: p = 0.724 (a look)Test 34, day 3: p = 0.519 (a look)Test 34, day 4: p = 0.408 (a look)Test 34, day 5: p = 0.295 (a look)Test 34, day 6: p = 0.552 (a look)Test 34, day 7: p = 0.562 (a look)Test 34, day 8: p = 0.552 (a look)Test 34, day 9: p = 0.750 (a look)Test 34, day 10: p = 0.673 (a look)Test 34, day 11: p = 0.721 (a look)Test 34, day 12: p = 0.895 (a look)Test 34, day 13: p = 0.849 (a look)Test 34, day 14: p = 0.831 (a look)Test 34, day 15: p = 0.760 (a look)Test 34, day 16: p = 0.683 (a look)Test 34, day 17: p = 0.904 (a look)Test 34, day 18: p = 0.823 (a look)Test 34, day 19: p = 0.787 (a look)Test 34, day 20: p = 0.792 (a look)Test 34, day 21: p = 0.561 (a look)Test 34, day 22: p = 0.335 (a look)Test 34, day 23: p = 0.634 (a look)Test 34, day 24: p = 0.681 (a look)Test 34, day 25: p = 0.533 (a look)Test 34, day 26: p = 0.314 (a look)Test 34, day 27: p = 0.341 (a look)Test 34, day 28: p = 0.317 (a look)Test 35, day 1: p = 0.259 (a look)Test 35, day 2: p = 0.011 (a look: p < 0.05, the test stops with a false winner)Test 35, day 3: p = 0.052 (after the stop)Test 35, day 4: p = 0.009 (after the stop)Test 35, day 5: p = 0.009 (after the stop)Test 35, day 6: p = 0.038 (after the stop)Test 35, day 7: p = 0.081 (after the stop)Test 35, day 8: p = 0.135 (after the stop)Test 35, day 9: p = 0.152 (after the stop)Test 35, day 10: p = 0.069 (after the stop)Test 35, day 11: p = 0.165 (after the stop)Test 35, day 12: p = 0.097 (after the stop)Test 35, day 13: p = 0.118 (after the stop)Test 35, day 14: p = 0.138 (after the stop)Test 35, day 15: p = 0.082 (after the stop)Test 35, day 16: p = 0.057 (after the stop)Test 35, day 17: p = 0.076 (after the stop)Test 35, day 18: p = 0.176 (after the stop)Test 35, day 19: p = 0.119 (after the stop)Test 35, day 20: p = 0.200 (after the stop)Test 35, day 21: p = 0.260 (after the stop)Test 35, day 22: p = 0.484 (after the stop)Test 35, day 23: p = 0.696 (after the stop)Test 35, day 24: p = 0.749 (after the stop)Test 35, day 25: p = 0.765 (after the stop)Test 35, day 26: p = 0.869 (after the stop)Test 35, day 27: p = 0.959 (after the stop)Test 35, day 28: p = 0.745 (after the stop)Test 36, day 1: p = 0.672 (a look)Test 36, day 2: p = 0.935 (a look)Test 36, day 3: p = 0.824 (a look)Test 36, day 4: p = 0.878 (a look)Test 36, day 5: p = 0.732 (a look)Test 36, day 6: p = 0.628 (a look)Test 36, day 7: p = 0.624 (a look)Test 36, day 8: p = 0.426 (a look)Test 36, day 9: p = 0.268 (a look)Test 36, day 10: p = 0.252 (a look)Test 36, day 11: p = 0.528 (a look)Test 36, day 12: p = 0.379 (a look)Test 36, day 13: p = 0.472 (a look)Test 36, day 14: p = 0.541 (a look)Test 36, day 15: p = 0.426 (a look)Test 36, day 16: p = 0.529 (a look)Test 36, day 17: p = 0.657 (a look)Test 36, day 18: p = 0.921 (a look)Test 36, day 19: p = 0.806 (a look)Test 36, day 20: p = 0.952 (a look)Test 36, day 21: p = 0.887 (a look)Test 36, day 22: p = 0.696 (a look)Test 36, day 23: p = 0.524 (a look)Test 36, day 24: p = 0.726 (a look)Test 36, day 25: p = 0.588 (a look)Test 36, day 26: p = 0.394 (a look)Test 36, day 27: p = 0.378 (a look)Test 36, day 28: p = 0.338 (a look)Test 37, day 1: p = 0.276 (a look)Test 37, day 2: p = 0.467 (a look)Test 37, day 3: p = 0.808 (a look)Test 37, day 4: p = 0.581 (a look)Test 37, day 5: p = 0.528 (a look)Test 37, day 6: p = 0.815 (a look)Test 37, day 7: p = 0.565 (a look)Test 37, day 8: p = 0.989 (a look)Test 37, day 9: p = 0.849 (a look)Test 37, day 10: p = 0.828 (a look)Test 37, day 11: p = 0.982 (a look)Test 37, day 12: p = 0.692 (a look)Test 37, day 13: p = 0.992 (a look)Test 37, day 14: p = 0.854 (a look)Test 37, day 15: p = 0.671 (a look)Test 37, day 16: p = 0.450 (a look)Test 37, day 17: p = 0.623 (a look)Test 37, day 18: p = 0.466 (a look)Test 37, day 19: p = 0.792 (a look)Test 37, day 20: p = 0.579 (a look)Test 37, day 21: p = 0.537 (a look)Test 37, day 22: p = 0.757 (a look)Test 37, day 23: p = 0.762 (a look)Test 37, day 24: p = 0.685 (a look)Test 37, day 25: p = 0.355 (a look)Test 37, day 26: p = 0.330 (a look)Test 37, day 27: p = 0.346 (a look)Test 37, day 28: p = 0.263 (a look)Test 38, day 1: p = 0.970 (a look)Test 38, day 2: p = 0.979 (a look)Test 38, day 3: p = 0.599 (a look)Test 38, day 4: p = 1.000 (a look)Test 38, day 5: p = 0.932 (a look)Test 38, day 6: p = 0.651 (a look)Test 38, day 7: p = 0.896 (a look)Test 38, day 8: p = 0.903 (a look)Test 38, day 9: p = 0.970 (a look)Test 38, day 10: p = 0.981 (a look)Test 38, day 11: p = 0.572 (a look)Test 38, day 12: p = 0.414 (a look)Test 38, day 13: p = 0.266 (a look)Test 38, day 14: p = 0.136 (a look)Test 38, day 15: p = 0.119 (a look)Test 38, day 16: p = 0.272 (a look)Test 38, day 17: p = 0.295 (a look)Test 38, day 18: p = 0.245 (a look)Test 38, day 19: p = 0.285 (a look)Test 38, day 20: p = 0.278 (a look)Test 38, day 21: p = 0.214 (a look)Test 38, day 22: p = 0.225 (a look)Test 38, day 23: p = 0.327 (a look)Test 38, day 24: p = 0.314 (a look)Test 38, day 25: p = 0.317 (a look)Test 38, day 26: p = 0.485 (a look)Test 38, day 27: p = 0.335 (a look)Test 38, day 28: p = 0.315 (a look)Test 39, day 1: p = 0.848 (a look)Test 39, day 2: p = 0.727 (a look)Test 39, day 3: p = 0.861 (a look)Test 39, day 4: p = 0.529 (a look)Test 39, day 5: p = 0.246 (a look)Test 39, day 6: p = 0.139 (a look)Test 39, day 7: p = 0.364 (a look)Test 39, day 8: p = 0.359 (a look)Test 39, day 9: p = 0.858 (a look)Test 39, day 10: p = 0.735 (a look)Test 39, day 11: p = 1.000 (a look)Test 39, day 12: p = 0.774 (a look)Test 39, day 13: p = 0.687 (a look)Test 39, day 14: p = 0.587 (a look)Test 39, day 15: p = 0.729 (a look)Test 39, day 16: p = 0.553 (a look)Test 39, day 17: p = 0.498 (a look)Test 39, day 18: p = 0.322 (a look)Test 39, day 19: p = 0.197 (a look)Test 39, day 20: p = 0.200 (a look)Test 39, day 21: p = 0.174 (a look)Test 39, day 22: p = 0.290 (a look)Test 39, day 23: p = 0.390 (a look)Test 39, day 24: p = 0.353 (a look)Test 39, day 25: p = 0.359 (a look)Test 39, day 26: p = 0.246 (a look)Test 39, day 27: p = 0.296 (a look)Test 39, day 28: p = 0.406 (a look)Test 40, day 1: p = 0.494 (a look)Test 40, day 2: p = 0.830 (a look)Test 40, day 3: p = 0.843 (a look)Test 40, day 4: p = 0.213 (a look)Test 40, day 5: p = 0.539 (a look)Test 40, day 6: p = 0.342 (a look)Test 40, day 7: p = 0.280 (a look)Test 40, day 8: p = 0.325 (a look)Test 40, day 9: p = 0.182 (a look)Test 40, day 10: p = 0.360 (a look)Test 40, day 11: p = 0.550 (a look)Test 40, day 12: p = 0.692 (a look)Test 40, day 13: p = 0.916 (a look)Test 40, day 14: p = 0.737 (a look)Test 40, day 15: p = 0.805 (a look)Test 40, day 16: p = 0.819 (a look)Test 40, day 17: p = 0.817 (a look)Test 40, day 18: p = 0.935 (a look)Test 40, day 19: p = 0.847 (a look)Test 40, day 20: p = 0.804 (a look)Test 40, day 21: p = 0.940 (a look)Test 40, day 22: p = 0.948 (a look)Test 40, day 23: p = 0.874 (a look)Test 40, day 24: p = 0.925 (a look)Test 40, day 25: p = 0.994 (a look)Test 40, day 26: p = 0.916 (a look)Test 40, day 27: p = 0.965 (a look)Test 40, day 28: p = 0.931 (a look)17142128Day of the test
  • Look every day
  • Sequential test
05102030Share of 10,000 A/A tests with a false winner17142128Day of the test5% (the promise)Look every daySequential testLook every day: 27.6%Sequential test: 1.8%Look every daySequential test27.6%
05102030False winners (%)17142128Day of the test5% (the promise)Look every daySequential testLook every day: 27.6%Sequential test: 1.8%27.6%
Forty simulated A/A tests, looked at day by day. With one look on day 28, 2 of 40 tests show a false winner. With a look every day, 10 of 40 stop on a false winner. The sequential test stops only 1.
Text description of the explorer

Controls: a choice of four rules (one look on day 28, once a week, every day, every day with the sequential test). A slider with − and + buttons sets the day of the test, from 1 to 28. The upper chart has 40 rows, one per A/A test, and 28 squares per row, one per day. A dark blue square is a look with p ≥ 0.05. An orange square is the first look with p < 0.05: the test stops there with a false winner, and the later squares turn pale orange. On day 28 with a look every day, 10 rows have an orange square: on day 1 (1 test), day 2 (3 tests), day 3, day 4, day 5, day 7, day 12 and day 18. Once a week, 5 tests stop; with one look, 2; with the sequential test, 1. The lower chart shows the share of 10,000 such tests with a false winner by each day. With a look every day, it rises from 5.1% on day 1 to 10.9% on day 3 and 27.6% on day 28. With the sequential test, it rises from 0.0% to 1.8%. A dashed line marks 5%.

Move the day slowly with a look every day. The false winners pile up: 1 by day 1, 5 by day 3, 10 by day 28. Then choose one look on day 28: only 2 tests cross. The data did not change. Only the rule for looking changed.

The course measured the same rules on 10,000 simulated A/A tests:

False winners in 10,000 simulated A/A tests of 28 days, by the rule for looking: once at day 28, weekly, every 2 days, daily, or daily with a sequential test. One look keeps the 5%. Daily peeking finds a false winner in more than a quarter of the tests. A sequential test, with a look every day, stays under 5%.
Text description of the chart

Five bars. One look at day 28: 5.2%. Once a week: 13.1%. Every 2 days: 22.3%. Every day: 27.6%. Sequential test, looking every day: 1.8%.

Daily peeking turns a 5% rule into a 27.6% rule. More than one test in four would “win” with a change that does nothing.

Sequential tests: look every day, keep the 5%

Section titled “Sequential tests: look every day, keep the 5%”

Teams want to look while a test runs. A broken change must stop quickly. A great change should not wait a month. So we need a test that allows a look every day.

Think of a doctor who checks a patient’s fever every hour. She does not call an alarm at the first reading of 37.6 °C, because readings jump. She uses a higher alarm level for frequent readings. Then she can check every hour without false alarms.

A sequential test does the same. At each look, it uses a wider interval than the usual one. It is wide by just enough that the chance of a false winner over all the looks together stays under 5%. This wider interval is called an always-valid interval, because it is valid at every look, not only at the end. Its p-value is an always-valid p-value.

The course project uses one sequential test: the mixture sequential probability ratio test (mSPRT), from Johari, Pekelis and Walsh (2015). Its code is 40 lines in larkfield_experiments/sequential.py. You do not need its formula. You need three facts:

  1. It has one setting, tau: a guess of the size of a real effect. The project uses 5% of the control group’s mean.
  2. Each day, it computes an always-valid interval, and it keeps the overlap of all the intervals so far. Once a value is out, it stays out.
  3. It decides “B is different” only when zero leaves that interval.

Look again at the chart of the checkout-button test. On day 3, the usual interval was +0.28 to +3.24 dollars: above zero, a “win”. The sequential interval was −0.63 to +3.72: it contains zero. The sequential test never decided, and that was right. For the derivation of the usual interval, see Compare alternatives in Essential Mathematics and Statistics for AI.

The project can simulate many A/A tests and apply each rule to them.

Terminal window
xp peek --aa-tests 1000
1,000 simulated A/A tests (no real effect), 28 days, 2,000 new customers per group a day:
fixed length, one look at day 28: 6.0% false winners
peeking once a week, stop at p < 0.05: 14.0% false winners
peeking every day, stop at p < 0.05: 28.6% false winners
sequential test, look every day: 2.0% false winners

With 1,000 tests, the rates are close to the 10,000-test rates: 6.0% against 5.2%, 28.6% against 27.6%. The sequential test stays under 5%.

The wider interval is not free. Suppose the change has a real effect. A sequential test usually needs more customers to find it than a fixed test with one look at its planned end. You pay with time for the right to look every day.

You want to Use Cost
Read the result once, at the planned end A fixed-length test, one look You must not stop early, even for a great result
Watch the test and stop early for a clear win or a clear harm A sequential test More customers to find the same effect
Look every day and stop at the first p < 0.05 Never About 28% false winners instead of 5%

You may always look at a running test to find bugs: a broken page, an SRM, an empty log. That is a check of the test, from Check the test before the result. The rule is about the decision. Do not decide “B wins” from a look that the plan did not allow.

Guided practice: simulate the false winners

Section titled “Guided practice: simulate the false winners”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The notebook makes its own data, so the data cell ends with Ready: 0 files in data/.
  3. In section 1, predict how many of 28 daily looks show p < 0.05 in one A/A test. Then run the cell. You see days with p < 0.05: 12: this one test is unlucky.
  4. In section 2, run 2,000 A/A tests. You see about 4.5% false winners with one look, and 27.2% with a look every day.
  5. In section 3, run the sequential test on the same tests. You see 1.5%.
  6. In section 4, compute weekly_rate: the share of tests with p < 0.05 on day 7, 14, 21 or 28.
  7. Run the check cell. It prints Check passed.
Show the answer to step 6
weekly_rate = round(float((p[:, [6, 13, 20, 27]] < 0.05).any(axis=1).mean()), 3)
print(weekly_rate)

It prints 0.12: 12.0% of the tests “win” at one of four weekly looks. Fewer looks give fewer false winners, but still more than 5%.

Tomás says: “Let’s run the test for 56 days instead of 28, so it has more customers. We still look every day.” Predict first: with twice as many days, will daily peeking find more false winners or fewer? Then run the simulation for 56 days.

Show the answer
Terminal window
xp peek --aa-tests 1000 --days 56
1,000 simulated A/A tests (no real effect), 56 days, 2,000 new customers per group a day:
fixed length, one look at day 56: 4.2% false winners
peeking once a week, stop at p < 0.05: 17.8% false winners
peeking every day, stop at p < 0.05: 33.7% false winners
sequential test, look every day: 1.9% false winners

More. With daily peeking, a longer test gives more chances to cross the line: 33.7% against 28.6% for 28 days. One look stays near 5%, and the sequential test stays under it. More customers do not fix peeking. Only the rule for looking does.

Common mistake

“It is significant on day 3, so it is real”

Symptom: Tomás sees p = 0.02 on day 3 and stops the test. The report says “+1.76 dollars per customer, significant”. The button ships. Over the next month, revenue does not move.

Cause: the p-value of 0.02 is valid for one look at the planned end. Tomás looked every day and stopped at the first crossing. With that rule, about 28% of tests with no effect “win”. The truth was zero, and the day-28 result says so: +0.85, p = 0.31.

Fix: write the length in the plan and look once at the end. If the team must watch the test, use a sequential test: xp peek shows its interval each day. Record any early stop in the test log, with the rule that allowed it.

Check your understanding

A team looks at every test once a day and stops at the first p < 0.05. Over a year, they run 40 tests of changes that, in truth, do nothing. About how many will they ship as winners? Use the course's simulation.

Show the answer

With daily looks over 28 days, about 27.6% of tests with no effect show a false winner. 27.6% of 40 is about 11 tests. With one look at the planned end, it would be about 5% of 40: 2 tests. So the daily rule ships about 9 extra changes that do nothing.

Check your understanding

Completion check. Explain to Omar, the product manager, in five sentences, why the checkout-button test cannot stop on day 3, where B was ahead with p = 0.02.

Show the answer
  1. The p-value of 0.02 is only valid for one look, at the end that the plan set: day 28.
  2. We looked every day, and each look is another chance for luck to cross the line.
  3. With daily looks, about 28% of tests with no real effect show a “win” on some day, not 5%.
  4. On day 28, the same test shows +0.85 dollars with p = 0.31, and the simulator says the true effect is zero.
  5. If we need to stop early, we use a sequential test, which allows a look every day; here it never found a winner.

Check your understanding

On day 10 of a test, the usual 95% interval is +0.40 to +2.10 dollars, and the sequential interval is −0.20 to +2.70 dollars. The team wants to stop and ship. What do you say?

Show the answer

Do not ship yet. The usual interval is only valid for one look at the planned end, so it cannot decide on day 10. The sequential interval is valid at every look, and it still contains zero: the data do not yet exclude “no effect”. Either wait for the planned end and read the usual interval once, or keep watching the sequential interval until zero leaves it.

Recap

  • Peeking means looking at a running test many times and stopping at the first p < 0.05. Each look is another chance for luck. In 10,000 simulated A/A tests of 28 days: one look 5.2% false winners, weekly 13.1%, daily 27.6%.
  • The checkout-button test (a practice problem we made on purpose) “won” on day 3 with p = 0.020. On day 28, it showed +0.85 dollars, with p = 0.31. The truth was zero.
  • A sequential test uses an always-valid interval: wider at each look, so the chance of a false winner over all looks stays under 5%. The course’s mSPRT gave 1.8% with a look every day.
  • It costs time: with a real effect, a sequential test usually needs more customers to decide.
  • You may look at a running test to find bugs. Decide “B wins” only at a look that the plan allows.

You can now decide when to look. Next, the same trap appears in another form: one look, but at 20 metrics.

Next: Many metrics and many tests