Skip to content
Français

Repeat runs and reproduction rate

  • Lesson 9
  • 40 min
  • Module 3: Explore and reproduce

Where we are

In the last two lessons, we found failures that appear in some runs but not in others. “It fails sometimes” is true, but nobody can plan a fix with it, and a developer who tries once may see it pass. In this lesson, we run the same input many times, count the failures, and report a number that others can trust.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn “it fails sometimes” into a number that others can trust. Run the same input many times, count the failures, and report the count with the number of runs.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa, with the app in ai-qa/practice-app, its virtual environment active, and PyYAML installed.

Tested on macOS (Apple silicon) with Python 3.13 and 3.14. The Windows commands are the same; they were not run for this lesson.

Your car makes a strange noise on some mornings. You take it to the garage. The mechanic drives it once around the block and hears nothing. He gives it back: “I cannot reproduce it.” You drive away, and the next morning you hear the noise again.

The mechanic did nothing wrong in that one drive. The problem is that one drive cannot show a noise that comes on some mornings only. So next time, you keep a record: “It happened on 3 of the last 10 mornings, always when the engine was cold.” Now the mechanic knows how often to expect it, and that one quiet drive proves nothing.

Esme reports the outdated return window from her session: “The assistant sometimes says 30 days instead of 45.” Tomás tries it at once:

Terminal window
python -m helpdesk ask "How many days do I have to return an unused item?" --seed 1

The reply, simulated (practice app): “You can return an unused item within 45 days of delivery for a full refund of the item price.” Tomás writes “Works on my machine” and closes the report.

With ordinary software, “works on my machine” is a useful clue. The same input and the same data give the same result, so a difference means that something else differs: a setting, the data, a version. With an AI feature, the same input, data and version can give another reply. Tomás’s run is not a clue. It is just one run, and most runs of this bug pass.

A real model makes it worse. The course recorded the real assistant: the same prompt, documents and tools as the practice app, on a hosted model. It answered 30 questions 10 times each. Only 3 of the 30 questions got the same text in all 10 runs. The median question got 8 different texts in 10 runs. Recorded (real model). And unlike the practice app, a real model has no seed that replays a reply exactly.

Here is one of those questions. A customer asks: “Do you deliver to Nunavut?” Larkfield delivers there, except large items. Two of the ten recorded replies:

Run 2: Yes. We deliver to Nunavut, except large items, which aren’t delivered to the territories. [1][3]
Run 4: No. Large items aren’t delivered to Nunavut or any other territory [1][3]. For other items, we deliver throughout Canada, including Nunavut [1].

Recorded (real model). Run 4 starts with “No”, and a customer who reads only the first word gets the wrong answer. This happened in 2 of 10 runs. A developer who runs the question once sees a good reply 80% of the time.

The idea: count failures, and say out of how many

Section titled “The idea: count failures, and say out of how many”

The answer is the same as for the car: run the input many times and count. The reproduction rate is the number of runs that failed, divided by the number of runs. Always report it as a count with its base: “5 of 30 runs”, not only “17%”.

The base matters because a share alone hides how much you know. “1 of 3” and “10 of 30” are both 33%. But one failure in three runs could easily be luck, while ten in thirty is a pattern. A tester who says “5 of 30 runs, seeds 1 to 30” gives three things: how often, how sure, and how to repeat it.

Worked example: the return window, measured

Section titled “Worked example: the return window, measured”

Esme writes the bug as a case with many runs. The case has the same checks as her property checks, and runs: 30.

  1. Create workspace/cases/repro-rate.yaml:

    workspace/cases/repro-rate.yaml
    # Measure a reproduction rate (Module 3): many runs, and a pass rule that shows the rate.
    id: repro-return-window-30-runs
    title: The return window is 45 days (30 runs)
    message: How many days do I have to return an unused item?
    checks:
    - contains: 45 days
    - cites: returns-policy@4
    runs: 30
  2. Run it:

    Terminal window
    python -m qa run ../workspace/cases/repro-rate.yaml

    The output, simulated (practice app):

    QA run: 1 case(s) from ../workspace/cases/repro-rate.yaml
    App: larkfield-helpdesk 2.3.0 | model sim-1 | prompt support-v8
    FAIL repro-return-window-30-runs 25/30 passed (needs 30; seeds 1-30)
    first failing run: seed 2, trace tr-0001-1
    - contains: missing ['45 days']
    - cites: expected returns-policy@4, cited ['returns-policy@3#return-window']
    1 case(s): 0 passed, 1 failed, 0 known bug(s) still failing, 0 known bug(s) now passing, 0 not run
    Report: reports/qa-YYYYMMDD-HHMMSS/report.md

25 of 30 runs passed, so 5 of 30 runs failed: 17%. The runner ran seeds 1 to 30, each in a new conversation on fresh test data. In the last line, YYYYMMDD-HHMMSS stands for the date and time of your run: that is the name of your report folder.

Open report.md in the report folder. It lists every failing run with its seed and its reply: seeds 2, 8, 11, 14 and 22. The folder conversations holds the export of each failing run, ready to attach to a bug report. Do all the failing replies fail in the same way? Here, all five say 30 days and cite version 3. That check matters: a rate counts one bug only if every failure is the same bug.

Now run the same case with fewer and with more runs. --runs-override makes --runs win over the runs: 30 in the file:

Terminal window
python -m qa run ../workspace/cases/repro-rate.yaml --runs 10 --runs-override
python -m qa run ../workspace/cases/repro-rate.yaml --runs 100 --runs-override

The result lines, simulated (practice app):

FAIL repro-return-window-30-runs 8/10 passed (needs 10; seeds 1-10)
FAIL repro-return-window-30-runs 70/100 passed (needs 100; seeds 1-100)
Runs Failed Rate A range of true rates that fit this count (95%)
10 2 20% 6% to 51%
30 5 17% 7% to 34%
100 30 30% 22% to 40%

The rate moves: 20%, then 17%, then 30%. None of them is wrong. Each is an estimate from a different number of runs. The last column says how far the estimate can be from the true rate. It is the range of true rates that could easily give the count you saw. Statisticians call it a 95% interval.

Read the first row. Two failures in 10 runs fit a true rate as low as 6% or as high as 51%. So after 10 runs, Esme can say “it fails often enough to matter”, but not “about 1 in 5”. After 100 runs, the range is 22% to 40%: about 1 run in 3.

  • measured rate
  • 95% range
The return question, seeds 1 to 200: the failure rate after each run, with the range of true rates that fit it. Simulated (practice app).
Text description of the chart

A line chart. The x axis is the number of runs, from 1 to 200; the y axis is the share of failing runs, from 0% to 100%. The measured rate jumps at first: after 2 runs it is 50%, because seed 2 failed. It settles slowly: 2 of 10, 5 of 30, 30 of 100 and 61 of 200, about 31%. Two dashed lines show the range of true rates that fit the count. After 10 runs the range is very wide, from 6% to 51%. It narrows as the runs grow, to about 24% to 37% after 200 runs.

The dashed lines come closer as the runs grow, but slowly. To make the range half as wide, you need about four times the runs. That is why “how many runs” is a decision, not a habit.

The seeds matter too. Here is the same case with 10 runs each, for two testers who start at different seeds. --seed-start sets the first seed:

Terminal window
python -m qa run ../workspace/cases/repro-rate.yaml --runs 10 --runs-override --seed-start 141
python -m qa run ../workspace/cases/repro-rate.yaml --runs 10 --runs-override --seed-start 61

The result lines, simulated (practice app):

PASS repro-return-window-30-runs 10/10 passed (needs 10; seeds 141-150)
FAIL repro-return-window-30-runs 4/10 passed (needs 10; seeds 61-70)

The first tester sees no failure in 10 runs and could write “cannot reproduce”. The second sees 6 failures in 10 and could write “fails more than half the time”. Same app, same version, same bug. Imagine 20 testers, each running 10 seeds in turn: seeds 1 to 10, 11 to 20, up to 191 to 200. The chart shows what they would see.

20 testers, 10 runs each (seeds 1 to 200 in blocks of 10): how many testers see 0, 1, 2 … failures. Simulated (practice app).
Text description of the chart

A column chart. Each column is a number of failures that one tester sees in 10 runs, from 0 to 6. Its height is how many of the 20 testers see it: 0 failures: 1; 1 failures: 1; 2 failures: 6; 3 failures: 4; 4 failures: 5; 5 failures: 2; 6 failures: 1. The column for 0 failures is orange: that tester would say “cannot reproduce”. Most testers see 2, 3 or 4 failures.

One tester in 20 saw nothing, and one saw 6 of 10. Neither is careless: this spread is what random choices do. So a single block of 10 runs can mislead in either direction. Report the seeds with the count, so that anyone can repeat your exact runs.

You now know that a short test can miss a bug that is really there. How many runs do you need before you are fairly sure to see it at least once? Predict first, then explore.

Explore

How often you see a bug that fails 1 run in 10

Chance to see no failure
34.9%
Chance to see it at least once
65.1%
Failures you can expect
1
Runs for a 90% chance to see it
22

A bug that fails 1 in 10, tested with 10 runs: you see no failure 34.9% of the time. About 3 of 10 testers would say "cannot reproduce". You see it at least once 65.1% of the time. To have a 90% chance to see it at least once, run it 22 times.

  • no failure seen
  • failure seen
0%10%20%30%40%012345Failures you see in 10 runsChance0 of 10 runs (no failure seen): 34.9%34.9%1 of 10 runs: 38.7%38.7%2 of 10 runs: 19.4%19.4%3 of 10 runs: 5.7%5.7%4 of 10 runs: 1.1%1.1%5 of 10 runs: 0.15%0.15%Other counts: below 0.05%
0%10%20%30%40%012345Failures you see in 10 runsChance0 of 10 runs (no failure seen): 34.9%34.9%1 of 10 runs: 38.7%38.7%2 of 10 runs: 19.4%19.4%3 of 10 runs: 5.7%5.7%4 of 10 runs: 1.1%1.1%5 of 10 runs: 0.15%0.15%Other counts: below 0.05%
The chance of each number of failures in N runs, for a bug that fails 1 run in 10. The orange bar is 'no failure seen'. The rate is an assumption you choose, not a measurement.
Text description of the explorer

A column chart with one slider: the number of runs, from 1 to 100, starting at 10. Each column is the chance to see 0, 1, 2 … failures in those runs, for a bug that fails 1 run in 10. The first column, orange, is the chance to see no failure at all. With 10 runs it is 34.9%: about 3 of 10 testers would say “cannot reproduce”. The chance to see the bug at least once is 65.1%. The readout also shows the failures you can expect (1) and the runs for a 90% chance to see the bug: 22. At 22 runs, the orange column falls to 9.8%.

Most people guess 10: “it fails 1 in 10, so 10 runs should show it.” But with 10 runs, the bug stays hidden 35% of the time. You need 22 runs for a 90% chance, and 29 for 95%. Rare bugs need many more runs than their “1 in” number suggests.

Now change the rate as well. Try a bug that fails 1 in 3, then 1 in 100.

Explore

Runs and failures for any rate

Chance to see no failure
1.7%
Chance to see it at least once
98.3%
Failures you can expect
3.3
Runs for a 90% chance to see it
6

A bug that fails 1 in 3, tested with 10 runs: you see no failure 1.7% of the time. Almost no tester would say "cannot reproduce". You see it at least once 98.3% of the time. That is already more than a 90% chance.

  • no failure seen
  • failure seen
0%10%20%30%012345678Failures you see in 10 runsChance0 of 10 runs (no failure seen): 1.7%1.7%1 of 10 runs: 8.7%8.7%2 of 10 runs: 19.5%19.5%3 of 10 runs: 26%26%4 of 10 runs: 22.8%22.8%5 of 10 runs: 13.7%13.7%6 of 10 runs: 5.7%5.7%7 of 10 runs: 1.6%1.6%8 of 10 runs: 0.3%0.3%Other counts: below 0.05%
0%10%20%30%02468Failures you see in 10 runsChance0 of 10 runs (no failure seen): 1.7%1 of 10 runs: 8.7%2 of 10 runs: 19.5%3 of 10 runs: 26%4 of 10 runs: 22.8%5 of 10 runs: 13.7%6 of 10 runs: 5.7%7 of 10 runs: 1.6%8 of 10 runs: 0.3%Other counts: below 0.05%
Two sliders: the number of runs and the bug's true rate. The rates are assumptions you choose.
Text description of the explorer

The same chart with a second slider: the bug’s true rate, from 1 in 100 to 1 in 2. It starts at 1 in 3 with 10 runs. Then you see no failure 1.7% of the time, and 6 runs give a 90% chance to see the bug. With 1 in 100 and 100 runs, you see no failure 36.6% of the time, and you need 230 runs for a 90% chance.

The bug fails Runs for a 90% chance to see it Runs for a 95% chance
1 in 2 4 5
1 in 3 6 8
1 in 10 22 29
1 in 20 45 59
1 in 100 230 299

The table also works the other way round. If you ran a case 30 times and saw no failure, the bug may still be there. Zero failures in 30 runs fits a true rate up to about 11%, so about 1 run in 9. Zero in 10 runs fits up to 28%. So “0 of 30 runs” is a good result, but it is not “never”.

The course’s recorded runs give real reproduction rates for the real assistant. Each question ran 10 times. Two questions failed in all 10 runs, and 22 never failed. The chart shows the questions that failed only sometimes, and the return question, with the range that 10 runs allow.

Recorded (real model): the assistant's prompt, documents and tools on chat-small, 10 runs per question, the course author's labels. Ranges: Wilson, 95%.
Text description of the chart

One row per question, with its failure rate in 10 recorded runs. The range shows the true rates that fit it: Order 8 days late (4 of 10): 40% (16.8% to 68.7%); Hose reel refund amount (3 of 10): 30% (10.8% to 60.3%); Delivery to Nunavut (2 of 10): 20% (5.7% to 51%); Hose reel warranty (1 of 10): 10% (1.8% to 40.4%); Refund time, in French (1 of 10): 10% (1.8% to 40.4%); Insists on a cancel (1 of 10): 10% (1.8% to 40.4%); Return window (0 of 10): 0% (0% to 27.8%). Every range is wide. A question with 1 failure in 10 fits a true rate from about 2% to 40%. The return question failed in none of its 10 runs, which still fits a true rate up to about 28%.

Look at how wide the ranges are. “Order 8 days late” failed 4 times in 10, but its true rate could be anywhere from about 17% to 69%. The three questions with 1 failure in 10 cannot be told apart, and each one could be rare or common. Ten runs were enough to show which questions fail. They are not enough to say how often, and not enough to rank them.

The return question shows the other side. The real model never said 30 days in 10 runs. But “0 of 10” fits a true rate up to 28%. The practice app fails this question in about 1 run in 3. So a real model can behave very differently from the simulation, and 10 runs cannot prove that a bug is absent.

These recordings cost little: 300 calls cost about US$0.04 on a small hosted model. Prices change, so check your provider’s current page. With a larger model, a longer prompt or a judge for each reply, runs cost more, and the number of runs becomes a budget decision.

A reproduction rate is evidence only with its context. Esme adds a section to her session notes after the session:

workspace/notes/session-01-refunds.md (part)
## Reproduction rates measured after the session (QA runner)
| Bug | Failing runs | Note |
|---|---|---|
| Outdated returns policy (30 days, version 3) | 5 of 30 (seeds 1-30: 2, 8, 11, 14, 22) | 30 of 100 with seeds 1-100. All failures cite version 3. |

A complete rate has five parts:

  1. The count and the base: 5 of 30 runs.
  2. How to repeat it: the seeds 1 to 30 and the failing seeds. For a real model, the settings and the date of the run in your run log.
  3. The build: app 2.3.0, model sim-1, prompt support-v8.
  4. What counts as a failure: the checks, here “says 45 days” and “cites version 4”.
  5. The evidence: the exports of the failing runs, from the report’s conversations folder.

For a real model, add that the runs cannot be replayed exactly, and attach every failing export. The bug report template in Module 4 has a field for each of these parts.

Measure a rate when a failure is intermittent, before you report it. The rate tells the team how serious the bug is and how hard it will be to see again. Measure again after a fix: Module 5 shows how many runs a fix needs before you accept it.

Choose the number of runs by what you need to show:

  • That the bug exists: run until the first failure, up to the number in the table above for the rate that you fear.
  • About how often it fails: 30 runs give a rough rate; 100 runs give a rate within about ±10 percentage points.
  • That a fix works: many runs with no failure, because “0 of N” only limits the rate to about 3 in N.

Runs of the practice app cost seconds. Runs of a real model cost money and time, and a large set of cases multiplies both. Spend the runs where the risk table says the harm is high.

Common mistake

Your count differs from the lesson's

Symptom: you run the case and get, for example, 7 of 30 failing runs, not 5 of 30.

Cause: you ran other seeds. Most often, --seed-start is set, or the case has another number of runs. Another cause is a different app: --app points to another release.

Diagnose: read the result line: it says which seeds ran, for example seeds 1-30. Read the App: line: it must say 2.3.0, sim-1 and support-v8.

Recover: run the case without --seed-start and without --app. With the same seeds and the same build, the practice app gives the same count on every computer. A real model does not: there, a different count is normal.

Common mistake

The rate mixes two bugs

Symptom: a case fails in 8 of 30 runs, but the failing replies are not alike.

Cause: the case has several checks, and different runs fail different checks. The rate then counts two failures together.

Diagnose: run the case with -v: it prints every failing run and the checks that failed. Or read report.md.

Recover: split the case into one case per failure, each with the checks for that failure only. Report one rate per bug.

Take the best finding of your own session from Test charters and sessions or Personas, edge cases and hostile inputs.

  1. Add a case for it to workspace/cases/repro-rate.yaml, after a line ---.
  2. Use the exact message and test customer from your notes. For a conversation, use steps: with lines - say: ....
  3. Write checks that fail when the bug happens, and only then.
  4. Set runs: 30.
  5. Run python -m qa check ../workspace/cases/repro-rate.yaml, then run the file.
  6. Read the failing replies in report.md. Check that they all show the same bug.
  7. Use the explorer: is 30 runs enough for the rate that you found?
  8. Write the rate in your session notes, with the five parts above.
Check your measurement

If your case fails in every run, check it once on a reply that you know is good: the check itself may be wrong. If it fails in no run, but you saw the bug in your session, compare the case with your notes. Check the message, the test customer, and a “yes” step if there was one. Then run the seed that failed in your session, with --seed-start and --runs 1 --runs-override. If that seed fails, the bug is rare: run 100 times.

A good result reads like this. “Fails in 5 of 30 runs (seeds 1 to 30; failing seeds 2, 8, 11, 14, 22). App 2.3.0, sim-1, support-v8. The range that fits this count is about 7% to 34%.” Then say what more runs would change. 100 runs would narrow the range to about ±10 points. They would not change the decision: Tomás must fix the bug.

Check your understanding

Predict: a bug fails 1 time in 10. You run the input 5 times. What is the chance that you see no failure? What do you write if you see none?

Show the answer

0.9 × 0.9 × 0.9 × 0.9 × 0.9 = 0.59, so 59%: more often than not, you see nothing. If you see no failure, write “0 of 5 runs (seeds 1 to 5)”, not “cannot reproduce”. Zero in 5 runs fits a true rate of up to about 43%, so it says little. Run more: 22 runs give a 90% chance to see a 1-in-10 bug.

Check your understanding

Tomás ran Esme's input once, it passed, and he wrote 'works on my machine'. What do you answer him, with numbers from this lesson?

Show the answer

One passing run is the expected result for this bug. It failed in 5 of 30 runs, so a single run passes about 5 times in 6. With an AI feature, the same input and version can give a different reply, so his run is not a clue that something differs. Ask him to run the case with the seeds that failed (2, 8, 11, 14, 22), or the whole case with 30 runs. With a real model, there is no seed: send him the exports of the failing runs and the count.

Check your understanding

Module check: report one finding with its reproduction rate and the number of runs, and say what more runs would change.

Show the answer

Example: “The assistant gives the outdated 30-day return window and cites returns policy version 3. It fails in 5 of 30 runs (seeds 1 to 30; failing seeds 2, 8, 11, 14, 22). App 2.3.0, model sim-1, prompt support-v8. Checks: the reply says 45 days and cites version 4. The exports of the failing runs are attached.” What more runs change: the estimate, not the finding. 5 of 30 fits a true rate of about 7% to 34%. With 100 runs, the count was 30 of 100, a range of about 22% to 40%. More runs would make the rate more precise. That helps to compare it before and after a fix. They would not make the bug less real.

Recap

  • The reproduction rate is failing runs divided by runs. Report it as a count with its base, “5 of 30 runs”, with the seeds and the build.
  • “It works on my machine” is weak for AI features: the same input and version can give a different reply. One passing run is often the expected result of an intermittent bug.
  • A rate from few runs is uncertain. 2 of 10 fits a true rate from about 6% to 51%. To halve the range, run about four times as many times.
  • To be 90% sure to see a bug that fails 1 time in 10, run it 22 times. “0 of 30” still fits a rate up to about 1 in 9.
  • The recorded real assistant failed some questions in 1 to 4 of 10 runs. Ten runs show which questions fail, but not how often.

Your workspace now: workspace/cases/repro-rate.yaml with Esme’s case and the case for your own best finding; in workspace/notes/, your session notes with a “reproduction rates” section. You have explored, varied and measured. The next module turns your findings into bug reports that people can act on.

Next: Capture the evidence