Skip to content
Français

Test charters and sessions

  • Lesson 7
  • 35 min
  • Module 3: Explore and reproduce

Where we are

In Module 2, we planned by risk and wrote checks for the failures that we expected. But many failures are ones that nobody expected, and a list of planned tests does not find them. In this lesson, we explore the app freely, but with a structure: a charter, a time box, notes with evidence, and a debrief.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Plan an exploratory session with a test charter, keep notes that hold evidence, and end with a short debrief. Then explore the bug-hunt app for 30 minutes with a charter of your own.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your course folder ai-qa, with the app in ai-qa/practice-app and its virtual environment .venv, as set up in the bug-hunt app lesson.
  • A text editor for Markdown files.

Tested on macOS (Apple silicon) with Python 3.13 and 3.14. The Windows commands were not run for this lesson.

You have two free hours in a city that you do not know. You can wander. You will see many streets, but at the end you cannot say what you saw or what you missed.

Or you can give yourself a mission: “Walk the old town, with the paper map, to find a bakery that is open on Sunday. Be back at the hotel at six.” The mission does not tell you which streets to take. You still choose as you go, and you still notice a good market on the way. But the walk has a goal, a limit and an end. Afterwards, you can tell a friend where you went, what you found and which streets are still to see.

This is a constructed example. On her first afternoon with the bug-hunt app, Esme does what many testers do with a new feature. She opens the web app and tries things for three hours. Some replies look odd. At the end of the day, her notes say this:

refund amount looked wrong once??
return window: 30 days sometimes, 45 other times
French reply in English? check

The next morning, Tomás, the AI engineer, asks three questions. Which test customer? Which seed? Which trace? Esme does not know, so he cannot see the failures. Then Omar, the product manager, asks: “Did you test cancellations?” Esme is not sure. She remembers a few, but not with which customer.

The afternoon had three problems:

  • No evidence. The notes have no trace IDs and no seeds. Nobody can see the failures again.
  • No coverage. Esme cannot say what she tested and what she did not.
  • No end. Nothing told her when to stop, or when to stop chasing one odd reply.

The problem is not that she explored. Exploring found three real problems in one afternoon. The problem is that the exploring left nothing that others can use.

Exploratory testing means that you design and run tests at the same time. Each result helps you choose the next test. It suits AI features well, because you cannot list every input and every acceptable reply in advance. But exploratory testing is not random clicking. It has a structure, in three parts.

  1. A test charter is the mission of one session, written before you start. It says what to explore, with what, and what you want to learn. Like the mission of the city walk, it gives a goal but not the route.
  2. An exploratory session is a block of time for one charter, often 30 to 90 minutes, without interruptions. The time box gives the session an end.
  3. Session notes are what you write while you test: each input, what you saw, and its evidence. A short debrief at the end turns the notes into decisions.

This way of working is called session-based test management. Jonathan and James Bach described it for testing ordinary software. It works the same for AI features. Only the evidence is different: for an AI reply, you keep the seed, the trace ID and the versions.

A charter fits in one sentence with three parts. This form comes from Elisabeth Hendrickson’s book on exploratory testing, Explore It!:

Explore a target with resources to discover information.

  • The target is one area of the app: a feature, a risk or a part of the conversation. It must be small enough for one session.
  • The resources are what you use: the test customers you play, the data, the seeds, and tools such as the trace and the export.
  • The information is the question that the session answers. “Whether refund amounts are ever wrong” is a question. “Test refunds” is not.

Here is Esme’s charter for her first planned session. She chose the target from her risk table: refunds have high harm and are used often.

A charter card titled 'Test charter: refunds and returns'. Explore: returns and refund offers, for delivered orders. Note: the target, one area of the app, small enough for one session. With: test customers C-1001, C-1002 and C-1003; fixed seeds 1 to 5; the trace and the export. Note: the resources, who you play, which data and which tools. To discover: whether refund amounts, policy versions and confirmations are ever wrong. Note: the information, the question that the session answers. At the bottom, the frame: Tester Esme, time box 30 minutes, app 2.3.0, prompt support-v8, model sim-1 (simulated); risks: refunds and returns, cancel an order, French answers; out of scope: other customers' orders. Note: who, how long, which build, which risks, and what is not in this session.
Esme's charter for her first session. The sentence on top is the mission; the frame below it makes the session repeatable.
Text description of the diagram

The card has four parts. Explore: returns and refund offers, for delivered orders. This is the target. With: three test customers, C-1001, C-1002 and C-1003, fixed seeds 1 to 5, the trace and the export. These are the resources. To discover: whether refund amounts, policy versions and confirmations are ever wrong. This is the information that the session must find.

Below the sentence is the frame. Esme is the tester and the time box is 30 minutes. The build is app 2.3.0, prompt support-v8 and model sim-1, which is simulated. The risks come from her risk table: refunds and returns, cancel an order and French answers. Other customers’ orders are out of scope, because another charter covers them.

Each test customer has a reason. C-1001 has no Rewards membership, C-1002 has a BBQ over 30 kg, and C-1003 is a French-speaking Rewards Gold member. Each one meets a different refund rule. The fixed seeds make every run repeatable, so any failure can be shown to Tomás. “Out of scope” protects the time box: other customers’ orders are a real risk, but they get their own session.

The hardest part of a charter is its size. Compare three charters for the same area:

Charter Problem
Explore the assistant to find bugs Too wide. It has no target and no question, so the session has no focus and no end.
Explore the return question with seed 2 to discover whether it says 30 days Too narrow. This is one test case, not a session. You already know the answer.
Explore returns and refund offers with three test customers and seeds 1 to 5 to discover whether amounts, versions and confirmations are ever wrong About right. One area, a few resources, one question that 30 minutes can answer in part.

A good charter leaves room for surprises. Esme does not list the messages that she will send. She lists ideas to try under the charter, and she adds new ideas while she tests.

Esme resets the test data, starts a timer and works through her ideas. She writes one line of notes for every message. The picture shows her 30 minutes. Five of her lines are real findings of the app’s planted defects. This lesson shows one of them; the others are yours to find.

A timeline from 0 to 30 minutes. A short grey band, setup, covers the first 3 minutes; a blue band, testing, investigating and taking notes, covers the rest. Events above and below the line: minute 2, OK, refund offer 51.05 dollars, trace tr-0001-1. Minute 3, OK, 'yes' starts the refund, tr-0001-2. Minute 6, a finding, yours to find. Minute 9, OK in French, a question for Grace. Minute 12, BUG, 30 days, version 3, tr-0004-1, with a red border. Minute 13, seed 3: 45 days, so 'sometimes', tr-0005-1. Minute 15, a finding, yours to find. Minute 18, OK, BBQ refund 600.00 dollars, tr-0007-1. Minute 21, a finding, yours to find. Minute 25, a finding outside the charter, yours to find. Minute 30, the time box ends: write the debrief.
Esme's 30-minute session on the refunds charter. Simulated (practice app). Dashed boxes are findings that this lesson does not show.
Text description of the diagram

The session starts with 3 minutes of setup: reset and the timer. Then testing, investigating and notes fill the rest. At minute 2, the refund offer for the hose reel is right: 51.05 dollars, trace tr-0001-1. At minute 3, the customer’s “yes” starts the refund, trace tr-0001-2. At minute 6 there is a finding that this lesson does not show. At minute 9, a French refund offer is right, but Esme writes a question for Grace. At minute 12 comes the bug that this lesson shows: the return window is 30 days, from policy version 3, trace tr-0004-1. At minute 13, seed 3 gives 45 days, so the bug happens only sometimes. Minute 15 has another hidden finding. At minute 18, the BBQ refund of 600.00 dollars is right. Minutes 21 and 25 have hidden findings; the one at minute 25 is outside the charter. At minute 30 the time box ends and Esme writes the debrief.

Three habits make this session useful.

Every line has its evidence. The trace ID and the seed are in the notes the moment Esme sees the reply. She does not need to remember anything later.

A finding gets a quick second look, not a long hunt. At minute 12, the reply says 30 days. One minute later, Esme sends the same question with seed 3, and the reply says 45 days. Now she knows that the bug happens only sometimes. She writes “How often?” and goes on. She measures the rate after the session, as the last lesson of this module shows.

A finding outside the charter is noted, not chased. At minute 25, Esme finds something that does not belong to refunds. It is too important to forget, so she writes one line and marks it “not in my charter”. It becomes a new charter, not a 40-minute detour.

Esme uses the session notes template from the course folder (templates/session-notes.md). These are four of her lines; the other lines hold the hidden findings.

workspace/notes/session-01-refunds.md (part)
| Min | What I did (customer, seed) | What I saw | Trace | Bug? Question? Idea? |
|---|---|---|---|---|
| 2 | C-1001, seed 1: "I want to return the hose reel from order LK-200102" | Asks first. 51.05 dollars = 64.00 - 12.95. | tr-0001-1 | OK |
| 3 | same conversation: "yes" | "Done: the return is started (refund 51.05 dollars)". Trace: `start_refund` with `confirmed: true`, result ok. | tr-0001-2 | OK |
| 12 | C-1001, seed 2: "How many days do I have to return an unused item?" | **"30 days", cites version 3.** | tr-0004-1 | **Bug: outdated policy.** |
| 13 | C-1001, seed 3: same question | "45 days", version 4. | tr-0005-1 | So it is "sometimes". |

The notes say what was expected, not only what happened. “51.05 dollars = 64.00 - 12.95” shows that Esme checked the amount against the rule: the item price minus 12.95 dollars return shipping. “OK” without a reason would not show that she checked anything.

At the top of the file, Esme also records how she spent the 30 minutes: setup 10%, testing 60%, investigating and noting bugs 30%. This split is a signal for the team. If investigating takes most of a session, the area has many bugs, and the charter covered less than planned.

Worked example: run the first minutes yourself

Section titled “Worked example: run the first minutes yourself”

Now run four lines of Esme’s session on your computer, and write them in your own notes. The IDs differ from hers in one place, because you skip her other lines.

  1. Open a terminal, move to ai-qa/practice-app, and activate the virtual environment as in the bug-hunt app lesson.

  2. Reset the test data, so that your IDs start at 1:

    Terminal window
    python -m helpdesk reset

    The output:

    The store is reset: 4 test customers, 8 orders, 8 reviews, no conversations.
  3. Start a chat as Alex Martin (C-1001, the default) with seed 1:

    Terminal window
    python -m helpdesk chat --seed 1
  4. Type I want to return the hose reel from order LK-200102 and press Enter. Then type yes, and then /quit.

    The output, simulated (practice app):

    Larkfield help desk 2.3.0 (simulated model sim-1).
    Signed in as test customer C-1001. Conversation c-0001, seed 1.
    Type a message. Commands: /trace, /export, /new, /quit
    you> I want to return the hose reel from order LK-200102
    assistant> I can start the return of the AquaFlow HR-30 hose reel from order LK-200102. Your refund will be 51.05 dollars (the item price minus 12.95 dollars return shipping). Reply yes to confirm.
    [trace tr-0001-1]
    you> yes
    assistant> Done: the return is started (refund 51.05 dollars). We email the return label within 1 business day.
    [trace tr-0001-2]
    you> /quit
  5. Send the return question with seed 2, and then with seed 3:

    Terminal window
    python -m helpdesk ask "How many days do I have to return an unused item?" --seed 2
    python -m helpdesk ask "How many days do I have to return an unused item?" --seed 3

    The output, simulated (practice app):

    Thanks for asking. You can return an unused item within 30 days of delivery for a full refund of the item price. [1] I hope this helps.
    Sources:
    [1] Returns policy, version 3: Return window (/policies/returns-policy/v3#return-window)
    [conversation c-0002 | trace tr-0002-1 | seed 2]
    Here is what our policy says. You can return an unused item within 45 days of delivery for a full refund of the item price. [1]
    Sources:
    [1] Returns policy, version 4: Return window (/policies/returns-policy/v4#return-window)
    [conversation c-0003 | trace tr-0003-1 | seed 3]

The replies are the same as Esme’s, because the seeds and the messages are the same. Only the conversation and trace numbers differ: Esme got tr-0004-1 and tr-0005-1, because she had sent other messages before. Now write your notes.

  1. Copy the two templates into your notes folder. Run this in ai-qa/practice-app:

    Terminal window
    cp ../templates/test-charter.md ../workspace/notes/charter-01-refunds.md
    cp ../templates/session-notes.md ../workspace/notes/session-01-refunds.md
  2. Fill in the charter with Esme’s charter from the card above.

  3. In the session notes, write one line for each of your four messages. Use your trace IDs.

Your four lines should look like this:

workspace/notes/session-01-refunds.md (part)
| Min | What I did (customer, seed) | What I saw | Trace | Bug? Question? Idea? |
|---|---|---|---|---|
| 2 | C-1001, seed 1: "I want to return the hose reel from order LK-200102" | Asks first. 51.05 dollars = 64.00 - 12.95. | tr-0001-1 | OK |
| 3 | same conversation: "yes" | "Done: the return is started (refund 51.05 dollars)". | tr-0001-2 | OK |
| 4 | C-1001, seed 2: "How many days do I have to return an unused item?" | **"30 days", cites version 3.** | tr-0002-1 | **Bug: outdated policy.** How often? |
| 5 | C-1001, seed 3: same question | "45 days", version 4. | tr-0003-1 | So it is "sometimes". |

Did the “yes” really start the refund? The reply says so, but the trace is the evidence. Run python -m helpdesk trace tr-0001-2. Its tool_calls shows start_refund with "confirmed": true and the result "ok": true. Write what you checked, not what you assumed.

When the time box ends, Esme stops, even in the middle of an idea. She writes the debrief summary at the end of her notes and meets Tomás and Grace for ten minutes. The debrief answers three questions:

  1. What did I cover, what did I not cover, and why? Esme covered refund offers for three customers, the confirmation, and the return window in two languages. She did not cover partial returns of several items or refunds for damaged items.
  2. What is the most important finding? She names one, so that the team knows what to look at first.
  3. What next? New charters. One is for order status and other customers’ orders, which were out of scope. One is for her finding at minute 25. A later session covers the two refund cases that she did not cover.

She also asks her open questions. Grace answers the first one at once: “Yes, in a French reply the product name can stay in English. It is the product’s name.” That is not a bug, so Esme closes it. Without the debrief, it could have become a bug report that wastes Tomás’s time.

A flow from top to bottom. Risk table: where the harm is. Then charter: explore a target, with resources, to discover information. Then session: a time box of 30 to 90 minutes, notes with trace IDs. Then debrief: what you covered, what you found, what to test next. From the debrief, two outputs with green borders: bug reports, with a reproduction rate; and new charters for what you did not cover. A dashed arrow goes from the new charters back to the session.
The cycle of session-based exploring. Each debrief produces bug reports and the next charters.
Text description of the diagram

The cycle starts from the risk table, which shows where the harm is. A charter names a target, its resources and the information to find. The session is a time box of 30 to 90 minutes, with notes that hold trace IDs. The debrief asks what was covered, what was found and what to test next. It has two outputs: bug reports, each with a reproduction rate, and new charters for what was not covered. The new charters lead to new sessions.

Use a charter-based session when you do not yet know where a feature fails. Examples are a new feature, a new model or prompt, or an area that the risk table rates high. Exploring finds the failures that nobody thought to write a test case for. That matters more for AI features, because their failures are hard to predict.

Do not use exploring to check something that you already know. Suppose the rule is clear: “the refund is the item price minus 12.95 dollars”. Then write a test case and let the QA runner repeat it. Exploring finds the bug; a saved case keeps it from coming back. You build those cases in later modules.

A session costs about 30 to 90 minutes of a tester’s time, plus about 10 minutes of debrief for two or three people. Shorter sessions spend too much of their time on setup. Longer ones lose focus. A session also depends on the tester: two testers with the same charter find different things. That is a strength, if each session leaves notes that others can use.

Common mistake

Notes without evidence

Symptom: a line in your notes says “wrong refund amount”, and nobody, including you, can show the reply again.

Cause: you wrote the observation but not the seed, the test customer or the trace ID. Or you ran reset before you exported the conversation, which deletes all traces.

Diagnose: check the line. Can someone repeat it from the line alone? They need the message, the test customer, the seed and the trace ID.

Recover: if you know the seed and the customer, send the same message again: the same seed gives the same reply. If you do not know the seed, run the message with seeds 1 to 10 until the failure comes back, and note that seed. Next time, copy the last line of each reply into your notes at once.

Common mistake

The session drifts away from the charter

Symptom: after 30 minutes, most of your notes are about something else, and your charter’s question has no answer.

Cause: you found something interesting and followed it. One finding led to the next.

Recover: stop and write the new area as one line, “not in my charter”. Go back to your ideas list. At the debrief, propose a new charter for the new area. If the finding is serious, for example an action without consent, tell the team at once. But do not test it further in this session.

Now plan and run a session of your own. Choose a target that is not refunds: for example order status, French answers, cancelling an order, or the summary of product reviews. Use your risk table to choose the one with the highest risk that you have not explored.

  1. Copy the charter template to workspace/notes/charter-02-<target>.md, for example charter-02-order-status.md.
  2. Write the charter sentence: explore, with, to discover. Then the frame: time box 30 minutes, the versions, the risks and what is out of scope.
  3. Write at least four ideas to try. Think of different test customers, different seeds, and a “yes” after a question.
  4. Copy the session notes template to workspace/notes/session-02-<target>.md.
  5. Run python -m helpdesk reset, then start a timer for 30 minutes.
  6. Test. Write one line for each message, with the customer, the seed and the trace ID.
  7. When a reply looks wrong, try one or two other seeds, then go on.
  8. When the timer ends, stop. Write the time split and the debrief summary.

Use ask for single questions, chat for conversations with a “yes”, and the web app if you prefer a browser. Choose your seeds yourself, for example 1 to 5, so that every line can be repeated.

Check your charter and notes

Read your charter as Omar would. Could another tester run the same session from it? It needs a target small enough for 30 minutes, the test customers and seeds, and a question that the session answers.

Read your notes as Tomás would. Can he repeat each line? It needs the message, the customer, the seed and the trace ID. Does each “OK” say what you checked?

Each of the suggested areas has at least one planted defect. If you found none, open the trace of every reply that you marked OK, and check its sources, versions and tool calls. Then look at your seeds: a failure that comes 1 time in 10 can easily hide in 5 runs. Keep your best finding: in Repeat runs and reproduction rate, you measure how often it happens.

Check your understanding

Rewrite this charter so that it can guide a 30-minute session: 'Test the French assistant.'

Show the answer

It has no target small enough, no resources and no question. One good version: “Explore French policy answers with C-1003, seeds 1 to 5 and the trace, to discover whether they are in French with the same facts as in English.” The frame adds the time box (30 minutes) and the versions (app 2.3.0, prompt support-v8, model sim-1). It also says what is out of scope, for example French refund offers.

Check your understanding

At minute 12, Esme sees '30 days, version 3'. Why does she send the same question once more with another seed, and why does she then go on instead of running it 30 times?

Show the answer

One more seed tells her quickly whether the failure happens every time or only sometimes. With seed 3 the reply is right, so the bug is intermittent, and that changes how she reports it. She goes on because the session’s time box belongs to the charter. Measuring a rate takes many runs and is easier with the QA runner after the session. She writes “How often?” in her notes so that she does not forget.

Check your understanding

Which three items must a line of session notes have so that a developer can see the same reply? Why does an AI feature need more than an ordinary web shop here?

Show the answer

The exact message, the test customer and the seed, and best of all the trace ID as well. In an ordinary web shop, the same input usually gives the same page, so the input is enough. An AI feature makes random choices, so the same message can give another reply. The seed lets the practice app repeat the run, and the trace shows what happened inside. With a real model, you also keep the export, because the reply cannot be repeated exactly.

Recap

  • Exploratory testing designs and runs tests at the same time. A structure keeps it useful: a charter, a time box, notes and a debrief.
  • A test charter is one sentence: explore a target, with resources, to discover information. Add the frame: tester, time box, versions, risks and what is out of scope.
  • An exploratory session is a time box of 30 to 90 minutes on one charter. Stop when it ends.
  • Every line of notes has its evidence: message, test customer, seed and trace ID, and what you checked.
  • Give a finding one quick second look, then go on. Note findings outside the charter and turn them into new charters.
  • The debrief says what you covered and did not cover, the most important finding, and what to test next.

Your workspace now: in workspace/notes/, Esme’s charter charter-01-refunds.md with your notes of its first minutes in session-01-refunds.md, and your own charter-02-<target>.md and session-02-<target>.md. Next, you vary who is typing and what they type: personas, edge cases and hostile inputs.

Next: Personas, edge cases and hostile inputs