Compare prompts

What you will be able to do

Find out whether a new prompt is really better. Run both versions on the same fixed tickets and score them with the same code. Then read the tickets that changed, instead of trusting one impressive answer.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Prompt v1 and v2, from Specify the task, and the injection ticket T-80002, from Provide context safely.
• Why a test set must look like the real work, from Build a representative test set, and why the same scoring rules must apply to every answer, from Score outputs consistently. Both are in the language models course. This lesson uses those ideas; it does not repeat them.

What you need on your computer

• The project after the guided practice of Provide context safely: python -m pytest gives 20 passed, 1 deselected.
• If something is missing, copy the snapshot end-of-m01 with the steps of Reset the project, and do the guided practice of the two earlier lessons of this module again.
• No account and no key.

Two recipes, one tasting

A café has two recipes for its tomato soup. The cook makes recipe B once, and a friend says: “Much better!” Is B better?

Not yet. Maybe that bowl was lucky. Maybe the friend was hungry. A fair test looks different. Both soups are made on the same day, with the same tomatoes. The same tasters try both, without knowing which is which, and they use the same score sheet. They taste many bowls, not one. And nobody changes recipe B during the test: otherwise the scores belong to a recipe that no longer exists.

Prompts are recipes for answers. Comparing them needs the same care.

The problem: one impressive answer

After the first lesson of this module, TomásTermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. was convinced. “Look at T-65083: v1 raised a false alarm, v2 got it right. v2 is better.”

But a different ticket tells a different story. On T-64849, a long ticket about a wobbly sale ladder, prompt v1 gave a wrong answer and v2 gave no answer at all. The model spent its whole budget of 1,000 tokens on hidden reasoning, and the text was empty. And on T-80008, both versions were equally good. If you pick the example, you can prove anything. One answer shows what a prompt can do, not what it usually does.

The idea: a fair comparison

A fair comparison of two prompts keeps everything the same except the prompt:

• Versioned prompts. Each version is a file that never changes after you use it: v1.md, v2.md. A change makes a new version. The output names the version, so every result says which prompt made it.
• A fixed set of tickets. The same 69 tickets every time: 60 from the deep learning course’s validation file, and 9 written for this course to test the rules. The language models course explains how to choose such a set.
• The same measures, computed by code. The team is right or wrong (68 tickets have an expected team; the empty ticket T-80006 has none). needs_human agrees with the expected value or not (69 tickets). Of the 9 tickets that need a person, how many were caught? And how many tickets got no usable answer?
• The same model and settings. Compare the prompts on chat-small, then separately on chat-strong. Never compare v1 on one model with v2 on another.
• The per-ticket view. The two totals tell you how much changed. The tickets that changed tell you why.

One honest warning about the expected answers. GraceTermGraceThe lead of Larkfield’s help desk. She writes the support policy and decides what a good answer to a customer is.’s rules H1 to H5TermRules H1 to H5The rules in Grace’s support policy that say when a person must take over a ticket: H1 safety or an injury, H2 a legal threat, H3 a request outside the policy, H4 another customer’s order or instructions to the assistant, H5 no readable request.Learn more Specify the taskBuilding Reliable Applications with LLM APIs are written in words, and someone had to apply them to each ticket. For this course, one person read every ticket and set needs_human by hand, then looked again wherever a model disagreed. One label changed on that review. So “wrong” sometimes means “disagrees with one careful person’s reading of a written rule”.

A tiny worked example: five tickets

Here are five tickets, with the needs_human answers of chat-small:

Ticket	Expected	v1	v2
T-64960 pool split after months outside	false	true ✗	false ✓
T-65047 pan handle came off	false	true ✗	false ✓
T-65083 promo code	false	true ✗	false ✓
T-65177 “warranty replacement lamp”	false	false ✓	false ✓
T-80008 where is my order?	false	false ✓	false ✓
Right		2 of 5	5 of 5

On these five tickets, v2 wins by 5 to 2. But who chose them? These five include the three tickets where v2 fixed a false alarm. A different five could show a tie. The choice of tickets decides the result, so the set must be fixed before you look at the answers.

The table also shows the useful way to read a comparison. Only the tickets that changed carry information about the difference. T-65177 and T-80008 were right in both versions: they say nothing about which prompt is better.

Larkfield’s assistant: the eval command

AmiraTermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. adds an eval command. It sends all 69 tickets with one prompt version to one model, scores every answer, and lists every ticket that is not right. The scoring lives in its own module, assistant/evaluate.py, so that a test can check it:

assistant/evaluate.py

"""Compare prompt versions on the same fixed tickets with the same measures."""

from collections import Counter
from dataclasses import dataclass

from .data import Ticket


@dataclass
class Row:
    ticket: Ticket
    answer: dict | None   # the parsed answer, or None when there was no usable answer
    note: str = ""        # why there is no answer, or which rule rejected it


def score(rows: list[Row]) -> dict:
    """Team accuracy (tickets with an expected team), needs_human agreement, and the failures."""
    teams = [r for r in rows if r.ticket.team]
    answered = [r for r in rows if r.answer]
    team_ok = sum(1 for r in teams if r.answer and r.answer.get("team") == r.ticket.team)
    human_ok = sum(1 for r in rows if r.answer and r.answer.get("needs_human") == r.ticket.needs_human)
    human_true = [r for r in rows if r.ticket.needs_human]
    caught = sum(1 for r in human_true if r.answer and r.answer.get("needs_human") is True)
    return {
        "tickets": len(rows),
        "no_answer": len(rows) - len(answered),
        "team_correct": team_ok,
        "team_scored": len(teams),
        "needs_human_agree": human_ok,
        "needs_human_true_caught": f"{caught}/{len(human_true)}",
        "notes": dict(Counter(r.note for r in rows if r.note)),
    }


def misses(rows: list[Row]) -> list[str]:
    out = []
    for r in rows:
        a = r.answer or {}
        if not r.answer:
            out.append(f"{r.ticket.ticket_id}  no answer ({r.note})")
            continue
        if r.ticket.team and a.get("team") != r.ticket.team:
            out.append(f"{r.ticket.ticket_id}  team {a.get('team')} (expected {r.ticket.team}, {r.ticket.style})")
        if a.get("needs_human") != r.ticket.needs_human:
            out.append(f"{r.ticket.ticket_id}  needs_human {a.get('needs_human')} (expected {r.ticket.needs_human})")
    return out

A missing answer counts as wrong on every measure. That is on purpose: a prompt that makes the model fail to answer is not better for it.

Here are the results for chat-small (gpt-6-luna on Azure), first with v1, then with v2:

python -m assistant eval --prompt v1
python -m assistant eval --prompt v2

The output:

{
  "tickets": 69,
  "no_answer": 0,
  "team_correct": 67,
  "team_scored": 68,
  "needs_human_agree": 65,
  "needs_human_true_caught": "9/9",
  "notes": {}
}
T-64849  team warranty (expected returns, long)
T-64849  needs_human True (expected False)
T-64960  needs_human True (expected False)
T-65047  needs_human True (expected False)
T-65083  needs_human True (expected False)
{
  "tickets": 69,
  "no_answer": 1,
  "team_correct": 67,
  "team_scored": 68,
  "needs_human_agree": 68,
  "needs_human_true_caught": "9/9",
  "notes": {
    "not JSON": 1
  }
}
T-64849  no answer (not JSON)

The team result did not change: 67 of 68 with both prompts. The routing words in these tickets are clear, so both prompts route almost every ticket correctly; this set is easy to route. needs_human went from 65 to 68 of 69, and both versions caught all 9 tickets that need a person. T-64849 is wrong in both: a wrong answer with v1, no answer with v2.

Read the comparison ticket by ticket

Predict first, then explore. The explorer shows the same recorded answers, one pair of squares per ticket.

Explore

Prompt v1 against prompt v2, ticket by ticket

Prompt v1

65 of 69

Prompt v2

68 of 69

Tickets that moved

3 fixed, 0 lost

chat-small, needs_human (of 69 tickets): v1 is right on 65, v2 on 68. The whole difference comes from 3 tickets: v2 fixed 3 and lost 0. 65 tickets are right in both versions, and 1 is wrong or without an answer in both.

• Right
• Wrong
• No answer

Each pair is one ticket: the top square is prompt v1, the bottom square is prompt v2. A dark frame marks a ticket whose result changed.

Tickets that are not right in at least one version (4)

Ticket	Expected needs_human	v1 answer and reason	v2 answer and reason
T-64849Hi. Bought a step ladder in the sale last month. Used it twice. Second time it wobbled, nothing broke. Got it from the big spring discount. Opened th…	false	trueThe ladder wobbled after use, making this a product fault and a safety concern that requires a person to handle.	no answer (no text: the token limit was used up)
T-64960Thanks again for sorting out my account password last week, everything works now! The paddling pool has split along a seam after several months outsi…	false	trueThis is a warranty request for a pool that split after use, but the stated warranty coverage only applies to garden tools and furniture, so a person must review the replacement request.	falseA product that splits after use is a warranty issue; the policy does not specify coverage for paddling pools.
T-65047We have cooked on our cast iron frying pan almost every night since it arrived last winter. It is heavy and it has been wonderful for searing and for…	false	trueThis is a handle failure after regular use, so it belongs to warranty, and the handle coming off while lifting the pan raises a safety concern that requires a person to handle it.	falseThe handle failed after the pan had been used, so this is a warranty issue.
T-65083Honestly this is annoying. It's not a delivery problem, the cabinet arrived fine, and I don't want a refund either. The promo code I entered at check…	false	trueThis is a payment issue involving a promo code, and the requested post-checkout discount adjustment is outside the stated policy, so a person must review it.	falseThe main request concerns a discount code at checkout, which is a payment issue.

Every answer is a real recorded response: chat-small is gpt-6-luna and chat-strong is gpt-6.1-sol, both on Azure. Scored as the eval command scores them.

Text description of the explorer

The explorer has two controls: the deploymentTermDeploymentA model that an account sets up at a provider, with its own name and its own limits. Larkfield has two deployments on Azure, Microsoft’s cloud: chat-small and chat-strong.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs (chat-smallTermchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs or chat-strongTermchat-strongThe name that Larkfield’s code gives to a larger hosted model: gpt-6.1-sol on Azure. It is often more accurate than chat-small, and it costs about 14 to 18 times more per call.Learn more Request anatomyBuilding Reliable Applications with LLM APIs) and the measure (needs_human or team). Each of the 69 tickets is a pair of small squares: the top square is prompt v1, the bottom square is prompt v2. Blue is right, orange is wrong, grey is no answer, and an empty square is not scored. A dark frame marks a ticket whose result changed. A table lists every ticket that is not right in at least one version, with the expected value and both answers and reasons.

chat-small, needs_human: v1 65 of 69, v2 68 of 69. Four tickets changed. v2 fixed three false alarms: T-64960 (pool split after months outside), T-65047 (pan handle came off) and T-65083 (promo code). T-64849 (sale ladder) was a false alarm with v1 and got no answer with v2, because the token limit was used up. The other 65 tickets are the same in both.

chat-small, team: 67 of 68 with both. Only T-64849 is not right: warranty with v1, no answer with v2.

chat-strong, needs_human: 67 of 69 with both versions; no ticket moved. T-64849 and T-65177 are false alarms in both. With v2, the reasons name a rule: H1 for the wobbly ladder, and H3 for the lamp, “outside the policy’s stated warranty coverage”.

chat-strong, team: 68 of 68 with both versions.

The whole difference on chat-small comes from four tickets. The other 65 got the same verdict from both prompts. On chat-strong, nothing moved at all: the stronger model guessed Grace’s rule even with v1. So v2 helps the smaller model with needs_human, and changes nothing on this set for the stronger one.

Is a difference of three tickets real? Ask again

Each ticket was recorded once per prompt. But these deployments sample their answers: they reject temperature: 0, as Request anatomy showed. If you send the same request again, some answers change. So Amira sent all 69 tickets twice more, with both prompts, to chat-small (US$0.08 for 276 calls). The chart counts the needs_human answers that are wrong or missing in each run.

• Prompt v1
• Prompt v2

The same 69 requests sent three times to chat-small (gpt-6-luna on Azure). Run 1 is the course recording; runs 2 and 3 were sent again 30 minutes later. Lower is better.

Text description of the chart

Six bars. Prompt v1, runs 1, 2 and 3: 4, 4, 3 tickets with a wrong or missing needs_human answer. Prompt v2, runs 1, 2 and 3: 1, 3, 3. Between runs of the same prompt, almost always the same five tickets changed: T-64849, T-64960, T-65047, T-65083 and T-65177 (and T-80006’s team once in v2).

The course recording happened to be v2’s best run. In runs 2 and 3, v2 had three misses, as many as v1’s run 3. The same five borderline tickets flipped back and forth within each prompt: the ladder, the pool, the pan, the promo code and the lamp. T-64849 got the right answer only once in six runs.

So the honest summary is not “v2 is better on needs_human”. It is: “v2 may lower the false alarms a little; three runs cannot tell.” On 69 tickets, a difference of one or two tickets, and even three, can come from sampling alone. To put a range around such a result, see How sure can 24 tickets be?, in the language models course. To see why only the tickets that changed count, see Compare alternatives in the mathematics course.

To see a small effect, you need more tickets, or more runs of each ticket. And measure what the change was designed to do. Prompt v2 added a rule for the replies: “If needs_human is true, say that a member of the team will contact the customer.” That is a measure where a prompt can make a large, clear difference, and it is the module practice below.

When to compare, and what it costs

Run the fixed set before every prompt change goes live, and keep the results with the version. With the mock, a run is free and takes a second. With a live model, one run of 69 tickets cost about US$0.02 on chat-small and US$0.28 on chat-strong, at the prices when we recorded. Check the provider’s current price page.

Compare the cost of each version too. v2 sends about 609 more tokens per ticket. Per 1,000 tickets, chat-small went from about US$0.27 to US$0.32 and chat-strong from about US$3.67 to US$4.59. The medianTermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One time per ticket stayed between 2.5 and 3.1 seconds.

A fixed set has limits. It cannot show problems that it does not contain. And if you tune a prompt against the same set for weeks, the prompt fits these 69 tickets, not the next 69. Add new real tickets to the set over time, and keep some that you never look at while you write prompts.

Guided practice: build the comparison

1. Make the file assistant/evaluate.py with the code from the section above.

2. Make the file tests/test_evaluate.py with this test. It scores three made-up rows whose results you can count by hand:

   tests/test_evaluate.py

   from assistant.data import Ticket
   from assistant.evaluate import Row, misses, score
   
   
   def test_score_counts_teams_needs_human_and_missing_answers():
       rows = [
           Row(Ticket("T-1", "C-1", "a", team="delivery", needs_human=False), {"team": "delivery", "needs_human": False}),
           Row(Ticket("T-2", "C-2", "b", team="payment", needs_human=True), {"team": "returns", "needs_human": True}),
           Row(Ticket("T-3", "C-3", "", team="", needs_human=True), None, "malformed_json"),
       ]
       s = score(rows)
       assert (s["team_correct"], s["team_scored"], s["needs_human_agree"], s["no_answer"]) == (1, 2, 2, 1)
       assert s["needs_human_true_caught"] == "1/2"
       assert len(misses(rows)) == 2

3. Replace all of assistant/__main__.py with this version. It keeps analyse, adds eval, and gives both commands --prompt and --model. The new function parse_answer reads the model’s text as JSON, or says why it cannot:

   assistant/__main__.py

   """Command line.
   
       python -m assistant analyse T-80008 --prompt v2    analyse one ticket with prompt v1 or v2
       python -m assistant eval --prompt v1               all 69 tickets of the evaluation set, scored
   """
   
   import argparse
   import json
   import sys
   
   from .config import Settings, make_provider
   from .context import build_request
   from .data import load_ticket, load_tickets
   from .evaluate import Row, misses, score
   from .providers import Completion, ProviderError
   
   
   def parse_answer(completion: Completion) -> tuple[dict | None, str]:
       """The model was asked for JSON. Return (the answer, "") or (None, why there is no answer)."""
       try:
           answer = json.loads(completion.text or "")
       except json.JSONDecodeError:
           return None, "not JSON"
       return (answer, "") if isinstance(answer, dict) else (None, "not a JSON object")
   
   
   def main(argv: list[str] | None = None) -> int:
       parser = argparse.ArgumentParser(prog="python -m assistant")
       commands = parser.add_subparsers(dest="command", required=True)
       one = commands.add_parser("analyse", help="analyse one ticket")
       one.add_argument("ticket_id")
       many = commands.add_parser("eval", help="analyse every ticket of the evaluation set and score it")
       for command in (one, many):
           command.add_argument("--prompt", default="v2", choices=["v1", "v2"])
           command.add_argument("--model", help="overrides ASSISTANT_MODEL")
       args = parser.parse_args(argv)
       settings = Settings.from_env()
       model = args.model or settings.model
       provider = make_provider(settings)
   
       if args.command == "analyse":
           try:
               ticket = load_ticket(args.ticket_id)
           except KeyError as error:
               print(error.args[0], file=sys.stderr)
               return 2
           try:
               completion = provider.complete(build_request(ticket, model, args.prompt))
           except ProviderError as error:
               print(f"No answer: {error}", file=sys.stderr)
               return 1
           answer, problem = parse_answer(completion)
           print(json.dumps(answer, indent=2, ensure_ascii=False) if answer else f"{completion.text}\n({problem})")
           print(f"{completion.model} | prompt {args.prompt} | finish_reason: {completion.finish_reason} | "
                 f"{completion.input_tokens} tokens in, {completion.output_tokens} out | {completion.latency_s} s")
           return 0
   
       rows = []
       for ticket in load_tickets().values():
           try:
               completion = provider.complete(build_request(ticket, model, args.prompt))
           except ProviderError as error:
               rows.append(Row(ticket, None, type(error).__name__))
               continue
           answer, problem = parse_answer(completion)
           rows.append(Row(ticket, answer, problem))
       print(json.dumps(score(rows), indent=2))
       print("\n".join(misses(rows)))
       return 0
   
   
   if __name__ == "__main__":
       sys.exit(main())

4. Run the tests. You should see 21 passed, 1 deselected.

5. Run the demo ticket. The last line should be gpt-6-luna-2026-09-22 | prompt v2 | finish_reason: stop | 1641 tokens in, 266 out | 5.094 s:

   python -m assistant analyse T-80008

6. Run python -m assistant eval --prompt v1 and then python -m assistant eval --prompt v2. Check that your output is the output in the section above.

7. In the explorer, select chat-small and needs_human. Find the four framed tickets, and read their reasons in the table.

Your project is now the snapshot end-of-m02.

Your turn: the stronger model

Before you run anything, predict: on chat-strong, will v2 improve needs_human as much as on chat-small? Then run both versions:

python -m assistant eval --prompt v1 --model chat-strong
python -m assistant eval --prompt v2 --model chat-strong

Show the answer

Both versions give the same scores: team_correct 68 of 68, needs_human_agree 67 of 69, needs_human_true_caught 9/9, and no_answer 0. The same two tickets miss in both:

T-64849  needs_human True (expected False)
T-65177  needs_human True (expected False)

v2 changed nothing on these measures. The stronger model already guessed Grace’s rule with v1, so the clearer contract had nothing to fix. The reasons did change: with v2, the model names the rule that it applies, H1 for the wobbly ladder and H3 for the lamp. That makes its answers easier to check. Are these two really errors? Both are judgment calls about the written rules. Grace could decide that a lamp is outside the warranty list (H3), and then the expected answer should change. A comparison often finds unclear rules, not only weak prompts.

For chat-strong, v2 costs more (about US$4.59 against US$3.67 per 1,000 tickets) for no gain on these measures. Its gain is in the replies, which the module practice measures.

Common mistake

A version that changed after it was measured

Symptom: Tomás likes Grace’s idea for injuries, and adds one sentence to v2.md: “If the customer is hurt, advise them to seek medical care.” Then he runs eval --prompt v2, and every ticket fails:

{
  "tickets": 69,
  "no_answer": 69,
  "team_correct": 0,
  "team_scored": 68,
  "needs_human_agree": 0,
  "needs_human_true_caught": "0/9",
  "notes": {
    "RecordingNotFound": 69
  }
}

Cause: the mock answers only the exact requests that were recorded. A changed prompt is a changed request. With a live model, the run would work, but its numbers would belong to a new prompt that is still called v2. The report “v2: 68 of 69” would then describe a prompt that no longer exists.

Fix:

1. Undo the change in v2.md. Run eval --prompt v2 again, and check that you get needs_human_agree 68.
2. Write the new idea as a new file, v3.md. Compare it with v2 on the same tickets, with a live model (OllamaTermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One or your own key). Add v3 to the choices of --prompt.
3. Never edit a prompt version after you have reported its results.

Module practice

The outline of this module asks you to compare two prompts for classification and draft replies on the same sample set. The eval command measures the classification. Now measure the replies too.

Grace’s reply rules in v2 give three checks that code can make:

1. The reply has 2 to 5 sentences.
2. The reply does not say that an action was done (“has been approved”, “has been sent”, “I’ve processed”…).
3. If needs_human is true, the reply says that a person will contact the customer.

Write a script compare_replies.py in the project folder. It sends all 69 tickets with v1 and then v2 to one model, the mock by default. It applies the three checks to each reply and prints the problems. Run it for chat-small and for chat-strong. Then write a short note for Grace, about half a page. Say what improved from v1 to v2 and what still fails, with ticket IDs and quotes. Say which differences are too small to trust.

The script is your own work: it is not part of the snapshot.

Show an example answer

compare_replies.py

"""Module practice: check the reply drafts of two prompt versions on the same 69 tickets.

    python compare_replies.py                  # chat-small
    python compare_replies.py chat-strong
"""

import re
import sys

from assistant.__main__ import parse_answer
from assistant.config import Settings, make_provider
from assistant.context import build_request
from assistant.data import load_tickets
from assistant.providers import ProviderError

DONE = re.compile(r"(?i)\b(has|have) been (approved|issued|processed|sent|refunded)\b|\bI'?ve (processed|issued|refunded)\b")
PERSON = re.compile(r"(?i)\b(contact|reach out|get in touch)\b")


def reply_problems(answer: dict) -> list[str]:
    """What is wrong with one reply draft, by Grace's three reply rules."""
    reply = answer.get("reply") or ""
    problems = []
    sentences = len(re.findall(r"[.!?](?:\s|$)", reply))
    if not 2 <= sentences <= 5:
        problems.append(f"{sentences} sentences")
    if DONE.search(reply):
        problems.append("says that an action was done")
    if answer.get("needs_human") is True and not PERSON.search(reply):
        problems.append("does not say that a person will contact the customer")
    return problems


def main() -> None:
    model = sys.argv[1] if len(sys.argv) > 1 else "chat-small"
    provider = make_provider(Settings.from_env())
    for version in ("v1", "v2"):
        found = []
        for ticket in load_tickets().values():
            try:
                answer, _ = parse_answer(provider.complete(build_request(ticket, model, version)))
            except ProviderError:
                answer = None
            if answer:
                found += [f"  {ticket.ticket_id}  {p}" for p in reply_problems(answer)]
        print(f"{model} {version}: {len(found)} reply problem(s)")
        print("\n".join(found))


if __name__ == "__main__":
    main()

python compare_replies.py
python compare_replies.py chat-strong

The output, shortened:

chat-small v1: 12 reply problem(s)
  T-64704  does not say that a person will contact the customer
  T-64849  does not say that a person will contact the customer
  …
  T-80007  does not say that a person will contact the customer
chat-small v2: 0 reply problem(s)

chat-strong v1: 10 reply problem(s)
  T-64704  does not say that a person will contact the customer
  T-64826  says that an action was done
  …
  T-80006  does not say that a person will contact the customer
chat-strong v2: 0 reply problem(s)

Read every hit before you count it. T-64826 is a false alarm of the phrase check: the reply says “so the payment team can check whether the refund has been sent”. That is a question about a refund, not a claim. Phrase checks are a safety net, not a judge. Module 3 meets the same problem.

A note for Grace (example):

“We compared prompt v1 and v2 on the same 69 tickets, with the recorded answers of 2026-10-08.

What improved. The replies: with v1, 12 replies on chat-small and 9 on chat-strong set needs_human to true. But they did not tell the customer that a person would contact them (for example T-80004, the hedge-trimmer burn, and T-80006, the empty ticket). With v2: none. (chat-strong v1 had one more hit, T-64826, a false alarm of the phrase check.) This is the rule that v2 added, and the difference is far too large to be chance. The reasons also improved: with v2 they name the rule (H1 to H5), so they are faster to check.

What did not clearly change. Routing: 67 of 68 on chat-small and 68 of 68 on chat-strong with both prompts. needs_human on chat-small went from 65 to 68 of 69 in the course recording. But when we sent the same tickets twice more, v2 got 66 and 66, and v1 65 and 66. The same five tickets flip in both versions, so this difference is mostly noise.

What still fails. T-64849 (the sale ladder) is right in only one of six runs; with v2 it sometimes gets no answer at all, because the model uses its whole token budget on reasoning (Module 3 handles this). T-64849 and T-65177 (the lamp) are flagged for a person by chat-strong with both prompts: please decide whether a wobbly ladder is a safety concern (H1) and whether a lamp is outside the warranty (H3). Then we update the expected answers or the policy. Finally, v2’s injury replies copy Example 2 and no longer advise medical care (T-80004). A v3 could add that; we would test it on the same tickets.”

Module check

Reset first if you need to. If your project does not give 21 passed, 1 deselected, copy the snapshot end-of-m02 with the steps of Reset the project, and run python -m pytest and python -m assistant analyse T-80008.

See it end to endClassify banking requests with an LLMCase study

The banking case study compares two prompt versions on 1,000 fixed messages, with a paired difference and its intervalTermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI.

Check your understanding

Completion check: explain the improvements that you observed from prompt v1 to v2, and the failures that remain. Give a ticket as an example of each, and say which differences you trust.

Show the answer

Improvements: the replies keep Grace’s rule. With v1, 12 replies on chat-small set needs_human to true but did not say that a person would contact the customer (for example T-80006); with v2, none. The reasons name the rule that applies. False alarms on needs_human fell in the course recording (T-64960, T-65047 and T-65083 were fixed on chat-small). Remaining failures: T-64849 (no answer or a wrong answer in five of six runs), T-65177 and T-64849 flagged by chat-strong with both prompts, and the copied Example 2 that drops “seek medical care” (T-80004). Trust: the reply difference (12 against 0) is large and consistent. The needs_human difference (65 against 68) shrank to about one ticket in repeat runs. The same five tickets flip in both versions, so it is not proof. Team accuracy did not change.

Check your understanding

Predict: you write prompt v3 and run it once on the 69 tickets. It gets needs_human right on 69, against v2's 68. Should Larkfield switch to v3?

Show the answer

Not on that evidence. One ticket of difference is within the noise: v2 itself moved between 66 and 68 in three runs of the same requests. First, look at which ticket changed and why. Then run both versions several times, or on more tickets, and compare only the tickets that change. Also compare the cost per ticket and the replies. Switch if v3 is better on the measures that matter, or equal and cheaper or simpler.

Check your understanding

Why does a missing answer count as wrong in score(), instead of being left out?

Show the answer

A prompt that makes the model fail to answer is worse for that ticket, not neutral. If missing answers were left out, a version could look better by failing on its hardest tickets. With v2, T-64849 got no answer: leaving it out would hide that the ladder ticket is still not handled.

Recap

• Compare prompts like recipes. Use versioned files that never change and a fixed set of tickets, chosen before you look. Use the same measures, computed by code, the same model, and the per-ticket view.
• On this easy-to-route set, v1 and v2 route the same: 67 of 68 on chat-small, 68 of 68 on chat-strong. needs_human went from 65 to 68 on chat-small in one recording, but repeat runs show that most of that is noise. The clear gain is in the replies: 12 problems with v1, none with v2.
• Only the tickets that change carry information. On 69 tickets, a difference of one to three tickets needs more runs or more tickets before you trust it.
• One impressive answer proves nothing, and neither does one run.

In Module 3, the answer stops being free text that the code hopes to parse. You define a schema, ask for structured outputTermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs, and check every answer before you store it.

Next: Define a schema