Experiment records

What you will be able to do

Write down each change that you test: what you changed, why, how you measured it, what you found and who decided. Then anyone can repeat the test, argue with it, or find out why a release was refused, months later.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Release manifests, ops trace and the evidence line, from Artifact lineage.
• Prompt fingerprints and ops release diff, from LLM-specific configuration.
• Controlled experiments, intervals and the release decision, from Controlled experiments and Trade-offs and regressions.

What you need on your computer

• Your project after the last lesson: python -m pytest prints 193 passed, and operations/releases/ has 1.5.0-99b6da and 1.5.0-054591.
• The saved files in data/telemetry/ and data/releases/.

Every step of this lesson reads files only: no Docker.

A notebook in the kitchen

Think of someone who bakes bread every weekend. One Saturday, she tries less salt, because a friend says it tastes better. The bread does not rise well. The next month, another friend gives the same advice. Did she try it already? What went wrong? She does not remember.

A careful baker keeps a notebook. Each attempt gets one page: what she changed, why, how she judged it, what happened, and what she decided. “Less salt, 15 g instead of 20. Friend’s idea. Same flour, same oven. Flat loaf. Back to 20 g. Maybe try again with a longer rise.”

The page does three jobs. She does not repeat a failed attempt by mistake. Someone else can try it exactly the same way. And when a friend says “less salt is better”, she can show what happened. The friend can then argue with the details, not with a memory.

The problem: the decision was in the wrong place

Larkfield had the evidence. In Evaluating and Testing AI Systems, Priya tested prompt v3 against prompt v2 on 271 tickets. The decision was DO NOT SHIP, with reasons. Then, last week, someone switched the live service to v3 with one setting.

How could that happen? The evaluation lived in the evaluation project: run files, a report and a decision printed in a terminal. The live service and the people who change its settings never saw it. Nothing in the operations folder said “v3 was tested and refused”. So, weeks later, the idea came back as new: “a shorter prompt, so it must be cheaper”.

There is a second problem, and it would remain even if nobody switched anything. Suppose that Omar asks, “Why exactly did we refuse v3? I think the test was unfair.” Without a record, the answer is somebody’s memory. Which tickets? Which runs? Which version of the set? How many repeats? Nobody can repeat the test, and nobody can challenge it in a useful way.

The idea: one record per change, next to the releases

An experiment here is one planned change, tested against what runs now, to answer one question. An experiment record is the page in the baker’s notebook: one file, kept with the service’s operations files, that holds:

• the question and the hypothesis: who asked, and what they expect to happen, in words that a test can prove wrong;
• the change: exactly one change, named by its versions;
• the evaluation: the set version, the runs, the repeats and the method, so someone can repeat it;
• the results: the numbers for both sides;
• the decision: the result, the reason, who decided and who was told;
• the next step.

Why “hypothesis” and not “goal”? A goal says what you want. A hypothesis says what you expect to measure: “fewer tokens in, so a lower cost”. The results then say whether it was true. If it was not, the record says so, and that is still a useful result.

The record links to the rest of the module through its run names. The candidate run in the record is the evidence line in a release manifest. So the path from a customer’s answer continues: answer → release ID → manifest → evidence → experiment record → decision.

The life of one experiment record, with the values of EXP-001. A new idea starts a new record; it never edits the old one.

Text description of the diagram

The record follows the change from left to right. Omar’s question becomes a hypothesis: warmer replies, fewer tokens in, lower cost. The change is one prompt, v2 to v3. The evaluation uses the set d4c96eaa10c0, two saved runs and five repeats on the development tickets. The results: the cost rose by 39%, the answers took about twice as long, and 25 tickets got no text. The decision, by Priya and Grace, was do not ship. The next step keeps v2 and asks that any new candidate pass the gate. A dotted arrow returns to the start: a new idea gets a new record, EXP-002, and EXP-001 stays as it is.

A tiny example: claim against measurement

Before you read the whole record, put the hypothesis next to the results. EXP-001 made three claims. The results come from the evaluation course’s real runs on its 271 tickets (run 1), and the decision numbers from its 98 holdout tickets:

Claim in the hypothesis	Prompt v2	Prompt v3	True?
v3 sends fewer tokens in (median per call)	1,902	1,477	yes
so it costs less (per 1,000 tickets)	US$0.343	US$0.477	no
and its decisions stay as good (holdout, team and needs a person)	95.9%	84.7%	no

The first claim was true, and the conclusion drawn from it was false. Fewer tokens went in, but the model wrote 262 tokens out with v2 and 627 with v3, at the median. Output tokens cost more than input tokens, so the total rose. A record shows that chain; a memory keeps only “it was cheaper, wasn’t it?”.

The evaluation course's real runs, as EXP-001 records them. The hypothesis expected a lower cost and the same quality; both went the other way.

Text description of the chart

The cost per 1,000 tickets was US$0.343 with prompt v2 and US$0.477 with prompt v3, about 39% more.

Correct decisions (team and needs a person) on the 98 holdout tickets. The difference is −0.112, with a 95% interval of −0.184 to −0.041: it does not include zero.

Text description of the chart

Prompt v2 made the correct decision on 95.9% of the holdout tickets, and prompt v3 on 84.7%. The difference is -0.112 [-0.184, -0.041].

What this tells us. v3 was not a cheaper prompt with a small quality risk. It was dearer and worse, and the interval shows that the drop in quality is not chance. Together with the release blockers, two safety tickets with no answer, the decision was easy. The point of the record is that the next person does not have to rebuild this argument from memory.

Read the project’s record

The record of the v3 test is in your project, as operations/experiments/EXP-001-prompt-v3.toml. List the records:

python -m ops experiments

id       title                                      owner  status    decision
-------  -----------------------------------------  -----  --------  -----------
EXP-001  Prompt v3: warmer replies, shorter prompt  Priya  rejected  do not ship

Then read it:

python -m ops experiments EXP-001

EXP-001: Prompt v3: warmer replies, shorter prompt (asked by Omar, owner Priya, rejected)
Hypothesis: A warmer prompt without examples makes replies that customers like more, and it costs less, because it sends fewer input tokens.
Change:     assistant-v2 -> assistant-v3. Nothing else: same model, output schema and token limit (1,000).

[evaluation]
  set                          d4c96eaa10c0
  baseline_run                 baseline-v2-chat-small-r1-b6a2b9
  candidate_run                candidate-v3-chat-small-r1-31b45f
  repeats                      5 runs of each prompt on the 102 development tickets
  how                          the evaluation course's harness: decisions (team and needs a person), reply rules, release blockers, 95% bootstrap intervals

[results]
  input_tokens_median          [1902, 1477]
  output_tokens_median         [262, 627]
  cost_per_1000_tickets_usd    [0.343, 0.477]
  latency_median_seconds       [2.92, 5.63]
  answers_without_text         [1, 25]
  decisions_correct            [0.959, 0.847]
  decisions_difference         -0.112 [-0.184, -0.041]
  blockers                     T-65663 (holdout): a hot, cracked pan; no answer, so no person was alerted. T-64740 (development): a parasol fell toward a child; no answer.

[decision]
  result                       do not ship
  reason                       A release blocker: safety tickets got no answer, so nobody was told to call the customer. It is also dearer (+39% per ticket) and slower (about 2 times), the opposite of the hypothesis.
  decided_by                   Priya and Grace
  informed                     Omar, Amira, Kwame
  next                         Keep v2. A new candidate must keep v2's rule 'if needs_human is true, say a member of the team will contact the customer', and must pass the release gate with no answer missing on any critical ticket.

Each part does one job:

• The change says “nothing else”. That is the controlled experiment of the evaluation course: one change, so any difference has one cause. The manifest diff of the last lesson proves it for the releases: 3 fields, all about the prompt.
• The evaluation names the set version and both run names. Anyone with the evaluation project can run the same harness on the same set and compare.
• The results keep both sides, [v2, v3]. A number without its comparison means little.
• The decision names people. decided_by is who can change it; informed is who must know. Omar asked for the change, and he is told why it was refused.
• The next step turns the refusal into a condition that a new candidate must meet.

Now connect it to last week’s answer. The v3 release’s evidence line was candidate-v3-chat-small-r1-31b45f. Find the record that tested it:

macOS / Linux

grep -l "candidate-v3-chat-small-r1-31b45f" operations/experiments/*.toml

Windows (PowerShell)

Select-String -List -Path operations\experiments\*.toml -Pattern "candidate-v3-chat-small-r1-31b45f" | ForEach-Object Path

operations/experiments/EXP-001-prompt-v3.toml

That closes the chain of Artifact lineage. From T-503354, the safety ticket with no answer, to its release, to its evidence, to this record, which said do not ship. Last week, the record and the release were in different places. Now both live in operations/, next to the service.

Records can be challenged: your turn

Tomás reads EXP-001 and objects: “v3 failed because of the token limit. Its answers without text used all 1,000 tokens. Give it 1,200 and it is fine.”

That might be true. The record makes the argument concrete. EXP-001’s results say what happened. Tomás proposes why, and a why can be tested. Priya’s answer is not “no”. It is “write it as an experiment”.

Write the record that Tomás’s idea needs.

1. Create the file operations/experiments/EXP-002-token-limit.toml. Use EXP-001 as the model: the same fields and sections.
2. Fill in the top: an ID, a title, who asked, the owner and status = "proposed".
3. Write the hypothesis so that a result can prove it wrong. Name what you expect to measure.
4. Write the change. Name exactly one change, and say what stays the same.
5. Fill in [evaluation]: the same set and baseline as EXP-001, and the candidate run as “not run yet”.
6. Leave [results] empty. Set [decision] to “not decided”, with who decides and who is told.
7. Run python -m ops experiments. Your record must be listed with proposed and not decided.

Worked solution

# An experiment record: what was changed, why, the evidence, and the decision.
id = "EXP-002"
title = "Prompt v3 with a token limit of 1,200"
asked_by = "Tomás"
owner = "Priya"
status = "proposed"
hypothesis = "Prompt v3 answers without text because they reach the limit of 1,000 tokens. With 1,200 tokens, fewer v3 answers are cut off, and v3 passes the release blockers."
change = "assistant-v3 with MAX_TOKENS 1200 instead of 1000. Nothing else: same model, schema and policy."

[evaluation]
set = "d4c96eaa10c0"
baseline_run = "baseline-v2-chat-small-r1-b6a2b9"
candidate_run = "not run yet"
repeats = "5 runs on the 102 development tickets, then 1 run on the holdout"
how = "the evaluation course's harness, the same as EXP-001"

[results]
# Empty until the runs exist.

[decision]
result = "not decided"
reason = "No evidence yet."
decided_by = "Priya and Grace"
informed = "Omar, Amira, Kwame, Tomás"
next = "Run the candidate. Ship only through the release gate; v3's other blockers (cost, latency) still apply."

python -m ops experiments then prints:

id       title                                      owner  status    decision
-------  -----------------------------------------  -----  --------  -----------
EXP-001  Prompt v3: warmer replies, shorter prompt  Priya  rejected  do not ship
EXP-002  Prompt v3 with a token limit of 1,200      Priya  proposed  not decided

Three things make it a good record. The hypothesis names a measurement: fewer answers cut off, and the blockers pass. The change is one value. The token limit is in the prompt fingerprint, so the candidate release gets a new ID, as in the last lesson’s “Your turn”. And the next step says that a cheaper or faster v3 does not follow. More tokens out cost more, so the cost blocker may still fail.

Delete the file when you have checked it, or keep it in your own project. The snapshot end-of-m02 has only EXP-001.

When to keep a record, and what it costs

Sam asks, “Do we need a record for every small change?”

Amira answers: “For every change that could change what customers get, and that someone tested to decide. A typo fix in a log message, no. A new prompt, a new model version, a new token limit, yes.”

Keep a record when a change has a question behind it, when people disagree, or when the result is a refusal. Refusals are the easiest to forget, and they are the most likely to come back as “a new idea”.

Do not edit a record after its decision to make it look better. If new evidence arrives, add a new record that names the old one. The old record shows what the team knew at the time.

What it costs: ten or twenty minutes of writing per experiment, and a place everyone reads. The real cost is honesty: a record of a failed idea can feel like a record of a mistake. It is not. EXP-001 is the most useful file in the folder. Last week it stopped nothing, but only because nobody read it.

Common mistakes

Common mistake

A record with two changes

Symptom: a record says “prompt v4 and model version 2026-11-04”, and the results show a drop. The team argues about which change caused it, and nobody can say.

Cause: two changes in one experiment. The results cannot separate them.

Fix: split it into two records, each with one change and the same baseline. Check with ops release diff that the candidate release differs from the baseline only in the lines of that one change.

Common mistake

ops experiments stops with a KeyError

Symptom: python -m ops experiments prints a Python traceback instead of the table. Its last line is:

KeyError: 'decision'

Cause: a record file in operations/experiments/ has no [decision] section, or no result in it. The list needs both. Another name in the error, such as 'owner', means that this field is missing.

Fix: add the section, with result = "not decided" until there is a decision. If the file is not valid TOML, the tool says where instead, for example ops experiments: Illegal character '\n' (at line 1, column 14): a quote is missing on that line.

Module practice: construct a release manifest

You built a manifest with a tool in this module. Now construct one by hand for a different, mock deployment, so that you know what each line is for. Everything in this task is made up for practice: the service, its versions and its numbers.

Larkfield’s warehouse team runs a small AI service, returns-helper. It reads a return request and fills in a return label. Here is what its team knows about the version that runs today:

What	Value
Code	version 0.3.0, Git commit 4e1c9a07d2b5, code fingerprint b71f0c22e9a4
Image	registry.example/returns-helper@sha256:9d3f5a
Database	3 migrations, the last is 003_return_labels.sql
Prompt	file returns-v1, prompt fingerprint 5c0d8e14aa71, token limit 400
Output schema	return_label, fingerprint e2a90b3c7d15
Model	deployment chat-small, the team tested it with gpt-6-luna-2026-09-22
Retrieval	none: it searches no documents
Evaluation	set version 7b2e04c1f9d8, the run that approved it: returns-v1-chat-small-r1-0a4c2e
Settings that change behaviour	PROVIDER_TIMEOUT = 10
Also on the server	the database password, the provider key, the server’s address

Your task:

1. Put each value into the right section of the project’s seven sections. Leave out what does not belong in a manifest.
2. Compute the release ID with the project’s method. It is the version, a dash and 6 characters of the SHA-256 of the sorted JSON.
3. Predict what happens to the ID if the provider moves chat-small to gpt-6-luna-2026-11-04 and the team writes that in the manifest.

Run Python in your browser

Construct the manifest of returns-helper

Fill in each section as a dictionary of "name": value pairs, with the field names of the course’s manifests where they exist. Run the box: it prints the release ID, and the check tells you if a section is missing.

Python code (you can change it)import hashlib import json # The mock deployment: fill in each section from the table above. manifest = { "code": {}, "database": {}, "assistant": {}, "model": {}, "retrieval": {}, "data": {}, "settings": {}, } text = json.dumps(manifest, sort_keys=True) release_id = "0.3.0-" + hashlib.sha256(text.encode()).hexdigest()[:6] print(release_id)

Worked solution

manifest = {
    "code": {"version": "0.3.0", "commit": "4e1c9a07d2b5", "fingerprint": "b71f0c22e9a4",
             "image": "registry.example/returns-helper@sha256:9d3f5a"},
    "database": {"migrations": 3, "last_migration": "003_return_labels.sql"},
    "assistant": {"prompt": "returns-v1", "prompt_fingerprint": "5c0d8e14aa71",
                  "schema": "return_label", "schema_fingerprint": "e2a90b3c7d15",
                  "max_completion_tokens": 400},
    "model": {"deployment": "chat-small", "expected_version": "gpt-6-luna-2026-09-22"},
    "retrieval": {"index_version": "none"},
    "data": {"evaluation_set": "7b2e04c1f9d8", "evidence": "returns-v1-chat-small-r1-0a4c2e"},
    "settings": {"PROVIDER_TIMEOUT": "10"},
}

The box prints 0.3.0-df62fa for exactly these values. Write "10" as text, as the course’s manifests do; the number 10 gives another ID. If you used other field names, your ID differs, and that is fine. Check the content instead:

• The password, the key and the address are not in it. They say where the release runs, not what it is, and a manifest is copied into releases and logs.
• The model has two lines: the name you ask for, and the version you expect behind it.
• Retrieval says “none” out loud. An empty section would not tell a reader whether someone forgot it.
• The evidence names the run that approved the release, so lineage reaches the evaluation.

Prediction: the ID changes, to 0.3.0-d9f5ac with these field names. Nothing in the team’s own files changed, but the release now expects another model. It is another release, and it needs its own evidence.

Completion check: trace a recorded answer

The module’s completion check: start from one recorded answer and find every version that made it, and where its input came from.

In “last week”, the same practice ticket T-65663 arrived several times: before the switch and after it. Ticket T-502406 is one copy that arrived before the switch. Trace it, and compare it with T-503354 from Artifact lineage:

python -m ops trace T-502406 data/telemetry/release-full.jsonl.gz --releases data/releases

Answer three questions in writing:

1. Which release made this answer, and which prompt, model version and evidence does it name?
2. How do you know that T-502406 and T-503354 had the same input? What does the tool show, and what does it not show?
3. Why did one get a valid answer and the other none? Name the lines of the two traces that prove it, and the experiment record that predicted it.

Worked solution

Ticket T-502406
  input     received +0.0 min, request 73de85044bbf, channel web_form, 292 characters, language guess en
            message fingerprint 3f8c9237238c5e1d = practice ticket T-65663 (made up for the evaluation course; split holdout, slice needs_person)
  result    ok after 3.347 s: team warranty, needs a person True, finish stop, 1920 tokens in, 297 out, US$0.000340
            answered by gpt-6-luna-2026-09-22 (replayed), prompt assistant-v2, trace 91ddac1f533db5fd08e1eeadef64b04f
  release   1.5.0-4fafd1 (data/releases/1.5.0-4fafd1.toml)
            code 1.5.0, commit none (the project folder is not a Git repository), fingerprint c8ee0759e085
            image not built (the course runs the service with Python)
            migrations 20, last 020_release_state.sql
            prompt assistant-v2 (fingerprint 82a15031e521), schema ticket_analysis (f66acdf35fe1), at most 1000 tokens
            model chat-small, expected version gpt-6-luna-2026-09-22, embeddings embed-small-384-v1
            retrieval answer-v1, index 75cf862216c3
            evaluation set d4c96eaa10c0 (inputs 3f90ab808275), evidence baseline-v2-chat-small-r1-b6a2b9

1. Release 1.5.0-4fafd1. It names the prompt assistant-v2 with fingerprint 82a15031e521 and the evidence baseline-v2-chat-small-r1-b6a2b9. The model is chat-small, expected version gpt-6-luna-2026-09-22, and the answer came from that version: no warning. Code fingerprint c8ee0759e085, 20 migrations, index 75cf862216c3.
2. Both traces show the message fingerprint 3f8c9237238c5e1d, 292 characters, and the match with practice ticket T-65663, a holdout ticket that needs a person. The tool never shows the text itself; the fingerprint proves that the two texts are equal without showing them.
3. The input, the code, the model and the index are the same. The release lines differ: 1.5.0-4fafd1 with prompt v2 against 1.5.0-767b56 with prompt v3. The result lines show the effect: finish stop, 297 tokens out, needs a person True for v2; finish length, 1000 tokens out and no decision for v3. EXP-001 had found the same failure on this ticket: its blocker line names T-65663, and its decision was do not ship.

A complete answer names the versions, the input’s provenance, and the evidence. “It was v3” is not enough.

Knowledge check

Check your understanding

Last week's evaluation result existed, and it said DO NOT SHIP. Explain why it still did not stop the change, and what this module changes so that it would.

Show the answer

The decision lived in the evaluation project, as run files and a terminal output. The service, its settings and the people who change them never saw it, and nothing tied a prompt name on the server to that decision. Now the experiment record lives in operations/experiments/, next to the release manifests. A release must name its evidence, and the evidence leads to the record and its decision. The service also refuses a prompt that is not the one in its release. The release gate of Module 5 makes the decision a check that runs before any rollout.

Check your understanding

Tomás says that EXP-001 was unfair because of the token limit. Why is the right answer a new record, EXP-002, and not an edit of EXP-001?

Show the answer

EXP-001 records what the team measured and decided with what it knew then. Editing it would hide that history and make the old decision look as if it had other evidence. Tomás’s idea is a new hypothesis with a new change, the token limit, so it needs its own runs and its own decision. EXP-002 names its baseline and refers to EXP-001, and both stay readable.

Check your understanding

Practical check: show the chain from a recorded answer to a decision. Which commands and lines do you use?

Show the answer

python -m ops trace T-503354 data/telemetry/release-full.jsonl.gz --releases data/releases shows the release 1.5.0-767b56 and the evidence candidate-v3-chat-small-r1-31b45f. grep -l "candidate-v3-chat-small-r1-31b45f" operations/experiments/*.toml finds EXP-001-prompt-v3.toml, and python -m ops experiments EXP-001 shows the decision: do not ship, by Priya and Grace, with its reason.

Recap

This module made the service traceable. Before it, an answer said only “prompt v3”. Now:

• Every AI result and every API answer carries a release ID. The release manifest behind it names the code, migrations, prompt, schema, model, retrieval, evaluation set, evidence and settings.
• The prompt fingerprint follows the text that the model reads, not the file name. The manifest also names the model version that the release expects, and ops trace warns when the provider answers with another one.
• ops release diff compares two releases field by field, and ops release verify and the service’s start refuse files that do not match.
• An experiment record keeps the hypothesis, the one change, the evaluation, the results and the decision, next to the releases. EXP-001 shows why prompt v3 was refused.
• From any recorded answer, you can now reach its versions, its input’s provenance and the decision that approved, or refused, its release.

Next, Module 3 makes the service tell you what it does while it runs. You add logs, metrics and one trace per ticket, each with this release ID.

Next: Logs, metrics and traces