LLM-specific configuration

What you will be able to do

Give every part of an LLM call that changes its answers a version that follows its content, and compare two releases setting by setting. A prompt’s name, or a model’s name, can stay the same while the thing behind it changes.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Release manifests, release IDs and python -m ops release build, from Artifact lineage.
• What a prompt, an output schema and a token limit are, from Building Reliable Applications with LLM APIs.
• Prompt v2 and the candidate prompt v3, and why the evaluation said DO NOT SHIP, from Trade-offs and regressions.

What you need on your computer

• Your project after Artifact lineage: python -m pytest prints 193 passed, and operations/releases/ has your manifest 1.5.0-99b6da (or your own ID).
• The saved files in data/telemetry/ and data/releases/.

The ops release commands read files only. Only the last steps, which start the API, need Docker.

The same name on the card

A café sells a “house latte”. The name on the menu has not changed in three years. In those three years, the café changed its coffee beans twice and its milk once. The barista also makes it with a smaller cup now.

A regular customer says: “The latte tastes different since last month.” The manager looks at the menu and says: “It is the same house latte.” Both are right. The name did not change, but the things behind the name did.

To answer the customer, the manager needs more than the name. She needs today’s recipe, and she needs to know which changes the supplier made without asking.

An LLM call has the same problem, twice.

The problem: two names that hide changes

The service asks the provider for one triage answer per ticket. The answer depends on more than the customer’s message. It depends on the prompt text, the support policy inside it, the output schema, the token limit, and the model that the provider runs. In Artifact lineage, each of these got a line in the manifest. Here is why each line records content, not a name.

First hidden change: the prompt file. Grace’s policy says that customers can return unused items within 30 days. Suppose that someone changes it to 60 days in ticket_api/prompts/policy.md, on the server, without a release. The file assistant-v2.md does not change. The setting still says ASSISTANT_PROMPT=assistant-v2. Every new row in ai_runs still says prompt_version = assistant-v2.

But the model now reads another policy, and it writes other replies. The evaluation that approved v2 tested the 30-day policy. Its result no longer applies, and nothing in the database shows it. The name is a label on a box; anyone can change what is in the box.

Second hidden change: the model. The service asks for "model": "chat-small". That is a deployment name: a name that the provider account gives to one model. The provider can put a newer model version behind the same name. The service’s request does not change at all, and the answers may change.

This happened in one of the course’s practice scenarios, a practice change we caused on purpose. For two and a half minutes, the simulator reported another model version behind chat-small. The service sent the same requests as before.

The idea: version the content, and expect the rest

The fix has two parts, one for each hidden change.

For what you control, version the content. In What is operated, a prompt version became a pair: the file name, for people, and a fingerprint, for programs. The name alone is not enough. The fingerprint covers everything that the service sends besides the customer’s message: the instructions, the policy, the output schema and the token limit. If any of these changes, the fingerprint changes. The project uses the first 12 characters of a SHA-256 hash, as for the dataset version in the evaluation course.

For what the provider controls, write down what you expect, and check it. The manifest names the deployment, chat-small, and the expected model version behind it. Each answer says which version answered. If the two differ, the provider changed the model, and the service can say so.

Together, these are the service’s LLM-specific configuration: the prompt, the schema, the provider and model identifiers, the retrieval settings and the evaluation set. Ordinary services have settings too. An LLM service has more of them. Some live outside your code: in text files that non-developers edit, and at the provider.

What decides one triage answer, and which field of the manifest records each part.

Text description of the diagram

The request has six parts. The system text comes from the prompt file, with the policy inside it. The user text holds the customer’s message. Then come the output schema, the token limit of 1,000 and the deployment name chat-small. The prompt fingerprint covers the system text, the policy, the user text’s template, the schema and the token limit. The schema also has its own fingerprint. The customer’s message is the input of one ticket, not part of the release. On the right, the provider chooses which model version runs behind chat-small. The answer reports that version, and the service compares it with the manifest’s expected version. We control the left side; the provider controls the right side.

A tiny example: what changes a fingerprint

The fingerprint must change when the model would read something different, and only then. The project computes it in ticket_api/assistant.py:

def fingerprint(version: str) -> str:
    """A short hash of everything the prompt version sends besides the ticket: the
    instructions, the policy and the output schema. The same fingerprint = the same prompt."""
    system, user = load_prompt(version)
    blob = json.dumps([system, user, response_format(), MAX_TOKENS], ensure_ascii=False)
    return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:12]

Four things go into the hash: the system text with the policy, the user text, the schema and the token limit. The prompt file also has notes for people above the [system] line. The service never sends them, so they are not in the hash.

Predict before you read on. Which of these three edits changes the fingerprint 82a15031e521 of prompt v2?

1. Add a note line at the top of assistant-v2.md, above [system].
2. Change “within 30 days” to “within 60 days” in policy.md.
3. Change the token limit MAX_TOKENS from 1,000 to 1,200 in assistant.py.

The author made each edit on a copy of the snapshot, ran python -m ops release verify 1.5.0-99b6da, and then undid the edit:

Edit	The service sends	Prompt fingerprint	verify says
None	the same text	82a15031e521	matches
1. A note above [system]	the same text	82a15031e521	matches
2. 30 → 60 days in the policy	another policy	24e380c8e37d	does NOT match: prompt
3. Token limit 1,200	another limit	e0acb7fc0400	does NOT match: prompt and code

Edit 2 is the café’s new beans: the name assistant-v2 stays, and the content changes. Edit 3 changes a Python file too, so the code fingerprint also changes. Edit 1 changes nothing that the model reads, so it is correctly not a new prompt.

Notice one more effect of edit 2. The policy is shared by both prompts, so the edit also changes the fingerprint of prompt v3. One file, two prompts changed.

Compare two releases, setting by setting

Now look at last week with this view. The course’s two manifests are the release with prompt v2 and the release with prompt v3, as the replay ran them. Compare them:

python -m ops release diff 1.5.0-4fafd1 1.5.0-767b56 --releases data/releases

3 of 25 settings differ:
setting                       1.5.0-4fafd1                      1.5.0-767b56
----------------------------  --------------------------------  ---------------------------------
assistant.prompt              assistant-v2                      assistant-v3
assistant.prompt_fingerprint  82a15031e521                      e91df8e7a92a
data.evidence                 baseline-v2-chat-small-r1-b6a2b9  candidate-v3-chat-small-r1-31b45f

The tool lists every field of both manifests, 25 in all, and prints only the fields that differ. Three differ:

• The prompt and its fingerprint. This is the change itself.
• The evidence. Each release names the evaluation run that supports it. The v3 release names the candidate run, the one whose decision was DO NOT SHIP.

Everything else is equal: the code fingerprint, the migrations, the schema, the token limit, the model and its expected version, the index and the settings. So Grace’s third question has an answer. Nothing else changed. If customers saw different answers, the prompt caused it.

Three lines in a diff can look small. Here is what those three lines did to the replayed tickets:

The replay of last week: the same service, the same model, only the prompt differs. Replayed answers; the switch to v3 for every ticket is constructed.

Text description of the chart

With prompt v2, 1 of 602 tickets got an answer without text (0.2%). With prompt v3, 186 of 1,798 tickets did (10.3%), and 10 more became dead letters.

• prompt v2
• prompt v3

Prompt v3 sends fewer tokens in, as Omar hoped, but the model writes more than twice as many tokens out.

Text description of the chart

Tokens in, the median per call: 1,902 with v2 and 1,477 with v3. Tokens out: 260 with v2 and 627 with v3.

What this tells us. The short prompt saved about 400 tokens in, but the model used about 370 more tokens out. Output tokens cost more, so the cost per 1,000 tickets rose from US$0.338 to US$0.474. The longer answers also took longer: the triage p95 went from 5.17 s to 8.34 s. Some answers used all 1,000 tokens before they wrote any text, and those tickets got no answer. One line of configuration changed the service’s quality, cost and speed. That is why a prompt is an artifact with a version, not “just a setting”.

The provider’s side: the model version

The diff compares what you released. It cannot see what the provider changed. For that, the service compares each answer with the manifest’s expected version. ops trace prints a warning when they differ.

The course’s stale-index scenario includes a practice model-version change that we caused on purpose. From minute 7.3 of the run, the simulator reported gpt-6-luna-2026-11-04 behind chat-small. The answers themselves were the same recordings as before; only the reported version changed. Trace a ticket from that time:

python -m ops trace T-500222 data/telemetry/stale-index.jsonl.gz --releases data/releases

Ticket T-500222
  input     received +7.4 min, request abcae4b8ae23, channel web_form, 269 characters, language guess en
            message fingerprint 2612adf43bc2f172 = practice ticket T-65224 (made up for the evaluation course; split dev, slice boundary)
  result    ok after 3.455 s: team account, needs a person False, finish stop, 1917 tokens in, 282 out, US$0.000335
            answered by gpt-6-luna-2026-11-04 (replayed), prompt assistant-v2, trace 48081b8d32f2658d626bdeb5bf8d1c26
  release   1.5.0-4fafd1 (data/releases/1.5.0-4fafd1.toml)
            code 1.5.0, commit none (the project folder is not a Git repository), fingerprint c8ee0759e085
            image not built (the course runs the service with Python)
            migrations 20, last 020_release_state.sql
            prompt assistant-v2 (fingerprint 82a15031e521), schema ticket_analysis (f66acdf35fe1), at most 1000 tokens
            model chat-small, expected version gpt-6-luna-2026-09-22, embeddings embed-small-384-v1
            retrieval answer-v1, index 75cf862216c3
            evaluation set d4c96eaa10c0 (inputs 3f90ab808275), evidence baseline-v2-chat-small-r1-b6a2b9
  WARNING   the answer came from gpt-6-luna-2026-11-04, but the release expects gpt-6-luna-2026-09-22: the provider changed the model

The release is the same, 1.5.0-4fafd1, and nothing in your diff would show a change. Only the comparison of each answer with the expected version catches it. In this scenario, 72 of the 450 tickets were answered under the new version name. Module 4 turns this check into an alert, in LLM/RAG regressions.

Kwame owns the provider contract. He adds one rule: where the provider allows it, ask for a pinned model version instead of a moving name. A pinned version changes only when you change it. Providers retire old versions, so the check stays necessary.

The other LLM-specific lines

Three more sections of the manifest are specific to AI services:

• The output schema (schema, schema_fingerprint). The schema tells the model the exact JSON shape to answer in. A changed schema can break the code that reads the answer, even with the same prompt. It has its own fingerprint, and it is inside the prompt fingerprint too.
• Retrieval (answer_prompt, search, index_version). Answers to questions search Larkfield’s documents. A new document version without a new index makes answers cite old text. At this stage, the line says none (answers search the documents table). Module 4 adds the index and its version.
• The evaluation set (evaluation_set, evaluation_inputs, evidence). The evidence is only valid for the set that it was measured on. The evaluation course gave the set two versions: d4c96eaa10c0 for the whole file, and 3f90ab808275 for the inputs only. If the inputs change, the old evidence no longer describes the release, and the evaluation must run again.

Try it on your project

You build the manifest of a v3 release, compare it with your v2 release, and catch an edited prompt.

1. Build the v3 manifest, with the candidate run as its evidence:

   python -m ops release build --prompt assistant-v3 --evidence candidate-v3-chat-small-r1-31b45f --write

   The last line says Saved operations/releases/1.5.0-054591.toml. Building a manifest does not deploy it: the service still runs 1.5.0-99b6da.

2. List your releases, then compare them:

   python -m ops release list
   python -m ops release diff 1.5.0-99b6da 1.5.0-054591

   release       prompt        model version          last migration       evidence
   ------------  ------------  ---------------------  -------------------  ---------------------------------
   1.5.0-054591  assistant-v3  gpt-6-luna-2026-09-22  016_release_ids.sql  candidate-v3-chat-small-r1-31b45f
   1.5.0-99b6da  assistant-v2  gpt-6-luna-2026-09-22  016_release_ids.sql  baseline-v2-chat-small-r1-b6a2b9
   3 of 20 settings differ:
   setting                       1.5.0-99b6da                      1.5.0-054591
   ----------------------------  --------------------------------  ---------------------------------
   assistant.prompt              assistant-v2                      assistant-v3
   assistant.prompt_fingerprint  82a15031e521                      e91df8e7a92a
   data.evidence                 baseline-v2-chat-small-r1-b6a2b9  candidate-v3-chat-small-r1-31b45f

   The same three lines as the course’s diff. Your manifests have 20 fields, not 25: your shell set no behaviour setting, so [settings] is empty.

3. Check that your files still match your v2 release:

   python -m ops release verify 1.5.0-99b6da

   Release 1.5.0-99b6da matches the files here (code, prompt, migrations).

4. Make the hidden change. Open ticket_api/prompts/policy.md. In rule 2, change within 30 days of delivery to within 60 days of delivery, and save the file.

5. Run verify again:

   Release 1.5.0-99b6da does NOT match the files here:
     - prompt assistant-v2: the manifest has fingerprint 82a15031e521, the file now has 24e380c8e37d

   The command ends with exit code 1, so a script or a CI job can stop on it.

6. Start the API with fastapi dev, as a server would after the edit. It refuses to start.

   The last line of its output:

   Settings error: RELEASE_ID 1.5.0-99b6da does not match: prompt assistant-v2: the manifest has fingerprint 82a15031e521, the file now has 24e380c8e37d

   This is the check in config.py from the last lesson. A release cannot run with a prompt that is not the one in its manifest.

7. Undo the edit: change 60 back to 30, and save. Run python -m ops release verify 1.5.0-99b6da again. It must say matches. Then fastapi dev starts as before; stop it with Ctrl+C.

If the policy must really change, that is fine. Make a new manifest, and the change gets a new ID, a new fingerprint and, before customers see it, new evidence.

Your turn: predict a diff

Change one condition and predict before you run anything. In ticket_api/assistant.py, the line MAX_TOKENS = 1000 sets the token limit of every triage call.

1. Predict: if you change it to 1200 and build a v2 manifest, which fields of the diff with 1.5.0-99b6da change? Write your list down.
2. Make the change and build: python -m ops release build --prompt assistant-v2 --evidence baseline-v2-chat-small-r1-b6a2b9 --write.
3. Run python -m ops release diff 1.5.0-99b6da with the new ID.
4. Undo the change, and delete the new manifest file from operations/releases/. verify 1.5.0-99b6da must say matches again.

Worked solution

The new ID is 1.5.0-086ef5, and three fields differ:

3 of 20 settings differ:
setting                          1.5.0-99b6da  1.5.0-086ef5
-------------------------------  ------------  ------------
assistant.max_completion_tokens  1000          1200
assistant.prompt_fingerprint     82a15031e521  e0acb7fc0400
code.fingerprint                 375564320c6c  eac625037c39

The token limit is recorded on its own line. It is also inside the prompt fingerprint, because the model gets the limit with every request. And it lives in a Python file, so the code fingerprint changes too. The prompt’s name, assistant-v2, did not change: the diff does not list it.

Why could this matter? In the replay, prompt v3 answers without text had used all 1,000 tokens. A higher limit might rescue some of them, and it would also raise the cost of every long answer. That is a change to evaluate as an experiment, not to make on a server, which is the subject of Experiment records.

When to version a setting, and what it costs

Tomás asks, “Do we need a new release for every comma in the policy? Grace edits that file every week.”

Priya answers: “Yes, if the model reads the comma. The policy is part of the prompt. A comma can change what the model says. A note above [system] is not read by the model, so it is free.”

Version by content anything that the model reads or that shapes its output. That is the prompt text, policy, examples, schema, token limit and model parameters such as temperature. Record and check the identifiers that the provider controls: the deployment and the model version. Record the evaluation set and the run that supports the release.

Do not put in the manifest values that change for every request, such as the customer’s message or the time. They belong to the input’s provenance, which ops trace shows.

What it costs: more releases, because small text edits now count. The team needs a way to make a release in minutes, or people will go around it. Fingerprints are also strict: a changed line ending or a trailing space gives a new fingerprint. That is the right answer, because the model reads those bytes, but it can surprise people.

Common mistakes

Common mistake

The service refuses to start after a small edit

Symptom: fastapi dev or the worker stops at once. The message starts with RELEASE_ID 1.5.0-99b6da does not match: prompt assistant-v2.

Cause: a file that the prompt fingerprint covers changed after the release was built: the prompt file, policy.md, the schema, or the token limit.

Diagnose: python -m ops release verify 1.5.0-99b6da names the part that differs. On a project in Git, git diff shows the edit.

Fix: undo the edit. If the edit is wanted, build a new manifest, test it, and set RELEASE_ID to the new ID.

Common mistake

The diff is empty, but the answers changed

Symptom: customers or reviews say that the answers changed. ops release diff between the release before and the release now shows no difference, or it is the same release.

Cause: something outside your release changed. Most often, the provider changed the model behind the deployment name.

Diagnose: run ops trace on a recent ticket and look for the WARNING line, or count the model versions in the results. On the replay, T-500222 shows it.

Fix: tell Kwame, who owns the provider contract. Pin the model version if the provider allows it. Run the evaluation again on the new version before you accept it.

Knowledge check

Check your understanding

The setting says ASSISTANT_PROMPT=assistant-v2, and every row of ai_runs says prompt_version = assistant-v2. Explain how the model can still read a different prompt, and how the project detects it.

Show the answer

assistant-v2 is only a file name. The policy (policy.md), the schema or the token limit can change without a change to that name. The project hashes everything that the service sends besides the message, the prompt fingerprint, and writes it in the manifest. ops release verify and the start of the service compare the manifest’s fingerprint with the files. A difference means “not the prompt of this release”, and the service refuses to start.

Check your understanding

ops release diff between last week's two releases lists 3 of 25 settings. Why is it useful that the other 22 are listed as equal?

Show the answer

It rules out the other causes. The code, the migrations, the schema, the token limit, the model and its expected version, the index and the settings were the same. So the change in quality, cost and speed came from the prompt. Without the full manifests, nobody could say that the model or the code did not also change.

Check your understanding

Practical check: show that your project catches an edited prompt. Which commands and outputs prove it?

Show the answer

After the edit of policy.md, python -m ops release verify 1.5.0-99b6da prints does NOT match and exits with code 1. The prompt line shows the fingerprints 82a15031e521 and 24e380c8e37d. fastapi dev refuses to start with the same reason. After the edit is undone, verify prints matches. ops release diff 1.5.0-99b6da 1.5.0-054591 lists the prompt, its fingerprint and the evidence.

Recap

• A prompt’s name is not its version. The prompt fingerprint hashes the system text with the policy, the user text, the schema and the token limit. Notes for people do not count, because the model never reads them.
• The provider controls the model version behind a deployment name. The manifest records the expected version, and ops trace warns when an answer came from another one.
• python -m ops release diff compares two releases field by field. Last week’s releases differ in 3 of 25 fields: the prompt, its fingerprint and the evidence. With those three lines, 10.3% of the replayed v3 tickets got no answer.
• python -m ops release verify and the service’s start refuse files that do not match the release.

Next, you look at the record that should have stopped last week’s change: the experiment that tested prompt v3, its evidence and its decision.

Next: Experiment records