What is operated

What you will be able to do

Last week, one part of the service changed and nobody looked at it, because nobody had written it down as a part. List everything that the ticket service depends on, where to read each part’s version, and the first sign that each part fails.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Run the live service: the triage step, prompt v2 and v3, replayed answers and python -m ops.
• The answers from documents (/v1/answers) of Scaling APIs and AI Workloads, and the retrieval idea from the evaluation course’s Retrieval metrics.

What you need on your computer

• Your project from the last lesson, at the snapshot end-of-m01, with its virtual environment. Docker is not needed in this lesson.

The hotel’s maintenance list

Think again of the small hotel. One morning, there is no hot water. The new manager does not know where the boiler is, which company services it, or how old it is. Guests complain for three hours while he searches.

An experienced manager keeps a list on the office wall. Each line is one thing that the hotel depends on: the boiler, the lifts, the internet, the laundry company, the booking website. Each line says who to call and which model or contract it is. It also gives the first sign of trouble: “the water in room 101 is cold after 7 a.m.”, “the lift stops between floors”. The list is short, but nobody searches in an emergency.

That list has two kinds of lines. Some are machines in the building. Others are not machines at all: a supplier, a contract, a booking system run by another company. They fail too, and they fail in their own way.

The problem: the part that nobody listed

Here is what changed last week, in the practice replay. One line in the service’s settings:

ASSISTANT_PROMPT=assistant-v3

No code changed. The tests passed, because they test the code. The container image was the same. The API kept answering 202 Accepted. From the point of view of the code, nothing had happened. But the service’s behaviour changed a lot:

Last week's one change, and where it showed. The only number that everyone watched did not move. A practice replay; the answers are real recordings.

Text description of the diagram

The change went into one part: the triage prompt. From there, three things moved. One answer in ten had no text. So the share of tickets with a valid draft fell to 91.79%, and ten tickets that needed a person reached nobody. The model wrote more than twice as many tokens, so the cost per 1,000 tickets rose from US$0.338 to US$0.474. The answers took longer, but still less than 10 minutes, so that number looked fine. The accept rate, the “uptime”, stayed at 100%: it did not depend on the prompt at all.

Nobody watched the prompt, because the prompt was not on anybody’s list. In people’s heads, “the service” was the code. In fact, the service is the code and everything that the code depends on to give a good answer.

The idea: an inventory of what is operated

To operate a service is to keep it doing its job for its users, every day, after it is built. What you operate is everything that can make it stop doing that job. An inventory is the written list of those parts, like the hotel’s list on the wall. For each part, it answers four questions:

1. Kind: what sort of part is it? Code, data, model, prompt, index, tool, provider, infrastructure or people.
2. Version: which version runs now, and where do you read it? A version that you cannot read is a version that you cannot compare.
3. Owner: who decides about this part, and who is called first when it fails? The next lessons go deeper.
4. First sign of a failure: what is the first thing that you would see when this part goes wrong? This is the signal to watch.

An AI service has more kinds of parts than a classic web service. A classic service changes when its code changes. An AI service also changes when:

• a prompt changes, as last week, with no code change;
• the provider changes its model, on its own schedule, behind the same name;
• the data changes: a policy text, the shop’s documents, the evaluation set;
• the retrieval index gets old, because the documents changed and the index was not built again.

Each of these needs its own line, its own version and its own first sign.

A tiny example: three lines

Start with three parts of the triage step, and write the four answers for each.

Part	Kind	Version: where to read it	Owner	First sign of a failure
ticket-api	code	the Git commit and the image digest	Amira	5xx answers; jobs that fail or wait too long
triage prompt	prompt	ASSISTANT_PROMPT and its fingerprint	Priya	answers without text; more tickets sent to a person
chat model	model	the model version in the provider’s answers	Priya	a new model version; changed tokens, time or decisions

Now replay last week against this small table. The change was in the second line. Its first sign, “answers without text”, is exactly what happened: 186 of 1,798 v3 answers. With this table on the wall, the question “what changed?” has a short list of places to look. Without it, people look at the code, and the code did not change.

The middle column needs care. The prompt’s name is not enough as a version: someone can edit assistant-v2.md and keep the name. So the project also computes a fingerprint: a short hash of everything that the prompt sends besides the ticket. That is the instructions, the policy, the output schema and the token limit. Run it in your project:

python -c "from ticket_api import assistant; print(assistant.fingerprint('assistant-v2'), assistant.fingerprint('assistant-v3'))"

82a15031e521 e91df8e7a92a

Two prompts, two fingerprints. If one word of the policy changes, the fingerprint changes too. A prompt version in this course is the pair: the file name, which people read, and the fingerprint, which a program can check. The rule in assistant.py is: to change a prompt, add a new file; never edit a file that a release used.

Larkfield’s inventory

Amira and Priya wrote the whole list in operations/inventory.toml. Each part is one block. Here is the prompt’s block:

[[component]]
name = "triage prompt"
kind = "prompt"
version = "ASSISTANT_PROMPT (a file in ticket_api/prompts/) and its fingerprint"
owner = "Priya"
first_sign = "answers without text; more tickets sent to a person; agents correct more teams"

The file is text, so it lives in Git with the code, and a change to it is reviewed like code. The ops tool prints it as a table:

python -m ops inventory

kind            component                                         owner  first sign of a failure
--------------  ------------------------------------------------  -----  --------------------------------------------------------------------------------
code            ticket-api (API and worker)                       Amira  5xx answers from POST /v1/tickets; jobs that fail or wait too long
prompt          triage prompt                                     Priya  answers without text; more tickets sent to a person; agents correct more teams
data            support policy (inside the prompt)                Grace  replies that state an old rule; agents rewrite the drafts
model           chat model (chat-small)                           Priya  a new model version in the answers; changed tokens, time or decisions
provider        AI provider                                       Kwame  HTTP 429 (quota) or 5xx and timeouts; the circuit breaker opens
data            evaluation set                                    Priya  a release decision that the set cannot support (a new kind of ticket is missing)
model           embedding model (embed-small)                     Priya  similar-ticket search returns unrelated tickets
data            shop documents                                    Grace  a document changes and answers still quote the old text
index           retrieval index (answers)                         Tomás  answers that cite an old document version; questions with no source found
tool            tools                                             Tomás  -
infrastructure  PostgreSQL (tickets, jobs, AI results)            Mei    503 database_busy; slow queries; a failed backup
infrastructure  Valkey (cache)                                    Amira  X-Cache: off (the service still works, but slower and dearer)
infrastructure  telemetry storage (collector, Grafana)            Mei    empty dashboards; alerts that cannot be evaluated
people          customer communication (status message, replies)  Grace  customers ask where their answer is

14 components, 6 owners: Amira, Grace, Kwame, Mei, Priya, Tomás.

Fourteen parts. Read a few lines that a classic web service would not have:

• Support policy (data). Grace’s rules are part of the prompt. A changed rule changes the answers, so it has its own line and its own owner: Grace writes the rules.
• Evaluation set (data). It does not serve any customer. But every release decision rests on it. If a new kind of ticket appears and the set has none, the set fails silently: it says “ship” about tickets that it never saw.
• Retrieval index. The answers to customers’ questions quote the shop’s documents through an index. Its version is the documents’ fingerprint at the moment it was built. If the documents change and the index does not, the answers cite old text.
• Tools: none. The assistant calls no tools today: it cannot look up an order. The line is still there, on purpose. “None” is a decision that someone made, and the day a tool is added, the line already exists and has an owner.
• Customer communication (people). When something fails, someone must tell the customers. That is a part of the service too, and its first sign is the worst one: customers ask where their answer is.

Two options help when the list is long. --versions shows the version column instead of the first signs. --owner shows one person’s parts:

python -m ops inventory --owner Priya

kind    component                      owner  first sign of a failure
------  -----------------------------  -----  --------------------------------------------------------------------------------
prompt  triage prompt                  Priya  answers without text; more tickets sent to a person; agents correct more teams
model   chat model (chat-small)        Priya  a new model version in the answers; changed tokens, time or decisions
data    evaluation set                 Priya  a release decision that the set cannot support (a new kind of ticket is missing)
model   embedding model (embed-small)  Priya  similar-ticket search returns unrelated tickets

4 components, 1 owners: Priya.

The map: what depends on what

A table lists the parts, but it does not show which part depends on which. When the AI provider fails, which parts fail with it? The map answers that question.

Larkfield's ticket service: what it depends on, and who owns each part. Drawn from operations/inventory.toml.

Text description of the diagram

Read the map from the top. Customers send tickets to ticket-api. Everything else hangs below it. The prompt includes the policy, and the evaluation set tests the prompt before a release (a dashed arrow: a check, not a call). The AI provider serves both models. The index is built from the shop’s documents. On the right, the telemetry storage is what you will read in later modules. Each box says its owner. The colours group the kinds, and the kind is also written in each box.

Follow one failure down the arrows. If the AI provider stops answering, the chat model and the embedding model stop with it. So the triage step stops, and so do new embeddings for similar-ticket search. PostgreSQL and the cache still work, so the API still accepts tickets. That is useful to know before it happens: an outage of the provider is not an outage of the API.

When to keep an inventory, and what it costs

Tomás is not convinced. “We have a README and a compose file. Isn’t that the inventory?”

Amira answers: “The compose file lists the containers. It does not list the prompt, the model version, the policy or the evaluation set. And it says nothing about who to call, or what failure looks like.”

Keep an inventory as soon as a service has real users and more than one person works on it. It is cheap to write: Larkfield’s is 14 blocks of text. The cost is keeping it true. An inventory that is out of date is worse than none, because people trust it. Two habits help. First, review it at every release, with the release’s changes in front of you. Second, keep it next to the code, so that a change to the service and a change to its list arrive in the same review.

Do not try to list everything. A line earns its place when its failure can hurt users, and when someone must decide about it. Python’s packages are not lines here: they are in the lock file, and the code’s tests cover them.

Your turn

Larkfield plans to let the assistant look up an order’s status. The tool calls the shop’s order system, which another team runs. Write the inventory block for it: name, kind, version, owner and first sign of a failure. Then decide which line of the current inventory changes too.

A worked solution

[[component]]
name = "order lookup (tool)"
kind = "tool"
version = "the order system's API version, and the tool's schema in ticket_api/prompts/"
owner = "Tomás"
first_sign = "lookups that fail or time out; replies that state an order status the lookup did not return"

The line tools changes: its version is no longer “none”. The order system belongs to another team. So the owner here is the person at Larkfield who answers for the tool: Tomás, who already owns the tools line. The first sign has two parts. One is technical: failed lookups. The other is about quality: an invented status. The prompt already forbids a status that is not from a lookup, but you still watch for it.

Common mistakes

Common mistake

ops inventory: no file operations/inventory.toml

Symptom:

ops inventory: no file operations/inventory.toml (run ops from the project folder)

Cause: you ran the command from another folder, or your project is still at the snapshot start, which has no operations/ folder.

Fix: go to the project folder (cd ~/projects/ticket-api). If ls operations shows nothing, copy the snapshot end-of-m01 over your project, as Run the live service shows.

Common mistake

An owner that is a team, a version that is a name

Symptom: a line says owner = "the AI team", or version = "latest". In an incident, everyone thinks that someone else is handling it, and nobody can say what changed.

Cause: the inventory was written as a description, not as a tool for a bad day.

Fix: write one person’s name for each owner. The next lesson adds a backup person. For each version, write where a program or a person reads the exact value. That is a fingerprint, a commit, a digest, or the model version in the provider’s answer. “latest” is not a version: it changes without telling you.

Knowledge check

Check your understanding

Last week's change passed every test, and the container image did not change. Which line of the inventory changed, and which of its four answers would have pointed to the problem first?

Show the answer

The triage prompt (kind: prompt) changed: ASSISTANT_PROMPT went from assistant-v2 to assistant-v3, and its fingerprint from 82a15031e521 to e91df8e7a92a. Its first sign, “answers without text”, matches what happened: 186 of 1,798 v3 answers had no text. The tests check the code, and the image holds the code, so neither could see a change in a setting.

Check your understanding

Predict: the AI provider is down for 10 minutes. Using the map, which parts stop working, and does the API still accept tickets?

Show the answer

The provider serves the chat model and the embedding model, so triage and new embeddings stop. The API depends on PostgreSQL to save tickets, not on the provider, so it still accepts them with 202. The jobs wait in the queue until the provider is back. The accept rate stays at 100% while no ticket gets a draft. A later module runs exactly this outage.

Check your understanding

Practical check: how many components and owners does the inventory have, and where do you read the version of the chat model?

Show the answer

python -m ops inventory ends with 14 components, 6 owners: Amira, Grace, Kwame, Mei, Priya, Tomás. With --versions, the chat model’s line says “the model version in the provider’s answers (gen_ai.response.model)”. The simulator reports it too: curl http://127.0.0.1:8300/admin/stats shows "model_version":"gpt-6-luna-2026-09-22".

Recap

• To operate a service is to keep it doing its job for its users. What you operate is everything that can stop it: code, data, models, prompts, indexes, tools, providers, infrastructure and people.
• An inventory gives each part a kind, a version and where to read it, an owner, and the first sign of a failure. Larkfield’s has 14 parts and 6 owners: python -m ops inventory.
• An AI service changes without a code change: a prompt, a provider’s model, a document, an old index. A prompt version is a file name and a fingerprint.
• The map shows what depends on what: a provider outage stops triage, but the API still accepts tickets.

Next, you decide what “working” means for each user journey, in numbers.

Next: Service objectives