Ownership and support

What you will be able to do

Last week’s incident had people who could have fixed it, but nobody whose job it was to notice. Give every part of the service and every objective one owner, across data, model, infrastructure and customer communication, and write down who is called first.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• What is operated: the inventory, its 14 parts and their first signs.
• Service objectives: the six SLOs, error budgets, burn rates and severity levels.

What you need on your computer

• Your project at the snapshot end-of-m01, with the saved telemetry in data/telemetry/. No Docker.

The sheet by the door

In a block of flats, there is a sheet by the front door. It says who to call. A water leak: the caretaker, Mr Diaz, and if he does not answer, the plumbing company’s emergency number. A smell of gas: the gas company, at once, at any hour. The lift: the lift company. And for every problem that affects the residents, the building manager writes a note in the hall.

The sheet is not about who caused the leak. It says who acts first, and who acts next if the first person cannot. Without it, ten residents see the water on the stairs, and each one thinks that somebody else has already called.

The problem: everyone saw it, nobody owned it

Here is last week again, told from the people’s side. It is constructed from the practice replay: the numbers are real, the conversations are not.

• Omar’s team changed ASSISTANT_PROMPT to assistant-v3 on a Monday. They saw that the API still answered 202, and they went home.
• Tomás looked at the dashboard on Tuesday. The valid-drafts panel was red, at 89%. He thought, “That is the model. Priya’s team watches the model.”
• Priya was busy with the evaluation set. Nobody had told her that the prompt had changed.
• Amira owned the API. The API was fine.
• Grace’s agents noticed fewer drafts, but tickets without a team do not appear in any agent’s list. They are invisible by design.
• On Wednesday, a customer phoned about a cracked frying pan.

Every person behaved sensibly. Nobody had a job that said “you act when the draft SLO burns”. The red panel had no owner.

The idea: one owner, called first

An owner of a part or an objective is the one person who decides about it, and who is called first when it fails. Not a team: a person, with a name. The owner does not have to fix everything alone. Their job is to act first: look, decide, fix or call the next person.

Three ideas go with ownership:

• Every part and every objective has exactly one owner. Two owners is the same as none: each waits for the other.
• Every owner has a backup. People sleep, go on holiday and get ill. The backup takes the calls when the owner cannot.
• An escalation path. When the cause is in another person’s part, the first owner calls that person. When customers are affected, someone tells them.

An AI service has four areas that need owners: data, model, infrastructure and customer communication. The fourth is easy to forget, and it is the one customers feel.

The four areas of ownership in Larkfield's service, and who holds each part. Omar owns the releases, not a component.

Text description of the diagram

The service has four areas. Data: Grace owns the policy and the shop’s documents, because she writes the rules; Priya owns the evaluation set; Tomás owns the retrieval index. Model: Priya owns the prompt and both models, because she measures their quality. Infrastructure: Amira owns the API, the worker and the cache; Mei owns PostgreSQL and the telemetry storage; Kwame owns the provider contract and the cost. Customer communication: Grace. Omar owns releases: he decides what ships, with the evidence that the gate gives, not instead of it.

A tiny example: one alert, three people

Take the draft SLO. Its owner, in operations/slos.toml, is Amira. Suppose its burn rate goes above 7: SEV-2, as the last lesson computed. Follow the call:

1. Amira is called first. She owns the SLO and the worker, so she can see the jobs. She runs ops summary and sees no_answer: 187. Many answers are empty.
2. The cause is in the prompt. Empty answers come from the model’s side, not from her code. She calls the owner of the prompt: Priya. Priya sees that assistant-v3 is live, and she knows that the evaluation said DO NOT SHIP.
3. Customers are affected. Ten tickets that needed a person reached nobody. That is SEV-1. Amira calls Grace, who finds those tickets and contacts the customers.

Three people, each called for one reason. Nobody had to guess. The diagram shows the full routing.

Who is called first for each objective, and who is called next. Every arrow ends with a person's name.

Text description of the diagram

Start at the top: an SLO burns or an alert fires. The SLO decides who is called first. Amira takes the three journeys that run through her code: accept, draft and answer time. Priya takes quality, Grace takes safety, Tomás takes answer sources, and Kwame takes cost. Amira is first for the most SLOs. So she often finds a cause in someone else’s part: the prompt or a model (Priya), the database or the telemetry (Mei), or the provider (Kwame). Whoever holds the incident calls Grace as soon as customers are affected.

Larkfield’s owners

The owners are already in the two files of the last lessons. The inventory gives each part an owner; the objectives give each SLO an owner. Ask the ops tool for one person:

python -m ops inventory --owner Grace

kind    component                                         owner  first sign of a failure
------  ------------------------------------------------  -----  ---------------------------------------------------------
data    support policy (inside the prompt)                Grace  replies that state an old rule; agents rewrite the drafts
data    shop documents                                    Grace  a document changes and answers still quote the old text
people  customer communication (status message, replies)  Grace  customers ask where their answer is

3 components, 1 owners: Grace.

Grace owns no code, and she is still an owner of three parts. The SLO table of ops slo has an owner column too. Put the two together, and you have the sheet by the door:

Person	Parts (inventory)	SLOs (called first)	Area
Amira	ticket-api, Valkey	accept, draft, answer_time	infrastructure
Priya	triage prompt, chat model, embedding model, evaluation set	quality	model, data
Grace	support policy, shop documents, customer communication	safety	data, customer communication
Tomás	retrieval index, tools	answer_sources	data
Mei	PostgreSQL, telemetry storage	–	infrastructure
Kwame	AI provider	cost budget	infrastructure
Omar	– (releases)	–	product

Mei owns no SLO. That is fine: when the database fails, the accept or draft SLO burns, Amira is called, and she calls Mei. Not every owner needs a pager, but every pager needs an owner.

Sam asks, “Why is Amira first for the draft SLO, when the problem last week was the prompt?”

“Because the first person must be able to look,” Amira says. “When the draft SLO burns, the cause can be the worker, the queue, the database, the provider or the prompt. I can see all of them from the worker. Priya can see only the model’s side. The first owner is the one with the widest view of that journey, not the one who is most likely to be guilty.”

The service brief

The inventory and the objectives are files for programs. People need one page. Larkfield’s service brief, operations/service-brief.md, is the module’s worked example. It has five short sections:

1. What the service does, in four sentences.
2. The user journeys and their objectives: the six SLOs and the cost budget, in a table.
3. Severity levels: SEV-1 to SEV-4, one line each.
4. Owners: who owns what, in one paragraph.
5. How the service can be up and still fail its users, the lesson of last week.

Read the last section of the file:

## How the service can be up and still fail its users

The accept SLO can be at 100% while the draft SLO fails: every ticket is saved, but the
assistant gives some of them no answer, so they wait for nobody. The dashboard's "uptime"
measures the first journey only.

Notice what the brief does not have: no architecture diagram, no code, no history. It answers the questions of a person who gets a call at night. What does this do? What does “working” mean? How bad is this? Who do I call?

When to write ownership down, and what it costs

Tomás asks, “Isn’t this a lot of process for six people?”

Grace answers: “It is one page. Last week, six people looked at a red panel and none of them acted. The page would have cost an hour to write.”

Write owners down as soon as two people share a service. Review them when people join, leave or change roles, and at every release. The cost is real but small: someone must keep the files true. The bigger cost is the backup: a backup must know the part well enough to act, so they need time to learn it. Ownership also has a risk: the owner can become the person who is blamed. In this course, an owner acts first; they are not the cause. Post-incident learning reviews incidents without blame.

Guided practice: read the brief against last week

Do these steps with the files of your project.

1. Open operations/service-brief.md in VS Code, and read the table of journeys.
2. Run python -m ops slo data/telemetry/release-full.jsonl.gz.
3. For each line of the output, find the journey in the brief, and write: met or not met, and who is called first.
4. Find the severity of the incident with the brief’s severity section.

Your notes should end like this: accept met (Amira); draft not met, burn 8.21 ≥ 7, SEV-2 (Amira); quality not met (Priya); safety misses, SEV-1 (Grace); answers: no data; cost over budget, SEV-3 (Kwame). The incident is SEV-1, the highest level of its parts.

Your turn: a service brief for the answers

Larkfield’s website also answers customers’ questions from the shop’s documents (POST /v1/answers). Treat it as its own small service, and write its service brief. Say what it does. Give two or three journeys, each with an SLI and an SLO. Say how you would use the severity levels. Name the owners, each with a backup. Use only parts that exist in the inventory. The normal week’s numbers are in the answer_time and answer_sources lines of ops slo.

A worked solution

# Service brief: answers to customers' questions

## What the service does
Customers ask a question on the website. The API finds the shop's documents that
match, through the retrieval index, and answers with sources. No person checks
the answer before the customer reads it.

## Journeys and objectives (window: one week of questions)
| Journey                           | SLI (good / all)                          | SLO  |
| A customer asks a question        | answers within 3 seconds                  | 95%  |
| The answer quotes current rules   | answers whose sources are the current     | 99%  |
|                                   | version of each document                  |      |
| The question finds a source       | answers with at least one source found    | 90%  |
Normal week (measured): 470 answers, 470 within 3 s, 470 with current sources.
The third SLO is a proposal: the project does not measure it yet.

## Severity
An answer that states a wrong rule about safety or money is SEV-1 (no person
checks it). An SLO that burns 7 times too fast is SEV-2. A slow burn is SEV-3.

## Owners (backup)
Answer time: Amira (backup Mei). Sources and the index: Tomás (backup Priya).
Shop documents: Grace (backup an agent she names). Customer messages: Grace.

## How it can be up and still fail its users
Every question gets a fast answer, but after a document changes and the
index is not rebuilt, the answers quote the old rule.

What makes it good: each SLI is good ÷ all, with a target and a window. The numbers say which are measured (470 of 470) and which are a proposal. Every owner is one person, with a backup. The SEV-1 line is specific to this service. Here, no agent stands between the answer and the customer, so a wrong rule reaches the customer directly. The last section names a failure that the uptime cannot see. LLM/RAG regressions replays exactly that failure.

Common mistakes

Common mistake

The owner is the person who made the change

Symptom: after last week, someone proposes: “Omar’s team changed the prompt, so Omar owns the draft SLO now.”

Cause: ownership is mixed up with blame. The person who made the last change is not the person who can see and act first next time.

Fix: keep the owner who has the widest view of the journey: Amira, for drafts. Fix the release process instead, so a prompt change goes through a gate and a canary. That is the work of the fifth module of this course.

Common mistake

The owner is away, and there is no backup

Symptom: an alert fires on a Saturday. The owner is on a plane. The alert waits until Monday.

Cause: the brief names one person for each part, and no backup.

Fix: add a backup for each owner, and make sure the backup can act: access to the dashboards, the ops tool and the runbook. Larkfield’s brief in the snapshot has owners only; adding the backups is part of the task above.

Knowledge check

Check your understanding

The quality SLO burns fast. Who is called first, and when must Grace be called?

Show the answer

Priya is called first: operations/slos.toml gives her the quality SLO, and she owns the prompt and the models, the most likely causes. If she finds that the cause is elsewhere (for example the database), she calls its owner (Mei). Grace is called as soon as customers are affected. For example, wrong decisions sent tickets to the wrong team. Or a ticket that needed a person did not reach one: that is SEV-1.

Check your understanding

Tomás says: 'uptime is green, so we're fine.' Answer him with one number from the normal week and one from last week.

Show the answer

The accept SLI, which is the uptime, was 100.00% in both weeks, so it cannot tell them apart. The draft SLI was 99.93% in the normal week and 91.79% last week. Last week, 197 tickets got no answer, and 10 that needed a person reached nobody. A green uptime says that tickets are accepted, not that customers are helped.

Check your understanding

Practical check: which SLOs does Amira own, and how many parts of the inventory does Grace own?

Show the answer

python -m ops slo data/telemetry/normal-week.jsonl.gz shows Amira as the owner of accept, draft and answer_time. python -m ops inventory --owner Grace ends with 3 components, 1 owners: Grace.: the support policy, the shop documents and customer communication.

Module recap and completion check

In this module, you ran the live service and read its saved telemetry. You listed the 14 parts that it depends on. You turned its user journeys into six SLOs with error budgets and severity levels. Then you gave every part and every SLO an owner. The module’s independent task was the service brief above.

Check your understanding

Completion check: explain how a system can be available but still fail its users. Use the ops tool's output on last week as your evidence, and say who should have been called.

Show the answer

Run python -m ops slo data/telemetry/release-full.jsonl.gz. The accept SLI is 100.00% (2,400/2,400): the service was available, and every ticket was saved. But the draft SLI is 91.79% (2,203/2,400) with a burn rate of 8.21, the safety SLI has 4 misses, and the cost is US$0.440 per 1,000 tickets (budget US$0.40, OVER). The availability SLI measured only the first journey (sending a ticket). The help desk’s journey failed after the API had answered. Prompt v3 gave answers without text, so tickets waited with no team, and 10 that needed a person reached nobody. Severity: SEV-1 (safety). Called first: Amira, owner of the draft SLO; then Priya, owner of the prompt; and Grace, for the customers.

Recap

• An owner is one person who decides about a part and is called first. Every part and every SLO has one; every owner needs a backup.
• The four areas are data, model, infrastructure and customer communication. Grace owns no code and still owns the most important message.
• The first person called is the one with the widest view of the journey, not the one who made the last change. They escalate by cause, and call Grace when customers are affected.
• A service brief puts the journeys, objectives, severity levels and owners on one page, for the person who is called at night.

Next module: last week, nobody could even say which tickets were answered with prompt v3. You give every answer a release ID that leads back to every version that made it.

Next: Artifact lineage