Quality attributes

What you will be able to do

Turn the words people use about a system, such as “fast”, “always on”, “private” and “cheap”, into targets that you can measure. A target that nobody can measure cannot be met, missed or improved.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• Larkfield’s five journeys and the out-of-scope list, from Users and workflows.
• The median and the 95th percentile (p95) of a set of times, from Latency and capacity in Scaling APIs and AI Workloads.
• Grace’s 67 test questions and how an answer is checked, from Answer measures in RAG: Building AI That Uses Your Data.

What you need on your computer

• Your folder design-pack, with the first half of requirements.md from the last lesson. You fill in the section Quality attributes. No command runs in this lesson.

“Fast delivery” or “30 minutes”

Two pizza shops are on the same street. The first one says: “Fast delivery.” The second one says: “95% of our deliveries arrive within 30 minutes. We count every week and put the number on the door.”

Which promise can you check? Only the second. “Fast” means 15 minutes to one customer and 45 to another, so the first shop can never be wrong. The second shop chose its words with care. It did not promise every pizza in 30 minutes, because one flat tyre would break that promise. It promised 95%, it said over what period, and it said how it counts.

Look also at what the promise costs. To keep it, the shop needs enough drivers at seven in the evening on a Friday, its busiest hour. “99% within 20 minutes” would need many more drivers, and the pizzas would cost more. A good promise is one that customers value and the shop can pay for.

The problem: everybody agrees, and nobody means the same thing

Kwame asks each person what the assistant must be like. Everybody answers at once. Grace says: “Fast. My agents have a customer on the phone.” Omar says: “Always on.” Ines says: “Private. My supplier terms must never reach another shop.” And Omar adds: “And cheap.”

Nobody disagrees, so the meeting feels like a success. Now watch what happens to “fast” when two people check it. Amira measures the small chat model on Grace’s 67 questions. The median answer takes 1.52 seconds, so half the answers are faster than that. “Fast,” says Amira, “done.” Grace’s agents still complain. Amira looked at the middle of the list; the agents remember the slow end. One answer in twenty took 3.79 seconds or more for the model alone: that is the 95th percentile, or p95.

Tomás has another idea: “Use the strong model for everything. Better answers.” Its median is 2.49 seconds, which still sounds fast. But its p95 is 6.56 seconds, for the model alone. Is that fast enough? Nobody can say, because “fast” has no number. The same is true of “always on”, “private” and “cheap”. Each word hides a target, and each person fills in their own.

The idea: quality attributes and SLOs

A quality attribute is a property of the whole system: how fast, how available, how private, how easy to change, how cheap. The last lesson’s functional requirements say what the system does. Quality attributes say how well it does it. They usually decide the architecture more than the features do. Answering in 6 seconds and answering in 0.6 seconds need very different designs.

To make a quality attribute checkable, you give it a target. A service level objective, or SLO, is a target for one measured number, over a fixed period. The name says it: an objective (a target) for the level of service that people get. The pizza shop’s promise is an SLO. A complete one answers five questions:

1. Which number? One measured number, such as “the time of one answer”. For times, name the percentile: p95, not the average.
2. What target? “6 s or less”.
3. Over what period? “Every week”. Without a period, one slow answer breaks the target forever.
4. How measured? Where the number comes from: “the API logs the time of each answer”.
5. Who agreed, and is it measured or assumed? A target that the users did not agree to is only the engineer’s wish.

From a word to an SLO in five questions. Each question removes one way that two people could read the word differently.

Text description of the diagram

The chain starts with Grace’s word: “It must be fast”. The first question asks which number: the time of one answer, at the 95th percentile. The second asks for the target: 6 seconds or less. The third asks for the period: every week. The fourth asks how it is measured: the API logs the time of each answer. The fifth asks who agreed and whether the number is measured or assumed. The result, in a box with a thick border, is SLO Q1: 95% of answers in 6 seconds or less, every week. The phone version shows the same chain without the fifth question.

Why p95 and not “every answer”? Because a few answers are always slow, for reasons outside your control: a busy provider, a slow network. A target of 100% would be broken by the first slow answer. With p95, you allow one answer in twenty to be slower, and you watch that the twentieth does not get worse. Why p95 and not the average? Because, as Amira found, the middle of the list hides the slow end that people remember.

A tiny example: what does the target allow?

Every target that is less than 100% allows some failures. That allowance is called the error budget. It turns a percentage into a count that people can picture.

Take a small day first. The assistant gets 2,000 questions. The availability target is 99.5%: the assistant must answer 99.5% of questions. Then:

• Answered at least: 2,000 × 0.995 = 1,990 questions.
• Allowed failures: 2,000 − 1,990 = 10 questions.

Ten unanswered questions in a day sounds small. Is it too many? Ask Grace: “Ten times a day, an agent sees an error and answers from memory. Is that acceptable?” Now the target is a conversation about people, not a percentage.

The next lesson estimates about 44,000 questions a month for both shops, in the base case. The chart shows the error budget of four targets at that volume:

The error budget at about 44,000 questions a month (1,465 a day × 30, the base case of the next lesson). Each extra 9 cuts the allowance by ten times. Q2's 99.5% allows about 220 failed answers a month.

Text description of the chart

A column chart with four columns: the number of failed answers that a target allows in a month of 43,950 questions. 99% allows 440. 99.5%, the target that Omar chose, allows 220. 99.9% allows 44. 99.99% allows 4.4. Each step adds one 9 and divides the allowance by about ten.

Each extra 9 divides the allowance by ten, and each one costs more to keep. At 99.99%, the assistant may fail four questions in a whole month. A single restart of the database server could use that up. Module 5 prices what a higher target needs, such as a standby database server. For now, the rule is: choose the lowest target that the users accept, because every 9 has a price.

A first estimate for Q1

Grace accepts “95% of answers in 6 seconds or less, every week”. Can the design meet it? The assistant does not exist yet as one system, but its steps were measured. Answering one question takes three steps:

• Embed the question: the embedding model turns it into a vector. Measured: p95 0.48 s, over 67 questions.
• Rerank the search results: a small model orders 20 candidate passages. Measured in the RAG course, on a laptop’s CPU: 0.32 s (median; no p95 was recorded).
• Answer: the small chat model writes the answer with citations. Measured: p95 3.79 s.

Kwame adds the three: 0.48 + 0.32 + 3.79 = 4.59 seconds, about 4.6.

The first estimate for Q1: the p95 of each measured step, added up (the rerank is a median). The sum, 4.59 s, is below the 6 s target. Adding p95s gives an upper estimate.

Text description of the chart

A column chart in seconds. Three steps: embed the question 0.48, rerank 0.32, answer with chat-small 3.79. A fourth column, in another colour, is their sum: 4.59. A fifth column, in a third colour, is the target: 6.00. The answer step is by far the tallest of the three steps. The sum is below the target, with about 1.4 seconds to spare.

Why is this an upper estimate? Because adding p95s assumes that the slowest embedding, the slowest rerank and the slowest answer all happen to the same question. That is rare, so the real p95 of the whole request is usually lower. An upper estimate is still useful: if even the pessimistic sum meets the target, the design starts with room to spare. The chart also shows where the time goes: the answer step is more than 80% of the sum. If Q1 is ever at risk, the model step is the place to look.

Now Tomás’s idea again. With the strong model, the sum is 0.48 + 0.32 + 6.56 = 7.36 seconds, above the target. A number now answers Tomás, not an opinion. Module 4 measures whether the strong model’s answers are even better.

Larkfield’s quality attributes

Kwame does the same work for every word from the meeting. Here is the result, the requirement-to-measurement table. Read each row from left to right. It gives the attribute and the word somebody used, the target, how it is measured, and the kind of number behind the target.

ID	Attribute (the word)	Target (SLO)	How we measure	Kind
Q1	Latency (“fast”)	95% of answers in 6 s or less, every week	the API logs the time of each answer; p95 per week	target agreed with Grace; first estimate 4.6 s (the p95 of each measured step added: an upper estimate)
Q2	Availability (“always on”)	99.5% of questions answered in shop hours, each month	the share of questions with a successful answer, from the logs	target agreed with Omar
Q3	Privacy (“private”)	no tenant ever sees another tenant’s document or answer	the access tests on every change; a cross-tenant check in the evaluation set	verified by tests, as in the authentication course
Q4	Answer quality (“good answers”)	at least 85% correct on the shop’s evaluation set; 0 answers without a citation	the evaluation set (67 questions for Larkfield) before every model or prompt change	first measurement: 86.6% (measured)
Q5	Auditability (“we must be able to prove it”)	every answer kept 1 year, with question, citations, document versions, tenant and user	a weekly query: answers without these fields = 0	target from Ines’s supplier contracts
Q6	Maintainability (“easy to change”)	one developer deploys a fix in one working day; a new tenant without a code change	the time from merge to production; the tenant checklist	target agreed with Amira
Q7	Recovery (“don’t lose our data”)	back within 4 hours; lose at most 1 hour of new data	a restore test every quarter	target agreed with Omar and Mei
Q8	Budget (“cheap”)	at most US$150 a month in the first year, all tenants together	the cost model now; the cloud bill every month	Omar’s budget

This table is the module’s main picture. Each row connects a person’s word to a number that someone will measure. When Module 3 computes the cost, it is checked against Q8. When Module 4 compares models, it is checked against Q1 and Q4. When Module 5 plans backups, it is checked against Q7.

A few rows need a closer look.

• Q3 has no percentage. Privacy between tenants is not “99.5% private”. One leak is a failure. So the target is absolute, and it is measured by tests that must all pass, on every change.
• Q4 has two parts. Correct answers can be counted, as Priya does with the 67 questions: the small model got 58 right in both passes, 86.6%. “0 answers without a citation” is absolute, like Q3. The 85% target leaves a small margin below today’s 86.6%. A change that drops below it does not ship.
• Q7 uses two numbers that Module 5 explains in full. The recovery time objective (RTO) is how long the assistant may be down after a disaster: 4 hours. The recovery point objective (RPO) is how much recent data you may lose: the last hour.
• Q8 is a quality attribute too. Cost is a property of the whole system, like speed. It has a target, a period and a way to measure it.

The Kind column is the honest part. Only Q4’s first value and Q1’s estimate rest on measurements so far. The other targets are agreements with people, and Q3 is verified by tests. None of them is a guess about demand; those come in the next lesson.

Guided practice: fill in the quality attributes

1. Open requirements.md in your pack. Find the section Quality attributes. It has a one-row table with the columns ID, Attribute, Target (SLO), How we measure and Kind.
2. Replace the example row with Q1. Use the five questions: which number, target, period, how measured, who agreed.
3. Add Q2 to Q8, one row each. The pack’s file does not keep the words in brackets: write only the attribute.
4. For each target with a percentage, check that it names a period: a week or a month.
5. For each time, check that it names a percentile, not an average.
6. In the Kind column, write “measured” only where a run produced the number. Write who agreed for the others.
7. Save the file. Compare it with requirements.md in the labs folder C26/snapshots/end-of-m01.

Leave the section Verified and assumed for the next lesson. It needs the demand numbers.

When to set quality targets

Set them before the architecture, together with the journeys. The architecture serves them: a 6-second target allows a simple design, and a 0.6-second target does not. Set each target with the person who feels it: Grace for latency, Ines for privacy, Omar for the budget.

Do not set a target for something nobody will measure. A target without a “how we measure” is decoration. And do not set targets higher than the users need. A 99.99% target sounds better than 99.5%. But it allows 4 failures a month instead of 220, and it needs a much more expensive design.

Targets have a cost: somebody must collect the numbers. The latency and availability targets need logs; privacy needs tests; recovery needs a restore test every quarter. Kwame lists each measurement in the requirements, so that the cost of measuring is visible too.

Common mistakes

Common mistake

A target without a period or a percentile

Symptom: the requirements say “answers in 6 seconds”. In the first week, one answer takes 9 seconds because the provider was busy. Is the target broken? Grace says yes; Amira says it was one answer.

Cause: “answers in 6 seconds” does not say which answers or over what period. Read strictly, it means every answer, forever, and the first slow answer breaks it.

Fix: name the percentile and the period: “95% of answers in 6 s or less, every week”. Now the 9-second answer is one of the 5% that the target allows, and the week is judged on its p95.

Common mistake

A target that nobody measures

Symptom: Q5 says “every answer kept 1 year”. Six months later, an audit finds answers without their document version. Nobody noticed, because nobody looked.

Cause: the target had no “how we measure”. A target that nobody checks drifts away without a sound.

Fix: add the measurement and its owner: “a weekly query: answers without these fields = 0”. Then a missing field shows up in a week, not in an audit.

Your turn: Bramble Books and the strong model

Change one condition. Ines says: “My customers ask about long supplier terms. Use the strong model for Bramble Books.” Bramble Books expects far fewer questions than Larkfield. Before you calculate, predict: does Q1 still hold for Bramble Books with the strong model? Use the measured p95s: embed 0.48 s, rerank 0.32 s, strong model 6.56 s. Then say which other target you must check, and why.

Show the answer

No. The first estimate is 0.48 + 0.32 + 6.56 = 7.36 seconds, above the 6-second target. The estimate is an upper one, so the real p95 may be a little lower. But the strong model alone is above 6 seconds at its p95, so Q1 is at risk even before search. Kwame cannot promise Q1 for Bramble Books with this model.

The other target to check is Q4, answer quality: Ines wants better answers, so measure them. On Grace’s 67 questions, the strong model was not better: 85.8% correct against 86.6% for the small model. Module 4 looks at why. Q8, the budget, also matters: the strong model cost US$0.003197 per question in the same run, about 19 times the small model. For a small shop that is affordable, but it must be in the cost model. One request changes three targets, and each one has a number.

Knowledge check

Check your understanding

Rewrite 'the assistant must be always on' as a complete SLO. Then say how many failed answers your target allows in a month of 44,000 questions.

Show the answer

For example: “The assistant answers 99.5% of questions in shop hours, each month. We measure the share of questions with a successful answer in the logs. Target agreed with Omar.” It names the number (the share of questions answered), the target (99.5%), the period (a month), how it is measured (the logs) and who agreed. At 44,000 questions, it allows 44,000 × 0.005 = 220 failed answers a month. A 99.9% target would allow 44.

Check your understanding

Predict and explain: the median time of an answer stays at 1.52 s, but the p95 grows from 3.79 s to 5.9 s. Is Q1 still met? What does the change tell you?

Show the answer

Q1 is about the whole request’s p95, so recompute the first estimate: 0.48 + 0.32 + 5.9 = 6.7 seconds, above 6. The estimate is an upper one, so the real p95 could still be just under the target, but the margin is gone. Measure the whole request’s p95 from the logs before you decide. The change tells you that the slow end got slower while the middle did not move. For example, the provider may be busy at some hours. The median would hide it. That is why Q1 names the p95.

Check your understanding

Practical check: your requirements.md now has a Quality attributes table. Which targets in it are absolute (no percentage), and why are they absolute?

Show the answer

Q3, privacy: “no tenant ever sees another tenant’s document or answer”. One leak is a failure, so the access tests must all pass on every change. Q4’s second part: “0 answers without a citation”, because an answer without a source cannot be checked. Q5 is measured as “answers without these fields = 0” each week. Your table has eight rows, Q1 to Q8, and each one has a target, a way to measure it and its kind.

Recap

• “Fast”, “always on”, “private” and “cheap” hide targets that each person fills in differently. The median of chat-small is 1.52 s, but its p95 is 3.79 s, and people remember the slow end.
• A quality attribute says how well the system works. An SLO is a target for one measured number over a period: which number, what target, what period, how measured, who agreed.
• An availability target allows an error budget: 99.5% of about 44,000 questions a month allows 220 failures. Each extra 9 divides it by ten and costs more.
• Q1’s first estimate adds the p95 of each step: 0.48 + 0.32 + 3.79 = 4.59 s, an upper estimate below the 6-second target. With the strong model it is 7.36 s.
• The requirement-to-measurement table connects Q1 to Q8, each with a target and a way to measure it.

Next, you estimate how many people will use the assistant, and how much, as ranges with their evidence.

Next: Demand assumptions