Failure scenarios

What you will be able to do

Find out what can go wrong in the design before anyone builds it. You walk through six failures of the shared assistant on paper. For each one, you write what users see, how the team notices, what limits the damage, and who acts. Then you find the part that everything depends on, and you choose a mitigation that fits its size.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The eight parts of the design and their owners, from Service boundaries: Larkfield’s parts.
• The quality attributes Q2 (availability), Q7 (recovery) and Q8 (budget), from Quality attributes.
• The workload of the busiest minute, from Workload arithmetic, and the answer cache’s assumed 10% of hits, from Caching and batching.
• How a circuit breaker works, from Failure containment in Scaling APIs and AI Workloads. It helps, but it is not required.

What you need on your computer

• Your folder design-pack with its active virtual environment, and python -m pytest -q printing 42 passed: the snapshot end-of-m04.
• The labs folder nextia-learning-labs next to it.

This lesson adds one document to the pack and no code. Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A road trip with a plan

A family plans a long drive. Before they leave, they ask a few “what if” questions. What if a tyre goes flat? They carry a spare, and the father knows how to change it. What if a road is closed? The phone has a map, and the mother checks it every hour. What if the car keys are lost? There is only one set. Without the keys, nothing else matters.

For the keys, the family has two choices. They can bring a second car, which is very expensive. Or they can give a spare key to the grandmother, who can bring it in two hours. The spare key costs a few dollars and accepts a two-hour delay. For a holiday, that is enough.

Each “what if” has four parts: what happens, how you notice, what you do about it, and who does it. The keys are a part that everything depends on. The family chose a cheap mitigation that fits the risk, not the most complete one.

The problem: “nothing will fail”

Kwame shows the design to the team. Tomás is relaxed. “The model provider is a big company. The database is a managed service. Nothing will fail, so we do not need a plan.”

The earlier courses already saw failures of this kind.

A provider outage, measured. In the scaling course, the model provider failed for 20 seconds. Thirty jobs were in the queue, and each job tried five times. The provider got 30 × 5 = 150 calls, and all of them failed. When the provider came back, all 30 jobs were already dead. With a circuit breaker, the same outage cost only 8 calls, and all 30 jobs succeeded afterwards.

A permission mistake, constructed. Suppose that Amira writes the answer cache of Module 4 with a key of only the question’s text, without the tenant. A Larkfield agent asks “How long do refunds take?”, and the answer is saved. Then a Bramble Books customer asks the same question. The cache returns Larkfield’s answer, with Larkfield’s refund policy and citations. No error appears anywhere, and a customer reads another shop’s private document. (This is a constructed case to show the risk. The design’s cache key includes the tenant.)

Neither failure is rare. Both happen in normal operation, and both are cheap to prevent once someone writes them down.

The idea: failure scenarios and a tabletop review

A failure scenario is a short story of one thing that goes wrong. For each scenario, the design states five things:

• What users see. “Answers fail”, “every answer is slow”, “a document is missing”.
• The detection signal. A measurement and a line, for example “the share of errors from the provider over 5 minutes is above 5%”. A customer’s complaint is not a signal: it arrives too late.
• The mitigation. What limits the damage: a timeout, a breaker, a queue, a transaction, a test.
• The owner. One person who acts. “The team” is not an owner.
• The recovery. How the system gets back to normal, and whether data is lost.

Engineers call the list of failures the failure modes of a system. A tabletop review is how you find and check them. The team sits around a table with the design and walks through one failure, minute by minute. “The provider answers 503. What happens next? Who notices?” Emergency planners use the same name for their drills on paper. Nothing is broken, and nothing costs money, but the gaps become visible.

Two more names. A single point of failure is a part with no backup: if it fails, the whole system stops. The car keys were one. A proportionate mitigation fits the risk and the budget, like the spare key instead of a second car.

A tiny example: five minutes without the provider

Take the first failure and walk through it with numbers. The provider fails for 5 minutes, at the busiest minute of a base day.

A tabletop walk-through of a provider outage, the failure F1 of the table below. Each box answers 'what happens next?'. The orange box is what users see.

Text description of the diagram

Minute 0: the model provider answers 503 to every call. After 5 failed calls in a row, the circuit breaker opens: for 30 seconds, the API makes no calls to the provider. Users see “try again in a minute”, and new uploads wait in the queue. At minute 5, the share of 5xx answers over 5 minutes is above 5%, so the alert calls Amira. She checks the provider’s status page and tells Grace, so the help desk knows. When the provider is back, one trial call succeeds and the breaker closes. The worker then processes the queued uploads, and no data is lost.

Now count what the users lose. All values come from the design pack:

1. The busiest minute of the base case has 0.268 questions per second: 0.268 × 60 = 16.1 questions a minute.
2. The answer cache answers 10% of them without the provider (the assumption of Module 4). So 16.1 × 0.9 = 14.5 questions a minute need the provider.
3. Five minutes of outage: 14.5 × 5 = about 72 questions get “try again in a minute”.
4. Q2 says that 99.5% of the month’s questions get an answer. The base month has 43,950 questions, so 0.5% may fail: 43,950 × 0.005 = 220 questions a month.
5. This outage uses 72 of the 220: 33% of the month’s allowance.

So the design survives one five-minute outage a month. Four such outages in one month would break Q2. The breaker does not save these 72 answers: nothing can, while the provider is down. But it keeps the queued jobs alive and stops useless calls. And the “try again” message tells people what to do.

Larkfield: the failure-mode table

Kwame runs the tabletop review with Amira, Mei and Grace. They take the five failures of the outline and add one: the database itself is lost. The result is a new file in the pack, decisions/failure-modes.md.

Guided practice: add the failure modes to the pack

1. In your folder design-pack, with the environment active, make the folder decisions and copy the file from the snapshot end-of-m05.

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m05
   mkdir decisions
   cp "$S/decisions/failure-modes.md" decisions/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m05"
   New-Item -ItemType Directory decisions
   Copy-Item "$S\decisions\failure-modes.md" decisions\

   The snapshot is end-of-m05, not end-of-m04: this module’s files come from the end of this module.

2. Count the failures in the table. Each row starts with | F:

   macOS / Linux

   grep -c "^| F" decisions/failure-modes.md

   Windows (PowerShell)

   (Select-String -Pattern '^\| F' decisions\failure-modes.md).Count

   6

3. Open decisions/failure-modes.md in your editor. Read one row from left to right: failure, what users see, detection signal, mitigation, owner, recovery. Then check that no cell is empty.

4. Run the tests. The file is a document, so the result does not change:

   python -m pytest -q

   ..........................................                               [100%]
   42 passed in 0.07s

   In Module 6, the pack’s check reads this file. It wants at least five failures, each with an owner, a mitigation and a signal.

Here is the table as a picture of the design. Each failure sits next to the part where it starts:

Where each failure starts, with its owner. PostgreSQL has a thick border: every answer and every job needs it.

Text description of the diagram

The help-desk tool calls the assistant API. The API uses PostgreSQL, which holds the records, the vectors, the job queue and the answer cache. It also calls the model provider. Uploads go to file storage and then to the ingestion worker, which writes to PostgreSQL and calls the model provider too. The failures sit next to their parts. F1, the provider outage or 429, is at the model provider, owned by Kwame. F2, the slow database, and F6, the lost database, are at PostgreSQL, owned by Mei. F3, the queue backlog, and F4, the partial ingestion, are at the worker, owned by Mei. F5, the permission mistake, is at the API, owned by Kwame. PostgreSQL has a thick border, because every answer and every job needs it.

Read the table, one failure at a time

The table is short on purpose. Each row has a number that a machine can watch, and a person’s name.

F1, provider outage or 429. This is the tiny example above. A 429 means “too many requests”: the provider’s quota is 100 calls and 100,000 tokens a minute. In the base case, the busiest minute uses about 14 calls, so a 429 is unlikely today. In the high case, the busiest minute already needs more than the quota; Growth plan plans for that. Owner: Kwame, because he owns the relationship with the model provider, as in the component table of Module 2: he talks to the provider and asks for a higher quota. Amira runs the fallback: she owns the API code with the timeout and the breaker, so the alert calls her.

F2, slow database. Users see that every answer is slow. The signal is the p95 of the search query above 100 ms. The databases course measured an HNSW search at 0.22 ms over 10,000 vectors. So 100 ms means that something is wrong, not just busy. A connection pool with a timeout stops one slow query from blocking every request. In a managed service, a bigger server is a setting. Owner: Mei.

F3, queue backlog. Users see that a new document is not searchable. The signal is the age of the oldest queued job above 15 minutes. In the base case, the worker is busy for about 25 seconds a month. So a backlog almost never means “too much work”; it means “the worker is stuck”. The upload answer says “processing”. Each tenant has a limit of queued jobs, so one shop cannot fill the queue for all. Owner: Mei.

F4, partial ingestion. The worker stops halfway through a document. Without care, some chunks are new and some are old. The mitigation is one transaction for the chunks, the vectors and the switch to the new version: either everything changes, or nothing does. The job’s lease then returns it to the queue, as in the scaling course. Owner: Mei.

F5, permission mistake. Users see a wrong, private answer, as in the constructed cache case above. Row-level security in the database is the second wall. It filters every query by the tenant, even when the code forgets. The authentication course measured a request at 4.3 ms without it and 4.4 ms with it. The release stops when an access test fails. Owner: Kwame, because a leak is a business problem as much as a code problem. Its recovery is the slowest: contain the leak, tell the tenant, and find every affected answer in the audit log.

F6, database lost or corrupted. The assistant stops completely. This row needs its own section.

The single point of failure

Look at the picture again. Every answer, every job, every cache hit and every permission check goes through PostgreSQL. If it is lost, nothing works. So PostgreSQL is the design’s single point of failure. The model provider is a second one for answers, but the breaker and the “try again” message already limit its damage.

The complete mitigation is high availability: a second database server waits as a standby and takes over when the first one fails. Tomás proposes it at once. Before you agree, look at the price. The provider’s documentation says two things. High availability needs a General Purpose server, not the small Burstable B1ms of the design. And it bills a standby server of the same size.

The database server's compute per month at list prices (Canada Central, 730 hours). High availability needs a General Purpose server and a same-size standby: about 21 times the design's server. The standby's storage would add more; it is not counted here.

Text description of the chart

A column chart with three columns: B1ms (now), one D2ds and two D2ds, short for D2ds_v5. The design’s server, one Burstable B1ms, costs US$13.50 a month. One General Purpose D2ds_v5 costs US$144.25. High availability needs two D2ds_v5: a server and its same-size standby. They cost US$288.50. Prices change: check the current page.

The numbers, every value visible:

• The design: one B1ms, 0.0185 × 730 hours = US$13.50 a month.
• High availability: two D2ds_v5, 2 × 0.1976 × 730 = US$288.50 a month, about 21 times more.
• The whole monthly cost would rise to 51.90 − 13.50 + 288.50 = US$326.90. Q8, Omar’s budget, is US$150 a month. High availability alone breaks it.

Now look at what the standby buys. Q2 asks for 99.5% of questions answered in shop hours. Q2 counts questions, but a restore stops the assistant for a stretch of time. As a rough guide, turn the 0.5% into time. The shops are open 12 hours a day, so a 30-day month has 360 shop hours. 0.5% of them is 1.8 hours: about 108 minutes a month. If a restore takes a little more than an hour, one restore a month still meets Q2. (The next lesson estimates that time.)

So the proportionate mitigation at this size is a tested restore, not a second server. It is the spare key, not the second car. Kwame writes it in the table as F6, with a trigger to look again: “Revisit when Q2 rises above 99.9%.” The next lesson sets the restore’s targets and checks whether a plan can meet them.

When to run a tabletop review

Run one before you build, when a change is still cheap. Run it again after every real incident, and when a new part joins the design. For a design of this size, one hour with the owners around a table is enough.

Do not try to list every possible failure. Start with the failures that the outline and the earlier courses showed: the provider, the database, the queue, ingestion and permissions. People read a table of six rows; nobody reads one of sixty.

What it costs. The review costs the owners’ time. Most mitigations here cost little: a timeout, a breaker, a transaction, a test. The expensive ones, such as a standby server, need a price and a target that justifies them.

Common mistakes

Common mistake

The customer is the detection signal

Symptom: a document has been missing from answers for two days. Nobody knew until Ines wrote an angry email.

Cause: the failure-mode row said “users report it” as its detection signal. Users report late, or never.

Fix: write a signal that a machine watches, with a line: “the age of the oldest queued job is above 15 minutes”. Name the person whom the alert calls.

Common mistake

'The team' owns the failure

Symptom: during an outage, Amira and Mei each think that the other one is handling it. Thirty minutes pass.

Cause: the owner column says “the team” or “engineering”.

Fix: write one name per row, and a second person for holidays. The owner does not do all the work. The owner makes sure that someone does it.

Common mistake

The file is not in the snapshot

Symptom: cp says No such file or directory for failure-modes.md.

Cause: S points to the snapshot end-of-m04, which has no folder decisions.

Fix: set S to ../nextia-learning-labs/C26/snapshots/end-of-m05 and copy again.

Reset. If your pack breaks, copy the snapshot end-of-m04 as the snapshots README shows, then repeat the steps above.

Your turn: a seventh failure

The table has no row for file storage. Each tenant’s original files are there, and the ingestion worker reads them. Write the row F7, “a file deleted by mistake, or file storage lost”, with all six columns. Keep your row in your notes, and leave the pack’s file as it is, so it stays equal to the snapshot.

Show a worked solution

ID	Failure	What users see	Detection signal	Mitigation	Owner	Recovery
F7	A file deleted by mistake, or file storage lost	answers still work (the chunks and vectors are in PostgreSQL), but a new ingestion or a download of the original fails	a weekly scan: every current document version has its file (the “missing files” check of the databases course’s scan)	soft delete and versions on the file storage; the database restore does not bring files back	Mei	restore the file’s earlier version; if none exists, ask the tenant to upload it again

The important part is in the mitigation. A database restore brings back rows, not files. The databases course kept files in object storage for this reason, and its lesson Recovery and retention checked them separately.

Knowledge check

Check your understanding

Name the five things that a failure scenario states, and explain why a customer's complaint is a poor detection signal.

Show the answer

A failure scenario states what users see, the detection signal, the mitigation, the owner and the recovery. A customer’s complaint arrives late, or never: many users just stop using the assistant. A good signal is a measurement with a line that a machine watches. Examples: “the 5xx share over 5 minutes is above 5%”, or “the oldest queued job is older than 15 minutes”. The alert calls a named person.

Check your understanding

Predict: the provider is down for 10 minutes at the busiest minute of a base day. How many questions fail, and how much of the month's Q2 allowance do they use?

Show the answer

About 14.5 questions a minute need the provider: 16.1 questions, minus the 10% that the cache answers. Ten minutes: 14.5 × 10 = 145 failed questions. The month allows 43,950 × 0.5% = 220 failures, so this outage uses 145 ÷ 220 = 66% of the allowance. A second outage of the same size in the same month would break Q2.

Check your understanding

Tomás wants high availability for the database 'because it is a single point of failure'. Give one fact for his proposal and two numbers against it, and say what you propose instead.

Show the answer

For: PostgreSQL is a single point of failure, because every answer and every job needs it. Against: two D2ds_v5 servers cost about US$288.50 a month of compute. The design’s B1ms costs US$13.50, about 21 times less. And the monthly total would rise from US$51.90 to about US$326.90, above Omar’s budget of US$150. Q2 allows about 108 minutes of downtime in shop hours a month, so one restore of about an hour fits. Propose a tested point-in-time restore now, and revisit high availability when Q2 rises above 99.9%.

Recap

• A failure scenario states what users see, a detection signal with a line, a mitigation, one owner and the recovery. A tabletop review walks through it on paper, minute by minute.
• The pack’s decisions/failure-modes.md has six: provider outage or 429 (Kwame; Amira runs the fallback); slow database, queue backlog, partial ingestion and lost database (Mei); permission mistake (Kwame).
• A five-minute provider outage at the busiest minute fails about 72 questions: 33% of the month’s Q2 allowance of 220. The breaker cannot save them, but it saves the queued jobs.
• PostgreSQL is the single point of failure. High availability costs about US$288.50 a month of compute against US$13.50, and it would break the budget. The proportionate mitigation is a tested restore.
• python -m pytest -q still prints 42 passed: this lesson added a document, not code.

Next: Recovery objectives