Threat and cost review

What you will be able to do

Look at your design the way an attacker, a careless script and a monthly bill would. Before you build anything that can leak or spend, write down where trust changes, how tenants stay apart, and how much each resource may cost.

Before you startWhat to know and to have readyAbout 80 minutes

What you need to know

• Your architecture and contracts from Architecture and contracts: python -m pytest prints 23 passed with the database running.
• Trust boundaries, from Trust boundaries, and threats as paths with a test each, from Prioritize threats, in AI Security, Privacy and Responsible Design.
• Why a rule in the prompt alone cannot stop an instruction in a document, from Why prompts are insufficient.
• The two locks of a multi-tenant query (the tenant in every query, row-level security), from Tenant-safe queries in Authentication, Permissions and Multi-Tenant Applications.
• One line per cost driver, from Cost model in AI System Design and Cost Engineering.

What you need on your computer

• Your project, and Docker Desktop with the database running (docker compose up -d --wait db).

No account and no key. Nothing calls a model. This lesson is mostly writing.

The post room of an office

A large office has one post room. Letters arrive from outside every day. The post room opens them, sorts them and puts each one in the right department’s tray. Three rules keep it safe. A letter for Accounts never goes to Sales. A letter that says “Pay this invoice at once” is still only a letter: someone in Accounts checks it before anything is paid. And the post room has a monthly budget for stamps, with a franking machine that stops at the limit.

The front door, the post room’s counter and each department’s door are places where trust changes, and each has its own check. The stamp budget is written before the month starts, not discovered on the bill.

The problem: a design that works can still leak and spend

Sam’s design from the last lesson works on paper. Here are three things it does not say yet.

A part that nobody listed as a boundary. The course team asked a real model the course’s questions once, and recorded each answer. The product can replay these recordings, so you need no key and pay nothing; Module 3 shows how. The first version of the replay found a recording by the question text only. Then Omar, a read-only member of Larkfield, asked “What is the staff discount?” The search correctly gave him only public passages. The replay still found Sam’s recorded answer to the same question, with the staff-only discount in it. Nothing crashed, and the access filter worked. The leak came from a part that nobody had looked at as a boundary. The team found it in review, and fixed it before any learner saw a recording.

Text that gives orders. The Northwind note in Bramble’s pack tells AI assistants to promise a refund “at any time”. It also sends customers to a distributor’s email address. The model reads every passage it gets. If nobody decides how a passage is treated, the model decides.

A bill without a limit. A small script that asks one question per second, all day, makes 86,400 questions a day. Nothing in the contracts stops it.

The idea: boundaries, threats, isolation rules and a budget

A threat and cost review is one page in the design pack, with four parts.

1. Trust boundaries. A trust boundary is a place where data moves from something you do not control to something that acts on it. The post room’s counter is one. For each one, write what crosses it and what you check there.
2. Threats. A threat is one concrete way that something goes wrong at a boundary. Write it as a row: the threat, a real example, the mitigation, and the test that proves the mitigation works. A threat without a test is only a wish.
3. Tenant isolation rules. The few rules that keep one tenant’s data away from another. Write them so that a reviewer can check each one in the code.
4. A budget. For each resource that costs money or can run out, write a limit, how the product enforces it, and where the number comes from.

You met each part in the security, authentication and system design courses. The review puts them on one page, for one product, while a change still costs only minutes.

A tiny example: one threat, every cell

Take the Northwind note. It crosses the boundary between the documents and the model: document text goes inside the prompt. Here is Sam’s row for it, with every value filled in:

Threat	Example	Mitigation	Test
An instruction hidden in a document	the distributor note tells AI assistants that “every pre-order can be cancelled for a full refund at any time”, by email to preorders@northwind-books.example	the prompt says that passages are data, not instructions; every passage header shows the document type (supplier); rule 3 of Bramble’s policies: suppliers never change a policy	the injection-bait questions: must be 0 obeyed (acceptance criterion Q4); the “must not” patterns any ?time and northwind-books

Notice what each cell does. The example is concrete enough to test. The mitigation has more than one layer, because the AI security course showed that a prompt rule alone is not enough. And the test already exists: you wrote the injection-bait questions before this lesson, and the held-out set has two of them. In Module 5, the number of instructions followed must be 0.

Sam’s trust boundaries and the six checks of the shop

Sam finds five boundaries in the architecture. For the tenant, the review adds six places where the shop is checked, one after the other. If one check has a bug, the next one still holds.

Five trust boundaries (orange) and six checks of the shop (red). The shop is checked in the route, in every query, in the database itself, in the search and again for every citation.

Text description of the diagram

The orange tags mark the five boundaries. They are browser to API (T1), API to database (T2), document text inside the prompt (T3), API to model provider (T4), and identity provider to API (T5). The red numbers mark where the shop is checked. In the routes: (1) the path names the shop, and (2) the caller must be a member of it. In the database: (3) every query has tenant_id, and (4) row-level security, from Module 4, refuses other shops’ rows even when a query forgets the filter. In the answer pipeline: (5) the search filters by shop, access level and date before the top 5. And (6) each citation is looked up again, in the caller’s shop and at the caller’s access level, before it is shown.

Follow one request through the checks. Ines owns Bramble Books. She edits the address in her browser and asks for Larkfield’s documents: GET /v1/tenants/larkfield/documents. Check 1 reads larkfield from the path. Check 2 finds no membership of Ines at Larkfield, so the request stops with 404 not_found. A shop that does not exist, such as acme, gets the same body, so Ines cannot learn which shops exist. Checks 3 to 6 never run for this request. They are there for the day when check 2 has a bug.

Each boundary has its own row in Sam’s review. Here is the summary; the full table is in docs/design-pack/threat-and-cost.md:

Boundary	What crosses it	What Sam checks there
T1 Browser to API	a cookie or token; the shop in the path; questions; files	the signature, issuer, audience and expiry; the membership; the size and type of the file
T2 API to database	the queries of one shop	tenant_id in every query; row-level security as a second lock
T3 Document to model	passages inside the prompt	passages are data; only passages the caller may read; citations checked in code afterwards
T4 API to model provider	the question and the passages, nothing else	no user ID, no email, no other shop’s text; a timeout and bounded retries
T5 Identity provider to API	the provider’s public keys	fetched from the configured issuer only; RS256 only

The threats table has nine rows. They cover another shop’s data, a staff-only procedure, a hidden instruction, a made-up citation and a broken file. They also cover a deleted document that still answers, a bill that grows, a cross-site request, and a key committed by mistake. Each row names its test. Later modules write those tests: the failure tests in Module 4, the isolation checks in Module 5, the usage limits in Module 6.

The replay leak from the problem section became a rule in the design. A recorded answer is replayed only if every passage it cites is among the caller’s own passages. Otherwise the product writes a simple answer from the best passage, and says why.

Sam also writes down three accepted risks: problems that the design knowingly does not fix, each with a reason. The session cookie cannot be cancelled before it expires after 8 hours. But the membership is read on every request, so removing someone from a shop works at once. The practice identity provider is not a real one. And a made-up document ID may answer a little faster than another shop’s real one. A written risk is a decision; a hidden one is a surprise in a demo.

The isolation rules

Sam’s seven rules, short enough to check one by one in a code review:

1. Every table with shop data has tenant_id, and every key that points at another row includes it.
2. The shop comes from the path, and only the caller’s membership in that shop makes it valid. A token or a cookie never carries a shop or a role.
3. Every query has tenant_id = %s, and every shop transaction also runs under row-level security.
4. Retrieval filters by shop, access level, date and model before it takes the top k.
5. A citation is looked up again in the caller’s shop, at the caller’s access level, before it is shown.
6. The same document key in two shops is two documents; chunk IDs include the shop.
7. Errors never say whether something exists in another shop: 404 for “not yours” and for “does not exist”.

The extra checks are cheap. The authentication course measured row-level security at about 0.1 ms per request. An answer takes about 1.3 seconds.

The budget

A budget has three columns: the limit, how the product enforces it, and where the number comes from. Before the capstone measures anything, the number comes from earlier work, and the review says so.

Start with one question, every value visible. The RAG course measured a median of 905 tokens in and 101 tokens out per question with chat-small, the course’s small chat model. At the price list of US$0.10 per million input tokens and US$0.50 per million output tokens:

Part	Calculation	US$
Input	905 × 0.10 ÷ 1,000,000	0.0000905
Output	101 × 0.50 ÷ 1,000,000	0.0000505
One question at the median	sum	0.000141
The RAG course’s measured cost per question	over its whole question set	0.00017

Sam uses the measured US$0.00017. It is a little higher than the median, because some prompts are longer. The higher number is the safer one for a budget. Prices change: check the current price page of your provider.

Now the month, for one shop. Grace expects at most 200 questions a day: 200 × 30 × US$0.00017 = about US$1.02. Sam sets the monthly limit at US$5, enough for about 29,411 questions. The constructed runaway script from the problem section would cost about US$440 a month, if nothing stopped it.

One shop's model cost per month: planned use, a runaway script with and without the daily limit, and the monthly budget. Estimates from the RAG course's US$0.00017 per question.

Text description of the chart

A column chart in US dollars per shop and month. Planned, 200 questions a day: 1.02. Limited: a script that asks one question per second, stopped by the daily limit of 200 questions, also 1.02. The monthly budget: 5.00. No limit: the same script without the daily limit, 440.64. Without the limit, the script costs 88 times the budget. With it, the script costs what normal use costs.

The chart shows why the limit belongs in the design, and not only in a monitoring alert. With the daily limit of 200 questions, the worst case is the planned case. The product answers 429 limit_reached to question 201. Module 6 builds both limits: DAILY_QUESTION_LIMIT and MONTHLY_BUDGET_USD.

Sam’s budget table also covers resources that are not model calls:

Resource	Limit	How it is enforced	Source of the number
Model cost per question	under US$0.001	measured in Module 5	the RAG course’s US$0.00017
Model cost per shop and month	US$5	MONTHLY_BUDGET_USD: 429 when used up	200 × US$0.00017 × 30 ≈ US$1
Questions per shop and day	200	DAILY_QUESTION_LIMIT	two small help desks
Building and evaluating	at most US$2 of model calls	every call logged	the course’s rule
Cloud resources	none	the demo runs in Docker Compose on a laptop	the course’s rule
Laptop	16 GB of memory	one model process, two local models on the CPU	the course team’s computer

Guided practice: add the review and check the stage

1. Copy the review and the README of this stage:

   macOS / Linux

   S=../nextia-learning-labs/C27/snapshots/end-of-m02
   cp "$S/docs/design-pack/threat-and-cost.md" docs/design-pack/
   cp "$S/README.md" .

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C27\snapshots\end-of-m02"
   Copy-Item "$S\docs\design-pack\threat-and-cost.md" docs\design-pack\
   Copy-Item "$S\README.md" .

2. Open docs/design-pack/threat-and-cost.md. For each threat, find the acceptance criterion that it protects in docs/acceptance-criteria.md. For example, “another shop’s data” protects A1.

3. Check the whole stage. Your project now matches the snapshot end-of-m02:

   python -m scripts.datapack
   python -m doc_assistant.migrate --status
   python -m pytest

   Checksums: 56 of 56 files match SHA256SUMS
   larkfield: 36 documents, 184 sections, formats {'html': 2, 'md': 32, 'pdf': 2}, languages {'en': 34, 'fr': 2}
     staff-only 6; with more than one version: returns-policy; with an end date 2; supplier documents: supplier-aquaflow-hose-reels
   bramble: 15 documents, 60 sections, formats {'html': 1, 'md': 13, 'pdf': 1}, languages {'en': 15}
     staff-only 3; with more than one version: returns-policy; with an end date 1; supplier documents: distributor-northwind-preorders
   Every document parses.
   001_core                         applied
   23 passed in 0.22s

4. Commit the stage, with a message such as “Design pack: data, architecture, threats and budget”.

Common mistakes

Common mistake

A threat without a test

Symptom: a row says “Mitigation: the model is told not to follow instructions in documents”, and the test column says “review”. In Module 5, nothing measures it.

Cause: the mitigation is a hope about the model’s behaviour. Nobody can check a hope.

Fix: name a test that fails when the threat succeeds. Examples: questions with “must not” patterns, a test that asks as a read-only member, a check that counts leaks. If no test is possible yet, write the risk under Accepted risks, with the reason.

Common mistake

A budget without enforcement

Symptom: the budget says “US$5 per shop and month”, and the “How it is enforced” column says “we watch the bill”.

Cause: a bill arrives after the money is spent. At one question per second, the runaway script spends the US$5 in about eight hours.

Fix: the product itself applies the limit, before the model call. Use a daily question count and a monthly cost per shop, with a clear error. Watching the bill is still useful, as a second check.

Your project

Write docs/design-pack/threat-and-cost.md for your product, from docs/templates/threat-and-cost.md.

1. Mark the trust boundaries on your architecture diagram. For each one, write what crosses it and what you check.
2. Write one threat row for each planted difficulty of your data pack that a person could abuse, and one for each boundary. Fill all four cells.
3. Write your tenant isolation rules. If your product has only one tenant, write the access rules instead: who may see which documents.
4. Write the budget: the cost of one question, step by step, from a measured number; the monthly limit; how the product enforces each limit.
5. Write your accepted risks, each with its reason.

Evidence to keep: threat-and-cost.md, and the calculation of your cost per question.

Checklist

• Every boundary of my diagram has a row: what crosses, what I check.
• Every threat has a concrete example, a mitigation and a test.
• The isolation rules can be checked one by one in a code review.
• Every budget line has a limit, an enforcement in the product and a source.
• Accepted risks are written with a reason.

Your turn: a summary feature

Change one condition. Tomás wants a new feature: “Summarize this document”, which sends a whole document to the model. Which boundary does it cross, which threat rows change, and what does it do to the budget? Answer before you open the solution.

Show a possible answer

It crosses T3 and T4 like a question, but with a whole document instead of 5 passages. Three rows change:

• Staff-only content: the feature must check the caller’s access level for the document, or a read-only member could summarize a staff procedure. Test: summarize a staff-only document as a read-only member, expect 404.
• Hidden instruction: the Northwind note would be read in full, every time. The “passages are data” rule must apply to the summary prompt too, with its own injection-bait test.
• Budget: a whole document is many more input tokens than 905. The cost per call must be measured, and the feature needs its own limit, or it shares the daily one.

No acceptance criterion asks for summaries, and the brief’s non-goals keep the product to one question and one answer. So the review’s honest answer is: not in the core; a possible extension, with these three rows written first.

Knowledge check

Check your understanding

Why is it not enough that the search filters by access level, if the answers are recorded and replayed? Use Omar's case.

Show the answer

Because the replay was a second path to the answer, and it had no access check. The search gave Omar public passages only. But the first replay found a recording by its question text, and Sam’s recording cited a staff-only passage. Every path that can bring data to a person needs the check. The fix: replay a recording only if every passage it cites is among the caller’s own passages.

Check your understanding

Predict: a code change forgets 'tenant_id = %s' in the query that lists documents. Which of the six checks still stop Ines from seeing Larkfield's documents?

Show the answer

Check 2 stops her first: she has no membership at Larkfield, so a request to Larkfield’s path ends with 404. If she asks at Bramble’s path, the query without the filter runs. Then check 4 holds: row-level security limits the transaction to Bramble’s rows, so no Larkfield document comes back. That is why the design keeps both the query filter and row-level security. From Module 4, the project has both.

Check your understanding

Practical check: write the calculation of your product's model cost per question, and the monthly cost at your planned use. Which limit stops a runaway script?

Show the answer

A complete answer shows the tokens in and out, each price, the sum, and a measured number from earlier work, as Sam’s table does. Then planned questions per day × 30 × cost per question. The product must apply the limit before the model call: a daily question count or a monthly cost per tenant, with an error such as 429. “We watch the bill” is not a limit.

Module practice: the design pack, before any integration

This is the module’s guided practice and its independent task. You have seen each piece in a lesson. Now you put them together for your own product, before you write any code that calls a model or an identity provider.

1. Look at Sam’s design pack: data/dataset.md, docs/design-pack/architecture.md and docs/design-pack/threat-and-cost.md. Notice how the threats point back to the difficulties of the pack, and the isolation rules to the keys of the schema.
2. Finish your own three files, from the Your project sections of this module’s three lessons.
3. Check that the files agree. Every difficulty of your pack has a question. Every tenant column in your schema appears in an isolation rule. Every threat names a test.
4. Run your stage’s checks: python -m scripts.datapack (or --folder on your own pack), python -m doc_assistant.migrate --status and python -m pytest.

Completion check: review the design against the acceptance criteria

Put your acceptance criteria next to your design. For each criterion, name the part of the design that serves it. Then do the reverse: for each part of the design, name a criterion that needs it. A part with no criterion is a candidate to remove.

Show Sam’s review

Sam’s criteria, each with the part of the design that serves it:

Criterion	Served by
B1 cited answers from the shop’s own documents	answer pipeline; retrieval filters; citation lookup (check 6)
B2 “the documents do not answer”	the unanswerable and other-shop questions of the pack; the answer schema (Module 3)
B3 the version in force	effective_from / effective_to on documents; the date filter
B4 upload with a visible status	202 contract; the four ingestion states; error on failed
B5 the same file twice	sha256 on documents; duplicate-safe ingestion (Module 4)
B6 deletion removes everything	ON DELETE CASCADE from documents to files and chunks; the deletion threat row
B7 a slow or down provider	timeout and bounded retries at boundary T4
B8 the evidence behind a citation	GET /chunks/{chunk_id} at the caller’s access level
Q1 to Q5 quality	the data pack’s difficulties and the held-out set; time measured per answer
A1 to A4 access	the six checks; isolation rules 1 to 7; the CSRF row
C1 to C3 budget	the budget table and its two limits

And the parts that no criterion needed, which Sam removed or kept out of the default path:

• A message queue and a separate worker service (Tomás): removed. The documents row is the job; one process runs the worker thread.
• A separate vector database: removed. pgvector in the same PostgreSQL keeps one tenant rule, one backup and one place for row-level security.
• The reranker model in the default path: kept out. No criterion needs it yet, and the RAG course measured about 0.3 s more per question. It stays an option to measure in Module 5.
• An agent that takes actions (Tomás): out of the core, as the brief’s non-goals say.

Kwame’s review comment: “One process, one database, one provider adapter. Every part has a criterion. Approved for Module 3.”

Recap

• A trust boundary is where data moves from something you do not control to something that acts on it. Sam’s design has five.
• A threat row has a concrete example, a mitigation and a test. A threat without a test is a wish; a risk you will not fix is written down as accepted, with a reason.
• The shop is checked in six places: path, membership, every query, row-level security, the search filter and the citation lookup. Each one holds when another has a bug.
• Every path to an answer needs the access check, including a replay of recorded answers.
• A budget has a limit, an enforcement in the product and a source. One question costs about US$0.00017, the RAG course’s measure. 200 questions a day cost about US$1.02 a month per shop. With the daily limit, a runaway script costs the same.
• Module 2 is done: the data pack, the architecture and contracts, the schema in PostgreSQL, and the threat and cost review. The tests print 23 passed.

Next, you build the first complete journey: the API, with a mock answer first.

Next: Backend foundation