Design review of an open document-question service

What you will be able to do

Review the design of a free public service that answers questions about a real collection of government documents. Count the collection instead of guessing it. Then size its storage and its loading time. Choose the model and the cache with the course’s measured numbers. Write the decisions and the failure plan. The result is a short review that a non-engineer can act on.

Case studyPortfolio piece

Dataset

OSHA's documents in the US Federal Register, 2021–2025 (628 documents) and January–September 2026 (83)

Licence

Public domain (US government works, 17 U.S.C. 105)

Time

About 3 hours

Runs in

VS Code, Python 3.12+

Skills used

• Counting tokens with a tokenizer
• Sizing storage and ingestion
• Cost per tenant with the calculator
• Routing and caching from measured facts
• Architecture decision records
• Failure table with RTO and RPO

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is the course’s design pack with a new folder, corpus/, which downloads and sizes the documents. The finished project is the result, for when you are stuck. The code is under the MIT licence.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses the whole course on a collection that is not Larkfield’s. These lessons matter most: Set up the design pack (the three kinds of numbers), Workload arithmetic, Unit economics, Model routing, Caching and batching, Recovery objectives and Architecture decisions. You do not need the other case study.

What you need on your computer

• Python 3.12 or newer, and VS Code.
• About 70 MB of free disk space: the virtual environment (about 56 MB), the documents (5 MB) and the tokenizer’s data.
• An internet connection for the setup. tiktoken downloads its tokenizer data once.

No account and no money. The model calls of this case study were made by the course team; their results are in the project’s files.

Tested on macOS (Apple M4, Python 3.12.13). The Windows and Linux commands are written, not tested.

The situation

Fernhollow Safety Line (“Fernhollow” for short) is a small advice line for employers in the United States. Three safety advisers answer the telephone. Callers ask things like “Do my roofers need a written heat plan?” or “What changed in the labels for chemicals?” The answers are in the rules of the Occupational Safety and Health Administration (OSHA). New rules, proposals and corrections appear in the Federal Register, the government’s daily journal.

Lena Ortiz runs Fernhollow. She has a grant of US$100 a month for a free public page. Anyone types a question, and the page answers with citations to OSHA’s documents. Niko Varga, a volunteer developer, drew a first design. Aisha Rahman, the senior adviser, will check the answers. The people and Fernhollow are invented; the documents are real.

Niko’s design has three ideas that worry Lena:

1. “There are thousands of pages. We need a vector database service.”
2. “Legal text is hard. Use the strong model for every answer.”
3. “Even simpler: put all the documents in the prompt. The provider caches long prompts.”

Lena asks you for a design review. “What should we build? What does it cost each month? What can go wrong, and who fixes it? Use numbers, and tell me which numbers are guesses.”

The objective. A review that Lena can act on: the collection sized by counting, the cost per tenant with a range, two or three decisions with their reasons, and a failure table with owners and recovery targets. Every number says if it is measured, a price or an assumption.

What you will build

The review's path. Your own counts replace the course's numbers where the collection changes them; the course's measured facts fill the rest.

Text description of the diagram

The collection is counted first. Its numbers go into a file that the course’s calculator reads. The calculator, with the demand and the design, gives the costs. The decisions and the failure table use both. Everything ends in one file for Lena.

Part	What you do	What you get
1	check the data	the documents are complete; the text needs cleaning
2	count tokens with a tokenizer	the real token count, and why a guess is risky
3	size storage and ingestion	megabytes, minutes and dollars, and corpus.toml
4	compare the baseline and the alternatives	one cost measure for every design
5	write the decisions	three ADRs
6	write the failure table	six failures with owners, RTO and RPO
7	cost per tenant and the quota	the range for Lena, and the first limit
8	new documents, end to end	the plan checked against 2026

The data

The Federal Register publishes every OSHA rule, proposed rule and notice. Its API needs no account. The documents are works of the US government, so they are in the public domain (17 U.S.C. 105). Some may quote copyrighted material, as GovInfo’s notice says, so quote short passages with their citation.

The course team took two fixed copies: OSHA’s documents from January 2021 to December 2025 (628), and the new ones from January to September 2026 (83). One row is one document: its number, type, title, date, printed pages and plain text. A fixed copy with a checksum gives every learner the same bytes. DATASET.md in the project is the dataset card.

These documents are not the rules in force. A proposed rule may never become a rule, and the rules in force are in the Code of Federal Regulations. The RAG course’s case study Answer from versioned regulations works with that code.

Set up the project

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder open-docs-review.

2. In VS Code, choose File > Open Folder and open open-docs-review. Then open a terminal: Terminal > New Terminal.

3. Make and activate a virtual environment, and install the packages:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt

   Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   The prompt starts with (.venv). The packages are tiktoken 0.14.0 (a tokenizer), pytest 9.1.1 and ruff 0.17.0. The calculator still needs only Python’s standard library.

4. Download the documents and check them:

   python -m corpus get

   Output

   Downloading https://github.com/Nextia-Intelligence/nextia-learning-labs/raw/main/C26/M07-L02-design-review/data/osha-fr-2021-2025.jsonl.gz
   Checked osha-fr-2021-2025.jsonl.gz: SHA-256 81b47f96634a... is correct
   Downloading https://github.com/Nextia-Intelligence/nextia-learning-labs/raw/main/C26/M07-L02-design-review/data/osha-fr-2026-new.jsonl.gz
   Checked osha-fr-2026-new.jsonl.gz: SHA-256 3ee6af82c139... is correct

5. Run the tests:

   python -m pytest -q

   Output (end)

   FAILED tests/test_corpus.py::test_tokens_are_counted_with_the_tokenizer_not_guessed
   1 failed, 11 passed in 0.04s

What this tells us. Both files arrived unchanged: their SHA-256 checksums are the expected ones. One test fails on purpose. The starter guesses the number of tokens, and Part 2 replaces the guess with a count.

Part 1: check the data

Before you size anything, check that the data is what you think it is:

python -m corpus check

Output

Checked osha-fr-2021-2025.jsonl.gz: SHA-256 81b47f96634a... is correct
Documents: 628; types: Notice 517, Proposed Rule 72, Rule 39
Published: 2021-01-11 to 2025-12-31
Without text: 0; repeated document numbers: 0; without a page count: 0
Raw pages: 20,451,255 characters; text after cleaning: 18,652,871 (8.8% was markup)
Shortest: C1-2021-26268 (Rule, 118 words, 1 page)

What this tells us.

• Complete. Every document has text and a page count, and no document appears twice.
• Mostly notices. 517 of the 628 documents are notices, for example a request for comments on a form. Only 39 are rules.
• The text is a web page. The API’s “raw text” is a small HTML page: the text sits inside <pre>, with links, <bullet> marks and subscripts such as LD<INF>50</INF>. 8.8% of the characters were markup. corpus/load.py removes it before anything is counted. Without that step, every count below would be about a tenth too high.

Part 2: count tokens, do not guess

The problem: the bill counts tokens

A parcel service charges by weight, not by the number of parcels. Ten boxes of books cost more than ten boxes of pillows.

A model provider charges by tokens, and its quotas count tokens too. A token is a piece of text that the model reads as one unit. It is often a whole common word, and sometimes a part of a word, a space or a punctuation mark. A tokenizer is the program that cuts text into tokens, and each model family has its own. (Request anatomy introduced tokens.)

The course used 1.48 tokens per word, measured on the RAG course’s shop policies. The starter uses that as a guess. Here is why a guess is risky for legal text:

Real cuts by the tokenizer cl100k_base. Ordinary words are about one or two tokens. A legal citation is more than four tokens a word.

Text description of the diagram

“Guardrails are 42 inches high.” has 5 words and 8 tokens. “29 CFR 1910.134(c)(2)(ii)” has 3 words and 13 tokens: every number, bracket and dot is its own piece.

Which tokenizer?

Use the tokenizer that the bill uses. The course team checked it with real calls. It sent 32 batches of this collection’s chunks to the embedding model, embed-small, and compared the tokens on the bill with two local counts. The tokenizer cl100k_base gave exactly the billed number in 32 of 32 calls (167,297 tokens; measured). The chat model, chat-small, counts with another tokenizer, o200k_base, which gave about 1.5% fewer tokens on the same text. This review uses cl100k_base everywhere, so the answer estimates are a little high: the safe side.

Count them

First, run the sizing with the guess:

python -m corpus size

Output (start)

Collection: 628 documents, 3,676 printed pages, 2,450,514 words
  tokens                   3,626,755  (1.48 per word, 987 per page; guessed: words x 1.48)

Now open corpus/size.py. Replace the line TOKENS_HOW = ... and the function count_tokens with this:

corpus/size.py (part)

TOKENS_HOW = "cl100k_base"   # how the tokens below are found


def count_tokens(text: str) -> int:
    """Count with the tokenizer: the same count as the provider's bill for embed-small (measured)."""
    import tiktoken  # the embedding model's tokenizer; get_encoding keeps it loaded after the first call

    return len(tiktoken.get_encoding("cl100k_base").encode(text, disallowed_special=()))

Run the sizing and the tests again:

python -m corpus size
python -m pytest -q

Output (start of the sizing, end of the tests)

Collection: 628 documents, 3,676 printed pages, 2,450,514 words
  tokens                   3,710,587  (1.51 per word, 1,009 per page; cl100k_base)
...
12 passed in 0.07s

Tokens per word for each of the 626 documents with at least 200 words, counted with cl100k_base. Most documents are near the guess. A few, full of numbers and citations, have up to 2.44.

Text description of the chart

Most documents have between 1.4 and 1.6 tokens per word, around the guess of 1.48. The lowest has 1.288 and the highest 2.443 tokens per word: tables of numbers and long citations.

What this tells us. For the whole collection, the guess was 2% low, because ordinary prose dominates. But one document has 65% more tokens than the guess. A design review counts, because quotas and failures depend on single documents, not on the average.

Part 3: size storage and ingestion

The rest of the sizing output:

Output of python -m corpus size (rest)

  chunks                      14,961  (6.1 per 1,000 words; 263 tokens each with the title)
  tokens to embed          3,941,704  (the chunks with their titles: +6% on the text)
  by type               Rule 39 documents, 34% of the tokens; Notice 517 documents, 34% of the tokens; Proposed Rule 72 documents, 32% of the tokens
  the 10 largest        49% of all tokens
Storage: vectors with table and index 71.8 MB, chunk text 18.5 MB, document files 18.7 MB
  in the database 90.4 MB = 0.3% of the smallest size (32 GB)
Ingest everything once: 3,941,704 tokens, US$0.08, 695 calls of up to 100 chunks
  one call at a time 8 min; at the token quota 39 min: about 39 min
New documents per month (60 months): low 0, median 10, mean 10.5, high 37

Each line multiplies a count by a measured value or a price:

• Chunks. corpus/chunk.py keeps the RAG course’s rule from Chunk deliberately: whole paragraphs, at most 200 words, each chunk starting with the document’s title. The titles add 6% to the tokens to embed.
• Storage. 14,961 chunks × 4,800 bytes per vector with its table and index (measured in the databases course, 384 numbers) = 71.8 MB. With the chunk text, the database holds 90.4 MB. The smallest storage size is 32 GB (price list), so storage is not a design question here.
• Ingestion. 3,941,704 tokens × US$0.02 per million (price list) = US$0.08. Time is the real limit. Each call takes about 0.43 s plus 0.04 s per 1,000 tokens (measured), so 695 calls one at a time take about 8 minutes. But the embedding model’s quota is 100,000 tokens a minute (measured: the provider’s rate-limit header). The whole collection needs at least 39 minutes. The course team’s sample of 167,297 tokens hit that quota four times in about two minutes (HTTP 429, measured).

Tokens per document, counted with cl100k_base: the three largest, the next seven, and the other 618. The 10 largest documents hold 49% of the collection's tokens; the largest alone is a 376-page proposal.

Text description of the chart

The largest document, 2024-14824 (Heat Injury and Illness Prevention in Outdoor and Indoor Work Settings, “Heat”), has 484,968 tokens. The next two are 2021-12428 (Occupational Exposure to COVID-19; Emergency Temporary Standard, “COVID”, 315,810 tokens) and 2024-08568 (Hazard Communication Standard, “HazCom”, 229,771 tokens). The next seven have 797,709 tokens together, and the other 618 documents 1,882,329.

What this tells us. The median document is 2 printed pages. The largest is 376 pages: the 2024 heat-injury proposal, 484,968 tokens. Plan the loading, the timeouts and the retries for that one, not for the average. Storage is tiny; time and the quota are what matter.

Put the counts into the pack

Two files tell the calculator about the collection:

1. Open demand.toml. The starter still has the course’s Larkfield values for the collection, marked TODO. Replace them with your counts, and change their evidence to verified::

   demand.toml (part)

   documents = [628, 628, 628]
   words_per_document = [3902, 3902, 3902]
   document_changes_per_month = [10, 10.5, 37]

   demand.toml (evidence, part)

   documents = "verified: python -m corpus check counted 628 documents, 2021 to 2025"
   words_per_document = "verified: python -m corpus size counted 2,450,514 words in 628 documents (mean 3,902)"
   document_changes_per_month = "verified: new documents per month, 60 months: median 10, mean 10.5, busiest month 37 (python -m corpus size)"

   The visitors’ rows stay assumed: Fernhollow has no traffic yet. Lena’s guess is 250 visitors who ask a day (50 to 1,000).

2. Write this collection’s measured values for the calculator:

   python -m corpus write

   Output

   Checked osha-fr-2021-2025.jsonl.gz: SHA-256 81b47f96634a... is correct
     ingest_embed_tokens_per_word         1.61 tokens per word
     chunks_per_1000_words                6.11 chunks
     chunk_text_bytes_per_word            7.57 bytes per word
     document_file_bytes             29,709.91 bytes per document (mean)
     answer_small_tokens_in           2,002.28 tokens per answer (estimate)
     answer_strong_tokens_in          2,002.28 tokens per answer (estimate)
   Wrote corpus.toml: the calculator now uses this collection's values

costmodel/inputs.py reads corpus.toml after measured.toml, so these six values replace the course’s. The last two need a word. The course measured 946.5 input tokens per answer, with five passages of about 52 tokens each. This collection’s chunks have 263 tokens. So the same prompt with five of them is about 946.5 − 5 × 52.31 + 5 × 263.47 = 2,002 tokens. That is an estimate from measured parts, and the file says so.

Part 4: the baseline and the alternatives

The baseline

The baseline is the course’s design for one tenant, with the course’s numbers: what Niko would get by copying the course. Every design gets one measure: the monthly cost for Fernhollow’s base workload. That is 13,050 questions a month, with the same price list. The quality bar is the course’s measured result, which comes from another question set.

Version	Collection values	Base cost a month
Baseline: the course’s design and numbers	Larkfield’s 36 short documents	US$47.04
+ this collection’s counts in demand.toml	628 documents, 3,902 words each	US$47.05
+ corpus.toml (chunks, prompt size)	this collection’s own	US$48.31

What this tells us. 628 long documents instead of 36 short ones changed the bill by one cent. The longer chunks changed it by 3%: the answer tokens went from US$2.01 to US$3.25 a month. Most of the cost is the always-on API replica and the database server. So the collection’s size is a question of time, quota and failure, not of money.

The alternatives, under one measure

python -m costmodel options

Output (start)

Model routing: the same 67 questions, the same search results (measured), base workload (cost model)
  design                    correct  US$/question   p50 s   p95 s  US$/month
  chat-small only             86.6%      0.000277    1.52    3.79      48.31
  chat-strong only            85.8%      0.005309    2.48    6.56     108.59
  routed (small first)        86.6%      0.001346    1.76    4.02      61.13
Cheapest design with the best measured quality: chat-small only

The “correct” column is the course’s Model routing result on the RAG course’s 67 questions about shop policies. It is not a measurement on OSHA’s documents. The cost columns are this collection’s: the longer prompt makes each answer dearer than in the course.

Niko’s third idea needs no run, only arithmetic. The whole collection is 3,710,587 tokens. chat-small’s quota is 100,000 tokens a minute, so one question would use 37 minutes of the quota. Suppose that the provider charged every token at its cached-input price (US$0.01 per million, price list). Then one question would cost US$0.037. For the 11,745 questions a month that reach the model, that is US$435.81 in tokens alone. In Caching and batching, the provider’s cache saved money on a long prompt, not time.

Design	Base cost a month	Quality evidence	Fits the limits?
Whole collection in every prompt	about US$481 (US$435.81 tokens + the rest)	none	no: 37 minutes of quota per question
chat-strong for every answer	US$108.59	85.8% (another question set)	yes
Routed, chat-small first	US$61.13	86.6% (another question set)	yes
chat-small only, shared answer cache	US$48.31	86.6% (another question set)	yes, up to the high scenario (Part 7)

What this tells us. On the evidence that exists, chat-small is the cheapest design with the best measured quality. But that evidence is weak for this collection. So the recommendation includes a measurement before launch, not after.

The cache: a public answer has no owner

In the course, every cache key held the tenant, because Larkfield’s answers must never reach Bramble Books. Fernhollow’s answers are public: every visitor may see every answer. So one cache entry can serve everyone, and the hit share may be higher than in the course. It is still an assumption (0%, 10% or 30%).

The danger is time, not people. About ten new documents arrive a month, and some are corrections, for example “Hazard Communication Standard; Corrections” in January 2026. So the key holds the collection version: the number of the last document loaded. A new document makes every key new, as the data version did in Keys and invalidation. The rest of the options output prices the cache:

Output of python -m costmodel options (caching)

Caching answers: the hit share is an ASSUMPTION; the time saved per hit is measured
  a miss takes 2.18 s (embed + search + answer, medians); a hit takes 0.014 s
   hit share  US$/month    saved  median wait s
          0%      48.68     0.00           2.18
         10%      48.31     0.37           2.18
         30%      47.58     1.10           2.18
         50%      46.84     1.84           2.18
  The median wait does not change until more than half of the questions are hits.

What this tells us. The cache saves about a dollar a month: 93% of the cost is fixed. Its value is in the quota: a hit makes no chat call. Part 7 shows why that matters.

Part 5: the decisions

Three decisions change the design, so each gets an architecture decision record (ADR) from decisions/template-adr.md, as in Architecture decisions. Copy the template three times and fill in every heading:

• ADR-0001: one PostgreSQL server holds the whole collection; load it once before launch. 90.4 MB is 0.3% of the smallest storage, so a vector database service adds a service to secure and back up for nothing. Load before launch at 70% of the embedding quota (about 56 minutes), so test questions still get their embeddings. Revisit at 80% storage, a search p95 over 100 ms, or ten times the collection.
• ADR-0002: chat-small answers with five passages; no routing until this collection has its own question set. Part of the record is below.
• ADR-0003: one shared answer cache, keyed by the question and the collection version. No visitor in the key, because the answers are public. Revisit if the measured hit share is above 30% (plan for it) or below 5% (remove it).

decisions/adr-0002-chat-small-with-five-passages.md (Decision and Revisit when)

## Decision

Option 1. Before launch, Aisha writes 50 questions with their answers from OSHA's documents, and we run the course's routing comparison on them.

## Revisit when

- the 50-question set shows chat-strong at least 5 points better than chat-small, or
- the busiest minute uses 70% of the token quota: first ask the provider for a higher quota; then try three passages (about 1,475 tokens a prompt, an estimate) and check the quality on the question set.

The finished project has all three records in full, with their options and consequences.

Common mistake

An ADR that quotes a number from the wrong question set as if it were this one's

A first draft of ADR-0002 said: “chat-small is 86.6% correct on our documents.” Aisha asked: “Which documents?” The number is real, but it was measured on shop policies, not on OSHA’s rules. A reader who trusts it skips the question set, and the first measurement on legal text happens in production. Fix: write where each measured number comes from (“on another question set”), and make the missing measurement a step in the decision. Here, the 50-question set comes before launch.

Part 6: the failure table

A public service without sign-in has failures that the course’s design did not have. Draw them on the design before you write the table:

The six failures on the design. F2 and F6 exist because the service is public; F3 and F4 because the source changes and some documents are huge.

Text description of the diagram

Visitors reach the API without signing in. The API uses the model provider and one PostgreSQL server. A worker loads new documents from the Federal Register. Each failure of the table below, F1 to F6, is drawn at the part it hits.

The table in decisions/failure-modes.md has one row per failure, with the course’s columns from Failure scenarios plus the recovery targets:

ID	Failure	Detection signal	Owner	RTO	RPO
F1	Provider outage or 429	5xx and 429 > 5% over 5 minutes	Niko	when the provider is back; our message within 15 minutes	none
F2	Bots use up the quota and the budget	questions per visitor per hour; daily spend > budget / 30	Niko	60 minutes (block, cap)	none
F3	A new rule or correction not loaded	daily: OSHA’s count in the API against ours	Aisha	1 day	none: the source keeps it
F4	The 376-page proposal fails halfway	a job past its lease; chunks stored ≠ expected	Niko	about 5 minutes for that document	none
F5	Database lost	availability alert	Niko	about 70 minutes (estimate)	5 minutes of the question log (assumed); documents: none
F6	An answer read as legal advice	“this answer is wrong” button; 20 answers reviewed a week	Lena	the cache entry deleted within 1 working day	none

The project’s file also has each mitigation. Two rows need numbers:

• F4. The largest document has 1,543 chunks: 16 calls of 100, and 484,968 tokens. At 100,000 tokens a minute, it alone needs about 5 minutes of the quota. One transaction per document means a failure leaves the old state, never half a document.
• F5. Lena’s targets are an RTO of 8 hours and an RPO of one day for the question log (assumed: a free service, no contract). Run the course’s recovery command:

python -m costmodel recovery

Output

Recovery (base): targets RTO 480 min, RPO 1440 min; database 184 MB restores in about 4.6 s (measured rate, a laptop)
  plan                              RPO min  RTO min  meets both?
  nightly dump to file storage        1,440       70  yes
  managed point-in-time restore           5       70  yes
Most of the recovery time is people and a new server, not copying data. Test the restore to know.

What this tells us. Both plans meet Lena’s targets. This collection also has a way back that Larkfield’s has not: the documents are public. If every backup failed, corpus get and a new load would rebuild the collection in about an hour. Only the question log would be lost.

Part 7: cost per tenant and the first limit

Fernhollow is the only tenant, so the cost per tenant is the whole cost:

python -m costmodel scenarios

Output

Scenarios (US$ per month)
                     low        base        high
  month 0          39.09       48.31       74.04   (1 tenants)
  per tenant       39.09       48.31       74.04
  month 12         39.09       49.72      115.32   (1 tenants)
  per tenant       39.09       49.72      115.32

python -m costmodel units adds the unit cost: US$0.0037 per question in the base case, all costs included, and 93% fixed. In the high case it falls to US$0.00078, because the fixed costs are shared by more questions, as in Unit economics. If Fernhollow joined a shared platform with other tenants, they would share the fixed part too.

Now the limits:

python -m costmodel growth --scenario high

• Tokens
• Calls (dashed)
• The quota

The high scenario's busiest minute against chat-small's quota of 100,000 tokens and 100 calls a minute. The tokens pass the quota in month 0; the calls pass it in month 6.

Text description of the chart

In the high scenario, the busiest minute needs 127% of the token quota in month 0 and 399% in month 12. The calls start at 59% and pass the quota in month 6. The cost passes Lena’s US$100 budget in month 9 (US$100.26).

What this tells us. The first limit is the provider’s token quota, already in month 0 of the high case: 127%. The course’s Larkfield design reached 114% with two shops. Here, one tenant gets there, because each prompt is twice as long. Three levers, in order. First, ask the provider for a higher quota before any publicity. Second, let the cache answer repeats. Third, try three passages instead of five (about 1,475 tokens, an estimate), after you measure the quality.

Part 8: new documents, end to end

The plan says about 10.5 new documents a month, with the collection’s average size. 2026 is new input that the plan never saw:

python -m corpus new

Output

Checked osha-fr-2026-new.jsonl.gz: SHA-256 3ee6af82c139... is correct
New documents: 83 from 2026-01 to 2026-09 (9.2 a month)
  239 pages, 132,766 words, 850 chunks, 216,557 tokens to embed
  embedding US$0.0043; database +5.1 MB
  largest: 2026-10735 (Notice, 11 pages, 14,958 tokens)

What this tells us. Nine months of new documents cost less than half a cent to embed and add 5 MB. The rate, 9.2 a month, is near the plan’s 10.5. The documents were smaller than the plan’s average (1,600 words, not 3,902), because the average includes the giant rules of 2021 to 2024. So the plan was on the safe side. One January document was a correction to the 2024 labelling rule. The collection version in the cache key (ADR-0003) and the daily check (F3) exist for exactly this case.

Check the review

Fill in review.md from the template: the service and its targets, the sizing, the recommendation, the cost per tenant, the assumptions and the risks. Name the one assumption that would change the recommendation. Then run the review’s check:

python -m costmodel check

Output

  PASS  demand: every value has evidence (verified or assumed)
  PASS  prices: every price has its source
  PASS  measured: every value names its run
  PASS  corpus: the calculator uses this collection's own numbers
  PASS  decisions: 2 or more ADRs with status, context, options, decision, consequences, revisit
  PASS  failures: at least 5, each with owner, mitigation, signal, RTO and RPO
  PASS  review: recommendation, cost per tenant, assumptions, risks and an unresolved assumption
Design review check: 7 of 7 passed

The check is the course’s pack check with this review’s rules. It sees the shape of the review, not its truth: Lena still reads it. The finished project adds a test that runs this check, so its python -m pytest -q prints 13 passed.

Insights and limits

• Count before you design. “Thousands of pages” sounded like a storage problem. Counted, it was 90 MB: 0.3% of the smallest database.
• The average hides the risk. The median document is 2 pages; the largest is 376. The loading time, the retries and one failure row are about that one document.
• The collection changes prompts, and prompts change the quota. Longer chunks made each answer prompt twice the course’s size. The money moved by 3%; the token quota became the first limit.
• Limits of this review. The demand is assumed. The answer prompt is an estimate from measured parts, counted with the embedding model’s tokenizer (chat-small’s counts about 1.5% fewer). The cache hit time comes from the scaling course. Every time was measured from one computer on one afternoon. The calculator still prices a summary for every new document, although the Federal Register gives an abstract (less than a cent a month). Windows and Linux are not tested.

Responsible use

• Not legal advice. The Federal Register records rules as they are made, proposed and corrected; the rules in force are in the Code of Federal Regulations. Every answer shows its documents, their dates and their types, and says that a proposed rule is not in force.
• Decline instead of guessing. When no passage supports an answer, the service says so and gives Fernhollow’s telephone number.
• People in public documents. The documents name officials and give their telephone numbers. Do not extract or republish contact details, and keep them out of answers.
• A public page is an open door. Limit questions per visitor, and cap the daily spending, before the page is announced.
• Be polite to the source. Check the Federal Register’s API once a day, one request at a time. Do not load all 628 documents again for every test: use the fixed copy.

Make it a portfolio piece

This project shows what employers look for in a solution architect: a design sized by counting, costs with their assumptions, and decisions that say when to look again.

What to put in the repository:

open-docs-review/
  README.md          what it is, the commands, the three kinds of numbers
  review.md          the one-page review: recommendation, cost per tenant, risks
  DATASET.md         the source, the licence, the checksums, the limits
  decisions/         ADR-0001 to 0003, failure-modes.md
  demand.toml        demand with its evidence
  corpus.toml        this collection's measured values (written by corpus write)
  corpus/            download, cleaning, chunks, token counts, storage, ingestion
  costmodel/         the course's calculator, reading corpus.toml
  tests/             13 tests, one of them the review's check

1. Make a new Git repository from your folder and push it to GitHub. .gitignore keeps out the data and the virtual environment.
2. Put the sizing table and the cost range from review.md at the top of the README.
3. Describe it in two sentences: “I reviewed a public question service over 628 government documents. Counting tokens showed that storage was tiny and that the token quota was the first limit. I recommended the small model, a shared cache and a quality test before launch.”

Go further. Size another agency’s documents with the same scripts, and compare the shapes. Write Aisha’s 50 questions, and run the course’s routing comparison on them. Add a second tenant, and see the cost per tenant fall.

Knowledge check

Check your understanding

The collection grew from 36 short documents to 628 long ones, yet the monthly cost rose by only one cent. Then corpus.toml raised it by about 3%. Why did the second change matter more than the first?

Show the answer

The documents change the cost only through ingestion and storage. Loading new documents costs fractions of a cent a month, and the database stays far below its smallest size, 32 GB. corpus.toml changed what every answer costs. This collection’s chunks are about five times longer, so a prompt with five passages has about 2,002 tokens, not 946. Answers happen thousands of times a month. Most of the cost is still fixed, so the total moved by only 3%. But the longer prompt made the token quota the first limit.

Check your understanding

Niko says: 'The course measured chat-small at 86.6%, so we know it works for OSHA's rules.' What do you answer, and what goes into the decision record?

Show the answer

The 86.6% is real, but it was measured on the RAG course’s shop policies, another question set. Legal text with long passages and cross-references may score differently. The decision can still be chat-small: it is the cheapest design with the best quality that was measured. But the record must say where the number comes from. It must also make the missing measurement a step: Aisha’s 50 questions on OSHA’s documents, run before launch. Its revisit trigger: chat-strong 5 or more points better.

Check your understanding

In the course, the answer cache's key held the tenant. In this review, it holds the collection version instead. Why each, and what goes wrong if the version is missing?

Show the answer

In the course, answers belong to a tenant, so a key without the tenant could show one shop’s private answer to another shop. Fernhollow’s answers are public, so every visitor can share one entry. But the documents change: a new rule or a correction can make an old answer wrong. With the collection version in the key, the first question after a load is a miss, and the answer is built from the new documents. Without it, the cache could keep citing a text that a correction replaced, until the entry expires.

Recap

• You checked the data and removed its web-page markup (8.8% of the characters) before counting.
• You counted tokens with the tokenizer that the bill uses: 3,710,587 for 628 documents. The guess was 2% low overall, and one document had 65% more tokens than the guess.
• Storage was 90 MB, 0.3% of the smallest database. Loading everything costs US$0.08 and takes at least 39 minutes, because of the embedding quota.
• chat-small and a shared cache cost US$48.31 a month in the base case (US$39.09 to US$74.04). 93% of it is fixed. The whole collection in every prompt would need 37 minutes of quota per question.
• You wrote three ADRs and six failures with owners, RTO and RPO. The first limit is chat-small’s token quota. The main unresolved assumption is answer quality on OSHA’s documents.

Next, the final assignment: write your own architecture and cost proposal for a small multi-tenant document assistant.

Next: Final assignment