Growth plan

What you will be able to do

Plan how the design grows without guessing its final shape. You choose a few signals, each with a line, and write the step that each signal starts. Then the pack’s calculator shows which line each scenario crosses first, and when. You finish the module with a tabletop review of two failures.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The provider quota and the busiest minute, from Workload arithmetic: Check the provider’s limit.
• The low, base and high scenarios and the budget Q8, from Unit economics: Read the scenarios.
• The failure-mode table, from Failure scenarios, and the recovery times, from Recovery objectives.
• Scaling signals and their lines, from Scaling signals in Scaling APIs and AI Workloads. It helps, but it is not required.

What you need on your computer

• Your folder design-pack from the last lesson, with python -m pytest -q printing 42 passed and python -m costmodel recovery working.

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A restaurant that grows table by table

A cook opens a small restaurant with twelve tables. A friend says: “Build for 200 guests now.” The cook does not. A hall for 200 costs rent every month, full or empty.

Instead, the cook writes three rules on the kitchen wall. “When guests wait more than 20 minutes on three weekends in a row, add tables in the back room. When orders wait more than 15 minutes at the kitchen, hire a second cook. When the fish supplier cannot deliver enough, find a second supplier.”

Each rule has a signal that someone can watch, a line, and a step. The restaurant grows when the signals say so, in the order that they say. Often the first problem is one that nobody guessed: here, the fish supplier, not the size of the hall.

The problem: “design for a million users”

Tomás has a proposal for the growth section. “Let’s design for a million users from the start: Kubernetes, a vector database service, several regions. Then we never need to change anything.”

The pack already knows the size of the base case. At its busiest minute, 0.58 requests are in flight. A million users would be thousands of times more. Every extra service would cost money every month and add a part that can fail, for users who do not exist.

And there is a worse problem. Look at the high case in the first month. Its busiest minute needs 104% of the provider’s call quota and 114% of its token quota. So the first limit is not the servers or the database. It is the provider’s quota: a number that a Kubernetes cluster does not change. A design for a million users would be expensive, and it would still miss the limit that comes first.

The idea: signals, lines and steps

A growth plan is a list of steps, each one started by a signal. It replaces one guessed final architecture. Each entry has three parts:

• The signal: a measurement of the running system, such as “tokens in the busiest minute”.
• The line, also called the trigger: the value at which you act, with time to spare. For example, “70% of the provider’s token quota”.
• The step: the smallest change that moves the signal back under its line, and its owner.

The scaling course used the same idea to add replicas. Why lines below the limit? A step takes time. Asking the provider for a higher quota, or changing the code, can take days. A line at 70% leaves 30% of room while the step happens.

The pack watches three signals each month: the provider quota, the database’s storage, and the monthly cost against the budget. The lines are in design.toml, in the section [growth]:

[growth]                 # Module 5, lesson 3: the signals that start each step
quota_alert_share = 0.7      # act when the busiest minute uses 70% of the provider quota
storage_alert_share = 0.8    # act when the database uses 80% of its provisioned storage
budget_usd_per_month = 150   # from requirements.md: Omar's budget for the first year

A tiny example: one signal by hand

Take the token quota in the base case, month 0. Every value comes from the pack:

1. The busiest minute has 14.46 chat calls (the workload prints 14.5): 16.1 questions, minus the 10% that the cache answers.
2. Each answer uses 946.5 tokens in and 152.75 out: 1,099.25 tokens (measured).
3. Tokens in the busiest minute: 14.46 × 1,099.25 = 15,896 tokens.
4. The quota is 100,000 tokens a minute (measured from the provider’s rate-limit headers in the scaling course). So the signal is 15,896 ÷ 100,000 = 16%.
5. The line is 70%. With the same daily shape, the busiest minute grows with the questions a day. The line is reached at 70 ÷ 15.90 × 1,465 = about 6,450 questions a day, about 4.4 times today’s base.

So in the base case, the quota is far away. The plan does not act now. It writes down the line and watches it.

• High: tokens
• Base: tokens
• Alert line, 70%

The busiest minute's share of the provider quota, month by month. The base case stays between 16% and 25%. The high case is above the 70% line, and above the quota itself, from month 0. The dashed line in the high case's colour is the share of calls; tokens are the tighter limit.

Text description of the chart

A line chart over 12 months, from 0% to 300% of the provider quota. A dashed line at 70% is the alert line. The base case’s tokens line starts at 16% and rises slowly to 25% in month 12, far below the alert line. The high case’s tokens line starts at 114% in month 0 and reaches 285% in month 12. It steps up in month 4 (168%) and month 9 (245%), when the third and fourth shops join. The high case’s calls line is a little lower: 104% in month 0 and 259% in month 12.

Larkfield: the growth signals in the pack

Guided practice: growth in the pack

1. In your folder design-pack, with the environment active, copy the growth code and the module’s tests from the snapshot end-of-m05.

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m05
   cp "$S/costmodel/growth.py" costmodel/
   cp "$S/tests/test_recovery_growth.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m05"
   Copy-Item "$S\costmodel\growth.py" costmodel\
   Copy-Item "$S\tests\test_recovery_growth.py" tests\

   The section [growth] is already in your design.toml: it came with the last lesson’s copy.

2. Run the growth signals in the base case:

   python -m costmodel growth

   Growth (base): busiest minute against the provider quota, storage, cost
     month tenants questions/day calls/quota tokens/quota  storage  US$/month  signals
         0       2         1,465         14%          16%       0%      51.90  -
         1       2         1,495         15%          16%       0%      52.04  -
         2       2         1,526         15%          17%       0%      52.19  -
         3       2         1,557         15%          17%       0%      52.33  -
         4       3         1,784         18%          19%       0%      53.41  -
         5       3         1,820         18%          20%       0%      53.58  -
         6       3         1,857         18%          20%       0%      53.75  -
         7       3         1,895         19%          21%       1%      53.93  -
         8       3         1,934         19%          21%       1%      54.11  -
         9       4         2,168         21%          24%       1%      55.22  -
        10       4         2,212         22%          24%       1%      55.43  -
        11       4         2,258         22%          24%       1%      55.64  -
        12       4         2,304         23%          25%       1%      55.86  -
   First signal: none in these months

   Month 0 is the tiny example: 16% of the token quota. The third shop joins in month 4 and the fourth in month 9 (assumed in demand.toml).

3. Run the high case:

   python -m costmodel growth --scenario high

   Growth (high): busiest minute against the provider quota, storage, cost
     month tenants questions/day calls/quota tokens/quota  storage  US$/month  signals
         0       2         5,580        104%         114%       0%      75.81  provider quota
         1       2         5,868        110%         120%       0%      76.90  provider quota
         2       2         6,171        115%         127%       1%      78.04  provider quota
         3       2         6,490        121%         133%       1%      79.25  provider quota
         4       3         8,206        153%         168%       1%      85.74  provider quota
         5       3         8,629        161%         177%       2%      87.34  provider quota
         6       3         9,073        169%         186%       2%      89.02  provider quota
         7       3         9,541        178%         196%       3%      90.79  provider quota
         8       3        10,034        187%         206%       3%      92.65  provider quota
         9       4        11,932        223%         245%       4%      99.83  provider quota
        10       4        12,547        234%         257%       5%     102.15  provider quota
        11       4        13,193        246%         271%       6%     104.60  provider quota
        12       4        13,874        259%         285%       6%     107.17  provider quota
   First signal: provider quota in month 0

4. Ask “what if”. Omar wonders what happens with a smaller budget. In design.toml, change the budget line to budget_usd_per_month = 100 and run python -m costmodel growth --scenario high again. The last lines now read:

       9       4        11,932        223%         245%       4%      99.83  provider quota
      10       4        12,547        234%         257%       5%     102.15  provider quota, budget
      11       4        13,193        246%         271%       6%     104.60  provider quota, budget
      12       4        13,874        259%         285%       6%     107.17  provider quota, budget
   First signal: provider quota in month 0, budget in month 10

   A second signal fires in month 10. Change the line back to 150.

5. Run the tests and the style check:

   python -m pytest -q
   ruff check .

   ..............................................                           [100%]
   46 passed in 0.08s
   All checks passed!

   Your pack is now the snapshot end-of-m05. One new test is called test_growth_high_hits_the_provider_quota_first.

Read the signals

The base case needs no step in the first year. The quota goes from 14% to 23% of the calls and from 16% to 25% of the tokens. Storage stays at 1% or less of the 32 GB. The cost rises from US$51.90 to US$55.86 with four shops, far under the budget of US$150.

The high case hits the provider quota first, in month 0. Its busiest minute needs 114% of the token quota. So about one token in eight is over the limit: 1 − 100 ÷ 114.5 = 13%. Some calls would get a 429 every busy day. Storage reaches only 6% in month 12, and the cost stays under the budget, at US$107.17.

Look at the cost of the three scenarios against the budget:

• High
• Base
• Low
• Budget (Q8)

The monthly cost in the three scenarios over 12 months, against Omar's budget of US$150. No scenario reaches the budget in the first year. The steps in months 4 and 9 are the third and fourth shops.

Text description of the chart

A line chart over 12 months, from US$0 to US$160 a month. A dashed line at US$150 is the budget. The low case is almost flat: US$40.54 in month 0 and US$40.79 in month 12. The base case rises from US$51.90 to US$55.86. The high case rises from US$75.81 to US$107.17. It steps up in month 4 (US$85.74) and month 9 (US$99.83). No line reaches the budget.

So the signal that matters is not on Tomás’s list. Only the provider’s quota needs a plan, and only if the high case comes true.

The growth plan: steps with their triggers

Kwame writes the plan as steps. Each step names its trigger and the decision that it changes. The first three triggers are the pack’s signals; the last two come from the failure table and the architecture.

The growth plan: five triggers, each starting one small step. Nothing changes until its signal crosses its line.

Text description of the diagram

The plan starts from today’s design: one application, PostgreSQL for everything, chat-small and one API replica. Five triggers lead to five steps. When the busiest minute uses 70% of the provider quota, ask for a higher quota and queue customers’ questions at the peak. When the monthly cost reaches the budget of US$150, find the biggest driver in the cost table and change that one first. When the database uses 80% of its storage, choose a bigger storage size, which is a setting. When the search p95 is over 100 ms, tune the index first, then consider a separate vector store. When the busiest minute needs more than 4 API replicas, split out the part that limits the API. The worker becomes its own service only on another trigger of ADR-0001: when ingestion needs a GPU or its own release rhythm.

The first step deserves numbers, because the high case needs it at once:

• How much quota? In month 12 of the high case, the busiest minute needs 259 calls and about 284,700 tokens. To stay at the 70% line, the quota must be 259 ÷ 0.7 = 370 requests a minute. For tokens, it must be 284,700 ÷ 0.7 = about 407,000.
• How? The provider’s documentation says that a deployment’s quota can be raised on request. This course did not make a request, so the time it takes is unknown. Ask early: the high case needs it in month 0.
• And meanwhile? Customers on the website can wait a few seconds in a queue at the peak. The scaling course showed this in Backpressure. Agents in the help desk go first. A larger cache share would also help: in the high case, the cache already answers 30%.

Kwame also writes what the plan does not contain: Kubernetes, a vector database service and a second region. Each one could become a step later. None of them has a trigger that any scenario crosses this year.

When to write a growth plan, and what it costs

Write one whenever demand is uncertain, which is almost always before launch. The plan is cheap: a few lines in a file and a monthly look at three numbers. The pack’s command prints them.

Do not build a step before its trigger fires. Its cost starts on the first day; its benefit starts only when the signal arrives. The exception is a step that takes longer than the room under its line. In the high case, the quota request is one, so Kwame asks for it now.

After launch, the real usage log replaces the assumed demand, and the plan’s months move.

Common mistakes

Common mistake

The tests arrive before the code

Symptom: you copied test_recovery_growth.py but not growth.py, and python -m pytest -q ends with:

E   ModuleNotFoundError: No module named 'costmodel.growth'
=========================== short test summary info ============================
ERROR tests/test_recovery_growth.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.05s

Cause: one test file checks recovery and growth together, and it imports the growth code.

Fix: copy costmodel/growth.py too, as step 1 shows.

Common mistake

A trigger at 100%

Symptom: the plan says “ask for a higher quota when we hit the quota”. On the first busy Monday above it, customers get errors for a week while the request waits.

Cause: the line was at the limit, so the step started too late.

Fix: put the line where the room that is left covers the time the step takes: 70% for a request that takes days.

Reset. If your pack breaks, copy the snapshot end-of-m05 as the snapshots README shows.

Module practice: a tabletop review of two failures

This module wrote the failure table, the recovery targets and the growth plan. A tabletop review puts them to work. The team picks a failure, a moment and a scenario. Then it answers five questions in order: what users see, which signal fires, who acts, what limits the damage, and how it recovers. Each answer uses a number from the pack.

A worked example: the quota on a busy day, high case, month 0. It is the busiest minute of a high day, and no quota step has happened yet.

1. What users see. The minute needs 104.2 chat calls and about 114,500 tokens, against 100 calls and 100,000 tokens (workload --scenario high, growth --scenario high). About 13% of the tokens are over the limit, so some answers get a 429. With retries, they arrive late; without them, users see “try again in a minute”.
2. The signal. The growth signal is already over its line: 114% against 70%. At run time, F1’s signal fires: the share of 429s over 5 minutes is above 5%.
3. Who acts. Kwame, the owner of F1, asks the provider for more quota. Amira runs the fallback: the breaker and the queue.
4. What limits the damage. The cache answers 30% of the questions in the high case, without the provider. A queue lets customers wait a few seconds; agents go first.
5. Recovery. Automatic after the busiest minutes; no data is lost. The real fix is the growth step: a quota of at least 370 requests and 407,000 tokens a minute for month 12.

Gap found: the failure table names the signal, but the quota request has no date. Kwame adds it to the plan: “Ask for the higher quota before launch, if Omar confirms the website’s home page (the high case).”

Your task. Run a tabletop review of two failures in the high case, and write the five answers for each, with numbers from the pack:

• A: the database is lost at 20:00 on a Saturday. Use python -m costmodel recovery --scenario high and the row F6.
• B: a tenant uploads 200 documents at once, and the worker stops halfway through the twelfth one. Use the rows F3 and F4.

End with one gap that you found and the change you would make to the pack.

Show a worked solution

A: the database is lost at 20:00 on a Saturday (high case).

1. What users see: the assistant is down for every shop. Every question fails until the restore is checked.

2. The signal: the availability alert. At night and at weekends, the pack’s high value for detection is 60 minutes.

3. Who acts: Mei, the owner of F6; Kwame if Mei cannot be reached.

4. What limits the damage: point-in-time restore to a new server. The output gives the times:

   Recovery (high): targets RTO 240 min, RPO 60 min; database 2,038 MB restores in about 50.9 s (measured rate, a laptop)
     plan                              RPO min  RTO min  meets both?
     nightly dump to file storage        1,440      191  no (RPO)
     managed point-in-time restore          15      191  yes

   That is 60 + 30 + 60 + 0.85 + 40 = 191 minutes: within the RTO of 240, with 49 minutes to spare. The data lost is at most 15 minutes, within the RPO of 60.

5. Recovery: the assistant is back at about 23:10.

B: 200 documents, the worker stops in the twelfth (high case).

1. What users see: the shop’s new documents are “processing”. The first 11 are searchable; the other 189 are not yet.
2. The signal: F4, the job “running” past its lease, and then F3, the oldest queued job older than 15 minutes.
3. Who acts: Mei, the owner of F3 and F4.
4. What limits the damage: each document is one transaction for its chunks, vectors and version switch. So the twelfth document is either fully old or fully new, never half. The lease returns its job to the queue. The limit of queued jobs per tenant keeps the other shops’ uploads moving.
5. Recovery: automatic when a worker takes the job again; no data is lost. If the worker stops again on the same document, the job becomes a dead letter that Mei looks at.

One gap and a change: the failure table says nothing about when people are available. The detection time at night is the largest part of A’s recovery time. Add a line to F6: “Outside shop hours, the alert calls the person on duty; Kwame is the backup.” Or, if the shops do not need the assistant at night, say so in Q2, which only counts shop hours.

Knowledge check

Check your understanding

Module completion check: identify a single point of failure in the design and a proportionate mitigation. Say why the mitigation is proportionate, with numbers.

Show the answer

PostgreSQL is a single point of failure: every answer, job, cache hit and permission check needs it. A proportionate mitigation is a point-in-time restore, tested every quarter. It meets Q7 in the pack. The RTO is about 70 minutes against 240 in the base case. The RPO is 5 minutes against 60, an assumption from the documentation that the team must test. It is included in the managed server’s price. High availability would cost about US$288.50 a month of compute against US$13.50. The total would rise to about US$326.90, above the budget of US$150. Q2 allows about 108 minutes of downtime in shop hours a month, so one restore fits. Revisit when Q2 rises above 99.9%.

Check your understanding

Predict: in the base case, how many questions a day would make the busiest minute reach the 70% line of the token quota? Use the month 0 values.

Show the answer

About 6,450 questions a day. In month 0, 1,465 questions a day use 15.90% of the token quota in the busiest minute. With the same daily shape and cache share, the share grows with the questions: 70 ÷ 15.90 × 1,465 ≈ 6,450. That is about 4.4 times today’s base, and almost three times the base case’s month 12 (2,304).

Check your understanding

Tomás says: 'If we build Kubernetes now, we are ready for the high case.' Use the growth output to answer him.

Show the answer

The high case’s first signal is the provider quota in month 0. Its busiest minute needs 104% of the calls and 114% of the tokens. Kubernetes runs more copies of our application, but every copy shares the same provider quota, so it does not help. Our own limits have room. Storage reaches only 6% in month 12, and the cost stays under US$150. The base case has 0.58 requests in flight at the busiest minute. The high case needs a higher quota: 370 requests and 407,000 tokens a minute by month 12. It also needs a queue for customers at the peak.

See it end to endCost and capacity plan from real demand dataCase study

The case study turns a real public demand series into peaks, growth and a cost range, and writes the growth signals for it.

Recap

• A growth plan is a list of steps, each started by a signal crossing its line (its trigger), instead of one guessed final architecture. Put the line below the limit, so the step has time.
• The pack watches three signals: the provider quota at 70%, storage at 80% and the budget of US$150 ([growth] in design.toml).
• Base case: no signal in 12 months. The tokens go from 16% to 25% of the quota, and storage stays at 1% or less. The cost goes from US$51.90 to US$55.86.
• High case: the provider quota from month 0: 104% of the calls and 114% of the tokens, 285% in month 12. Storage reaches 6% and the cost US$107.17. The step: ask for a higher quota early, and queue customers at the peak.
• A tabletop review of two failures found a gap: nobody is on duty at night. Night detection is the largest part of the recovery time.
• python -m pytest -q prints 46 passed. Your pack is now the snapshot end-of-m05.

Module 5 in one paragraph. You asked what fails and how the design grows, with numbers instead of hopes. Six failure scenarios each got a signal, a mitigation and one owner. A five-minute provider outage at the busiest minute costs about 72 answers, 33% of the month’s allowance. PostgreSQL is the single point of failure. High availability would cost US$288.50 a month of compute against US$13.50. So the proportionate answer is a tested point-in-time restore. On paper, it meets Q7: about 70 minutes to recover and 5 minutes of data lost. A restore test must measure both. And the growth plan replaces “design for a million users” with signals. In the base case, nothing needs to change this year. In the high case, the first limit is the provider’s quota, already in month 0.

Next, Module 6 writes the decisions of the proposal as records, prepares the presentation for Omar, and orders the work into a roadmap.

Next: Architecture decisions