Build versus buy

What you will be able to do

Decide whether to run a part yourself or to pay a provider to run it. Put the whole cost on the table: the machine at its list price, and the people’s time as a stated assumption. You compare a managed database with a server you run, and a hosted model with one on your own GPU. Then you finish the module with a fair comparison of three designs.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The cost-driver table, from Cost model, and fixed and variable costs, from Unit economics.
• The option matrix, from Model routing, and the cache’s assumption, from Caching and batching.
• What a GPU server for a model involves, from CPU, GPU and managed endpoints in Deploying AI Models for Real Users. It helps, but it is not required.

What you need on your computer

• Your folder design-pack from the last lesson, with python -m pytest -q printing 35 passed.

No local model runs in this lesson. Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

Bread from the shop or from your oven

You can buy a loaf of bread for 4 dollars, or bake one at home. The flour, yeast and power cost about 1 dollar. So home bread is 75% cheaper, if you only count the ingredients.

But a loaf takes about three hours, with one hour of your work: mixing, shaping, cleaning up. If your hour is worth 20 dollars, the home loaf costs 21 dollars. And if the oven breaks, you fix it. Home baking pays when you bake many loaves at once, when you love it, or when no shop sells the bread you need. For one loaf a week, the shop wins.

Every build versus buy question looks like this. “Build” means that you run the part yourself on machines that you rent. “Buy” means that a provider runs it and sells you the result. The price list shows the ingredients. Your own time is not on any price list.

The problem: the cheaper hour

Tomás has two proposals, each with a real list price.

“A Linux virtual machine B2s costs US$0.0464 an hour. Managed PostgreSQL of the same size, B2s, costs US$0.074 an hour. The VM is 37% cheaper. Let’s run PostgreSQL ourselves.”

“And the model: one GPU machine with an NVIDIA T4 costs US$0.584 an hour, and then the answers are free. No token bill, ever.”

Both statements are true, and both lead to a bigger bill. The first one forgets the disk and the work that a managed service does for you. The second one forgets that “free per answer” needs a machine that runs all month, and people to keep it running. To see this, the comparison needs every part of the cost.

The idea: the whole cost, with people

The total operating cost of an option is its machines at list prices plus the people’s time to run it. People’s time is an assumption: hours per month × a cost per hour. Nobody can measure Amira’s hours before the system exists, so the pack states them as a range, with the reason for each value. You set your own rate.

A managed service is not free of work. Somebody still reads the alerts and tests a restore. But the provider does the rest: the operating system, the updates, the backups, the point-in-time restore.

Who does what. Buying a managed database moves most of the operating work to the provider; building moves it to your team. The hours are the pack's assumptions.

Text description of the diagram

On the left, managed PostgreSQL. The provider runs the server, its operating system and updates, the backups and the point-in-time restore. Mei reads the alerts and tests a restore every quarter: about 1 hour a month, an assumption. On the right, PostgreSQL on a VM that the team runs. The provider gives only the machine and the disk. Mei and Amira do the operating system, the PostgreSQL updates, the backups, the monitoring and the restore tests: about 6 hours a month, an assumption.

A second idea helps with the model: the break-even point. It is the volume at which two options cost the same. A hosted model costs per token: zero when nobody asks, more with every question. Your own GPU costs the same every month, busy or not. Below the break-even point, the hosted model is cheaper; above it, your own GPU. But only if one GPU can do that much work.

A tiny example: the database

Here is the database comparison with every value visible. The prices are list prices for Canada Central; the hours and the rate (US$60 an hour) are the pack’s base assumptions.

Buy: managed PostgreSQL, B1ms (1 vCore, 2 GiB, the design’s choice since Module 3)

• Machine: 0.0185 × 730 hours = 13.50, plus 32 GB of storage × 0.1265 = 4.05. Total US$17.55.
• People: 1 hour × 60 = US$60.00.
• Total: US$77.55 a month.

Build: PostgreSQL on a VM, B2s (2 vCPUs, 4 GiB, with a 128 GiB premium disk P10)

• Machine: 0.0464 × 730 = 33.87, plus the disk 19.71. Total US$53.58.
• People: 6 hours × 60 = US$360.00.
• Total: US$413.58 a month.

The VM’s hour is cheaper, but the managed server is smaller, and the disk alone costs more than the managed storage. Even at the same size, a managed B2s costs 0.074 × 730 + 4.05 = US$58.07, close to the VM’s US$53.58. The difference that matters is the people: five more hours a month, US$300.

Guided practice: build versus buy in the pack

1. In your folder design-pack, with the environment active, copy the last code file of the module and the module’s tests.

   macOS / Linux

   S=../nextia-learning-labs/C26/snapshots/end-of-m04
   cp "$S/costmodel/buildbuy.py" costmodel/
   cp "$S/tests/test_options.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C26\snapshots\end-of-m04"
   Copy-Item "$S\costmodel\buildbuy.py" costmodel\
   Copy-Item "$S\tests\test_options.py" tests\

   Open the section [labour] at the end of design.toml. Each value is a range (low, base, high), and each one has a reason in [labour.evidence] that starts with assumed:.

2. Compare the options in the base case:

   python -m costmodel buildbuy

   Build versus buy (base; machine = list prices, people = your labour assumptions)
     option                               machine    people     total  US$ per month
     managed PostgreSQL (B1ms)              17.55     60.00     77.55
     PostgreSQL on your own VM (B2s)        53.58    360.00    413.58
     hosted model (chat-small, tokens)       7.52      0.00      7.52
     own model on a T4 GPU VM              426.32    600.00  1,026.32
   The own model pays for itself above 6,000,994 questions a month (now: 43,950); one GPU writes at most about 509,067 answers a month at the ASSUMED speed, and its quality is not measured.

   The first two rows are the tiny example. The hosted model’s line is the answer tokens of all 43,950 questions; it does not take off the cache’s hits.

3. Look at the high case. The pack’s high case also takes the high labour values. That means more hours, a rate of US$100 and a faster GPU, 60 tokens per second.

   python -m costmodel buildbuy --scenario high

   Build versus buy (high; machine = list prices, people = your labour assumptions)
     option                               machine    people     total  US$ per month
     managed PostgreSQL (B1ms)              17.55    200.00    217.55
     PostgreSQL on your own VM (B2s)        53.58  1,200.00  1,253.58
     hosted model (chat-small, tokens)      28.63      0.00     28.63
     own model on a T4 GPU VM              426.32  2,000.00  2,426.32
   The own model pays for itself above 14,186,932 questions a month (now: 167,400); one GPU writes at most about 1,018,134 answers a month at the ASSUMED speed, and its quality is not measured.

4. Use your own rate. In design.toml, change the rate to a range around your team’s real cost of one hour, for example:

   rate_per_hour = [20, 30, 50]

   Run python -m costmodel buildbuy again:

   Build versus buy (base; machine = list prices, people = your labour assumptions)
     option                               machine    people     total  US$ per month
     managed PostgreSQL (B1ms)              17.55     30.00     47.55
     PostgreSQL on your own VM (B2s)        53.58    180.00    233.58
     hosted model (chat-small, tokens)       7.52      0.00      7.52
     own model on a T4 GPU VM              426.32    300.00    726.32
   The own model pays for itself above 4,246,864 questions a month (now: 43,950); one GPU writes at most about 509,067 answers a month at the ASSUMED speed, and its quality is not measured.

   Half the rate halves the people column, and the order of the options does not change. Change the line back to [40, 60, 100].

5. Run the tests and the style check:

   python -m pytest -q
   ruff check .

   ..........................................                               [100%]
   42 passed in 0.07s
   All checks passed!

   Your pack is now the snapshot end-of-m04. One of the new tests is called test_the_cheapest_machine_is_not_the_cheapest_total: it checks that the people’s time is the biggest part of running your own database.

Read the comparison

The four totals of the base case, side by side:

Total operating cost in the base case: buy or build the database (managed PostgreSQL or your own VM) and the model (hosted tokens or your own T4 GPU). List prices plus people's time at an assumed US$60 an hour. Building costs more in both pairs, mostly in people.

Text description of the chart

A column chart with four columns. Managed PostgreSQL costs US$77.55 a month, machine and people. PostgreSQL on your own VM costs US$413.58. The hosted model’s tokens cost US$7.52. Your own model on a T4 GPU machine costs US$1,026.32. In each pair, the “build” column is much taller.

The database. Buying costs US$77.55 a month, building US$413.58. With your own rate, the numbers change, but the order stays. Even with zero hours, the VM’s machine line (US$53.58) is above the managed one (US$17.55) for these sizes. Kwame keeps the managed database. He writes the reason with its assumption: “about 5 more hours a month of our time, and a restore that we would build ourselves”.

The model. The hosted model costs US$7.52 a month in tokens. A T4 machine costs US$426.32 a month before anyone works on it. Its true cost per answer is easy to miss. At the assumed 30 tokens per second, one GPU that writes all month long gives about 509,000 answers. That is 426.32 ÷ 509,067 = US$0.00084 per answer for the machine alone, five times chat-small’s US$0.000171. Even at 60 tokens per second, it would be US$0.00042, more than twice the hosted price.

The break-even point needs a picture, because the volumes span three powers of ten:

• Buy model
• Own model, one GPU
• Own model, enough GPUs

Monthly cost of the answers at volumes from 10,000 to 15.8 million questions, both axes on a log scale. On paper, one GPU breaks even at about 6.0 million questions. But one GPU writes only about 509,000 answers a month at the assumed speed. With enough GPUs, the own model stays above the hosted one.

Text description of the chart

A line chart with both axes on a log scale. The x axis shows the questions a month, from 10,000 to about 15.8 million; the base case today is 43,950. The hosted model’s line rises steadily: US$1.71 at 10,000 questions, US$171 at 1 million, about US$2,711 at 15.8 million. The dashed line for one T4 GPU is flat at US$1,026.32 and crosses the hosted line at about 6.0 million questions. But one GPU writes at most about 509,000 answers a month, so a third line adds a GPU for each 509,000 questions: US$1,026.32 up to 509,000, US$1,452.64 at 1 million, and US$6,142.16 at 6.3 million. That third line stays above the hosted line at every volume.

The dashed line shows the paper answer: the own model “pays for itself above 6.0 million questions a month”. The design has 43,950, about 137 times fewer. And the paper answer is not reachable: at 6 million questions, you need 12 GPUs, not one. With enough GPUs, the own model costs more than the hosted model at every volume in the chart, with the assumed speed and labour. Its quality is also unknown: the course measured only chat-small and chat-strong.

So self-hosting is not about price at this size. Other reasons can make it worth a measurement: data that must not leave your network, or a model that no provider sells.

When to build and when to buy

Buy a part that is not what makes your product different, when a provider runs it well at your size. Here, that is the database and the model. The managed price includes work that you would otherwise do and maybe forget, such as tested backups.

Build a part when the provider cannot meet a requirement, or when the part is the product itself. Large, steady volume can also justify it, if it keeps your machines busy. Then measure your own throughput first; do not take a speed from a blog.

The comparison itself is cheap: one table and one hour of honest assumptions. Write the people’s hours and the rate next to the result, so that Omar can change them. And write a trigger to look again. For the model, it is a volume near the break-even point. For the database, it is a need that the managed service cannot meet.

Common mistakes

Common mistake

A range in the wrong order

Symptom: you set the rate to [40, 30, 100], and python -m costmodel buildbuy prints:

Input problem: 'rate_per_hour': low <= base <= high is not true for [40, 30, 100]

Cause: every range in the pack is low, base, high. Here the base (30) is below the low value (40).

Fix: write the three values in order, for example [20, 30, 50]. The check exists so that a typing mistake cannot turn the low case into the expensive one.

Common mistake

The cheaper hour that costs more

Symptom: the proposal moves the database to a VM to save 37% per hour. After three months, the team has spent 18 hours on updates and a failed backup.

Cause: the comparison counted only the machine’s hourly price. It left out the disk (US$19.71 a month) and the people’s time.

Fix: compare total operating costs: machine plus people, each with its source. Put the people’s hours in the design file as an assumption with a range, as [labour] does.

Reset. If your pack breaks, copy the snapshot end-of-m04 as the snapshots README shows.

Module practice: compare three designs fairly

This module compared options one at a time. A proposal compares whole designs, against the same workload and the same targets. The rule of a fair comparison: one workload, one quality check, one price list, and the people’s time for every option.

A worked example: three designs, base case. The targets are Q4, at least 85% correct, and Q1, a p95 of at most 6 s for the whole answer. Q8 limits the cloud bill to US$150 a month. The operating cost adds the people for the database.

• A, the course-end design: chat-small, 10% cache hits, managed PostgreSQL. Cloud bill US$51.90 (costs). People 60.00. Operating cost 51.90 + 60.00 = US$111.90.
• B, routed: the same with model = "routed". Cloud bill US$77.89 (options). People 60.00. Operating cost US$137.89.
• C, own database: design A with PostgreSQL on a VM. Cloud bill 51.90 − 17.55 + 53.58 = US$87.93: the managed server and storage out, the VM and disk in. People 360.00. Operating cost US$447.93.

Design	Q4 correct	Q1 p95 estimate	Q8 cloud bill	Operating cost
A: chat-small, managed DB	86.6%: meets	4.59 s: meets	US$51.90: meets	US$111.90
B: routed, managed DB	86.6%: meets	4.82 s: meets	US$77.89: meets	US$137.89
C: chat-small, own DB	86.6%: meets	4.59 s: meets	US$87.93: meets	US$447.93

All three meet every target, so the choice is the cost: A. Note what the table does not show. Design C also needs a backup and restore plan that the team builds. Recovery objectives in Module 5 plans the restore.

Your task. Do the same comparison for the high case, month 0, with the pack’s high assumptions. Use python -m costmodel options --scenario high and python -m costmodel buildbuy --scenario high. For design C, take the managed database’s machine line out and the VM’s line in, as above. Say which designs meet each target, and which one you recommend.

Show a worked solution

The commands give the cloud bills and the people:

Model routing: the same 67 questions, the same search results (measured), high workload (cost model)
  design                    correct  US$/question   p50 s   p95 s  US$/month
  chat-small only             86.6%      0.000171    1.52    3.79      75.81
  chat-strong only            85.8%      0.003197    2.48    6.56     448.12
  routed (small first)        86.6%      0.000815    1.76    4.02     155.07
Cheapest design with the best measured quality: chat-small only

From buildbuy --scenario high: managed database people US$200.00, own database people US$1,200.00, machines 17.55 and 53.58.

Design	Q4	Q1	Q8 cloud bill	Operating cost
A: chat-small, managed DB	86.6%: meets	4.59 s: meets	US$75.81: meets	75.81 + 200.00 = US$275.81
B: routed, managed DB	86.6%: meets	4.82 s: meets	US$155.07: fails	155.07 + 200.00 = US$355.07
C: chat-small, own DB	86.6%: meets	4.59 s: meets	75.81 − 17.55 + 53.58 = US$111.84: meets	111.84 + 1,200.00 = US$1,311.84

Design B now fails the budget: in the high case, the strong model’s share of the tokens pushes the cloud bill over US$150. Design C stays within the budget, because Q8 counts only the cloud bill, but its operating cost is almost five times A’s. The recommendation is still A.

One more number belongs in the answer. In the high case, design A’s busiest minute needs 104.2 chat calls against a quota of 100 (workload --scenario high). Design B would add chat-strong calls on top. No design in this table solves the quota; Growth plan in Module 5 plans that.

Knowledge check

Check your understanding

Module completion check: explain, with two examples from this module, why the cheapest per-call or per-hour option may not be the lowest total operating cost.

Show the answer

A price per call or per hour leaves out the fixed parts and the people. First, your own model on a T4 GPU has no token bill, so each answer is “free”. But the machine costs US$426.32 a month whether anyone asks or not, and people’s time adds about US$600. That is US$1,026.32, against US$7.52 of hosted tokens for the same 43,950 questions. Second, a B2s VM costs US$0.0464 an hour, 37% less than a managed B2s server. But the VM needs a disk (US$19.71) and about 6 hours of work a month. That is US$413.58, against US$77.55 for the managed database. The total operating cost adds machines, fixed amounts and people over a month, at the real volume.

Check your understanding

Predict: you lower the labour rate to US$0 an hour (the team works for free). Does the own database on a VM become cheaper than the managed one? Use the machine lines.

Show the answer

No. With no people’s cost, the totals are the machine lines. The B2s VM with its P10 disk costs US$53.58; the managed B1ms with 32 GB of storage costs US$17.55. The managed option is still cheaper for these sizes. A smaller VM and disk could change that, but the people’s time is the bigger difference in any real case.

Check your understanding

Tomás says: 'The pack says the own model pays for itself above 6.0 million questions a month. When we reach that, we switch.' What is missing from his plan?

Show the answer

Two things. First, capacity. One T4 GPU writes at most about 509,000 answers a month at the assumed 30 tokens per second. So 6 million questions need about 12 GPUs, and then the cost is far above the hosted model. At the assumed speed, the GPU’s machine cost alone is US$0.00084 per answer, five times chat-small’s tokens. Second, the speed and the quality are assumptions: nobody has measured a self-hosted model on Grace’s questions. The plan needs a measured speed and a quality run before any switch.

See it end to endDesign review of an open document-question serviceCase study

The case study chooses routing and caching for a real open document collection with measured numbers, and writes the decision records.

Recap

• Build versus buy: compare the total operating cost: machines at list prices plus people’s time, an assumption with a range ([labour] in design.toml). Set your own rate.
• Database, base case: managed B1ms US$77.55 (17.55 + 60.00) against PostgreSQL on a B2s VM US$413.58 (53.58 + 360.00). The VM’s hour is cheaper; the total is not.
• Model: hosted tokens US$7.52 against a T4 GPU US$1,026.32 (426.32 + 600.00). On paper, break-even at about 6.0 million questions a month. One GPU writes only about 509,000 at the assumed speed, and nobody measured its quality.
• A fair comparison of designs uses one workload, one quality check, one price list and the people’s time. The course-end design A, chat-small with a managed database, meets Q1, Q4 and Q8 at the lowest cost. The routed design fails Q8 in the high case.
• python -m pytest -q prints 42 passed. Your pack is now the snapshot end-of-m04.

Module 4 in one paragraph. You tested three ways to change the biggest lines of the bill, each with a measurement. A stronger model did not improve Grace’s 67 answers: chat-small 86.6%, chat-strong 85.8%. Routing changed no result and added 50% to the monthly cost. The misses come from search, documents and prompts, not from the model. An answer cache costs almost nothing and saves little money. With an assumed 10% of hits, the base case is US$51.90. Its real value is headroom under the provider’s quota. Batching made embeddings 6.4 times faster for the same price. And building your own database or model costs more than buying, once people’s time is on the table. The course-end design is chat-small, a cache, and managed services.

Next, Module 5 asks what fails: a provider outage, a slow database, a partial ingestion. It sets recovery targets and a growth plan with triggers.

Next: Failure scenarios