Service boundaries

What you will be able to do

Split the document assistant into a few parts, each with one job and one owner, and draw them on one page. Then every later question has an address: what this part costs, what fails here, and who fixes it.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The five journeys and the eight quality targets, from Users and workflows and Quality attributes.
• What the assistant does for one question: embed, search, rerank, answer with citations, from What RAG is in RAG: Building AI That Uses Your Data.
• A queue and a worker for slow work, from Job lifecycle in Scaling APIs and AI Workloads.

What you need on your computer

• Your folder design-pack at the end of Module 1, with the virtual environment active. python -m pytest -q prints 12 passed. If your pack is broken, copy the snapshot end-of-m01 from the labs folder, as in Set up the design pack.
• The labs folder nextia-learning-labs next to design-pack. Run git -C nextia-learning-labs pull once to get the latest snapshots.
• Optional: draw.io, as the desktop app or at app.diagrams.net in a browser. It needs no account.

Tested with Python 3.12 on macOS (Apple silicon). The Windows (PowerShell) and Linux commands are written, not tested.

A restaurant at lunch time

Think of a small restaurant at lunch time. A waiter takes orders at the front and brings the food. The cook makes the dishes. In the afternoon, a helper peels potatoes and makes soup for tomorrow. The pantry holds the food, and a supplier delivers it every morning.

Each station has one job and one person in charge. The split follows a simple question: who is waiting? A customer waits for a lunch order, so the cook works on it now. Nobody waits for tomorrow’s soup, so the helper makes it in a quiet hour.

Now imagine that the waiter also peels the potatoes, between orders. On a busy day, customers wait at the door while he peels. Nothing is broken, and every job gets done. But the slow job sits in the path of the fast one.

The problem: one part that does everything

Amira’s first sketch of the shared assistant has one program. It answers questions. It also takes uploads, and it reads, chunks, embeds and summarizes each new document before it replies to the upload. It is simple, and for one small file it works.

Then Ines wants to load Larkfield’s whole collection to test the assistant. The course team measured each step of an ingestion, one call at a time. For one Markdown document, the medians are 0.11 ms to parse and chunk it, 0.39 s to embed its chunks and 1.89 s to summarize it. That is 2.28 s per document. For the 36 documents, one after the other, the upload request lasts 36 × 2.28 = 82 seconds.

During those 82 seconds, the program makes 36 embedding calls and 36 summary calls: 72 model calls. Agents ask questions at the same time, through the same program and the same provider. The provider’s quota is 100 chat requests a minute, measured in the scaling course. A slow upload now competes with Grace’s agents for the quota and for the program itself.

And if the upload breaks at document 20, what is saved? Nobody can say, because nobody decided. This is a constructed case: we did not run the 82-second upload, but every number in it is a measured median.

The idea: service boundaries

A service boundary is a line around one part of the system. Inside the line, the part has one responsibility: one job that you can say in one sentence, with one person who owns it. Across the line, other parts ask for that job in a clear way, and they do not reach inside.

Three questions find good lines, as in the restaurant:

1. Does a person wait for this work? Answers are interactive: a person waits seconds. Ingestion is background work: Ines can wait minutes. Work with a different “who waits” belongs in a different part.
2. Which data does it keep? A part that stores data (documents, vectors, answers) has different needs from a part that only computes. It needs backups, a size limit and access rules.
3. Is it ours? The model provider and the sign-in service belong to somebody else. We cannot change them, so we draw them as outside parts and plan for their failures.

The picture of these parts is a container diagram. Here “container” does not mean a Docker container. It means a part that runs on its own, or stores data on its own: an application, a database, a file store. The name comes from the C4 model, a common way to draw software at four levels of zoom. The container level is the one that a product manager and a developer can both read.

A tiny example: sort the jobs

Here are the steps of the two main journeys, with their measured median times. Sort them with the first question: who waits?

Step	Journey	Who waits?	Median time
Embed the question	J1, J2: a question	the agent or customer	0.34 s
Search and rerank	J1, J2	the agent or customer	0.32 s
Write the answer (chat-small)	J1, J2	the agent or customer	1.52 s
Parse and chunk a document	J3: an upload	nobody: Ines checks later	0.11 ms
Embed the chunks (one call per document)	J3	nobody	0.39 s
Summarize the document	J3	nobody	1.89 s

The two groups take about the same time: 2.18 s for an answer and 2.28 s for one document. So the split is not about speed. It is about who waits, and how the work arrives. Questions arrive all day, one at a time, and each person waits. Documents arrive in batches, and nobody stands in front of the screen.

The steps of an answer (blue) and of an ingestion (orange), measured one call at a time. Both journeys take about two seconds; the difference is that a person waits for the first one.

Text description of the chart

Five columns, in seconds. The answer steps, where a person waits: embed the question 0.34, search and rerank 0.32, write the answer 1.52. The ingestion steps, where nobody waits: embed the chunks 0.39, summarize 1.89. Parse and chunk (0.11 ms) is too small to draw. The model calls are the tall columns in both groups. The reranker time comes from the RAG course, on a laptop; the other times are calls to the model provider.

So the answer goes in one part, the Assistant API, and the ingestion goes in another, the Ingestion worker. The API saves an upload and a job, and replies at once. The worker takes the job when it can, as in the scaling course’s queue. Now an upload of 36 documents costs Ines one short request. The 72 model calls move to a part that can slow down when the quota is busy.

Larkfield’s parts

Kwame applies the three questions to the whole assistant, with Amira and Mei. The course outline names five kinds of job: the interactive API, background ingestion, retrieval, model access and storage. They become eight parts, because the people’s tool and the outside services count too.

The assistant at container level. The API and the worker never call each other: the API adds a job row in PostgreSQL, and the worker takes it.

Text description of the diagram

The help-desk tool signs the person in at the identity provider and sends the question with the token to the Assistant API. The API uses three parts in the middle column: the model provider, PostgreSQL and file storage. Its arrows are solid blue, because a person waits for them. The Ingestion worker, on the right, uses the same three parts, with dashed grey arrows, because nobody waits. The API and the worker have no arrow between them. The API writes a job row in PostgreSQL, and the worker reads it. Logs and alerts receive logs from every part. On a phone, the diagram shows the same parts stacked, without the arrow labels.

Each part has one sentence, an owner and a list of what it stores:

Part	Kind	Owner	Its one job
Help-desk tool	client	Amira	shows the assistant to agents, and to customers on the shop’s website
Identity provider	outside	Kwame	signs people in; gives each request a token with its tenant and role
Assistant API	ours	Amira	answers one tenant’s question: checks the token, searches, calls the model, saves the answer
Ingestion worker	ours	Mei	turns an uploaded document version into chunks, vectors and a summary, in the background
PostgreSQL	store	Mei	keeps every record, with row-level security per tenant
File storage	store	Mei	keeps the original file of every document version
Model provider	outside	Kwame	runs the chat and embedding models
Logs and alerts	store	Amira	collects logs and traces, and alerts a person when a signal crosses its line

Where did retrieval and model access go? They are not separate boxes. Retrieval is code that both the API and the worker use, and its data lives in PostgreSQL. Model access is one module that every model call goes through, so that Kwame can change a model in one place. They have boundaries, but inside the application, not between machines. The lesson Simplicity and evolution explains why.

Tomás asks: “Why one PostgreSQL for records, vectors, the job queue and the answer cache? Isn’t that four jobs?” (The answer cache keeps saved answers, so that a repeated question needs no model call. The next lesson shows what its key must hold.) Mei answers that it is one job: keep the data, safely, for each tenant. One store means one backup, one restore test and one place for the tenant rule. The databases course measured a vector search over 10,000 vectors in about 0.22 ms. Our base case stores about 18 MB in its first month, as Module 3 calculates. A second store would double the work for no measured gain.

Guided practice: list the parts in the pack

The pack checks the architecture as it checks the prices: every part needs a responsibility and an owner, and it must be on the diagram. You add the check, then the parts, then the diagram.

1. Try the command first. It does not exist yet in your pack:

   python -m costmodel components

   This command arrives in Module 2. Download the end-of-module-2 snapshot or keep going.

2. Copy the check and its tests from the snapshot end-of-m02. You read them; you do not write them.

   macOS / Linux

   cp ../nextia-learning-labs/C26/snapshots/end-of-m02/costmodel/components.py costmodel/
   cp ../nextia-learning-labs/C26/snapshots/end-of-m02/tests/test_components.py tests/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C26\snapshots\end-of-m02\costmodel\components.py costmodel\
   Copy-Item ..\nextia-learning-labs\C26\snapshots\end-of-m02\tests\test_components.py tests\

3. Run the command again. Now it asks for the list of parts:

   Input problem: components.toml is missing (expected at …/design-pack/architecture/components.toml)

4. Make the folder architecture with mkdir architecture; the command is the same in both shells. In it, create a new file components.toml, and write the first part:

   # Every part of the design: one responsibility, one owner, what it stores.
   # kind: client | app | store | external. The check: python -m costmodel components
   
   [components.api]
   name = "Assistant API"
   kind = "app"
   responsibility = "Answers one tenant's question: checks the token, searches, calls the model, saves the answer."
   owner = "Amira"
   stores = ""

   stores is empty on purpose: the API keeps nothing itself. Everything it saves goes to PostgreSQL.

5. Add the other seven parts from the table above, one [components.<key>] table each. Use the keys client, identity, worker, database, files, models and monitoring. Write what each part stores, for example "uploaded files, one folder per tenant". The command lists the parts in the order of the file, so your rows can come in another order than the output below.

6. Run the command. The parts are complete, but the diagram does not exist yet. The last line says:

   8 components, each with one responsibility and an owner; NOT on the diagram: Help-desk tool, Identity provider, Assistant API, Ingestion worker, PostgreSQL, File storage, Model provider, Logs and alerts

   The lines above it are the table of the eight parts. The check reads the diagram’s source file, architecture/container.drawio, and looks for each part’s name.

7. Draw the diagram in draw.io, with one box per part and the arrows of the diagram above. Start each label with the part’s name. Save it as architecture/container.drawio. Then use File › Export as › SVG to save architecture/container.svg next to it. If you have no draw.io, copy both files from the snapshot instead:

   macOS / Linux

   cp ../nextia-learning-labs/C26/snapshots/end-of-m02/architecture/container.* architecture/

   Windows (PowerShell)

   Copy-Item ..\nextia-learning-labs\C26\snapshots\end-of-m02\architecture\container.* architecture\

8. Run the command again. The output is now:

     component              kind       owner    responsibility
     Help-desk tool         client     Amira    Shows the assistant to agents and to customers on the shop's website.
     Identity provider      external   Kwame    Signs people in and gives each request a token with its tenant and role.
     Assistant API          app        Amira    Answers one tenant's question: checks the token, searches, calls the model, saves the answer.
     Ingestion worker       app        Mei      Turns an uploaded document version into chunks, vectors and a summary, in the background.
     PostgreSQL             store      Mei      Keeps every record of the system, with row-level security per tenant.
     File storage           store      Mei      Keeps the original file of every document version.
     Model provider         external   Kwame    Runs the chat and embedding models that the API and the worker call.
     Logs and alerts        store      Amira    Collects logs and traces, and alerts a person when a signal crosses its line.
   8 components, each with one responsibility and an owner; all are on the container diagram

9. Run the tests:

   python -m pytest -q

   FAILED tests/test_components.py::test_the_sequences_name_the_parts_of_the_diagram
   FAILED tests/test_components.py::test_the_tenant_travels_with_every_flow - Fi...
   2 failed, 14 passed in 0.05s

   This is expected. The two failing tests read the flow files that you write in the next two lessons. The two tests of this lesson pass.

Compare your components.toml with the one in the snapshot end-of-m02. Your sentences can differ. Check that each part has one job, that no job appears in two parts, and that every name matches a box on your diagram.

When to draw boundaries, and what they cost

Draw the parts before you estimate any cost. The cost model in Module 3 has one line per driver. Each driver belongs to a part: the API’s compute, the database server, the model tokens. A part without a line in the cost model is a part you forgot to price.

Do not draw a box for every function. Eight parts fit on one page, and Omar can read them. If your diagram needs a long legend, the parts are too small. Code modules, such as search, belong inside a box, not as boxes of their own.

Each boundary has a cost too. Two processes need two ways to start and two sets of logs. The job row between the API and the worker is one more thing that can get stuck. So the worker needs the retries and status checks of the scaling course. For Larkfield, that cost is small and the gain is clear: an upload never blocks an answer.

Common mistakes

Common mistake

A part that is not on the diagram

Symptom: python -m costmodel components ends with NOT on the diagram: Monitoring, and the tests print 3 failed, 13 passed. The first failure is test_every_component_has_a_job_an_owner_and_is_on_the_diagram.

Cause: someone renamed “Logs and alerts” to “Monitoring” in components.toml, but not on the diagram. Now the list and the picture describe two different systems, and a reader trusts the wrong one.

Fix: use one name in both places. Change the label on the diagram, or change the name back. Run the command until the last line says all are on the container diagram.

Common mistake

A part with no owner

Symptom: the command stops with Input problem: component 'worker' has no owner.

Cause: the owner of the worker is an empty string. Perhaps nobody wanted it. A part without an owner is the part that nobody watches when its queue grows.

Fix: decide the owner with the team, and write the name. For the worker, Mei owns it, because it writes the data that she backs up. If two people share a part, name the one who answers the alert.

Your turn: Tomás’s search service

Change one thing. Tomás adds a ninth part at the end of components.toml:

[components.search]
name = "Search service"
kind = "app"
responsibility = "Searches a tenant's chunks."
owner = "Amira"
stores = ""

Predict what the command and the tests say. Then run them. Should the search be its own part? Use the three questions.

Show the answer

The command’s last line is 9 components, each with one responsibility and an owner; NOT on the diagram: Search service. The tests print 3 failed, 13 passed, and the first failure shows assert 9 == 8: the pack’s test expects Kwame’s eight parts.

Now the three questions. Who waits? The same person who waits for the answer: search is one step of the answer, 0.32 s of 2.18 s. Which data? None of its own: the chunks and vectors are in PostgreSQL. Is it ours? Yes, but so is the API. So search has no reason of its own to run apart. It is a module that the API and the worker both call, inside the application. A separate service would add a network call to every answer and one more thing to deploy. Remove the ninth table. The lesson Simplicity and evolution lists the measured signals that would change this answer.

Knowledge check

Check your understanding

Why does the upload of a document go to a background worker, when one ingestion (2.28 s) takes about as long as one answer (2.18 s)?

Show the answer

Because the split is about who waits, not about speed. A person waits for each answer. Nobody waits for an ingestion: Ines checks later. Documents also arrive in batches: 36 documents meant 82 s of model calls in one request, in the constructed case. With the worker, the upload returns at once. A batch of uploads never holds a request open, and the worker can slow down when the model quota is busy.

Check your understanding

Predict: you rename 'File storage' to 'Blob storage' in components.toml only. What does python -m costmodel components print on its last line, and why does the pack treat this as a problem?

Show the answer

8 components, each with one responsibility and an owner; NOT on the diagram: Blob storage. The list and the diagram now disagree, so one of them is wrong, and a reader cannot tell which. The pack checks the names because Omar and Grace read the diagram, and the cost model and the failure table use the list.

Check your understanding

Practical check: for three parts of your choice, say the one job, the owner, what it stores, and why it exists as its own part.

Show the answer

For example:

• Ingestion worker (Mei): turns an uploaded document version into chunks, vectors and a summary. It stores nothing itself. It exists because nobody waits for ingestion, and it must not block answers.
• PostgreSQL (Mei): keeps every record, with row-level security per tenant. It stores tenants, document versions, chunks, vectors, answers, the job queue and the answer cache. It exists because the data needs one place with backups and one tenant rule.
• Model provider (Kwame): runs the chat and embedding models. We keep nothing there. It is its own part because it belongs to somebody else: we plan for its quota and its outages instead of changing it.

A good answer gives a reason from the three questions: who waits, which data, and whether it is ours.

Recap

• A service boundary is a line around a part with one responsibility and one owner. Three questions find it: does a person wait, which data does it keep, and is it ours?
• An answer (2.18 s) and an ingestion (2.28 s) take about the same time. They are still two parts, because a person waits for one and nobody waits for the other.
• Larkfield’s container diagram has eight parts: the help-desk tool, the identity provider, the Assistant API, the Ingestion worker, PostgreSQL, file storage, the model provider, and logs and alerts.
• Retrieval and model access are modules inside the application, not boxes of their own.
• python -m costmodel components checks that every part has a job and an owner and is on the diagram. The tests print 2 failed, 14 passed until you write the flow files.

Next, you follow the tenant through every arrow of this diagram.

Next: Data and identity flows