Skip to content
Français

Roll it out to an organisation

  • Lesson 21
  • 40 min
  • Module 7: Build, buy and roll out

Where we are

In the last lesson, we compared our gateway with four products and found that the features do not decide alone. But a gateway that nobody moves to protects nothing, and one that blocks everyone on day one makes people go around it. In this lesson, we plan Larkfield's rollout phase by phase, we measure adoption and friction, and we decide build or buy. We start with a new expense system.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Decide whether your organisation needs its own gateway at all. Then roll the platform out in phases: registration first, new agents through the gateway, existing apps one by one, then close the side doors. Give exceptions an owner and an end date, and measure two numbers: adoption and friction.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project as the last lesson left it (194 tests pass), in ~/projects/ai-estate, with the virtual environment active.
  • The estate running (python -m estate up), with the registrations and credentials of the earlier modules.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The month’s numbers are replayed traffic with a simulated bill and egress log.

A company replaces paper expense forms with an app. On Monday, the finance team does not throw the paper forms away. New employees start with the app from their first day. Then each department moves in turn, with a person from finance next to them for a week. One department has a trip abroad that the app cannot handle yet. It gets written permission to use paper until the end of the month, signed and dated.

Finance counts two things every week. How many claims came through the app? And how many people called the help desk because the app refused a claim? When almost every claim comes through the app, finance stops accepting paper.

There are two ways to get a rollout wrong, and Larkfield’s replayed month shows the size of both.

Block everything on day one. In the replayed month, the egress rule came on 12 September, after the last app had moved. Before each move, the apps sent 12,977 requests straight to the provider. If the rule had come on 1 September, every one of those requests would have been blocked. The triage service would have sent its tickets to people, and the workflow would have stopped. This is a what-if, computed from the month’s records. People who see their work break on day one learn one lesson: go around the platform.

Never close the side doors. The digest script ran on 7 September with a personal key, to a provider without a contract. Nothing stopped it until the egress rule. Priya’s personal key made 202 judge calls that nobody saw until discovery. A platform that is only optional is a second way to call models, not a control.

First, the question that comes before any rollout: does this organisation need a gateway of its own?

A gateway pays when several teams share models, providers and budgets. Then one place for keys, limits and records saves each team from building its own, differently. One team with one provider and one or two apps is a different case. The provider’s own project keys, quotas and cost alerts may be enough; check what your provider offers. A managed gateway is the next step. Even then, register every app: the registry is about knowing what runs, not about traffic.

A decision flow. How many teams share models and budgets? One team: how many providers and apps? One provider and one or two apps: the provider's own project keys, quotas and cost alerts may be enough; register every app anyway. Several: a managed or an open-source gateway. Several teams: is there a must-have that no product has? No: may the requests pass through a vendor's service? Yes: a managed gateway, or a self-hosted product. No: a self-hosted product. Yes, and you can staff it for years: build that part yourself, next to a product if you can. Every gateway path ends with: keep the registry, the catalog, the policy and the evidence as your own files.
Do you need your own gateway? Build only the part that no product has, and only if you can run it for years.
Text description of the diagram

Start with the number of teams that share models and budgets. With one team, ask how many providers and apps it has. With one provider and one or two apps, the provider’s own project keys, quotas and cost alerts may be enough. Register every app anyway. With more, use a managed or an open-source gateway. With several teams, ask whether a must-have exists that no product has. If not, ask whether the requests may pass through a vendor’s service. If yes, use a managed gateway or a self-hosted product. If no, use a self-hosted product. Is a must-have missing everywhere, and can you staff the work for years? Then build that part yourself, next to a product if you can. Every gateway path ends at the same box: keep the registry, the catalog, the policy and the evidence as your own files.

Larkfield has four teams and its own hardware for gemma3:4b. It has a contract with one provider in two regions; the second region is simulated in this course. Its model calls through the gateway cost US$21.31 in the replayed month. The workflow spent US$14.28 and triage US$6.48. The judge spent US$0.42, the document assistant US$0.13. Money is not why Larkfield needs a platform. The digest, the refunds and the shared keys are.

The idea: phases, exceptions, adoption and friction

Section titled “The idea: phases, exceptions, adoption and friction”

A phased rollout moves the organisation in steps that each have a measure. Larkfield’s plan is in docs/rollout-plan.md:

Phase What happens What we measure
0. Inventory every AI use: the known apps, the provider bills, the egress log, a scan of the code findings outside the registry
1. Registration first every app fills in the form; high risk also gets a review registrations; time from request to active
2. New agents through the gateway every new agent gets its own credentials and calls only the gateway share of new agents through the gateway
3. Move existing apps one by one each changes its base URL and credential provider calls through the gateway (the bill)
4. Close the side doors revoke the old keys; the egress rule lets only the gateway out calls around the gateway; blocked connections
5. Run it budgets, drills, reviews, the monthly report, exceptions friction: refusals per agent, and why

Registration comes first because it costs a team almost nothing and changes no traffic. New agents come before old apps because they have no old habits to move.

An exception is written permission to skip one rule, for one agent, until a date, with a reason and an approver. It is the paper form for the trip abroad. Without an exceptions process, people make their own exceptions in secret.

Adoption is the share of provider calls that came through the gateway. Count it from the provider’s bill, not from the gateway’s records: the gateway cannot count the calls that go around it. Friction is the share of each agent’s calls that the gateway refused, and why. Adoption shows whether people use the platform. Friction shows what it costs them.

Six phases in order, each with what happened in the replayed month and what to measure. 0, Inventory: 1 to 7 September, every app calls the provider straight; measure what is outside the registry. 1, Register first: 8 September, every app registered with 7 forms; measure registrations and time to active. 2, New agents, the gateway only: every new agent gets its own credentials; measure the share of new agents through the gateway. 3, Move apps one by one: 8 triage, 9 documents, 10 workflow, 11 judge; measure the bill: 75.9% of calls through it. 4, Close the side doors: 12 September, the egress rule; the digest blocked 3 times; measure calls around it and blocked connections. 5, Run it: 16 the remap gate, 17 the outage, 22 the runaway; measure friction: 19 normal calls refused on 22 September.
The rollout by phase, as the replayed month ran it in fast time. A real organisation needs weeks for each phase.
Text description of the diagram

Phase 0, inventory: from 1 to 7 September, every app called the provider straight; the measure is what is outside the registry. Phase 1, register first: on 8 September, every app was registered, 7 forms; the measure is the number of registrations and the time to active. Phase 2, new agents only through the gateway, each with its own credentials; the measure is the share of new agents through the gateway. Phase 3 moves the apps one by one. Triage moved on the 8th, documents on the 9th, the workflow on the 10th and the judge on the 11th. The bill showed 75.9% of the month’s calls through the gateway. Phase 4, close the side doors: the egress rule on 12 September blocked the digest 3 times. The measure is calls around the gateway and blocked connections. Phase 5, run it: the remap gate on the 16th, the outage on the 17th, the runaway on the 22nd. The friction was 19 normal calls refused on the 22nd.

A tiny worked example: adoption from the bill

Section titled “A tiny worked example: adoption from the bill”
  1. Copy this lesson’s files from the snapshot end-of-m07, then run the tests:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m07
    cp $S/docs/rollout-plan.md docs/
    cp $S/estate/adoption.py estate/
    cp $S/estate/cli/adoption.py estate/cli/
    cp $S/tests/test_adoption.py tests/
    cp $S/README.md .
    python -m pytest -q

    The last line says 195 passed, 1 deselected. The new test counts calls through and around the gateway on a small bill.

  2. Open evidence/azure-bill-2026-09.csv, the replayed month’s simulated bill. It has one line per key and deployment. Add up the lines by key:

    Key on the bill Whose key (config/provider-keys.yaml) Requests
    az-key-9 (6 lines, both regions) the gateway’s own key 19,563 + 2,677 + 18,391 + 55 + 13 + 55 = 40,754
    az-key-1 (3 lines) the service team’s old shared key 5,880 + 769 + 5,793 = 12,442
    az-key-2 the data team’s old key 324
    az-key-3 Priya’s personal key: not in the inventory 202
    around the gateway 12,442 + 324 + 202 = 12,968
  3. Let the command do the same:

    Terminal window
    python -m estate adoption --month 2026-09 --bill evidence/azure-bill-2026-09.csv
    Provider calls through the gateway: 40,754 of 53,722 (75.9%); around it: 12,968.
    Friction: the share of each agent's calls that the gateway refused.

The command finds the same numbers as your table. It reads each key’s status in provider-keys.yaml: only a key with status: gateway counts as “through”. So an unknown key, like Priya’s, counts as “around”, which is the safe side. The friction part is empty: your own records have no calls in September. The month’s records are in the report of Module 6.

75.9% is the share for the whole month, and the month is mostly the days after the moves. Per app, the share depends on the day that the app moved:

The replayed month (replayed traffic): each app's requests through the gateway as a share of all its requests, with the day it moved. The judge ran three evaluations, two of them before it moved.
Text description of the chart

Four bars. ticket-triage, moved on 8 September: 76.1% (37,226 of 48,899 requests). doc-assistant, moved on the 9th: 73%. support-workflow, moved on the 10th: 69.7%. release-judge, moved on the 11th: 33.3%, because two of its three evaluations ran before the move, with Priya’s personal key.

Now measure friction on your own records. Make one refusal on purpose: the triage service asks for a capability that its registration does not list, as in Module 4.

  1. Send the request:

    Terminal window
    TOKEN=$(python -m estate token ticket-triage)
    curl -s -X POST http://127.0.0.1:8322/v1/chat/completions \
    -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
    -d '{"model": "strong-reasoning", "messages": [{"role": "user", "content": "Hello"}]}'
    {"error":{"code":"capability_not_allowed","message":"ticket-triage's registration does not list strong-reasoning.","type":"gateway","check":"capability"}}
  2. Run the report without a month, so that it reads all your records:

    Terminal window
    python -m estate adoption --bill evidence/azure-bill-2026-09.csv

    Our run, on a fresh state with three triage tickets and two document questions before the refusal:

    Provider calls through the gateway: 40,754 of 53,722 (75.9%); around it: 12,968.
    Friction: the share of each agent's calls that the gateway refused.
    ticket-triage 1 of 7 refused (14.3%): capability 1
    doc-assistant 0 of 2 refused (0.0%)

    Your lines are longer and your numbers differ: your records hold every call that you made in this course.

Each friction line names the agent, the share refused and the check that refused. capability 1 says what to change: the registration, or the app. A high share is a signal to talk to the team, not to blame it.

Two limits of this number. First, it counts model calls only. A suspended agent gets no token at all, and a tool call refused by the tool gateway is in another table, so neither appears here. Second, a refusal is not always friction. In the replayed month, the workflow had 35,182 refused calls. Of them, 35,163 were the runaway loop, which the limits stopped on purpose. Only 19 were normal work: 19 of 1,787 normal workflow calls (1.1%), refused by the daily quota after the loop had used it up. Those tasks went to people. So read the “why”, and separate the controls that worked from the work that they blocked.

Larkfield: the order of the moves, and one exception

Section titled “Larkfield: the order of the moves, and one exception”

Kwame moved the apps in an order with reasons. Triage first: the most calls, medium risk, and a move that is only a base URL and a credential. The document assistant next: low risk, a small team. The workflow third: high risk, with approvals; its owner Amira had seen triage move first. The judge last: its pin and its evaluation needed Priya’s time.

Mei asks for an exception. The document assistant’s second tenant cannot leave its old key until its tenant tests pass. Kwame records it:

Terminal window
python -m estate exception add doc-assistant --rule "the gateway-only egress rule" --until 2026-11-30 --reason "its second tenant moves after the tenant tests pass"
Exception 1: doc-assistant may skip the gateway-only egress rule until 2026-11-30.

The course ran this on a fresh state. If you kept the exception from Evidence for governance, yours is exception 2, and the list below has both.

Terminal window
python -m estate exception list
1 doc-assistant the gateway-only egress rule until 2026-11-30: its second tenant moves after the tenant tests pass (approved by usr-kwame)

The exception is a record: it names the agent, the rule, the reason, the end date and the approver, and the monthly report lists it. It does not open the network by itself. The network team adds the matching rule, and removes it on the date. The end date is the important field: an exception without one becomes the new rule.

Common mistake

75.9% adoption, but of which bills?

Symptom: the monthly report says that 75.9% of provider calls came through the gateway. Kwame reports it as “three calls in four, at every provider”.

Cause: the command counts only the bills that you give it. Add the Fernway expense claim:

Terminal window
python -m estate adoption --month 2026-09 --bill evidence/azure-bill-2026-09.csv --bill evidence/fernway-expense-2026-09.csv
Provider calls through the gateway: 40,754 of 53,723 (75.9%); around it: 12,969.
Friction: the share of each agent's calls that the gateway refused.

One more call around the gateway: the digest’s call to Fernway. The share hardly moves, but the risk does: that call sent customer messages to a provider without a contract. And some AI use is on no bill at all. The document assistant’s embeddings run on the CPU of its own machine. The three blocked digest runs are only in the egress log.

Diagnose: list every provider that the inventory, the expense claims and the egress log name. Check that each one has a bill in the command.

Recover: pass every bill and expense export with --bill. Report the share together with the list of findings from estate discover, never alone.

Your turn: build or buy, for Larkfield and for a larger company

Section titled “Your turn: build or buy, for Larkfield and for a larger company”

This is the independent task of the module. Write a recommendation of about one page, for two companies:

  • Larkfield: the company of this course, its four teams, its one provider in two regions, its six systems and its platform owner, Kwame.
  • A larger company: 40 teams, three providers, a security team that collects every trace, and data that some countries require to stay in the country.

For each company, answer four questions, with reasons from the comparison sheet, the rollout and the month’s numbers:

  1. Does it need its own gateway, or are the provider’s own controls enough? Use the decision flow.
  2. Build, adopt an open-source product, or buy a managed one? Name the must-haves that decide, and run score_gateways.py with them. If you deleted it, create it again from Gateways compared.
  3. What stays the company’s own, whatever the product?
  4. In which order does it roll out, and which two numbers does it watch?
One possible answer

Larkfield. Four teams share one provider and budgets, so a gateway pays: the provider’s controls alone would not stop the next digest. Larkfield should not run the course’s gateway in production: it is a teaching project with no security review, and Kwame alone would maintain it. With Larkfield’s must-haves, only LiteLLM and our gateway passed. A self-hosted LiteLLM means running PostgreSQL and Redis. Azure API Management fits Larkfield’s one provider, if Larkfield accepts token quotas per capability instead of money budgets. It then redoes them after every remap. Either way, Larkfield keeps its registrations, catalog, remap gate, approval policy and evidence as its own files, and keeps the tool gateway’s approval grants. It rolls out in the order of this lesson and watches the bill’s adoption share and the friction per agent.

The larger company. Many teams and providers: a gateway is needed. OpenTelemetry and guardrails are must-haves, so only LiteLLM passed with the sheet. Its guardrails per team are in the enterprise edition, so it needs a licence. The data-residency rule asks for a self-hosted gateway in each country, or a managed one with regional processing: check the current documentation. No product holds the registry with risk tiers and review dates. So the company builds that part as its control plane, and a platform team runs it. It moves new agents first, one business unit at a time, with exceptions that end, and it reads friction per team every week.

Registration first costs each team a form and a review, and changes no traffic. Start there even if the gateway is months away.

Moving apps one by one costs time: in the replayed month one app a day, in a real company weeks each. It keeps every incident small and teaches each team from the one before.

Closing the side doors costs a network rule and the revoked keys. Do it only when adoption is near complete and the exceptions are written down. Otherwise you break work and teach people to go around.

Exceptions cost a record and a review on their end date. Without the end date, they cost the rule itself.

Adoption and friction cost one command a month. Report them together. High adoption with high friction means that people are forced, not helped.

Check your understanding

The friction line for support-workflow says that 35,182 of its calls were refused in September. Is the platform too strict for Amira's team?

Show the answer

No: read the “why” first. 35,163 of those refusals were the runaway loop on 22 September, stopped by the rate limit and the daily quota, as designed. Only 19 were normal work, refused by the quota after the loop had used it up: 19 of 1,787 normal workflow calls, 1.1%. Those 19 are the real friction. The fix is about the loop, for example a step limit that catches it earlier, not a weaker limit.

Check your understanding

A team asks to keep its old provider key 'for a while' because its app is not ready to move. What does a good exception record, and why is the end date the most important field?

Show the answer

It records the agent, the rule it may skip, the reason, the approver and the end date, as estate exception add does. The monthly report lists it. Someone must also change the real control, for example the egress rule, and change it back on the date. Without an end date, the exception never ends: it becomes a second, silent rule, and the side door stays open.

Check your understanding

Larkfield picks a gateway product. In two years, it wants to change to another one. What would the change cost, and what in the design keeps that cost low?

Show the answer

The cost: each app that calls models changes its base URL and gets a new credential. The platform writes a new translation of its files into the new product. That means the catalog’s mappings and fallbacks, each registration’s keys, budgets and limits, the guardrails and the tool rules. It runs each consumer’s evaluation through the new product before it moves, as the remap gate does. It exports the old usage rows, so that the year’s evidence continues. Then it moves the apps one by one, with exceptions for the late ones, and repeats the kill-switch drill. What keeps it low: the apps speak the OpenAI-compatible API and ask for capability names, never a product’s route or a model. Larkfield’s own files are the source of truth, and the product holds only a translation. And the usage evidence is in Larkfield’s own format.

Recap

  • One team with one provider may need only the provider’s own controls; several teams sharing models and budgets need a gateway. Build only what no product has, and only if you can run it for years.
  • Roll out in phases: inventory, registration, new agents, apps one by one, then close the side doors.
  • An exception names the agent, the rule, the reason, the approver and an end date.
  • Adoption comes from the bills: 40,754 of 53,722 calls (75.9%) in the replayed month. Friction comes from the refusals and their reasons: 19 normal workflow calls, not 35,182.
  • The cost of changing product stays low when the apps ask for capabilities and the knowledge stays in your own files.

Your project now matches the snapshot end-of-m07, the course’s end: 195 tests pass. You can build the platform, compare it, roll it out and measure it. But every number in this course came from Larkfield’s own recordings. Next, in the case studies, you build a capability catalog from public evaluation results that anyone can check.

Next: A capability catalog from public evaluation results