Skip to content
Français

One door for model calls

  • Lesson 10
  • 40 min
  • Module 4: The model gateway

Where we are

In Module 3, we registered every app, gave each one its own identity, and built a first gateway that refuses a call without a valid registration. But that gateway only says yes or no: it records nothing, limits nothing and does not look at the data. In this lesson, we build the whole path of the model gateway, with eleven checks and one usage record for every call. We start in a company's post room.

What you will be able to do

Put the whole model gateway in front of Larkfield’s apps. It is one door that checks every call in a fixed order, sends it to a model and records it. The apps keep the API they already use and change one setting.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project ~/projects/ai-estate as Module 3 left it: the same as the snapshot end-of-m03, with its .venv, every app registered and the credentials of the five apps issued. If you start here, follow the README of end-of-m03: “Start the estate”.
  • The labs repository in ~/projects/nextia-learning-labs, for the snapshot end-of-m04.
  • Two terminals, with the virtual environment active in both. Ports 8320 to 8322 must be free.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them.

Think of a company where every letter leaves through one post room. A clerk checks that the sender works there. She checks that the sender’s department may send this kind of letter, and that a letter with a bank card inside does not leave. She weighs it, picks a courier and writes one line in a ledger: who sent it, which courier, what it cost.

The staff do not change how they write letters. They only drop them in a new tray. And at the end of the month, the ledger says what each department sent and spent. The courier’s bill says only one thing: “the company sent 4,000 letters”.

The problem: a door that remembers nothing

Section titled “The problem: a door that remembers nothing”

The first gateway of Module 3 is a door with a guard, but no ledger. Start the estate and send two apps through it, as you did in that module:

Terminal window
python -m estate up
python -m apps.replay ticket-triage --units 5 --via gateway
python -m apps.replay support-workflow --units 2 --via gateway
python -m provsim bill

The bill, simulated from your replayed requests:

month,provider,key_id,deployment,requests,tokens_in,tokens_out,cost_usd
2026-10,azure,az-key-9,chat-small,5,9594,2025,0.001972
2026-10,azure,az-key-9,chat-strong,6,14776,738,0.036932
2026-10,azure,az-key-9,embed-small,5,424,0,0.000008

Every line now says az-key-9: the gateway’s own key. Before the gateway, the bill at least split the calls by key. Now it cannot say which app made a call at all. Ask the estate instead:

Terminal window
python -m estate usage
python -m estate: error: argument command: invalid choice: 'usage' (choose from init, up, down, status, serve, reset, catalog, remap, check, register, list, show, suspend, resume, retire, drain, review, review-due, events, credentials, token, config)

There is no record. The first gateway also has no limit, so an agent in a loop could call a model all night. And it never looks at the data in a request. Stop the estate before you change it: python -m estate down.

An AI gateway is a service that every model call of an organisation passes through. It is the post room: it checks the caller, decides where the call goes, and records it. This course calls it the model gateway, because Module 5 adds a second one for tools.

The gateway runs the same checks on every request, always in the same order. The first check that fails answers, and the request goes no further. Every request, answered or refused, gets one row in the usage store: the gateway’s ledger.

A flow, top to bottom. The app sends a request with a capability, its token and its data. Check 1, configuration: no snapshot gives 503 control_unavailable. Check 2, identity: no valid token gives 401. Check 3, registration: not registered or suspended gives 403. Check 4, capability: not in the registration or not in the contract gives 403 capability_not_allowed or 400 feature_not_in_contract. Check 5, data class: not allowed gives 403 data_class_not_allowed. Check 6, kill switches: a switch gives 403 stopped. Check 7, failure mode: if a control cannot run and the risk is high, 503 control_unavailable. Check 8, limits: over a limit gives 429 rate_limited, run_step_limit, quota_exhausted or budget_exhausted. Check 9, guardrails: a secret gives 400 blocked_by_guardrail; a card number is redacted. Check 10, route: no provider answers gives 503 no_provider_available. Otherwise 200 with headers that name the model. Check 11, record: one usage row for every request.
The model gateway's path. Each check either passes the request on or answers with its own code, and every request gets a usage row.
Text description of the diagram

The app sends a request with a capability, its token and its data. Check 1 asks whether the replica has its configuration. Check 2 checks the token, and check 3 the registration. Check 4 asks whether the registration lists the capability and whether the capability’s contract has what the request needs, such as tools. Check 5 checks the data class. Check 6 asks whether a kill switch stops the call. Check 7 asks whether every control can run, and what to do if one cannot. Check 8 compares the agent’s use with its limits. Check 9 looks for secrets and card numbers in the text. Check 10 sends the request to the mapping’s provider, or to a fallback. The answer carries headers that name the model. Check 11 writes one usage row, for a refusal too.

The order is not random. Each check needs what the checks before it found out. The gateway cannot look up a registration before it knows who is calling. It cannot apply an agent’s limits before it knows which agent it is. The cheap checks come first, and the one that costs money, the provider call, comes last. So a refused request costs a fraction of a millisecond and nothing on the bill.

This lesson builds the whole path. You use checks 1 to 4 and 10 to 11 now. The limits are the next lesson, the data classes and guardrails the one after, and the failure mode the last lesson of this module.

Keep the interface that the apps already use

Section titled “Keep the interface that the apps already use”

The gateway speaks the same API as the provider: OpenAI-compatible chat completions and embeddings. So an app does not change its code. It changes three values. The base URL is the gateway’s, not the provider’s. The credential is its own token, not a provider key. The model name is a capability, such as strong-reasoning, not a deployment.

The support workflow of the agents course shows this. Its model calls go through one adapter class. This lesson adds a small file, resolver/platform.py, with an adapter that points the same OpenAI client at the gateway with the workflow’s token. One environment variable, RESOLVER_PROVIDER=gateway, chooses it. Nothing else in the workflow changes.

The whole gateway arrives at once: its code for every check, the usage store and the gateway’s settings. The next lessons switch parts of it on and explain them.

  1. Copy this lesson’s files from the snapshot end-of-m04. In ~/projects/ai-estate:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m04
    cp $S/gateway/app.py $S/gateway/guardrails.py $S/gateway/telemetry.py gateway/
    cp $S/estate/usage.py $S/estate/switches.py estate/
    cp $S/estate/cli/usage.py estate/cli/
    cp $S/config/gateway.yaml config/
    cp $S/resolver/platform.py $S/resolver/config.py $S/resolver/__main__.py resolver/
    cp $S/apps/client.py apps/
    cp $S/tests/test_usage.py tests/

    The new apps/client.py gives each replay its own run IDs, so that a replay of the same tickets is a new run.

  2. Run the tests:

    Terminal window
    python -m pytest -q

    The last line says 137 passed, 1 deselected. The two new tests check the usage reports and the counters.

  3. Open config/gateway.yaml in your editor. It holds the gateway’s settings. They include the failure mode of each risk tier, the budget warning, two guardrails, prompts: keep: never and the rules for a staged change. The next lessons explain each part.

  4. Publish the configuration, so that the gateway gets the new settings:

    Terminal window
    python -m estate config publish --note "the gateway's settings"
    Published version 2.
    Told 0 of 1 gateways to refresh now.

    The estate is stopped, so no gateway was there to tell. It reads the new version when it starts.

  5. Start the estate:

    Terminal window
    python -m estate up
    provsim up (process 70161) http://127.0.0.1:8320
    control up (process 70163) http://127.0.0.1:8321
    gateway up (process 70165) http://127.0.0.1:8322

    Your process numbers differ. The apps’ address is the same as in Module 3: http://127.0.0.1:8322/v1.

Worked example: one call through the whole path

Section titled “Worked example: one call through the whole path”

Send one triage ticket through the gateway, and print each request:

Terminal window
python -m apps.replay ticket-triage --units 1 --via gateway --verbose

The output, replayed (the MLOps course’s triage request and this course’s embedding):

200 fast-small -> chat-small (gpt-6-luna-2026-09-22)
200 embed -> embed-small (text-embedding-3-small)
ticket-triage: 2 requests through the gateway | status {200: 2}
answered by: chat-small (gpt-6-luna-2026-09-22) x1, embed-small (text-embedding-3-small) x1
tokens: 1,954 in, 375 out | cost: US$0.0004

The app asked for two capabilities, fast-small and embed. The gateway chose the model for each, and its answer header x-gateway-model named the model that answered. Now read the gateway’s record of the last call:

Terminal window
python -m estate usage --last 1
ts (the time of the call)
request_id 274f4fc12975486d
replica a
config_version v2.r15
agent_id ticket-triage
team service
capability embed
provider azure
deployment embed-small
model_version text-embedding-3-small
data_class personal
tokens_in 62
tokens_out 0
cost_usd 1.24e-06
gateway_ms 0.35
upstream_ms 0.93
recorded_ms 329.0
status 200
outcome ok
run_id ticket-triage:T-64856:r1:6f6b3d
fallback 0
degraded 0

Your request ID and the last part of the run ID differ, and the course leaves the time out. Read it as the post room’s ledger line:

  • Who: agent_id and team come from the token and the registration, not from anything that the app says about itself.
  • What for: the capability that the app asked for, and the data_class of the request. The triage service sent no X-Data-Class header, so the gateway took the most sensitive class of its registration, personal.
  • What answered: the provider, the deployment and the exact model_version. When Module 2’s remap moves a capability to another model, this field shows it on the first call.
  • What it cost: the tokens, and cost_usd at the catalog’s prices: 62 tokens × US$0.02 per million = US$0.00000124.
  • How long: gateway_ms is the gateway’s own work (0.35 ms), upstream_ms the provider’s answer time. The simulator answers at once; recorded_ms is the real model’s recorded time, 329 ms.
  • Which configuration: config_version v2.r15 is the published version 2, with revision 15 of the registry. Module 3 explained the revisions; the last lesson of this module uses the version.

The row has no prompt and no answer text. prompts: keep: never in config/gateway.yaml is the rule: the gateway sees every request, so it is the worst place to keep their text.

Ask for a capability that the triage service’s registration does not list. Get the service’s token first:

Terminal window
TOKEN=$(python -m estate token ticket-triage)
curl -s -i -X POST http://127.0.0.1:8322/v1/chat/completions \
-H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN" \
-d '{"model": "strong-reasoning", "messages": [{"role": "user", "content": "Hello"}]}'
HTTP/1.1 403 Forbidden
server: uvicorn
x-gateway-request-id: 3ea457c0fbb643cc
x-gateway-replica: a
x-gateway-config: v2.r15
content-length: 154
content-type: application/json
{"error":{"code":"capability_not_allowed","message":"ticket-triage's registration does not list strong-reasoning.","type":"gateway","check":"capability"}}

The token is valid for 10 minutes; in a new terminal, get it again. The answer names the check (capability) and says what to change: the registration, not the gateway. Run python -m estate usage --last 1 again. The row now has outcome refused, check_name capability, the same message, and no provider: the request never left the gateway. Refusals are evidence too. In Module 6, they show which agents try to do what their registration does not allow.

Explore

Where does this request stop?

Result
200 chat-small
Checks passed
10 / 10
Replica
a, b or c

200: passes every check, and chat-small (gpt-6-luna-2026-09-22) answers.

  • passes
  • passes, degraded or with a fallback
  • stops the request
  • not reached
Document assistant, its usual request, nothing is down
#CheckWhat the gateway found
1ConfigurationThe replica has a configuration snapshot (it polls the control plane every 2 s).
2IdentityA valid access token for the model gateway (sub: doc-assistant).
3Registrationdoc-assistant is registered and active (low risk).
4Capabilityfast-small is in the registration and in the catalog; the contract has what the request needs.
5Data classNo X-Data-Class header: the gateway takes internal, the most sensitive class of the registration.
6Kill switchesNo kill switch stops this agent or this capability.
7Failure modeEvery control can run.
8LimitsUnder its limits: 60 requests a minute, its daily tokens, US$3 a month, and its team's budget.
9GuardrailsNo card number and no secret in the text.
10Routeazure/chat-small answers: chat-small (gpt-6-luna-2026-09-22).
11RecordOne usage row: agent, team, capability, model, tokens, cost, time, outcome. No prompt text.

Computed from the gateway’s rules; the outcomes match the measured runs.

Computed from the gateway's rules in gateway/app.py and the registrations; the error codes and messages are the gateway's own.
Text description of the explorer

Choose one of five agents: the triage service, the support workflow, the document assistant, the release judge, or the weekly digest, which is not registered. Then choose what it sends: its usual request, a request for strong-reasoning, a request with tools, personal data, payment data, or a request with a key in the prompt. A table shows the 11 checks in order. Each check passes, stops the request with its code and message, or is not reached. For example, the document assistant with personal data stops at check 5, the data class: “doc-assistant is not registered for personal data”. The triage service asking for strong-reasoning stops at check 4: its registration does not list it. The workflow with tools also stops at check 4, because the contract of strong-reasoning has no tools. The weekly digest stops at check 2: it has no token, only a personal key, and the gateway answers 401 invalid_token. Payment data stops every agent at check 5. A key in the prompt stops at check 9, the guardrails.

  1. Send the four apps’ requests through the gateway:

    Terminal window
    python -m apps.replay ticket-triage --units 5 --via gateway
    python -m apps.replay support-workflow --units 2 --via gateway
    python -m apps.replay doc-assistant --units 3 --via gateway
    python -m apps.replay release-judge --units 3 --via gateway

    Each one ends with status {200: …} and the same tokens and costs as in Module 1. The answers are the same recordings. Only the door changed.

  2. Read the usage per agent:

    Terminal window
    python -m estate usage
    agent calls ok refused failed tokens in tokens out US$
    ------------------------------------------------------------------------------------------------------
    support-workflow 6 6 0 0 14,776 738 0.0369
    release-judge 3 3 0 0 4,559 309 0.0122
    ticket-triage 13 12 1 0 11,972 2,400 0.0024
    doc-assistant 3 3 0 0 2,385 681 0.0006
    total 25 24 1 0 33,692 4,128 0.0521

    The triage line counts the worked example too: one ticket (2 calls), the refused call, and the five tickets (10 calls).

  3. Read it per model, and per outcome:

    Terminal window
    python -m estate usage --by model
    python -m estate usage --by outcome

    --by model names each model version that answered, and a line - for the refused call, which no model answered. --by outcome shows refused:capability 1.

This is the question that Module 1 could not answer from the bill: which app used what, and what it cost. The answer now comes from the gateway’s own records, per agent and per team, and does not depend on keys.

Run the support workflow’s real agent on ticket T-90103, with its model calls through the gateway:

Terminal window
RESOLVER_PROVIDER=gateway python -m resolver run T-90103 --variant agent_structured --model strong-reasoning

The first lines, replayed (the agents course’s structured agent on chat-strong):

Run T-90103-agent_structured-strong-reasoning-1 | single agent with one JSON step per call (for models without tool calling) | strong-reasoning
Outcome: resolve (rule W5) | 2 model call(s), 1 tool call(s) | 4647 tokens in, 266 out | US$0.01195

The agent proposes a reship and a free return label for a cracked planter, and waits for a person. Approve it as the agents course did, with the same setting: RESOLVER_PROVIDER=gateway python -m resolver approve T-90103-agent_structured-strong-reasoning-1 --as agent (in PowerShell, the variable is still set). The two changes are made in the practice database. Run python -m estate usage again: support-workflow now has 2 more calls and US$0.0120 more. The workflow’s tool calls still run inside the app. The gateway did not see them, which is the next point.

Tomás, an AI engineer, says what many teams think: “We have a gateway now, so we are governed.” The usage store shows why that is not true:

  • It sees model calls, not the agent’s work. The workflow’s get_order call and the two writes that you approved never passed the model gateway. Module 5 puts a tool gateway in front of them.
  • It sees only what goes through it. The document assistant embeds its documents on its own CPU, with a local model: those embeddings never reach the gateway. An app that keeps a provider key can still go around the door, as the failure case below shows.
  • It sees text, not meaning. It can find a card number in a prompt. It cannot know whether a refund is right for this ticket. The guardrails lesson of this module returns to this.

A door on every call adds time. The course team measured it on a laptop with 271 recorded triage requests, sent three times each. They went straight to the simulator, through one gateway replica, and through the front door of the last lesson of this module. The simulator answered at once (speed 0), so only the door’s own time is left.

Measured on the course team's laptop (271 triage requests × 3 passes, the simulator at speed 0); not a capacity. The same requests' recorded model latency: 2,922 ms at the median.
Text description of the chart

Three bars, the median time of one request in milliseconds. Straight to the simulator: 0.61 ms. Through one gateway replica: 1.82 ms. Through the front door and a replica: 2.66 ms. So the gateway adds about 1.2 ms, and 2.0 ms with the front door. The real model took 2,922 ms at the median for the same requests.

The gateway adds about 1.2 ms at the median, and 2.0 ms with the front door (p95: 3.06 ms in all). The model itself took 2,922 ms for the same requests, more than a thousand times longer. The replica’s own work, the usage row included, was 0.52 ms. These are times on one laptop, not a capacity: a real gateway runs on other machines, with a network between them. But the order of size holds. For an app that waits seconds for a model, a gateway’s milliseconds do not decide the design.

What does decide it:

  • You do not need a gateway for one app with one provider. Microsoft’s gateway guide says that a gateway serves “specific scenarios that might not be present in every workload”. It pays when several teams and agents share models, providers and budgets, as at Larkfield.
  • It becomes a shared dependency. If the door fails, every app fails, unless it is built for that. The last lesson of this module is about that.
  • It holds every provider key and sees every request. So it needs strict access, pinned dependencies and no prompt text in its records.

Common mistake

The usage store does not show an app's calls

Symptom: you send the document assistant’s question, and the replay works:

doc-assistant: 1 requests straight to the provider | status {200: 1}
answered by: chat-small (gpt-6-luna-2026-09-22) x1
tokens: 779 in, 439 out | cost: US$0.0003

But python -m estate usage shows no new row for doc-assistant.

Cause: the app did not use the gateway. The first line says it: “straight to the provider”. The replay ran without --via gateway, so the app used its old provider key, sim-az-data-team-0002. In a real estate, this is an app that someone forgot to move, or a copy that still holds the old key.

Diagnose: the egress log shows the call. Its last line names the machine and the key:

Terminal window
python -m provsim egress
...,docs-01,larkfield-ai.openai.azure.example,azure,az-key-2,200,3472

docs-01 connected to the provider with az-key-2, not through the gateway.

Recover: send the app through the gateway (--via gateway; for the workflow, RESOLVER_PROVIDER=gateway). The real fix is that the old key stops working and the network lets only the gateway reach the provider. Find unregistered AI finds such calls and switches on that network rule. The rollout lesson of Module 7 plans when the old keys are revoked.

Your turn: predict which check stops each request

Section titled “Your turn: predict which check stops each request”

Each request below breaks one rule. Before you send it, write down the check that stops it and the code that you expect. Use the request path diagram. Then send them, with the tokens of the right apps.

  1. Get the tokens of the document assistant and the workflow:

    Terminal window
    DOC=$(python -m estate token doc-assistant)
    WF=$(python -m estate token support-workflow)
  2. The document assistant sends personal data:

    Terminal window
    curl -s -X POST http://127.0.0.1:8322/v1/chat/completions -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $DOC" -H 'X-Data-Class: personal' \
    -d '{"model": "fast-small", "messages": [{"role": "user", "content": "Hello"}]}'
  3. The workflow asks strong-reasoning for a tool call:

    Terminal window
    curl -s -X POST http://127.0.0.1:8322/v1/chat/completions -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $WF" \
    -d '{"model": "strong-reasoning", "messages": [{"role": "user", "content": "Hello"}], "tools": [{"type": "function", "function": {"name": "get_order", "parameters": {"type": "object"}}}]}'
  4. The triage service asks for vision, with its token from the worked example in $TOKEN:

    Terminal window
    curl -s -X POST http://127.0.0.1:8322/v1/chat/completions -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $TOKEN" \
    -d '{"model": "vision", "messages": [{"role": "user", "content": "Hello"}]}'
  5. The weekly digest tries the gateway with its Fernway key:

    Terminal window
    curl -s -X POST http://127.0.0.1:8322/v1/chat/completions -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer sim-fw-personal-7Q2K9XW4' \
    -d '{"model": "fast-small", "messages": [{"role": "user", "content": "Hello"}]}'
  6. Count the refusals per check with python -m estate usage --by outcome.

On Windows, write the bodies with escaped quotes, as in the worked example (not tested).

Check your predictions
{"error":{"code":"data_class_not_allowed","message":"doc-assistant is not registered for personal data.","type":"gateway","check":"data_class"}}
{"error":{"code":"feature_not_in_contract","message":"strong-reasoning does not offer tool calling (its contract has no tools).","type":"gateway","check":"capability"}}
{"error":{"code":"capability_not_allowed","message":"ticket-triage's registration does not list vision.","type":"gateway","check":"capability"}}
{"error":{"code":"invalid_token","message":"The token is not a JWT.","type":"gateway","check":"identity"}}

The document assistant stops at check 5: its registration allows only internal data. The workflow stops at check 4, before any provider sees the request. Module 2’s contract for strong-reasoning has no tools, because chat-strong refuses tools through this API. vision is in the catalog, but the triage registration does not list it. So check 4 refuses it before it finds that nothing serves vision yet. The digest stops at check 2: a provider key is not a token. The registry will not give the digest a token while its owner has left. --by outcome shows each refusal under its check, and none of them cost anything.

Check your understanding

Why does the gateway check the token and the registration before the limits and the guardrails, and the provider last?

Show the answer

Each check needs what the earlier ones found out. The limits and the allowed data classes belong to an agent. So the gateway must first know which agent is calling (the token) and whether it may call (the registration). The cheap checks come first, so a refused request costs a fraction of a millisecond and nothing on the bill. The provider call is the only step that costs money, so it comes last, when every check has passed.

Check your understanding

After the move, the provider's bill shows every call on az-key-9. How does Larkfield now know what the triage service spent, and why can't a misconfigured app hide its calls in that number?

Show the answer

From the usage store. The gateway writes one row per request, with the agent and team, the model that answered and the cost. python -m estate usage adds them up per agent. An app cannot write another agent’s name in its rows, because the gateway takes the name from the signed token, not from the request. But the store only sees calls that pass the gateway. An app that still uses an old provider key is not in it, and the egress log or the bill must find it.

Check your understanding

Practical check: send one question of the document assistant through the gateway, then show its usage row. Which fields tell you who called, which model answered and what it cost?

Show the answer

python -m apps.replay doc-assistant –units 1 –via gateway, then python -m estate usage –last 1. Who: agent_id doc-assistant and team data. Which model: provider azure, deployment chat-small and model_version gpt-6-luna-2026-09-22. What it cost: tokens_in, tokens_out and cost_usd, at the catalog’s prices. The data class is internal, the most sensitive class of its registration, because the request sent no X-Data-Class header. The row has no prompt text.

Recap

  • An AI gateway is one door for every model call. Larkfield’s model gateway runs 11 checks in a fixed order, and the first check that fails answers with its own code.
  • Every request, answered or refused, gets one usage row: who called, what for, which model answered, the tokens, the cost and the time. No prompt text is kept.
  • Apps keep the OpenAI-compatible API and change three values: the base URL, their credential and the model name. The workflow changes one setting, RESOLVER_PROVIDER=gateway.
  • The gateway added about 1.2 ms at the median on the course team’s laptop, against 2.9 s for the model. Its real costs are elsewhere: a shared dependency, and the keys it holds.
  • The gateway sees model calls only, and only the calls that pass through it.

Your project now: the whole gateway in gateway/, the usage store in estate/usage.py, config/gateway.yaml, the workflow’s gateway adapter in resolver/platform.py, and 137 tests that pass. You can now see who called which model and what it cost. But nothing stops an agent that calls a model in a loop: the record only shows the cost afterwards. Next, you set the limits that stop it.

Next: Quotas, budgets and rate limits