Skip to content
Français

When the gateway fails

  • Lesson 13
  • 45 min
  • Module 4: The model gateway

Where we are

In the last three lessons, we put every model call behind one door that checks identity, limits and data. So Tomás asks the fair question: if the gateway stops, does every agent stop with it? And one bad rule reached two apps at once. In this lesson, we stop a replica, the control plane and a provider during a replayed hour, and we see which design choices decide what still works. We start at a bank's cash machines.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Build the gateway so that it is not a single point of failure. Use several replicas behind one address, a cached configuration, a failure mode per risk tier, provider fallbacks and staged changes. Then measure what still works when each part stops.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

  • The gateway’s path, limits and guardrails, from the three earlier lessons of this module, starting with One door for model calls.
  • What a risk tier is, from What a registration holds. The workflow is high risk, triage and the judge are medium, the document assistant is low.

What you need on your computer

  • The project as the last lesson left it (159 tests pass), with the guardrails back to the two real rules.
  • Two terminals in ~/projects/ai-estate, with the virtual environment active in both. Ports 8320 to 8325 must be free.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

One bank has many cash machines in a city. If one machine breaks, you walk to the next one: nobody says the bank is down. Each machine also keeps working for a while when its line to the head office drops. It pays out small amounts from what it already knows. But it refuses a large transfer, because that needs a check that only the head office can do.

When the bank changes the machines’ software, it does not change every machine at the same minute. It tries one branch first. If the new software fails there, only that branch’s customers notice, and the bank goes back.

One bank, one way to get cash, but not one machine. That is the whole idea of this lesson.

Tomás says: “A gateway is a single point of failure, so do not use one.” With the gateway that you built so far, he is right. Stop it, and send one ticket:

Terminal window
python -m estate down gateway
python -m apps.replay ticket-triage --units 1 --via gateway

The replay ends with a long traceback, whose last line is:

httpx.ConnectError: [Errno 61] Connection refused

On Linux the number is 111; on Windows, 10061 (not tested). One process stopped, and every app lost its models. In the last lesson, one bad line in config/gateway.yaml stopped two apps at once. That is the blast radius of a change: how many requests it reaches before someone stops it.

But without a gateway, each app needs its own retries, fallback region, key and limits. Five teams build five versions, and the weekly digest builds none. Their failures are smaller, but each is different, and nobody sees them together. So the question is how to build the gateway so that one failure does not stop everything. Stop the estate now: python -m estate down.

The idea: one logical door, not one machine

Section titled “The idea: one logical door, not one machine”

The gateway is one logical door: every app knows one address, and one set of rules applies. Behind that address, it is not one machine. Microsoft’s guide to gateways in front of several model deployments says the same: “deploy multiple instances of the gateway”. Larkfield’s gateway uses five design choices:

The apps know one address, 127.0.0.1:8322/v1. It is the front door, inside a box named 'One logical door: the model gateway'. The front door does health checks and sends each request to a healthy replica: a, b or c, each with its own cached configuration. The control plane, with the registry, tokens and configuration versions, sends the configuration to the gateway: each replica polls every 2 s, and a change is pushed. The gateway writes to one usage store, with one shared count per agent and team. It calls azure in East US 2, with a dashed fallback to azure-canada, a simulated second region.
One logical door: one address and one set of rules, served by several replicas that each keep a copy of the configuration.
Text description of the diagram

The apps know one address. Behind it, a front door sends each request to one of three replicas, a, b and c. It skips a replica that fails its health check. Each replica keeps its own copy of the configuration. The control plane publishes the configuration: each replica polls it every 2 seconds, and a change is also pushed to them. All replicas write to one usage store, so they share one count per agent and team. They call the provider in East US 2, and fall back to a second region.

  1. Several stateless replicas behind a front door. The counters are in the shared usage store, and the rules come from the control plane. So any replica can serve any request, and one replica can stop.
  2. Configuration cached at each replica. A replica polls the control plane every 2 seconds and keeps the last good copy, in memory and on disk. If the control plane stops, the replica goes on with its copy, up to a limit, max_stale_seconds.
  3. A failure mode per risk tier. When a control cannot run, fail open means: serve the call without that control, marked as degraded. Fail closed means: refuse the call. Larkfield chose by risk tier.
  4. Provider fallback. When a provider fails, the gateway tries the next provider of the mapping.
  5. Staged configuration changes. A new version goes to one replica first, the canary, and goes back if its error rate is too high.

The control plane stops at 12:00:00. Replica a’s last good poll was at 11:59:59. It keeps polling every 2 seconds, and each poll fails. The age of its copy decides what happens:

Time Age of the copy State Ticket triage (medium) Support workflow (high)
12:00:03 4 s fresh (at most 2 polls + 1 s) served served
12:00:30 31 s stale (under 60 s) served, as usual served, as usual
12:01:01 62 s expired (over 60 s) served, marked degraded (fail open) refused: 503 control_unavailable (fail closed)

For the first minute, a control-plane outage is invisible to the apps. After the limit, each tier gets its own answer. Sixty seconds is a practice value. A real estate weighs two risks: an old copy that misses a suspension, against a failure mode that starts too soon.

Fail open is not free, and the course measured each cost on the real code:

Risk tier Mode While the control cannot run What fail open means here
low (document assistant) fail open calls pass, marked x-gateway-degraded each replica counts from the last total it read plus its own calls: with three replicas, up to three times what was left of the budget. The guardrails still run (they are code in the replica). A new suspension does not arrive
medium (ticket triage, release judge) fail open the same the same
high (support workflow, tool servers) fail closed 503 control_unavailable not used: no write without every control. The workflow hands its tasks to people

This is docs/failure-modes.md in the project. The same modes apply when the usage store is down. A registration may set its own failure mode.

  1. Copy the front door, the failure-mode table, the staged configuration commands, the traffic tool, the tests, and the new README:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m04
    cp $S/gateway/front.py gateway/
    cp $S/docs/failure-modes.md docs/
    cp $S/estate/cli/config.py estate/cli/
    cp $S/apps/traffic.py apps/
    cp $S/tests/test_front.py $S/tests/test_config_cache.py $S/tests/test_rollout.py $S/tests/test_services.py tests/
    cp $S/README.md $S/.env.example .
  2. Run the tests: python -m pytest -q. The last line says 172 passed, 1 deselected. Your project is now the same as the snapshot end-of-m04.

  3. Start the estate:

    Terminal window
    python -m estate up
    provsim up (process 71680) http://127.0.0.1:8320
    control up (process 71681) http://127.0.0.1:8321
    replica-a up (process 71682) http://127.0.0.1:8323
    replica-b up (process 71683) http://127.0.0.1:8324
    replica-c up (process 71684) http://127.0.0.1:8325
    front up (process 71685) http://127.0.0.1:8322/front

    The estate now starts three replicas and a front door, because the project has gateway/front.py. The apps’ address, port 8322, did not change: the front door took it.

  4. Send six tickets, and see which replicas served them:

    Terminal window
    python -m apps.replay ticket-triage --units 6 --via gateway
    python -m estate usage --by replica

    The replay ends with status {200: 12}. In --by replica, replicas b and c now have about 4 calls each, and replica a has the rest. Earlier lessons had only replica a.

  5. Ask one replica how fresh its copy of the configuration is:

    Terminal window
    curl -s http://127.0.0.1:8323/ready
    {"replica":"a","config":"v6.r19","freshness":"fresh","age_s":1.0,"last_error":"","usage_store":"up","pending_rows":0}

    Your version numbers differ if you published more often.

Guided practice: stop a replica and the control plane during an hour

Section titled “Guided practice: stop a replica and the control plane during an hour”

The tool apps/traffic.py sends a compressed hour of Larkfield’s traffic through the gateway. The triage service, the workflow and the document assistant send their real recorded requests, about 2 a second. Every 10 seconds it prints what each app got back. You stop parts of the estate in the second terminal, at the times below.

  1. In terminal 1, start the traffic:

    Terminal window
    python -m apps.traffic --seconds 150
  2. At about 20 seconds, in terminal 2, stop replica b:

    Terminal window
    python -m estate down replica-b
  3. At about 45 seconds, stop the control plane:

    Terminal window
    python -m estate down control
  4. At about 60 seconds, ask replica a how old its copy is, with curl -s http://127.0.0.1:8323/ready. Then send one document question, which needs a new token: python -m apps.replay doc-assistant --units 1 --via gateway.

  5. At about 110 seconds, ask replica a again.

  6. At about 140 seconds, start the control plane again:

    Terminal window
    python -m estate up control
  7. When the traffic ends, start replica b again with python -m estate up.

The course’s run. Terminal 1, the traffic (replayed requests; the hour is compressed, so read the order of events, not the rate):

Traffic through the gateway for 150 s, about 2 requests a second (Ctrl+C stops)
10 s | triage ok 20 | workflow - | documents - | replicas a 7, b 7, c 6
20 s | triage ok 16 | workflow ok 4 | documents - | replicas a 7, b 6, c 7
30 s | triage ok 16 | workflow ok 3 | documents - | replicas a 9, c 10
40 s | triage ok 19 | workflow ok 1 | documents - | replicas a 10, c 10
50 s | triage ok 17 | workflow ok 1 | documents ok 1 | replicas a 10, c 9
60 s | triage ok 18 | workflow - | documents ok 2 | replicas a 10, c 10
70 s | triage ok 16 | workflow - | documents ok 3 | replicas a 9, c 10
80 s | triage ok 16 | workflow - | documents ok 4 | replicas a 10, c 10
90 s | triage ok 14 | workflow ok 3 | documents ok 2 | replicas a 10, c 9
100 s | triage ok 20 | workflow - | documents - | replicas a 10, c 10
110 s | triage degraded 9, ok 8 | workflow - | documents degraded 2 | replicas a 9, c 10
120 s | triage degraded 16 | workflow control_unavailable 2 | documents degraded 2 | replicas a 10, c 10
130 s | triage degraded 17 | workflow control_unavailable 1 | documents degraded 1 | replicas a 10, c 9
140 s | triage degraded 20 | workflow - | documents - | replicas a 10, c 10
150 s | triage degraded 5, ok 12 | workflow - | documents ok 2 | replicas a 9, c 10
Total: triage degraded 67, ok 192 | workflow control_unavailable 3, ok 12 | documents degraded 5, ok 14

Terminal 2, the two answers of replica a, at about 61 and 111 seconds:

{"replica":"a","config":"v6.r19","freshness":"stale","age_s":16.4,"last_error":"ConnectError: All connection attempts failed","usage_store":"up","pending_rows":0}
{"replica":"a","config":"v6.r19","freshness":"expired","age_s":66.5,"last_error":"ConnectError: All connection attempts failed","usage_store":"up","pending_rows":6}

Read it phase by phase:

  • Replica b stops (20 s). Only a and c answer from then on, and no call failed: the front door skips a replica that fails its health check.
  • The control plane stops (45 s). For 60 seconds, every app is served, the workflow too, from the cached copies. Replica a’s copy is stale, 16.4 s old.
  • The copy expires (about 105 s). Triage and the document assistant fail open: served, marked degraded. The workflow fails closed: control_unavailable, and its tasks go to people. Replica a keeps 6 usage rows that wait for the store (pending_rows).
  • A new token (60 s). The document question of step 4 ended with httpx.ConnectError: [Errno 61] Connection refused: a new token comes from the control plane. Apps that already had a token went on.
  • The control plane is back (140 s). One poll later, the copies are fresh again.

The course team also ran this with a crash instead of a clean stop. It killed replica b with SIGKILL, and the simulator answered at the recorded speed of the real models, so that calls were in flight. The chart shows that run, one point per 10 seconds:

  • answered
  • answered, degraded (fail open)
  • refused or failed
Measured: real processes on the course team's laptop, 160 s, replayed requests at their recorded speed; replica b killed at 20 s, the control plane stopped at 45 s and started again at 140 s; max_stale_seconds 60.
Text description of the chart

Three lines over 160 seconds, calls per 10 seconds. The answered line stays near 20 from 0 to 100 seconds, through the replica’s crash at 20 seconds and the control plane’s stop at 45 seconds. Between 100 and 140 seconds it falls to 0, and the degraded line rises to about 20: the triage service and the document assistant failed open. After 140 seconds, the answered line is back near 20. The refused-or-failed line stays at 0 to 2: 2 calls in flight on replica b when it crashed (502 replica_failed, not sent again), 5 workflow calls refused when the configuration was too old (fail closed), and 2 recorded content-filter refusals of the provider. Replicas: a 175, b 14, c 174 calls.

Two calls failed, and only those: they were on replica b when it crashed. The front door does not send such a request again, because it may already have reached the provider. A model call is not a write, but it costs money, and a second answer may differ. So the app gets 502 replica_failed and decides. A refused connection is different: the request never reached the replica, so the front door sends it to the next one.

Explore

What still works when one part stops?

Result
503 control_unavailable
Checks passed
6 / 10
Replica
a, b or c

Stopped at check 7 of 11: Failure mode. 503 control_unavailable: A control cannot run (the configuration is 90 s old (limit 60 s)): high risk fails closed. The workflow hands the task to a person. Measured: after max_stale_seconds, triage 70 and documents 1 passed degraded (fail open); the workflow’s 5 calls were refused (fail closed).

  • passes
  • passes, degraded or with a fallback
  • stops the request
  • not reached
Support workflow, its usual request, control plane down 90 s
#CheckWhat the gateway found
1ConfigurationThe control plane has not answered for 90 s: the replica uses its cached snapshot.
2IdentityA valid access token for the model gateway (sub: support-workflow).
3Registrationsupport-workflow is registered and active (high risk).
4Capabilitystrong-reasoning is in the registration and in the catalog; the contract has what the request needs.
5Data classNo X-Data-Class header: the gateway takes personal, the most sensitive class of the registration.
6Kill switchesNo kill switch stops this agent or this capability.
7Failure mode503 control_unavailable: A control cannot run (the configuration is 90 s old (limit 60 s)): high risk fails closed.
8LimitsNot reached: an earlier check answered.
9GuardrailsNot reached: an earlier check answered.
10RouteNot reached: an earlier check answered.
11RecordOne usage row for the refusal too: outcome refused, check failure_mode.

Computed from the gateway’s rules; the outcomes match the measured runs.

Computed from the gateway's rules, the registrations and config/gateway.yaml; the measured notes quote the course's runs.
Text description of the explorer

Choose an agent and a request, as in the first lesson of this module, and then what is down: nothing, replica b, the control plane for 30 or 90 seconds, the usage store, the East US 2 region, both regions, a kill switch on the capability, the monthly budget, or the rate limit. The 11 checks light up in order. With the control plane down for 90 seconds, the workflow stops at check 7, the failure mode: 503 control_unavailable, because high risk fails closed. The document assistant and the triage service pass, marked degraded, because they fail open. With the control plane down for 30 seconds, everyone passes: the cached copy is still within its limit. With East US 2 down, every agent passes through the second region. With both regions down, every agent stops at check 10, the route, with 503 no_provider_available, and each app does its own fallback: the triage service sends the ticket to a person, the workflow hands the task to a person, the document assistant says “try again later”.

A provider outage is not a gateway failure

Section titled “A provider outage is not a gateway failure”

When the provider fails, the gateway tries the mapping’s fallbacks, which each app would otherwise build itself. It falls back on 429, 5xx errors and timeouts, not on a 4xx answer: another provider would refuse the same bad request. After 3 failures in a row, a circuit breaker skips a provider for 10 seconds.

  1. Take down the East US 2 region of the simulator, and send three tickets:

    Terminal window
    python -m provsim control azure down
    python -m apps.replay ticket-triage --units 3 --via gateway
    python -m estate usage --last 1

    The replay ends with status {200: 6}, and the usage row says provider azure-canada and fallback 1. The second region is simulated: the same recordings answer, as they would from the same models in another region.

  2. Take down every provider, and send three tickets and one workflow task:

    Terminal window
    python -m provsim control '*' down
    python -m apps.replay ticket-triage --units 3 --via gateway

    The replay ends with status {503: 6} and no_provider_available x6. The usage row of such a call says outcome failed, check_name route and No provider answered for embed: last error 503 azure-canada is unavailable (a simulated outage). Run the workflow on T-90104 with RESOLVER_PROVIDER=gateway, as in the second lesson of this module. It ends with Stopped: provider_error. No proposal: the ticket goes to a person.

  3. Bring the providers back, and send three tickets at once:

    Terminal window
    python -m provsim control '*' normal
    python -m apps.replay ticket-triage --units 3 --via gateway

    In the course’s run, all 6 calls still got no_provider_available. The breaker skips each provider for 10 seconds after its last failure. Wait 10 seconds and replay again: status {200: 6}. If 10 seconds have already passed when you replay, the calls pass at once.

The course found a real bug here. In the replayed month, the breaker first counted real seconds while the replay moved through hours. It refused 1,280 calls after the outage had ended. A breaker on the wrong clock is a real class of bug. The fix was the estate’s clock everywhere.

In the replayed month, East US 2 was down from 10:00 to 11:00 on 17 September, a simulated outage. The second region served 123 answers, and no call failed. From 11:00 to 11:30, both regions were down. Then 35 triage calls, 2 workflow calls and 3 document questions had no provider. The triage service sent its tickets to a person’s queue, the workflow handed its tasks to people, and the document assistant said “try again later”.

Staged changes: the blast radius of a bad rule

Section titled “Staged changes: the blast radius of a bad rule”

The last lesson’s bad rule reached every app at once. Now send it to the canary replica only, and let the watch decide:

  1. Add the same long-numbers line to config/gateway.yaml, as in the last lesson. Publish it to the canary only:

    Terminal window
    python -m estate config publish --stage canary --note "a new guardrail: long numbers"
    python -m estate config status
    Published version 7 (canary replicas first).
    Told 3 of 3 gateway replicas to refresh now.
    stable version 6 | candidate 7 on the canary since …
    replica a runs v7.r20 last seen …
    replica b runs v6.r20 last seen …
    replica c runs v6.r20 last seen …

    The course leaves the times out. The status also lists the last five versions. Only replica a runs version 7.

  2. Send ten tickets: python -m apps.replay ticket-triage --units 10 --via gateway. Only the calls that the front door sends to replica a can be blocked. The course’s run ended with status {200: 16, 400: 4}: 4 blocked of 20, against 12 of 20 in the last lesson.

  3. Let the watch decide:

    Terminal window
    python -m estate config watch
    {
    "candidate": 7,
    "canary": [
    "a"
    ],
    "seen": 6,
    "errors": 4,
    "error_rate": 0.6667,
    "state": "rolled back to v6"
    }

    The watch counts the canary’s errors with the candidate version. It promotes the version after 40 requests with at most 5% errors. It rolls it back as soon as the errors pass 5% of 40, here after 6 requests.

  4. Remove the line from config/gateway.yaml, and publish again: python -m estate config publish --note "the two guardrails, after the canary".

The course measured the same bad rule in five ways, on two replayed hours of Larkfield’s traffic. The rule is simulated, the requests are replayed, and the rule went out at 09:30. “A person” means that someone reads an alert and rolls back after 15 minutes, an assumption.

Simulated rule (block 6 or more digits), replayed traffic at Larkfield's rates; 'a person' rolls back after 15 minutes (an assumption); 'automatic' is a check on the last 40 requests.
Text description of the chart

Five bars. A, all replicas at once, a person rolls back after 15 minutes: 18 requests blocked (triage 14, workflow 4). B, all at once, an automatic rollback: 5. C, staged on one replica of three, the automatic watch: 4. D, staged, a person after 15 minutes: 6. E, no shared gateway, the same rule in the triage service’s own code, a person after 15 minutes: 14, all of them triage.

A shared gateway makes a bad change reach every app: 18 against 14. But staging and an automatic watch make it smaller than one app’s own mistake: 4. Without a gateway, there is no canary for five apps’ code, and each team finds its own bugs alone.

The gateway holds the provider keys of the whole estate, and it sees every request. It is the most valuable thing to attack:

  • Only the gateway reads the keys in state/secrets.json. The apps hold only their own credentials. In Find unregistered AI, an egress rule blocks the other network paths to the provider. The rollout lesson of Module 7 plans when the apps’ old provider keys are revoked.
  • Strict access to the door itself. Here, /admin/refresh and /admin/faults answer anyone on your computer. In a real estate, only the control plane may call them, with its own credential.
  • Pinned and checked dependencies. requirements.txt pins every version. A change to the gateway’s code goes through review and the canary.
  • No prompt text: prompts: keep: never.

Then the gateway is a critical shared dependency, like the identity provider or DNS, and it is run like one. It has many replicas, a cached copy of what it needs, a known answer for every failure, and changes in stages.

Common mistake

A replica restarts while the control plane is down

Symptom: you restart replica a (python -m estate down replica-a, then python -m estate up replica-a) while the control plane is stopped. At once, the workflow’s calls to it are refused:

{"error":{"code":"control_unavailable","message":"A control cannot run (the configuration came from disk, and the store has not answered since): high risk fails closed.","type":"gateway","check":"failure_mode"}}

The document assistant’s calls pass, with x-gateway-degraded: fail open (low risk): the configuration came from disk, and the store has not answered since.

Cause: a replica that starts without the control plane starts from its copy on disk, state/gateway/a-config.json. It cannot know how old that copy is, so it counts it as expired at once: high risk fails closed, low and medium fail open. The other replicas still have their copies in memory, and serve as before for up to 60 seconds.

Diagnose: curl -s http://127.0.0.1:8323/ready says "freshness":"expired" and "age_s":null, and last_error names the failed poll.

Recover: python -m estate up control. Two seconds later, /ready says "freshness":"fresh". Do not restart replicas while the control plane is down, unless you must.

Your turn: a failure mode for a provider outage

Section titled “Your turn: a failure mode for a provider outage”

Both regions are down for half an hour. This is not a failure of the gateway’s controls, so config/gateway.yaml does not decide it: each app’s owner decides what the app does with 503 no_provider_available.

  1. Read docs/failure-modes.md and the two registrations, registrations/ticket-triage.yaml and registrations/support-workflow.yaml: their risk tiers and their autonomy.
  2. For the triage service and for the refund workflow, write down your choice. May the app use another model that Larkfield has, such as the local gemma3:4b? Or does it hand its work to people?
  3. Write one reason for each, from the course’s evidence.
  4. Compare with the knowledge check below.

Check your understanding

Tomás says: 'A shared gateway is a single point of failure, so do not use one.' When is he right, and what makes him wrong for Larkfield's gateway? Use the course's measurements.

Show the answer

He is right for a gateway built as one process: when it stopped, every app got “Connection refused”. He is wrong for replicas behind one address, with a cached configuration and a failure mode per tier. In the measured run, replica b crashed and only its 2 calls in flight failed. Without the control plane, every call passed for 60 seconds; then low and medium risk passed degraded, and only the workflow’s 5 calls were refused, on purpose. A bad change still reaches every app (18 blocked, against 14 in one app), so changes go out in stages: 4 with the canary.

Check your understanding

During the store outage, the document assistant fails open. What exactly does it lose, and why is that acceptable for it but not for the workflow?

Show the answer

Its calls pass with the last configuration, marked degraded. Each replica counts its budget from the last total it read, plus its own calls. So with three replicas, it could spend up to three times what was left. A suspension made meanwhile does not arrive. The guardrails still run, because they are code in the replica. That is acceptable for a low-risk agent that reads internal documents and changes nothing. The workflow proposes refunds and reships: a write without every control is not allowed, so it fails closed and hands its tasks to people.

Check your understanding

Completion check: choose and justify the failure mode of the triage service and of the refund workflow during a provider outage, when the second region is down too.

Show the answer

First, both use the fallback to the second region. It runs the same models, so their evaluations still hold: 123 answers on 17 September, and no call failed. When no region is left, the triage service fails open in the business sense. It is medium risk and only suggests a team, so the ticket goes to a person’s queue without AI, and only speed is lost (35 tickets that day). The refund workflow is high risk: it proposes refunds and reships. So it fails closed: no proposal, and the task goes to a person. Neither should move to a model without evidence for its task: on the agents course’s tasks, gemma3:4b passed 28 of 70, and the workflow’s minimum is 0.95.

Recap

  • One logical door is not one machine: stateless replicas behind one address, a cached configuration at each, and a shared usage store.
  • Without the control plane, every call passed for 60 s; then low and medium risk failed open (degraded) and high risk failed closed. Fail open costs budgets counted per replica and missed suspensions; the guardrails still run.
  • A provider outage goes to the fallbacks. When nothing is left, each app’s owner decides: a person’s queue for triage, no proposal for the workflow.
  • A bad change through a shared gateway: 18 blocked all at once, 4 with the canary, 14 in one app alone.
  • The gateway holds every key: protect its access, pin its dependencies, keep no prompt text.

Your project now: the same as the snapshot end-of-m04, with a front door, three replicas, staged configuration and 172 tests that pass. Every model call is now checked, limited and recorded, and the gateway survives its own failures. But the agents do more than call models: the workflow’s refunds and reships are tool calls, and the model gateway never sees them. Next, in Module 5, you put a gateway in front of the tools.

Next: A gateway for tools