Gateways compared
- Lesson 20
- 40 min
- Module 7: Build, buy and roll out
Where we are
In Module 6, we turned the gateway's records into evidence, and we found the AI that nobody registered. Every part of that platform is code that we wrote, so we know what each part does. But Larkfield does not have to run its own gateway: open-source and managed products exist. In this lesson, we compare ours with four of them, from their own documentation. We start at a workshop that built a go-kart.
Lesson 20 of 21 · Governing AI Agents: Gateways, Registries and ControlsGateways compared
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Compare your own gateway with open-source and managed products, feature by feature, from each product’s own documentation. Separate the must-haves from the nice-to-haves. Then look at what a feature list hides: the licence, what the product needs to run, its supply-chain risk and who runs it.
Before you startWhat to know and to have ready
What you need to know
- Every control of the platform, from Modules 2 to 6:
- the catalog, from A capability catalog;
- the identities, from One identity per agent;
- the limits, from Quotas, budgets and rate limits;
- the failure modes, from When the gateway fails;
- the tool gateway, from A gateway for tools.
What you need on your computer
- The project as Module 6 left it (194 tests pass), in
~/projects/ai-estate, with the virtual environment active. - No account, key or trial of any product. You read the products’ documentation; you install nothing.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
A go-kart in the workshop
Section titled “A go-kart in the workshop”A delivery company needs vans. Its mechanics built a go-kart in the workshop first, to learn how an engine, brakes and steering work. Now they can read a van’s specification sheet and know what each line means.
The go-kart will never deliver parcels. So the company makes a sheet: one row for each thing that its drivers need, and one column for each van. It fills the columns from the makers’ own specification sheets, not from a salesman or from memory. Some rows are must-haves: a van without a tail lift is out, however good the rest is. And the buyer asks two more questions that no specification sheet answers. Who services it, and what does it cost to change the brand in three years?
The answer to the last question depends on the company, not on the van. If the drivers’ routes and the delivery records are the company’s own, a new brand of van is a small change.
The problem: a comparison from memory
Section titled “The problem: a comparison from memory”Kwame asks Tomás for a first comparison of the products that the platform team talked about. Tomás writes it in ten minutes, from what he remembers. This draft is constructed for the course:
Envoy AI Gateway: an Envoy add-on, no limits that know tokensAzure API Management: has budgets per teamLiteLLM: budgets and keys; no product counts the steps of an agent runCloudflare AI Gateway: logs and caching onlyEach line is wrong, or no longer true, when you read the products’ documentation:
- Envoy AI Gateway has a new name and home: Agent Router. Its documentation describes token-based rate limits and token quotas.
- Azure API Management’s AI policies limit tokens, per minute and per period. A limit in money is not in its documentation.
- LiteLLM’s documentation describes “agent iteration budgets”: a cap on the model calls of one session.
- Cloudflare AI Gateway’s documentation describes spend limits, guardrails and fallbacks.
Products change every few weeks. A comparison from memory decides with old facts, and nobody can check it later, because it has no sources. Worse, it compares what the products are famous for, not what Larkfield needs.
The idea: a comparison sheet from the documentation
Section titled “The idea: a comparison sheet from the documentation”A comparison sheet is a table with one row per control that you need and one column per product. Each cell comes from that product’s own current documentation, with a link to the page. Three rules make it trustworthy:
- The rows are your controls, not the products’ features. You built every control of the course, so you know what each row means. That is the go-kart’s value.
- Every cell starts with one word:
yes,partly,noornot found, then what the documentation says, then a source. “Not found” is not “no”. It means that you did not find it, and it is a question to ask the vendor. - The sheet names the version you read. Then everyone knows when to check it again.
The decision is called build or buy: run your own code, or use a product, open-source or managed. The cost of changing products later is the switching cost. A product that is hard to leave gives you lock-in.
Open the sheet in your project. Copy it from the snapshot end-of-m07:
cp ../nextia-learning-labs/C32/snapshots/end-of-m07/docs/gateway-comparison.md docs/Copy-Item ..\nextia-learning-labs\C32\snapshots\end-of-m07\docs\gateway-comparison.md docs\cp ../nextia-learning-labs/C32/snapshots/end-of-m07/docs/gateway-comparison.md docs/The sheet has 19 feature rows, one row that names the version read, and a list of sources under the table. The picture shows the first word of each cell, plus the licence and what each product needs to run.
Text description of the diagram
Rows are features, columns are gateways. Capability names, an OpenAI-compatible endpoint and fallbacks are yes everywhere, with Azure API Management’s model aliases in preview. Feature checks in a contract are yes only for ours and not found for all four products. Registration before the first call: ours yes, LiteLLM partly, Agent Router not found, Cloudflare not found, Azure partly. One identity per agent: ours, LiteLLM and Azure yes, Agent Router and Cloudflare partly. Budgets in money: ours, LiteLLM and Cloudflare yes, Agent Router and Azure partly, because they count tokens. Rate limits and token quotas: Cloudflare partly, the others yes. Step limit per run: ours and LiteLLM yes, the others not found. Guardrails: Agent Router not found, the others yes. Tool control: Cloudflare no, because it is a separate product, the others yes. Kill switches: ours and Azure yes, LiteLLM and Cloudflare partly, Agent Router not found. OpenTelemetry: ours and Azure partly, the others yes. Self-hosted: Cloudflare no, Azure partly, the others yes. Licences: ours MIT, LiteLLM MIT with commercial parts, Agent Router Apache-2.0, Cloudflare and Azure proprietary services.
Read the matrix by its rows
Section titled “Read the matrix by its rows”Three patterns matter more than any one cell.
The data path is the same everywhere. Every product takes OpenAI-compatible requests, hides the provider’s model behind a name of your choice, and falls back to another backend. You can buy this part.
The records about agents are rare. No product’s documentation shows a registration with an owner, a purpose, a risk tier and a review date before the first call. LiteLLM’s keys have an owner and a team. Azure’s subscriptions can need an administrator’s approval. Neither holds what Module 3’s form holds. No product checks a contract’s features, as estate catalog check does. These parts are Larkfield’s own knowledge, not traffic.
The differences are in limits, tools and hosting. Budgets in money are in LiteLLM and Cloudflare. Agent Router and Azure count tokens. Tool control through MCP is in three products, and in a separate Cloudflare product. Cloudflare runs only as a service. Azure’s self-hosted gateway exists in two tiers and still gets its configuration from Azure.
Sam asks: “Our column is green almost everywhere. So is ours the best?” Kwame answers: “The rows are the controls that we built. A sheet made from your own design always favours your design. Look at the rows the picture leaves out.”
What the feature list hides
Section titled “What the feature list hides”Four rows of the sheet are not features. They decide who does the work.
- Licence. LiteLLM’s core is MIT, but its
enterprise/folder has a separate commercial licence (licence). Its documentation marks some features as enterprise, such as sign-in with JWT and guardrails per key. Agent Router is Apache-2.0 throughout. Cloudflare and Azure are proprietary services. Read the licence of each feature you need, not only of the project. - What it needs to run. LiteLLM needs PostgreSQL for keys and budgets, and Redis for more than one instance (production guide). Agent Router needs Kubernetes and Envoy Gateway (prerequisites). Each one is another service that your team runs, upgrades and secures.
- What happens when its own store is down. You chose this per risk tier in Module 4. LiteLLM’s documentation says that calls fail by default when its database cannot be reached, with a setting that keeps keys working. Agent Router’s quota checks fail open by default when the rate-limit service is down. Azure’s self-hosted gateway keeps running on its last configuration. Each is a choice that you must know before an outage, not during it.
- Supply-chain risk. A gateway holds every provider key, so its code is a target. LiteLLM’s own security update describes two versions on PyPI that were compromised: 1.82.7 and 1.82.8. Both open-source projects publish security advisories (LiteLLM, Agent Router). Our gateway has 10 pinned packages, 35 when installed with what they need. It is a teaching project: it was tested on one laptop and never had a security review.
The last point is the honest cost of “build”. The course’s code shows every part of a gateway. Running it for Larkfield means someone patches it, reviews it, wakes up when it fails and answers for it, for years.
A tiny worked example: score the sheet for Larkfield
Section titled “A tiny worked example: score the sheet for Larkfield”A sheet with 19 rows and five columns is hard to read at once. A small script makes the decision visible, step by step. It is a practice file, not part of the project.
-
Create the file
score_gateways.pyin the project folder, with this content:score_gateways.py """Score the gateways of docs/gateway-comparison.md for one company's needs (a practice file: not part of the project)."""import argparsePOINTS = {"yes": 2, "partly": 1} # "no", "not found" and everything else: 0 pointsSHORT = ["ours", "LiteLLM", "Agent Router", "Cloudflare", "Azure APIM"]ap = argparse.ArgumentParser()ap.add_argument("--must", action="append", default=[], help="a feature that must be 'yes'")ap.add_argument("--want", action="append", default=[], help="a feature that earns points")a = ap.parse_args()with open("docs/gateway-comparison.md", encoding="utf-8") as f:rows = [line.strip().strip("|").split(" | ") for line in f if line.startswith("| ")]table = {r[0].strip(): r[1:] for r in rows[1:]}assert len(rows[0]) - 1 == len(SHORT), "the sheet has another set of products"def points(cell: str) -> int:return POINTS.get(cell.split(":")[0].split()[0], 0)print(f"{'':<42}" + "".join(f"{s:>13}" for s in SHORT))for kind, features in (("must", a.must), ("want", a.want)):for feature in features:print(f"{kind} {feature[:36]:<36}" + "".join(f"{points(c):>13}" for c in table[feature]))passes = [all(points(table[f][i]) == 2 for f in a.must) for i in range(len(SHORT))]print("fails a must-have:", ", ".join(s for s, ok in zip(SHORT, passes) if not ok) or "none")scores = {s: sum(points(table[f][i]) for f in a.want) for i, s in enumerate(SHORT) if passes[i]}print("score of the rest:", ", ".join(f"{s} {n} of {2 * len(a.want)}" for s, n in scores.items()))points()reads only the first word of a cell. A must-have needsyes(2 points); anything less removes the product. The wanted rows then give 2, 1 or 0 points to the products that are left. -
Score the sheet with Larkfield’s four must-haves and five wanted features:
Terminal window python score_gateways.py \--must "One identity per agent" --must "Budgets per agent and team (money)" \--must "Tool control (MCP)" --must "An OpenAI-compatible endpoint" \--want "Capability names instead of models" --want "Fallbacks" --want "Rate limits and token quotas" \--want "Kill switches" --want "OpenTelemetry"Terminal window python score_gateways.py `--must "One identity per agent" --must "Budgets per agent and team (money)" `--must "Tool control (MCP)" --must "An OpenAI-compatible endpoint" `--want "Capability names instead of models" --want "Fallbacks" --want "Rate limits and token quotas" `--want "Kill switches" --want "OpenTelemetry"Terminal window python score_gateways.py \--must "One identity per agent" --must "Budgets per agent and team (money)" \--must "Tool control (MCP)" --must "An OpenAI-compatible endpoint" \--want "Capability names instead of models" --want "Fallbacks" --want "Rate limits and token quotas" \--want "Kill switches" --want "OpenTelemetry"ours LiteLLM Agent Router Cloudflare Azure APIMmust One identity per agent 2 2 1 1 2must Budgets per agent and team (money) 2 2 1 2 1must Tool control (MCP) 2 2 2 0 2must An OpenAI-compatible endpoint 2 2 2 2 2want Capability names instead of models 2 2 2 2 2want Fallbacks 2 2 2 2 2want Rate limits and token quotas 2 2 2 1 2want Kill switches 2 1 0 1 2want OpenTelemetry 1 2 2 2 1fails a must-have: Agent Router, Cloudflare, Azure APIMscore of the rest: ours 9 of 10, LiteLLM 9 of 10
Read the output from the top. The must rows remove three products. Agent Router has a partial identity and a token-only budget. Cloudflare’s identity is partial, and its tool control is a separate product. Azure API Management fails on one row only: its quotas count tokens, not money. The want rows then score the two products that are left. The last line is a tie: ours 9, LiteLLM 9. Each loses one point: ours on OpenTelemetry, which the course never tested, LiteLLM on kill switches, which block one key at a time.
A tie is a useful result. It says that the features do not decide between these two. The decision moves to the rows that the script does not score. They are the licence, what it needs to run, the supply chain and who runs it. That is the next lesson’s question.
Larkfield: why these must-haves
Section titled “Larkfield: why these must-haves”Each of Kwame’s must-haves comes from something that the course saw happen.
- One identity per agent: the shared key
az-key-1hid which app spent what. Priya’s personal key was found only by discovery in Module 6. - Budgets in money: the teams’ budgets are in US$ (the service team has US$40 a month). A token quota is not a money budget when one capability maps to models with different prices. Chat-strong costs 20 times more per input token than chat-small. So after a remap, the same token quota is a different amount of money.
- Tool control: the workflow’s refunds are tool calls, and the model gateway never sees them.
- An OpenAI-compatible endpoint: every app moved by changing its base URL and its credential, nothing else.
Kwame keeps Larkfield’s own files in any case: the registrations, the catalog and its contracts, the remap gate, the approval policy and the evidence. No product holds them, as the matrix shows. The product, bought or built, only needs to apply them.
Text description of the diagram
At the top are the apps. Each knows only a base URL, its own credential and capability names such as fast-small. Every call goes to the gateway product, the part that Larkfield can replace. It routes the requests, applies keys, limits and guardrails, writes its own logs and calls the model providers. On the left are Larkfield’s own files, the source of truth. They are the 7 registrations and the catalog with its contracts. They also hold the evaluations, the remap gate, the approval policy with its 12 tests, the tools file and the usage evidence. They are translated into the product’s configuration. The product’s usage rows come back, one row per call, and the evidence is made from them. To change the product, Larkfield writes a new translation, runs each consumer’s evaluation through it and moves the apps one by one.
Common mistake
A total score hides a missing must-have
Symptom: a first run scores eleven rows as wanted, with no --must: the nine rows of the worked example, plus Guardrails and Registration before the first call. The last two lines are:
fails a must-have: nonescore of the rest: ours 21 of 22, LiteLLM 20 of 22, Agent Router 14 of 22, Cloudflare 15 of 22, Azure APIM 19 of 22Azure API Management looks close to the top, with 19 of 22.
Cause: a total adds points across rows, so a strong row pays for a missing one. Azure scores 2 on identity, kill switches, tool control and guardrails. That hides its 1 on budgets in money, a row that Larkfield cannot do without.
Diagnose: ask, for each row, “would we choose a product without this?” If the answer is no, the row is a must-have.
Recover: move those rows to --must. A product that fails one is out, whatever its total. If no product passes, the must-have needs work: build that part, or change the need. For example, convert each team’s US$ budget into token quotas per capability, and redo it after every remap.
Your turn: a larger company
Section titled “Your turn: a larger company”Change one condition: the company. A larger company has 40 teams, three providers and a security team that collects every trace. Its must-haves are one identity per agent, rate limits and token quotas, fallbacks, OpenTelemetry and guardrails. Budgets, tool control, kill switches and registration are wanted.
Predict first: which products pass all five must-haves? And does our gateway pass?
-
Score the sheet with these needs:
Terminal window python score_gateways.py \--must "One identity per agent" --must "Rate limits and token quotas" --must "Fallbacks" \--must "OpenTelemetry" --must "Guardrails" \--want "Budgets per agent and team (money)" --want "Tool control (MCP)" --want "Kill switches" \--want "Registration before the first call"On Windows, end the lines with a backtick, not a backslash.
-
Run it again with OpenTelemetry as a wanted feature, not a must-have.
-
Delete the practice file:
rm score_gateways.py(Windows:Remove-Item score_gateways.py).
Check your prediction
ours LiteLLM Agent Router Cloudflare Azure APIMmust One identity per agent 2 2 1 1 2must Rate limits and token quotas 2 2 2 1 2must Fallbacks 2 2 2 2 2must OpenTelemetry 1 2 2 2 1must Guardrails 2 2 0 2 2want Budgets per agent and team (money) 2 2 1 2 1want Tool control (MCP) 2 2 2 0 2want Kill switches 2 1 0 1 2want Registration before the first call 2 1 0 0 1fails a must-have: ours, Agent Router, Cloudflare, Azure APIMscore of the rest: LiteLLM 6 of 8Only LiteLLM passes, and our gateway fails: its OpenTelemetry is partial. With OpenTelemetry as wanted, three products pass:
fails a must-have: Agent Router, Cloudflarescore of the rest: ours 9 of 10, LiteLLM 8 of 10, Azure APIM 7 of 10One row moved from must to want, and the answer changed from one product to three. So the must-haves are the real decision, and they deserve an argument with reasons, not a guess. Also check what “yes” means for a large company: LiteLLM’s guardrails per key or team are in its enterprise edition.
When to compare, and what it costs
Section titled “When to compare, and what it costs”Make the sheet before any build-or-buy decision, and again before each yearly review. It costs a few hours of reading documentation. The rows come from your controls, so most of the work is already done.
Do not trust a score alone. The script counts features. It cannot count who runs the product at night, what the licence costs, or a supply-chain incident. Use it to remove products that fail a must-have, then decide with people.
Check again. One product in this lesson changed its name while the course was written. Before you decide, open the current documentation for every cell you rely on, and update the sheet.
Ask the vendor about “not found”. A missing cell may be a missing page, not a missing feature. A short question to the vendor is cheaper than a wrong decision.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'Azure API Management has budgets, it says so on the sheet: partly.' Why does Larkfield treat that cell as a fail?
Show the answer
The cell says partly because Azure’s AI policies limit tokens per minute and per period, not money. Larkfield’s budgets are in US$ per agent and team. A token quota on strong-reasoning stands for a different amount of money after a remap, because the models’ prices differ. Chat-strong costs 20 times more per input token than chat-small. So for a must-have row, partly is not enough.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two cells say 'not found' and 'no'. What is the difference, and what do you do with each?
Show the answer
“No” means that the documentation says the product does not do it, for example Cloudflare’s tool control, which is in a separate product. “Not found” means that the course team did not find it in the documentation. It is an open question, not a fact. For a “no” on a must-have, the product is out. For a “not found” on a must-have, ask the vendor or test it before you decide.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Our gateway and LiteLLM tie at 9 of 10 for Larkfield. Name two things the score does not count that should decide between them.
Show the answer
Any two of these. Who runs and secures it: ours is a teaching project with no security review, and Larkfield would maintain it alone. What it needs to run: LiteLLM needs PostgreSQL and, for several instances, Redis. The licence of the features you need: some LiteLLM features are in its commercial enterprise edition. The supply chain: a gateway holds every provider key, so how its packages are built and signed matters, as LiteLLM’s compromised PyPI versions showed. And the failure behaviour of its store, which you must choose per risk tier.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A comparison sheet has your controls as rows and products as columns. Each product’s current documentation fills its column, with a source per cell and the version read.
not foundis a question, not ano. A must-have removes a product; only then do points count.- The data path (OpenAI-compatible endpoint, model names, fallbacks) is in every product. Registration records and contract checks were in none: keep them as your own files.
- A feature list hides the licence, what the product needs to run, its failure behaviour and its supply-chain risk. “Build” also means that you run it for years.
Your project now has docs/gateway-comparison.md, filled for four products. You can score it for any company’s needs. But a sheet does not move one app, and the best product protects nothing if the teams go around it. Next, you plan the rollout at Larkfield, from inventory to the last side door, and you measure the friction it causes.