Skip to content
Français

Gateways compared

  • Lesson 20
  • 40 min
  • Module 7: Build, buy and roll out

Where we are

In Module 6, we turned the gateway's records into evidence, and we found the AI that nobody registered. Every part of that platform is code that we wrote, so we know what each part does. But Larkfield does not have to run its own gateway: open-source and managed products exist. In this lesson, we compare ours with four of them, from their own documentation. We start at a workshop that built a go-kart.

What you will be able to do

Compare your own gateway with open-source and managed products, feature by feature, from each product’s own documentation. Separate the must-haves from the nice-to-haves. Then look at what a feature list hides: the licence, what the product needs to run, its supply-chain risk and who runs it.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project as Module 6 left it (194 tests pass), in ~/projects/ai-estate, with the virtual environment active.
  • No account, key or trial of any product. You read the products’ documentation; you install nothing.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.

A delivery company needs vans. Its mechanics built a go-kart in the workshop first, to learn how an engine, brakes and steering work. Now they can read a van’s specification sheet and know what each line means.

The go-kart will never deliver parcels. So the company makes a sheet: one row for each thing that its drivers need, and one column for each van. It fills the columns from the makers’ own specification sheets, not from a salesman or from memory. Some rows are must-haves: a van without a tail lift is out, however good the rest is. And the buyer asks two more questions that no specification sheet answers. Who services it, and what does it cost to change the brand in three years?

The answer to the last question depends on the company, not on the van. If the drivers’ routes and the delivery records are the company’s own, a new brand of van is a small change.

Kwame asks Tomás for a first comparison of the products that the platform team talked about. Tomás writes it in ten minutes, from what he remembers. This draft is constructed for the course:

Envoy AI Gateway: an Envoy add-on, no limits that know tokens
Azure API Management: has budgets per team
LiteLLM: budgets and keys; no product counts the steps of an agent run
Cloudflare AI Gateway: logs and caching only

Each line is wrong, or no longer true, when you read the products’ documentation:

Products change every few weeks. A comparison from memory decides with old facts, and nobody can check it later, because it has no sources. Worse, it compares what the products are famous for, not what Larkfield needs.

The idea: a comparison sheet from the documentation

Section titled “The idea: a comparison sheet from the documentation”

A comparison sheet is a table with one row per control that you need and one column per product. Each cell comes from that product’s own current documentation, with a link to the page. Three rules make it trustworthy:

  1. The rows are your controls, not the products’ features. You built every control of the course, so you know what each row means. That is the go-kart’s value.
  2. Every cell starts with one word: yes, partly, no or not found, then what the documentation says, then a source. “Not found” is not “no”. It means that you did not find it, and it is a question to ask the vendor.
  3. The sheet names the version you read. Then everyone knows when to check it again.

The decision is called build or buy: run your own code, or use a product, open-source or managed. The cost of changing products later is the switching cost. A product that is hard to leave gives you lock-in.

Open the sheet in your project. Copy it from the snapshot end-of-m07:

Terminal window
cp ../nextia-learning-labs/C32/snapshots/end-of-m07/docs/gateway-comparison.md docs/

The sheet has 19 feature rows, one row that names the version read, and a list of sources under the table. The picture shows the first word of each cell, plus the licence and what each product needs to run.

A matrix of 14 features by five gateways: ours, LiteLLM, Agent Router, Cloudflare AI Gateway and Azure API Management. Every product has capability names (Azure in preview), an OpenAI-compatible endpoint and fallbacks. Feature checks in a contract: only ours; not found for the four products. Registration before the first call: ours yes, LiteLLM and Azure partly, Agent Router and Cloudflare not found. Budgets in money: ours, LiteLLM and Cloudflare yes, Agent Router and Azure partly (tokens). Step limit per run: ours and LiteLLM yes, the others not found. Tool control: Cloudflare no, the others yes. Self-hosted: Cloudflare no, Azure partly. OpenTelemetry: ours and Azure partly. Licences: MIT, MIT core with commercial parts, Apache-2.0, two proprietary services.
The comparison sheet in one picture. Every product cell comes from that product's documentation, with its source in docs/gateway-comparison.md. Check the current documentation before you decide.
Text description of the diagram

Rows are features, columns are gateways. Capability names, an OpenAI-compatible endpoint and fallbacks are yes everywhere, with Azure API Management’s model aliases in preview. Feature checks in a contract are yes only for ours and not found for all four products. Registration before the first call: ours yes, LiteLLM partly, Agent Router not found, Cloudflare not found, Azure partly. One identity per agent: ours, LiteLLM and Azure yes, Agent Router and Cloudflare partly. Budgets in money: ours, LiteLLM and Cloudflare yes, Agent Router and Azure partly, because they count tokens. Rate limits and token quotas: Cloudflare partly, the others yes. Step limit per run: ours and LiteLLM yes, the others not found. Guardrails: Agent Router not found, the others yes. Tool control: Cloudflare no, because it is a separate product, the others yes. Kill switches: ours and Azure yes, LiteLLM and Cloudflare partly, Agent Router not found. OpenTelemetry: ours and Azure partly, the others yes. Self-hosted: Cloudflare no, Azure partly, the others yes. Licences: ours MIT, LiteLLM MIT with commercial parts, Agent Router Apache-2.0, Cloudflare and Azure proprietary services.

Three patterns matter more than any one cell.

The data path is the same everywhere. Every product takes OpenAI-compatible requests, hides the provider’s model behind a name of your choice, and falls back to another backend. You can buy this part.

The records about agents are rare. No product’s documentation shows a registration with an owner, a purpose, a risk tier and a review date before the first call. LiteLLM’s keys have an owner and a team. Azure’s subscriptions can need an administrator’s approval. Neither holds what Module 3’s form holds. No product checks a contract’s features, as estate catalog check does. These parts are Larkfield’s own knowledge, not traffic.

The differences are in limits, tools and hosting. Budgets in money are in LiteLLM and Cloudflare. Agent Router and Azure count tokens. Tool control through MCP is in three products, and in a separate Cloudflare product. Cloudflare runs only as a service. Azure’s self-hosted gateway exists in two tiers and still gets its configuration from Azure.

Sam asks: “Our column is green almost everywhere. So is ours the best?” Kwame answers: “The rows are the controls that we built. A sheet made from your own design always favours your design. Look at the rows the picture leaves out.”

Four rows of the sheet are not features. They decide who does the work.

  • Licence. LiteLLM’s core is MIT, but its enterprise/ folder has a separate commercial licence (licence). Its documentation marks some features as enterprise, such as sign-in with JWT and guardrails per key. Agent Router is Apache-2.0 throughout. Cloudflare and Azure are proprietary services. Read the licence of each feature you need, not only of the project.
  • What it needs to run. LiteLLM needs PostgreSQL for keys and budgets, and Redis for more than one instance (production guide). Agent Router needs Kubernetes and Envoy Gateway (prerequisites). Each one is another service that your team runs, upgrades and secures.
  • What happens when its own store is down. You chose this per risk tier in Module 4. LiteLLM’s documentation says that calls fail by default when its database cannot be reached, with a setting that keeps keys working. Agent Router’s quota checks fail open by default when the rate-limit service is down. Azure’s self-hosted gateway keeps running on its last configuration. Each is a choice that you must know before an outage, not during it.
  • Supply-chain risk. A gateway holds every provider key, so its code is a target. LiteLLM’s own security update describes two versions on PyPI that were compromised: 1.82.7 and 1.82.8. Both open-source projects publish security advisories (LiteLLM, Agent Router). Our gateway has 10 pinned packages, 35 when installed with what they need. It is a teaching project: it was tested on one laptop and never had a security review.

The last point is the honest cost of “build”. The course’s code shows every part of a gateway. Running it for Larkfield means someone patches it, reviews it, wakes up when it fails and answers for it, for years.

A tiny worked example: score the sheet for Larkfield

Section titled “A tiny worked example: score the sheet for Larkfield”

A sheet with 19 rows and five columns is hard to read at once. A small script makes the decision visible, step by step. It is a practice file, not part of the project.

  1. Create the file score_gateways.py in the project folder, with this content:

    score_gateways.py
    """Score the gateways of docs/gateway-comparison.md for one company's needs (a practice file: not part of the project)."""
    import argparse
    POINTS = {"yes": 2, "partly": 1} # "no", "not found" and everything else: 0 points
    SHORT = ["ours", "LiteLLM", "Agent Router", "Cloudflare", "Azure APIM"]
    ap = argparse.ArgumentParser()
    ap.add_argument("--must", action="append", default=[], help="a feature that must be 'yes'")
    ap.add_argument("--want", action="append", default=[], help="a feature that earns points")
    a = ap.parse_args()
    with open("docs/gateway-comparison.md", encoding="utf-8") as f:
    rows = [line.strip().strip("|").split(" | ") for line in f if line.startswith("| ")]
    table = {r[0].strip(): r[1:] for r in rows[1:]}
    assert len(rows[0]) - 1 == len(SHORT), "the sheet has another set of products"
    def points(cell: str) -> int:
    return POINTS.get(cell.split(":")[0].split()[0], 0)
    print(f"{'':<42}" + "".join(f"{s:>13}" for s in SHORT))
    for kind, features in (("must", a.must), ("want", a.want)):
    for feature in features:
    print(f"{kind} {feature[:36]:<36}" + "".join(f"{points(c):>13}" for c in table[feature]))
    passes = [all(points(table[f][i]) == 2 for f in a.must) for i in range(len(SHORT))]
    print("fails a must-have:", ", ".join(s for s, ok in zip(SHORT, passes) if not ok) or "none")
    scores = {s: sum(points(table[f][i]) for f in a.want) for i, s in enumerate(SHORT) if passes[i]}
    print("score of the rest:", ", ".join(f"{s} {n} of {2 * len(a.want)}" for s, n in scores.items()))

    points() reads only the first word of a cell. A must-have needs yes (2 points); anything less removes the product. The wanted rows then give 2, 1 or 0 points to the products that are left.

  2. Score the sheet with Larkfield’s four must-haves and five wanted features:

    Terminal window
    python score_gateways.py \
    --must "One identity per agent" --must "Budgets per agent and team (money)" \
    --must "Tool control (MCP)" --must "An OpenAI-compatible endpoint" \
    --want "Capability names instead of models" --want "Fallbacks" --want "Rate limits and token quotas" \
    --want "Kill switches" --want "OpenTelemetry"
    ours LiteLLM Agent Router Cloudflare Azure APIM
    must One identity per agent 2 2 1 1 2
    must Budgets per agent and team (money) 2 2 1 2 1
    must Tool control (MCP) 2 2 2 0 2
    must An OpenAI-compatible endpoint 2 2 2 2 2
    want Capability names instead of models 2 2 2 2 2
    want Fallbacks 2 2 2 2 2
    want Rate limits and token quotas 2 2 2 1 2
    want Kill switches 2 1 0 1 2
    want OpenTelemetry 1 2 2 2 1
    fails a must-have: Agent Router, Cloudflare, Azure APIM
    score of the rest: ours 9 of 10, LiteLLM 9 of 10

Read the output from the top. The must rows remove three products. Agent Router has a partial identity and a token-only budget. Cloudflare’s identity is partial, and its tool control is a separate product. Azure API Management fails on one row only: its quotas count tokens, not money. The want rows then score the two products that are left. The last line is a tie: ours 9, LiteLLM 9. Each loses one point: ours on OpenTelemetry, which the course never tested, LiteLLM on kill switches, which block one key at a time.

A tie is a useful result. It says that the features do not decide between these two. The decision moves to the rows that the script does not score. They are the licence, what it needs to run, the supply chain and who runs it. That is the next lesson’s question.

Each of Kwame’s must-haves comes from something that the course saw happen.

  • One identity per agent: the shared key az-key-1 hid which app spent what. Priya’s personal key was found only by discovery in Module 6.
  • Budgets in money: the teams’ budgets are in US$ (the service team has US$40 a month). A token quota is not a money budget when one capability maps to models with different prices. Chat-strong costs 20 times more per input token than chat-small. So after a remap, the same token quota is a different amount of money.
  • Tool control: the workflow’s refunds are tool calls, and the model gateway never sees them.
  • An OpenAI-compatible endpoint: every app moved by changing its base URL and its credential, nothing else.

Kwame keeps Larkfield’s own files in any case: the registrations, the catalog and its contracts, the remap gate, the approval policy and the evidence. No product holds them, as the matrix shows. The product, bought or built, only needs to apply them.

Three layers. At the top, the apps: a base URL, their own credential and capability names. They send every call to the gateway product, the part you can replace: it takes the requests and routes them, applies keys, limits and guardrails, and writes its own logs. It calls the model providers: azure, azure-canada and local. On the left, Larkfield's own files, the source of truth: registrations (7 forms), the catalog, the evaluations and the remap gate, the policy with 12 tests, the tools file and the usage evidence. An arrow from the files to the product: translated into the product's configuration. A dashed arrow back: usage rows exported, one row per call. Below: to change product, write a new translation, run each consumer's evaluation through it, move the apps one by one.
Own the knowledge, rent or replace the machine. The files on the left are Larkfield's whatever the product; only their translation is product-specific.
Text description of the diagram

At the top are the apps. Each knows only a base URL, its own credential and capability names such as fast-small. Every call goes to the gateway product, the part that Larkfield can replace. It routes the requests, applies keys, limits and guardrails, writes its own logs and calls the model providers. On the left are Larkfield’s own files, the source of truth. They are the 7 registrations and the catalog with its contracts. They also hold the evaluations, the remap gate, the approval policy with its 12 tests, the tools file and the usage evidence. They are translated into the product’s configuration. The product’s usage rows come back, one row per call, and the evidence is made from them. To change the product, Larkfield writes a new translation, runs each consumer’s evaluation through it and moves the apps one by one.

Common mistake

A total score hides a missing must-have

Symptom: a first run scores eleven rows as wanted, with no --must: the nine rows of the worked example, plus Guardrails and Registration before the first call. The last two lines are:

fails a must-have: none
score of the rest: ours 21 of 22, LiteLLM 20 of 22, Agent Router 14 of 22, Cloudflare 15 of 22, Azure APIM 19 of 22

Azure API Management looks close to the top, with 19 of 22.

Cause: a total adds points across rows, so a strong row pays for a missing one. Azure scores 2 on identity, kill switches, tool control and guardrails. That hides its 1 on budgets in money, a row that Larkfield cannot do without.

Diagnose: ask, for each row, “would we choose a product without this?” If the answer is no, the row is a must-have.

Recover: move those rows to --must. A product that fails one is out, whatever its total. If no product passes, the must-have needs work: build that part, or change the need. For example, convert each team’s US$ budget into token quotas per capability, and redo it after every remap.

Change one condition: the company. A larger company has 40 teams, three providers and a security team that collects every trace. Its must-haves are one identity per agent, rate limits and token quotas, fallbacks, OpenTelemetry and guardrails. Budgets, tool control, kill switches and registration are wanted.

Predict first: which products pass all five must-haves? And does our gateway pass?

  1. Score the sheet with these needs:

    Terminal window
    python score_gateways.py \
    --must "One identity per agent" --must "Rate limits and token quotas" --must "Fallbacks" \
    --must "OpenTelemetry" --must "Guardrails" \
    --want "Budgets per agent and team (money)" --want "Tool control (MCP)" --want "Kill switches" \
    --want "Registration before the first call"

    On Windows, end the lines with a backtick, not a backslash.

  2. Run it again with OpenTelemetry as a wanted feature, not a must-have.

  3. Delete the practice file: rm score_gateways.py (Windows: Remove-Item score_gateways.py).

Check your prediction
ours LiteLLM Agent Router Cloudflare Azure APIM
must One identity per agent 2 2 1 1 2
must Rate limits and token quotas 2 2 2 1 2
must Fallbacks 2 2 2 2 2
must OpenTelemetry 1 2 2 2 1
must Guardrails 2 2 0 2 2
want Budgets per agent and team (money) 2 2 1 2 1
want Tool control (MCP) 2 2 2 0 2
want Kill switches 2 1 0 1 2
want Registration before the first call 2 1 0 0 1
fails a must-have: ours, Agent Router, Cloudflare, Azure APIM
score of the rest: LiteLLM 6 of 8

Only LiteLLM passes, and our gateway fails: its OpenTelemetry is partial. With OpenTelemetry as wanted, three products pass:

fails a must-have: Agent Router, Cloudflare
score of the rest: ours 9 of 10, LiteLLM 8 of 10, Azure APIM 7 of 10

One row moved from must to want, and the answer changed from one product to three. So the must-haves are the real decision, and they deserve an argument with reasons, not a guess. Also check what “yes” means for a large company: LiteLLM’s guardrails per key or team are in its enterprise edition.

Make the sheet before any build-or-buy decision, and again before each yearly review. It costs a few hours of reading documentation. The rows come from your controls, so most of the work is already done.

Do not trust a score alone. The script counts features. It cannot count who runs the product at night, what the licence costs, or a supply-chain incident. Use it to remove products that fail a must-have, then decide with people.

Check again. One product in this lesson changed its name while the course was written. Before you decide, open the current documentation for every cell you rely on, and update the sheet.

Ask the vendor about “not found”. A missing cell may be a missing page, not a missing feature. A short question to the vendor is cheaper than a wrong decision.

Check your understanding

Tomás says: 'Azure API Management has budgets, it says so on the sheet: partly.' Why does Larkfield treat that cell as a fail?

Show the answer

The cell says partly because Azure’s AI policies limit tokens per minute and per period, not money. Larkfield’s budgets are in US$ per agent and team. A token quota on strong-reasoning stands for a different amount of money after a remap, because the models’ prices differ. Chat-strong costs 20 times more per input token than chat-small. So for a must-have row, partly is not enough.

Check your understanding

Two cells say 'not found' and 'no'. What is the difference, and what do you do with each?

Show the answer

“No” means that the documentation says the product does not do it, for example Cloudflare’s tool control, which is in a separate product. “Not found” means that the course team did not find it in the documentation. It is an open question, not a fact. For a “no” on a must-have, the product is out. For a “not found” on a must-have, ask the vendor or test it before you decide.

Check your understanding

Our gateway and LiteLLM tie at 9 of 10 for Larkfield. Name two things the score does not count that should decide between them.

Show the answer

Any two of these. Who runs and secures it: ours is a teaching project with no security review, and Larkfield would maintain it alone. What it needs to run: LiteLLM needs PostgreSQL and, for several instances, Redis. The licence of the features you need: some LiteLLM features are in its commercial enterprise edition. The supply chain: a gateway holds every provider key, so how its packages are built and signed matters, as LiteLLM’s compromised PyPI versions showed. And the failure behaviour of its store, which you must choose per risk tier.

Recap

  • A comparison sheet has your controls as rows and products as columns. Each product’s current documentation fills its column, with a source per cell and the version read.
  • not found is a question, not a no. A must-have removes a product; only then do points count.
  • The data path (OpenAI-compatible endpoint, model names, fallbacks) is in every product. Registration records and contract checks were in none: keep them as your own files.
  • A feature list hides the licence, what the product needs to run, its failure behaviour and its supply-chain risk. “Build” also means that you run it for years.

Your project now has docs/gateway-comparison.md, filled for four products. You can score it for any company’s needs. But a sheet does not move one app, and the best product protects nothing if the teams go around it. Next, you plan the rollout at Larkfield, from inventory to the last side door, and you measure the friction it causes.

Next: Roll it out to an organisation