A capability catalog
- Lesson 4
- 40 min
- Module 2: Ask for capabilities, not models
Where we are
In Module 1, we started Larkfield's estate and saw that every app calls a model by its own name, with its own key. So when a model changes, every app must change, and nobody can see which apps depend on which model. In this lesson, apps ask for a capability instead, and we write down what each capability promises, including the features that differ between models. A car rental shows the idea.
Lesson 4 of 21 · Governing AI Agents: Gateways, Registries and ControlsA capability catalog
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Give every model call a name for what it needs: a capability. Then write down what each capability promises: the features, a quality floor, a latency, the data it may receive and a price. You will see, with real recorded errors, why the features must be part of that promise.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s apps, their owners and their keys, from Five agents and nobody knows.
- How to start the estate and replay an app, from Meet Larkfield’s AI estate.
- What a tool call is: the model asks your code to run a function, as in the agents course.
What you need on your computer
- Your project
~/projects/ai-estate, the same as the snapshotend-of-m01:python -m pytest -qprints91 passed, 1 deselected. - The labs repository in
~/projects/nextia-learning-labs, with the snapshotend-of-m02. - One terminal with the virtual environment active. Port 8320 must be free.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them. Every model answer in this lesson is replayed: a real recording from an earlier course.
A car rental
Section titled “A car rental”You book a rental car for a family trip. You do not book “the grey Toyota with plate KX 512”. You book a class: “a mid-size car, five seats, automatic”. The company decides which car you get. When the grey Toyota is in the garage, you get a blue Hyundai, and your trip still works.
But one detail can ruin the trip. Your family tows a small trailer. Most mid-size cars at this branch have a tow bar, but not all. If your booking says only “mid-size”, you may get a car without one, and you find out in the car park. So you write it in the booking: “mid-size, five seats, automatic, with a tow bar”. Now the company must give you a car that has one, or tell you before the trip.
A booking class is a promise with conditions. You must write in it the conditions that matter to you.
The problem: an app that names a model
Section titled “The problem: an app that names a model”Today, each of Larkfield’s apps names a model deployment in its code: chat-small or chat-strong. That is like booking a car by its plate. When Kwame wants to move the triage service to a newer or cheaper model, Amira must change the triage code. When he wants to know which apps depend on chat-strong, he must read every app.
The plate problem is not the worst part. Models are not interchangeable, and the difference is not only quality. Try it yourself. The agents course built a support agent that calls tools: it reads an order with get_order, then proposes an action. Run the same agent, with the same code, on both deployments, with and without one setting.
-
Start the estate:
Terminal window python -m estate initpython -m estate up -
Point the agents course’s workflow at the provider simulator. These three variables are its own settings:
Terminal window export RESOLVER_PROVIDER=openai_compatibleexport RESOLVER_BASE_URL=http://127.0.0.1:8320/azure/v1export RESOLVER_API_KEY=sim-az-service-team-0001Terminal window $env:RESOLVER_PROVIDER = "openai_compatible"$env:RESOLVER_BASE_URL = "http://127.0.0.1:8320/azure/v1"$env:RESOLVER_API_KEY = "sim-az-service-team-0001"Terminal window export RESOLVER_PROVIDER=openai_compatibleexport RESOLVER_BASE_URL=http://127.0.0.1:8320/azure/v1export RESOLVER_API_KEY=sim-az-service-team-0001 -
Run the tool-calling agent on ticket T-90103 with
chat-small:Terminal window RESOLVER_MODEL=chat-small python -m resolver run T-90103 --variant agentTerminal window $env:RESOLVER_MODEL = "chat-small"; python -m resolver run T-90103 --variant agentTerminal window RESOLVER_MODEL=chat-small python -m resolver run T-90103 --variant agentRun T-90103-agent-chat-small-1 | single agent with tool calling: the model chooses every step | chat-smallStopped: provider_error. No proposal: the ticket goes to a person.Last error: HTTP 400: Error code: 400 - {'error': {'message': "Function tools with reasoning_effort are not supported for chat-small in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': None}} -
Run it again with the setting that the error asks for,
reasoning_effortset tonone:Terminal window RESOLVER_MODEL=chat-small RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agentTerminal window $env:RESOLVER_REASONING_EFFORT = "none"; python -m resolver run T-90103 --variant agentTerminal window RESOLVER_MODEL=chat-small RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agentThe first lines (the output goes on with the proposal and a reply draft):
Run T-90103-agent-chat-small-2 | single agent with tool calling: the model chooses every step | chat-smallOutcome: resolve (rule W5) | 2 model call(s), 1 tool call(s) | 4618 tokens in, 220 out | US$0.00057 -
Now the same two runs with
chat-strong. On Windows, first runRemove-Item Env:RESOLVER_REASONING_EFFORT:Terminal window RESOLVER_MODEL=chat-strong python -m resolver run T-90103 --variant agentRESOLVER_MODEL=chat-strong RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agentTerminal window $env:RESOLVER_MODEL = "chat-strong"; python -m resolver run T-90103 --variant agent$env:RESOLVER_REASONING_EFFORT = "none"; python -m resolver run T-90103 --variant agentTerminal window RESOLVER_MODEL=chat-strong python -m resolver run T-90103 --variant agentRESOLVER_MODEL=chat-strong RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agentThe last line of each run:
Last error: HTTP 400: Error code: 400 - {'error': {'message': "Function tools with reasoning_effort are not supported for chat-strong in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': None}}Last error: HTTP 400: Error code: 400 - {'error': {'message': "Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', 'medium', 'high', and 'xhigh'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': 'unsupported_value'}} -
Remove the workflow’s settings, so they do not change later commands:
Terminal window unset RESOLVER_PROVIDER RESOLVER_BASE_URL RESOLVER_API_KEYpython -m resolver resetTerminal window Remove-Item Env:RESOLVER_PROVIDER, Env:RESOLVER_BASE_URL, Env:RESOLVER_API_KEY, Env:RESOLVER_MODEL, Env:RESOLVER_REASONING_EFFORTpython -m resolver resetTerminal window unset RESOLVER_PROVIDER RESOLVER_BASE_URL RESOLVER_API_KEYpython -m resolver resetresetprintsFresh practice database: …/state/work/larkfield.sqlite. The runs changed nothing in the practice data: the one proposal only waited for an approval that you did not give.
Every error here is real. The simulator refuses these requests with the text that the real Azure deployments gave in the earlier courses. This course recorded the first one; the LLM applications course recorded the two for chat-strong. The one answer that worked is the agents course’s recording.
Text description of the diagram
The grid has two columns, chat-small and chat-strong, and two rows: no setting, and reasoning_effort set to none. Three of the four cells are red. chat-small without the setting answers HTTP 400: function tools are not supported with its default reasoning setting. chat-small with reasoning_effort none answers, and the agent proposes a reship and a free return label for T-90103, which wait for approval. chat-strong without the setting answers HTTP 400 too. chat-strong with reasoning_effort none answers HTTP 400 for another reason: this model does not accept none. So chat-strong cannot call function tools through the Chat Completions API at all.
Read the chat-strong errors closely. The first one says “set reasoning_effort to ‘none’”. You do, and the second one says that this model does not support none. The advice in the first error is right for chat-small and wrong for chat-strong. So the agent works on one deployment and fails on the other, whatever you set.
The failed calls cost nothing: python -m provsim bill shows only the two answered calls of the run that worked. But each failure sent a ticket to a person. Now imagine that Kwame moves an app with tool calls from chat-small to chat-strong because it is “better”. The app does not get better answers. It gets no answers.
The idea: ask for a capability, with a contract
Section titled “The idea: ask for a capability, with a contract”The fix is the booking class. An app asks for a capability: a name for what the call needs, such as fast-small or strong-reasoning. The platform decides which model serves it. A capability catalog is the list of these capabilities, each with a contract: the promise that every model behind it must keep. The name describes the need, not the model, so the model can change and the name stays.
Larkfield’s contract has five parts:
| Part | What it promises | The car rental |
|---|---|---|
| Features | what a request may use: chat, structured_output (a JSON answer that follows a schema), tools, images, embeddings |
the tow bar |
| Quality floor | a minimum score on a named evaluation | “rated four stars or more” |
| Latency | the time in which 95% of answers arrive (p95) | “ready at 9:00” |
| Data classes | the most sensitive data that a request may carry | “may cross the border” |
| Price band | the most that the platform may pay per million tokens | “up to 60 a day” |
A data class is a label for how sensitive data is. Larkfield has four, in config/data-classes.yaml: public, internal, personal (anything about a customer) and payment (card numbers: never sent to any model). Mei owns them.
Why name the evaluation, and not only write “good quality”? Because “good” means nothing that someone can check. “task_success of at least 0.95 on the agents course’s 70 resolution tasks” can run again on a new model. The answer is yes or no. You use exactly that in the last lesson of this module.
Some catalogs also record the context size: how many tokens a request may hold. Larkfield’s first catalog leaves it out. Add it when a consumer sends long inputs, for example many documents in one request.
A tiny worked example: one card
Section titled “A tiny worked example: one card”Kwame’s catalog has four capabilities. Here is the card of one of them, printed by the command that you add in a moment:
Capability: strong-reasoning purpose: multi-step decisions that need the strongest model - resolution plans, judging answers features: chat, structured_output quality: task_success >= 0.95 on agents course, 70 resolution tasks latency p95: 12.0 s data classes: public, internal, personal price band: up to US$2.5 in / US$12.0 out per 1M tokens serves today: azure/chat-strong (gpt-6.1-sol-2026-09-29) fallback: azure-canada/chat-strongRead it from the top:
- features:
chatandstructured_output, but nottools. That is the recorded fact from the experiment: chat-strong cannot call tools here. So the contract does not promise tools, and an app that needs tools must ask for another capability. - quality: the support workflow’s evaluation, with its minimum. chat-strong scored 70 of 70 on it.
- latency and price band: chat-strong costs US$2 in and US$10 out per million tokens, at the recordings’ prices. The band allows up to US$2.50 and US$12. So a slightly dearer model would still fit, but not one that costs five times more. Prices change: check the provider’s current price page.
- data classes: up to
personal, because the workflow reads customer tickets. Neverpayment. - serves today and fallback: the mapping, the subject of the next lesson. The model version in brackets is the exact model behind the deployment name.
Text description of the diagram
At the top, two apps, the support workflow and the release judge, ask for strong-reasoning. They name the capability, never a model. The card has two halves. The left half is the contract. It lists the features chat and structured output, and a quality floor of task success 0.95 on the agents course’s 70 tasks. It also gives a p95 latency of 12 seconds and the data classes public, internal and personal. Its price band is up to US$2.50 in and US$12 out per million tokens. The right half is the mapping: today azure chat-strong, model version gpt-6.1-sol-2026-09-29. Its fallback is the same deployment in a simulated second region. The command python -m estate catalog check makes sure that every mapping keeps the contract.
Larkfield: four capabilities
Section titled “Larkfield: four capabilities”Kwame names the capabilities by the need of the apps, not by the vendor’s product names:
| Capability | For | Features | Quality floor | p95 | Price band (in / out) |
|---|---|---|---|---|---|
fast-small |
triage, routing, short drafts | chat, structured_output, tools | decisions ≥ 0.85, evaluation course’s triage set | 6 s | US$0.50 / US$2.00 |
strong-reasoning |
resolution plans, judging answers | chat, structured_output | task_success ≥ 0.95, agents course’s 70 tasks | 12 s | US$2.50 / US$12 |
embed |
vectors for finding similar tickets | embeddings | none yet | 1 s | US$0.05 / – |
vision |
photos that customers attach | chat, images | none yet | 10 s | US$2.50 / US$12 |
Each line has a reason. fast-small promises tools, because chat-small can call tools with the right setting. The triage requests’ recorded model latency had a p95 of 5.16 seconds (replayed), inside its 6 seconds. vision is in the catalog, but no model serves it yet: nothing in the recordings reads images. Listing it says “we plan this” instead of letting a team find its own provider. embed and vision have no quality floor yet. That is a gap, and the catalog shows it openly.
Sam asks: “Why not call them chat-small and chat-strong, like the deployments?” Kwame answers: “Because then the name tells you the model. When chat-small is served by another model next year, the name lies.” Tomás has the opposite worry: “So a capability name means the behaviour never changes?” No. The contract keeps the features, the data rules and the price. The answers of a new model will still differ. That is why the contract names an evaluation. And it is why changing the model is a release, as the last lesson of this module shows.
Add the catalog to your project
Section titled “Add the catalog to your project”-
Copy this lesson’s files from the snapshot
end-of-m02: the folderconfig, the catalog module and its command. In~/projects/ai-estate:Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m02cp -R $S/config .cp $S/estate/catalog.py estate/cp $S/estate/cli/catalog.py estate/cli/Terminal window $S = "..\nextia-learning-labs\C32\snapshots\end-of-m02"Copy-Item -Recurse $S\config .Copy-Item $S\estate\catalog.py estate\Copy-Item $S\estate\cli\catalog.py estate\cli\Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m02cp -R $S/config .cp $S/estate/catalog.py estate/cp $S/estate/cli/catalog.py estate/cli/ -
Look at the four files in
config/.catalog.yamlholds the contracts.models.yamlsays what each deployment can do.providers.yamlanddata-classes.yamlare for the next lesson. Inmodels.yaml, find the linefeatures: [chat, structured_output]of chat-strong, with the commentno function tools in Chat Completions (real HTTP 400s). -
List the catalog:
Terminal window python -m estate catalog listfast-small azure/chat-small short structured answers at low cost - triage, routing, short draftsstrong-reasoning azure/chat-strong multi-step decisions that need the strongest model - resolution plans, judging answersembed azure/embed-small vectors for finding similar ticketsvision nothing yet (planned) read photos that customers attach -
Print one card:
python -m estate catalog show strong-reasoning. It prints the card of the worked example above. -
Print the card of
vision:python -m estate catalog show vision. The last line saysserves today: nothing (planned), and the quality line saysnone >= None on none yet. That line is the gap. -
Check that every mapping keeps its contract:
Terminal window python -m estate catalog checkThe catalog is consistent: every mapping keeps its contract. -
Run the tests. The last line still says
91 passed, 1 deselected: this lesson’s tests come with the next lesson.
catalog check compares each contract with models.yaml. It finds three kinds of problem. A contract feature that the model does not have. A price above the band. A data class that the provider’s contract does not allow. The check prints each problem and exits with code 1, so a build or a review can stop on it.
Your turn: promise tools on strong-reasoning
Section titled “Your turn: promise tools on strong-reasoning”Amira would like the support workflow’s tool-calling agent on strong-reasoning. Predict first: what happens if Kwame adds tools to the contract of strong-reasoning? Which lines will catalog check print?
Work on a copy, so that the real catalog does not change:
-
Copy the catalog:
cp config/catalog.yaml my-catalog.yaml(Windows:Copy-Item config\catalog.yaml my-catalog.yaml). -
In
my-catalog.yaml, understrong-reasoning:andcontract:, changefeatures: [chat, structured_output]tofeatures: [chat, structured_output, tools]. Save the file. -
Check the copy:
Terminal window python -m estate catalog check --file my-catalog.yaml -
Compare the output with your prediction.
Check your prediction
strong-reasoning -> azure/chat-strong: the contract promises tools, the model does not have itstrong-reasoning -> azure-canada/chat-strong: the contract promises tools, the model does not have itTwo lines, one for each place that the mapping may send a call: the primary and the fallback. The check found in a second what the agent found as a 400 in the experiment. The right answer to Amira is not to change the contract. Her tool-calling agent asks for fast-small, whose contract promises tools, or the team keeps its structured agent on strong-reasoning, which needs no function tools. Delete the copy when you are done: rm my-catalog.yaml (Windows: Remove-Item my-catalog.yaml).
Common mistake
A typo in a deployment name
Symptom: after an edit of the catalog, python -m estate catalog check prints this line and exits with code 1:
fast-small -> azure/chat-smal: no deployment chat-smal in config/models.yamlCause: the mapping of fast-small names a deployment, chat-smal, that models.yaml does not list. Without the check, the mistake would show only when the first call fails, for every app that asks for fast-small at once.
Diagnose: the line names the capability, the provider and the deployment. Search the catalog for that name, for example grep -n chat-smal config/catalog.yaml (Windows: Select-String chat-smal config\catalog.yaml).
Recover: correct the name, or add the deployment to models.yaml with the features that its real calls showed. Then run catalog check again until it prints The catalog is consistent. Run the check before every change goes out; Module 3 makes the configuration’s publish step run it.
When to use a catalog, and what it costs
Section titled “When to use a catalog, and what it costs”Use it when several apps or agents share models and providers, as at Larkfield: six AI systems, owned by four teams. The catalog is then the one place that says what each model may be used for.
Do not use it for one app that calls one model. A catalog of one capability is a name for a name. Microsoft’s guidance on a gateway in front of several deployments says the same about the gateway itself. It helps in some situations, not in all.
Keep it small. Four capabilities are easy to reason about. One capability per app turns the catalog back into a list of models with new names. Add a capability only when a real consumer needs a different promise.
What it costs. Someone must own it: Kwame. models.yaml must stay true, so every new model’s features must come from real calls, as the 400s did here. And a contract is only as good as its evaluation. A capability without a quality floor, like embed today, has no evidence when its model changes.
Knowledge check
Section titled “Knowledge check”Check your understanding
Amira's tool-calling agent works on chat-small with reasoning_effort none. Kwame wants to serve all chat calls from chat-strong for better quality. What happens to her agent, and how should the catalog prevent it?
Show the answer
Her agent stops at its first tool call with HTTP 400. chat-strong cannot call function tools through the Chat Completions API. Without a setting it refuses the tools. With reasoning_effort none it refuses the setting. Every ticket then goes to a person. The catalog prevents this because the agent asks for fast-small, whose contract lists tools. catalog check refuses any mapping of fast-small to a model without tools, so the change fails before it reaches any app.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the contract of strong-reasoning name the agents course's 70 resolution tasks and a minimum of 0.95, instead of saying 'the strongest model'?
Show the answer
“The strongest model” cannot be checked, and it changes as new models appear. A named evaluation with a minimum can run on any candidate model. The result is yes or no: task success of at least 0.95 on the same 70 tasks, or not. It also says whose evidence counts: the support workflow’s. When the model behind the capability changes, the platform runs that evaluation again.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The vision capability is in the catalog, but nothing serves it. Is that useful or confusing? Give one reason for each side.
Show the answer
Useful: it tells teams that the platform plans to offer image reading, with its data classes and price band. So a team does not find its own provider for photos. Confusing: an app that asks for vision today gets no model, and the card has no quality floor yet. The card says “nothing (planned)” and “none yet” openly, so the gap is visible instead of hidden.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An app asks for a capability (
fast-small,strong-reasoning,embed,vision), not a model. The capability catalog lists them. - Each capability has a contract: features, a quality floor on a named evaluation, a p95 latency, allowed data classes and a price band.
- Features belong in the contract: the same agent got HTTP 400 on chat-small without
reasoning_effort: none, and on chat-strong with or without it. Only chat-small with the setting called tools. python -m estate catalog list | show | checklists the catalog, prints one card and refuses a mapping that breaks its contract.embedandvisionhave no quality floor yet: a gap that the catalog shows.
Your project now has config/ and the estate catalog command; the tests still say 91 passed. Each capability now has a promise. But a promise does not say which model keeps it, where, or what happens when that provider is down. Next, you map each capability to a model and a provider, with a fallback and data rules.