Skip to content
Français

A capability catalog

  • Lesson 4
  • 40 min
  • Module 2: Ask for capabilities, not models

Where we are

In Module 1, we started Larkfield's estate and saw that every app calls a model by its own name, with its own key. So when a model changes, every app must change, and nobody can see which apps depend on which model. In this lesson, apps ask for a capability instead, and we write down what each capability promises, including the features that differ between models. A car rental shows the idea.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Give every model call a name for what it needs: a capability. Then write down what each capability promises: the features, a quality floor, a latency, the data it may receive and a price. You will see, with real recorded errors, why the features must be part of that promise.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • Your project ~/projects/ai-estate, the same as the snapshot end-of-m01: python -m pytest -q prints 91 passed, 1 deselected.
  • The labs repository in ~/projects/nextia-learning-labs, with the snapshot end-of-m02.
  • One terminal with the virtual environment active. Port 8320 must be free.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them. Every model answer in this lesson is replayed: a real recording from an earlier course.

You book a rental car for a family trip. You do not book “the grey Toyota with plate KX 512”. You book a class: “a mid-size car, five seats, automatic”. The company decides which car you get. When the grey Toyota is in the garage, you get a blue Hyundai, and your trip still works.

But one detail can ruin the trip. Your family tows a small trailer. Most mid-size cars at this branch have a tow bar, but not all. If your booking says only “mid-size”, you may get a car without one, and you find out in the car park. So you write it in the booking: “mid-size, five seats, automatic, with a tow bar”. Now the company must give you a car that has one, or tell you before the trip.

A booking class is a promise with conditions. You must write in it the conditions that matter to you.

Today, each of Larkfield’s apps names a model deployment in its code: chat-small or chat-strong. That is like booking a car by its plate. When Kwame wants to move the triage service to a newer or cheaper model, Amira must change the triage code. When he wants to know which apps depend on chat-strong, he must read every app.

The plate problem is not the worst part. Models are not interchangeable, and the difference is not only quality. Try it yourself. The agents course built a support agent that calls tools: it reads an order with get_order, then proposes an action. Run the same agent, with the same code, on both deployments, with and without one setting.

  1. Start the estate:

    Terminal window
    python -m estate init
    python -m estate up
  2. Point the agents course’s workflow at the provider simulator. These three variables are its own settings:

    Terminal window
    export RESOLVER_PROVIDER=openai_compatible
    export RESOLVER_BASE_URL=http://127.0.0.1:8320/azure/v1
    export RESOLVER_API_KEY=sim-az-service-team-0001
  3. Run the tool-calling agent on ticket T-90103 with chat-small:

    Terminal window
    RESOLVER_MODEL=chat-small python -m resolver run T-90103 --variant agent
    Run T-90103-agent-chat-small-1 | single agent with tool calling: the model chooses every step | chat-small
    Stopped: provider_error. No proposal: the ticket goes to a person.
    Last error: HTTP 400: Error code: 400 - {'error': {'message': "Function tools with reasoning_effort are not supported for chat-small in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': None}}
  4. Run it again with the setting that the error asks for, reasoning_effort set to none:

    Terminal window
    RESOLVER_MODEL=chat-small RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agent

    The first lines (the output goes on with the proposal and a reply draft):

    Run T-90103-agent-chat-small-2 | single agent with tool calling: the model chooses every step | chat-small
    Outcome: resolve (rule W5) | 2 model call(s), 1 tool call(s) | 4618 tokens in, 220 out | US$0.00057
  5. Now the same two runs with chat-strong. On Windows, first run Remove-Item Env:RESOLVER_REASONING_EFFORT:

    Terminal window
    RESOLVER_MODEL=chat-strong python -m resolver run T-90103 --variant agent
    RESOLVER_MODEL=chat-strong RESOLVER_REASONING_EFFORT=none python -m resolver run T-90103 --variant agent

    The last line of each run:

    Last error: HTTP 400: Error code: 400 - {'error': {'message': "Function tools with reasoning_effort are not supported for chat-strong in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': None}}
    Last error: HTTP 400: Error code: 400 - {'error': {'message': "Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', 'medium', 'high', and 'xhigh'.", 'type': 'invalid_request_error', 'param': 'reasoning_effort', 'code': 'unsupported_value'}}
  6. Remove the workflow’s settings, so they do not change later commands:

    Terminal window
    unset RESOLVER_PROVIDER RESOLVER_BASE_URL RESOLVER_API_KEY
    python -m resolver reset

    reset prints Fresh practice database: …/state/work/larkfield.sqlite. The runs changed nothing in the practice data: the one proposal only waited for an approval that you did not give.

Every error here is real. The simulator refuses these requests with the text that the real Azure deployments gave in the earlier courses. This course recorded the first one; the LLM applications course recorded the two for chat-strong. The one answer that worked is the agents course’s recording.

A grid of four results for the agents course's tool-calling agent on ticket T-90103, with the same code. Columns: chat-small and chat-strong. Rows: no setting, and reasoning_effort none. chat-small with no setting: HTTP 400, function tools with reasoning_effort are not supported for chat-small, the ticket goes to a person. chat-small with reasoning_effort none: HTTP 200, resolve by rule W5, 2 model calls and 1 tool call, a reship and a free return label that wait for approval. chat-strong with no setting: HTTP 400, function tools are not supported for chat-strong. chat-strong with reasoning_effort none: HTTP 400, reasoning_effort does not support none with this model. Only one of four runs works.
The same agent on two deployments. Only one combination of model and setting can call tools through this API.
Text description of the diagram

The grid has two columns, chat-small and chat-strong, and two rows: no setting, and reasoning_effort set to none. Three of the four cells are red. chat-small without the setting answers HTTP 400: function tools are not supported with its default reasoning setting. chat-small with reasoning_effort none answers, and the agent proposes a reship and a free return label for T-90103, which wait for approval. chat-strong without the setting answers HTTP 400 too. chat-strong with reasoning_effort none answers HTTP 400 for another reason: this model does not accept none. So chat-strong cannot call function tools through the Chat Completions API at all.

Read the chat-strong errors closely. The first one says “set reasoning_effort to ‘none’”. You do, and the second one says that this model does not support none. The advice in the first error is right for chat-small and wrong for chat-strong. So the agent works on one deployment and fails on the other, whatever you set.

The failed calls cost nothing: python -m provsim bill shows only the two answered calls of the run that worked. But each failure sent a ticket to a person. Now imagine that Kwame moves an app with tool calls from chat-small to chat-strong because it is “better”. The app does not get better answers. It gets no answers.

The idea: ask for a capability, with a contract

Section titled “The idea: ask for a capability, with a contract”

The fix is the booking class. An app asks for a capability: a name for what the call needs, such as fast-small or strong-reasoning. The platform decides which model serves it. A capability catalog is the list of these capabilities, each with a contract: the promise that every model behind it must keep. The name describes the need, not the model, so the model can change and the name stays.

Larkfield’s contract has five parts:

Part What it promises The car rental
Features what a request may use: chat, structured_output (a JSON answer that follows a schema), tools, images, embeddings the tow bar
Quality floor a minimum score on a named evaluation “rated four stars or more”
Latency the time in which 95% of answers arrive (p95) “ready at 9:00”
Data classes the most sensitive data that a request may carry “may cross the border”
Price band the most that the platform may pay per million tokens “up to 60 a day”

A data class is a label for how sensitive data is. Larkfield has four, in config/data-classes.yaml: public, internal, personal (anything about a customer) and payment (card numbers: never sent to any model). Mei owns them.

Why name the evaluation, and not only write “good quality”? Because “good” means nothing that someone can check. “task_success of at least 0.95 on the agents course’s 70 resolution tasks” can run again on a new model. The answer is yes or no. You use exactly that in the last lesson of this module.

Some catalogs also record the context size: how many tokens a request may hold. Larkfield’s first catalog leaves it out. Add it when a consumer sends long inputs, for example many documents in one request.

Kwame’s catalog has four capabilities. Here is the card of one of them, printed by the command that you add in a moment:

Capability: strong-reasoning
purpose: multi-step decisions that need the strongest model - resolution plans, judging answers
features: chat, structured_output
quality: task_success >= 0.95 on agents course, 70 resolution tasks
latency p95: 12.0 s
data classes: public, internal, personal
price band: up to US$2.5 in / US$12.0 out per 1M tokens
serves today: azure/chat-strong (gpt-6.1-sol-2026-09-29)
fallback: azure-canada/chat-strong

Read it from the top:

  • features: chat and structured_output, but not tools. That is the recorded fact from the experiment: chat-strong cannot call tools here. So the contract does not promise tools, and an app that needs tools must ask for another capability.
  • quality: the support workflow’s evaluation, with its minimum. chat-strong scored 70 of 70 on it.
  • latency and price band: chat-strong costs US$2 in and US$10 out per million tokens, at the recordings’ prices. The band allows up to US$2.50 and US$12. So a slightly dearer model would still fit, but not one that costs five times more. Prices change: check the provider’s current price page.
  • data classes: up to personal, because the workflow reads customer tickets. Never payment.
  • serves today and fallback: the mapping, the subject of the next lesson. The model version in brackets is the exact model behind the deployment name.
A catalog card. At the top, support-workflow and release-judge ask for strong-reasoning: the app names a capability, never a model. The card is titled Capability: strong-reasoning. On the left, the contract, what it promises: features chat and structured_output; quality floor task_success at least 0.95 on the agents course's 70 tasks; latency p95 12 seconds; data classes public, internal and personal; price band up to US$2.50 in and US$12 out per million tokens. On the right, the mapping, what serves it: serves today azure chat-strong, model version gpt-6.1-sol-2026-09-29; below it a fallback, azure-canada chat-strong, a simulated second region, used if the first fails with 429, 5xx or a time-out. An arrow from the contract to the mapping says kept by. At the bottom: python -m estate catalog check, every mapping must keep the contract.
The catalog card of strong-reasoning: the contract is the promise; the mapping is what keeps it today.
Text description of the diagram

At the top, two apps, the support workflow and the release judge, ask for strong-reasoning. They name the capability, never a model. The card has two halves. The left half is the contract. It lists the features chat and structured output, and a quality floor of task success 0.95 on the agents course’s 70 tasks. It also gives a p95 latency of 12 seconds and the data classes public, internal and personal. Its price band is up to US$2.50 in and US$12 out per million tokens. The right half is the mapping: today azure chat-strong, model version gpt-6.1-sol-2026-09-29. Its fallback is the same deployment in a simulated second region. The command python -m estate catalog check makes sure that every mapping keeps the contract.

Kwame names the capabilities by the need of the apps, not by the vendor’s product names:

Capability For Features Quality floor p95 Price band (in / out)
fast-small triage, routing, short drafts chat, structured_output, tools decisions ≥ 0.85, evaluation course’s triage set 6 s US$0.50 / US$2.00
strong-reasoning resolution plans, judging answers chat, structured_output task_success ≥ 0.95, agents course’s 70 tasks 12 s US$2.50 / US$12
embed vectors for finding similar tickets embeddings none yet 1 s US$0.05 / –
vision photos that customers attach chat, images none yet 10 s US$2.50 / US$12

Each line has a reason. fast-small promises tools, because chat-small can call tools with the right setting. The triage requests’ recorded model latency had a p95 of 5.16 seconds (replayed), inside its 6 seconds. vision is in the catalog, but no model serves it yet: nothing in the recordings reads images. Listing it says “we plan this” instead of letting a team find its own provider. embed and vision have no quality floor yet. That is a gap, and the catalog shows it openly.

Sam asks: “Why not call them chat-small and chat-strong, like the deployments?” Kwame answers: “Because then the name tells you the model. When chat-small is served by another model next year, the name lies.” Tomás has the opposite worry: “So a capability name means the behaviour never changes?” No. The contract keeps the features, the data rules and the price. The answers of a new model will still differ. That is why the contract names an evaluation. And it is why changing the model is a release, as the last lesson of this module shows.

  1. Copy this lesson’s files from the snapshot end-of-m02: the folder config, the catalog module and its command. In ~/projects/ai-estate:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m02
    cp -R $S/config .
    cp $S/estate/catalog.py estate/
    cp $S/estate/cli/catalog.py estate/cli/
  2. Look at the four files in config/. catalog.yaml holds the contracts. models.yaml says what each deployment can do. providers.yaml and data-classes.yaml are for the next lesson. In models.yaml, find the line features: [chat, structured_output] of chat-strong, with the comment no function tools in Chat Completions (real HTTP 400s).

  3. List the catalog:

    Terminal window
    python -m estate catalog list
    fast-small azure/chat-small short structured answers at low cost - triage, routing, short drafts
    strong-reasoning azure/chat-strong multi-step decisions that need the strongest model - resolution plans, judging answers
    embed azure/embed-small vectors for finding similar tickets
    vision nothing yet (planned) read photos that customers attach
  4. Print one card: python -m estate catalog show strong-reasoning. It prints the card of the worked example above.

  5. Print the card of vision: python -m estate catalog show vision. The last line says serves today: nothing (planned), and the quality line says none >= None on none yet. That line is the gap.

  6. Check that every mapping keeps its contract:

    Terminal window
    python -m estate catalog check
    The catalog is consistent: every mapping keeps its contract.
  7. Run the tests. The last line still says 91 passed, 1 deselected: this lesson’s tests come with the next lesson.

catalog check compares each contract with models.yaml. It finds three kinds of problem. A contract feature that the model does not have. A price above the band. A data class that the provider’s contract does not allow. The check prints each problem and exits with code 1, so a build or a review can stop on it.

Your turn: promise tools on strong-reasoning

Section titled “Your turn: promise tools on strong-reasoning”

Amira would like the support workflow’s tool-calling agent on strong-reasoning. Predict first: what happens if Kwame adds tools to the contract of strong-reasoning? Which lines will catalog check print?

Work on a copy, so that the real catalog does not change:

  1. Copy the catalog: cp config/catalog.yaml my-catalog.yaml (Windows: Copy-Item config\catalog.yaml my-catalog.yaml).

  2. In my-catalog.yaml, under strong-reasoning: and contract:, change features: [chat, structured_output] to features: [chat, structured_output, tools]. Save the file.

  3. Check the copy:

    Terminal window
    python -m estate catalog check --file my-catalog.yaml
  4. Compare the output with your prediction.

Check your prediction
strong-reasoning -> azure/chat-strong: the contract promises tools, the model does not have it
strong-reasoning -> azure-canada/chat-strong: the contract promises tools, the model does not have it

Two lines, one for each place that the mapping may send a call: the primary and the fallback. The check found in a second what the agent found as a 400 in the experiment. The right answer to Amira is not to change the contract. Her tool-calling agent asks for fast-small, whose contract promises tools, or the team keeps its structured agent on strong-reasoning, which needs no function tools. Delete the copy when you are done: rm my-catalog.yaml (Windows: Remove-Item my-catalog.yaml).

Common mistake

A typo in a deployment name

Symptom: after an edit of the catalog, python -m estate catalog check prints this line and exits with code 1:

fast-small -> azure/chat-smal: no deployment chat-smal in config/models.yaml

Cause: the mapping of fast-small names a deployment, chat-smal, that models.yaml does not list. Without the check, the mistake would show only when the first call fails, for every app that asks for fast-small at once.

Diagnose: the line names the capability, the provider and the deployment. Search the catalog for that name, for example grep -n chat-smal config/catalog.yaml (Windows: Select-String chat-smal config\catalog.yaml).

Recover: correct the name, or add the deployment to models.yaml with the features that its real calls showed. Then run catalog check again until it prints The catalog is consistent. Run the check before every change goes out; Module 3 makes the configuration’s publish step run it.

Use it when several apps or agents share models and providers, as at Larkfield: six AI systems, owned by four teams. The catalog is then the one place that says what each model may be used for.

Do not use it for one app that calls one model. A catalog of one capability is a name for a name. Microsoft’s guidance on a gateway in front of several deployments says the same about the gateway itself. It helps in some situations, not in all.

Keep it small. Four capabilities are easy to reason about. One capability per app turns the catalog back into a list of models with new names. Add a capability only when a real consumer needs a different promise.

What it costs. Someone must own it: Kwame. models.yaml must stay true, so every new model’s features must come from real calls, as the 400s did here. And a contract is only as good as its evaluation. A capability without a quality floor, like embed today, has no evidence when its model changes.

Check your understanding

Amira's tool-calling agent works on chat-small with reasoning_effort none. Kwame wants to serve all chat calls from chat-strong for better quality. What happens to her agent, and how should the catalog prevent it?

Show the answer

Her agent stops at its first tool call with HTTP 400. chat-strong cannot call function tools through the Chat Completions API. Without a setting it refuses the tools. With reasoning_effort none it refuses the setting. Every ticket then goes to a person. The catalog prevents this because the agent asks for fast-small, whose contract lists tools. catalog check refuses any mapping of fast-small to a model without tools, so the change fails before it reaches any app.

Check your understanding

Why does the contract of strong-reasoning name the agents course's 70 resolution tasks and a minimum of 0.95, instead of saying 'the strongest model'?

Show the answer

“The strongest model” cannot be checked, and it changes as new models appear. A named evaluation with a minimum can run on any candidate model. The result is yes or no: task success of at least 0.95 on the same 70 tasks, or not. It also says whose evidence counts: the support workflow’s. When the model behind the capability changes, the platform runs that evaluation again.

Check your understanding

The vision capability is in the catalog, but nothing serves it. Is that useful or confusing? Give one reason for each side.

Show the answer

Useful: it tells teams that the platform plans to offer image reading, with its data classes and price band. So a team does not find its own provider for photos. Confusing: an app that asks for vision today gets no model, and the card has no quality floor yet. The card says “nothing (planned)” and “none yet” openly, so the gap is visible instead of hidden.

Recap

  • An app asks for a capability (fast-small, strong-reasoning, embed, vision), not a model. The capability catalog lists them.
  • Each capability has a contract: features, a quality floor on a named evaluation, a p95 latency, allowed data classes and a price band.
  • Features belong in the contract: the same agent got HTTP 400 on chat-small without reasoning_effort: none, and on chat-strong with or without it. Only chat-small with the setting called tools.
  • python -m estate catalog list | show | check lists the catalog, prints one card and refuses a mapping that breaks its contract.
  • embed and vision have no quality floor yet: a gap that the catalog shows.

Your project now has config/ and the estate catalog command; the tests still say 91 passed. Each capability now has a promise. But a promise does not say which model keeps it, where, or what happens when that provider is down. Next, you map each capability to a model and a provider, with a fallback and data rules.

Next: Map capabilities to models