A capability catalog from public evaluation results
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In this course, we wrote Larkfield's capability contracts from its own recorded evaluations, and the remap gate blocked a cheaper model because one consumer failed. Most organisations do not have evaluations of their own on the first day. In this case study, a company whose only model is about to retire builds its first catalog from public benchmark results and a public price file. The contracts, the gate and the rules are the course's; only the evidence is new.
Case study 1 of 3 · Governing AI Agents: Gateways, Registries and ControlsA capability catalog from public evaluation results
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Build a company’s first capability catalog from public evidence. You write the contracts and set each app’s own minimum. Then rules choose the mapping from a public price file and openly licensed benchmark results. You see how the mapping changes when the company’s only model retires, and when a second one would. You finish with a recommendation that says what the public data cannot tell you.
This case study is a small project folder, public-catalog, in the course’s own style: Python, YAML and one command line. The starter has the data, the loaders and the checks. You write a contract, a minimum and the gate’s decision, then let the project choose and write the mapping. The finished project is the result, for when you are stuck or want to compare. The code is under the MIT licence, so you can publish your version.
Before you startWhat to know and to have ready
What you need to know
This case study uses the catalog and the gate of the course. These lessons matter most:
- Capabilities and their contracts, from A capability catalog.
- Mappings, fallbacks and data rules, from Map capabilities to models.
- The remap gate per consumer, and why the average is the wrong check, from Change a mapping safely.
- The registration that names each consumer’s evaluation, from What a registration holds, and the review window of 60 days, from The agent lifecycle.
The page teaches one new idea: a public score belongs to one setting of the model.
What you need on your computer
- Python 3.12 or newer, VS Code and a terminal, as in the course.
- About 40 MB of free disk space for the virtual environment.
- No account, no key and no money. Everything runs on your computer from files in the project. One optional step downloads the latest public data (about 5 MB).
Tested on macOS (Apple silicon) with Python 3.12.13, PyYAML 6.0.3 and pytest 9.1.1, from a fresh copy of the starter. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them.
The situation
Section titled “The situation”Hollin Energy supplies electricity and gas to about 300,000 homes. Three teams built an AI app each, and each team copied the first team’s code:
- The help assistant (Joel, customer care) drafts answers to customers’ questions about their bills. An agent reads every draft before it goes out.
- The billing agent (Ada, billing) works out corrected bills and payment plans with tools. A person approves every change.
- The research assistant (Lena, network engineering) answers engineers’ questions about standards and equipment from memory, without a web search.
All three call the same model straight, with the same key: o3-2025-04-16 on the OpenAI API. Noor, the platform engineer, reads a public price file and finds a line next to it: deprecation_date: 2026-12-11. The provider retires the model in 50 days. On that day, all three apps stop at once.
Ben, from finance, has a quick answer: “Take the top model of a leaderboard and move everyone to it.” Noor has another plan. She wants the apps to ask for a capability, as the course taught, so that the next retirement changes one line instead of three apps. But Hollin has no evaluations of its own yet. So the first catalog must stand on public evidence, and it must say clearly what that evidence cannot show.
Hollin Energy, its people, apps, contracts and workloads are made up for this case study. The models, providers, scores, prices and dates are real public data, from the two sources below.
What you will build
Section titled “What you will build”| File | What it holds | Part |
|---|---|---|
config/models.yaml |
the public evidence for 61 models: scores by setting, offers by provider | The data |
config/catalog.yaml |
two contracts, fast-small and strong-reasoning, and their mapping |
The contracts |
config/consumers/*.yaml |
the three apps, with their own benchmark and minimum | Each app’s own minimum |
pubcat/gate.py |
the gate’s decision | The gate on public evidence |
docs/RECOMMENDATION.md |
the decision, the evidence, the cost and the limits | The recommendation |
Text description of the diagram
Two public files feed the project. Epoch AI’s results give a score per model and setting. LiteLLM’s price file gives the price, features and retirement date of each model at each provider. They become config/models.yaml, 61 models with 102 offers. With Hollin’s four providers, that makes 227 targets: a model, at a provider, with a setting. Each target goes through the contract check and then the gate, where every app must reach its own minimum. The cheapest target that passes becomes the mapping, with a fallback. A retirement date removes targets, and the same rules choose again.
Set up the project
Section titled “Set up the project”-
Download the starter project from the box at the top of this page. Unzip it into your projects folder, for example
~/projects. You get the folderpublic-catalog. Open it in VS Code. -
In the VS Code terminal, make a virtual environment and install the two packages:
Terminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements.txtTerminal window py -m venv .venv.venv\Scripts\Activate.ps1python -m pip install -r requirements.txtTerminal window python3 -m venv .venvsource .venv/bin/activatepython -m pip install -r requirements.txtThe packages are PyYAML 6.0.3 and pytest 9.1.1, the versions of the course project. Your course virtual environment works too.
-
Run the tests:
python -m pytest -q --tb=no. The last line says13 failed, 11 passed. That is expected: each failing test belongs to one of the four tasks, and its message names the task. At the end, all 24 pass.
The data
Section titled “The data”The project joins two public sources. Both are in data/ as a frozen copy, unchanged, with a file SHA256SUMS.
Epoch AI’s benchmark results. Epoch AI, a research organisation, runs public benchmarks on many models and publishes every result. The project uses three of its benchmark files. GPQA Diamond has 198 hard, graduate-level science questions. OTIS Mock AIME 2024–2025 has competition mathematics problems with exact answers. SimpleQA Verified has short factual questions, answered from memory without a search. One row is one model at one setting on one benchmark: the share of questions it answered correctly (mean_score) and its standard error (stderr).
LiteLLM’s price file. LiteLLM is an open-source gateway. Its project keeps one large file with an entry for each model at each provider. An entry has the list price per token and the features that LiteLLM records, such as function calling. It sometimes has a deprecation_date, the day the provider retires the model.
Noor compared other sources first. The LMArena leaderboard (CC BY 4.0) gives relative ratings from people’s votes, so a fixed minimum would drift as new models join. The Open LLM Leaderboard results have no licence that we could find, and cover only open-weight models. The providers’ own price pages have no open licence, so the case study links to them.
First checks
Section titled “First checks”Check that the files are the ones the case study used, and count what they hold:
python -m pubcat dataChecksums: 6 files match SHA256SUMS gpqa_diamond 325 results, 225 models otis_mock_aime_2024_2025 305 results, 204 models simpleqa_verified 92 results, 83 modelsModels with a public score: 234; with an offer at one of our providers: 61; without: 173What this tells us. The files are unchanged. Epoch AI tested many more models on GPQA Diamond than on SimpleQA Verified: a missing score will be common, and it is not a zero. Only 61 of the 234 models are offered by one of Hollin’s four providers. The other 173 are models of other providers, older models, or open-weight models without a hosted price. Hollin’s providers are in config/providers.yaml: OpenAI, Azure OpenAI, Google Cloud Vertex AI and Anthropic. Each has a contract with a data processing agreement, which is an assumption of the case study. A provider that is not in the file is never called, whatever its scores, as in Map capabilities to models.
The evidence file
Section titled “The evidence file”Build config/models.yaml, the evidence that the catalog reads, and look at today’s model:
python -m pubcat modelspython -m pubcat show o3-2025-04-16Wrote config/models.yaml: 61 models, 102 offersLeft out: 173 models with a public score but no offer at our providersModel: o3-2025-04-16 (OpenAI) gpqa_diamond low 0.798 ±0.029, medium 0.808 ±0.028, high 0.818 ±0.021 otis_mock_aime_2024_2025 low 0.600 ±0.074, medium 0.844 ±0.055, high 0.839 ±0.044 simpleqa_verified high 0.494 ±0.016 offer azure US$2.0 in / US$8.0 out per 1M tokens; chat, structured_output, tools, images; retires 2026-11-19 offer openai US$2.0 in / US$8.0 out per 1M tokens; chat, structured_output, tools, images; retires 2026-12-11What this tells us. Three things. First, one model has several scores: o3 answered 60% of the mathematics problems at low and 84% at high. Second, the same model retires on different days at different providers: 11 December 2026 at OpenAI, 19 November 2026 at Azure. A retirement date belongs to an offer, not to a model. Third, each score has a standard error. On 198 questions, ±0.029 is about six questions, so a difference of 0.01 between two models means nothing.
models.yaml is the course’s models.yaml with two additions: the public scores, and a price and retirement date per provider. Do not edit it by hand. Change the data or the providers, and build it again.
A public score belongs to one setting
Section titled “A public score belongs to one setting”Think of a car’s fuel use. The brochure says “4.2 litres per 100 km”. The small print says “at a steady 50 km/h”. In the city, with stops and starts, the same car uses 7 litres. The number is true, but only for the condition it was measured in.
Model scores work the same way. Many models take a reasoning setting, such as reasoning_effort: none, low, high or max. A higher setting lets the model reason longer before it answers. It answers better on hard questions, more slowly, and with more output tokens, which you pay for. Epoch AI writes the setting after the model’s name: gpt-5.6-luna_low is the model gpt-5.6-luna at low.
python -m pubcat show gpt-5.6-lunaModel: gpt-5.6-luna (OpenAI) gpqa_diamond none 0.636 ±0.034, low 0.823 ±0.027, max 0.916 ±0.017 otis_mock_aime_2024_2025 none 0.400 ±0.074, low 0.667 ±0.071, max 0.983 ±0.012 simpleqa_verified max 0.410 ±0.016 offer azure US$0.2 in / US$1.2 out per 1M tokens; chat, structured_output, tools, images; retires 2028-01-11 offer openai US$0.2 in / US$1.2 out per 1M tokens; chat, structured_output, tools, images; retires no date in the price fileSo the rule is: a score counts only for a mapping that sends the same setting. In the course, a mapping already carries its settings (params, from Map capabilities to models). The project looks up the score at the mapping’s setting. A missing score at that setting is no evidence.
- gpt-5.6-luna
- gpt-5.4-nano
- o3
Text description of the chart
Nine columns in three colours, one colour per model. gpt-5.6-luna: none 0.64, low 0.82, max 0.92. gpt-5.4-nano: none 0.56, low 0.72, high 0.78. o3: low 0.80, medium 0.81, high 0.82. The axis starts at 0.5. For the two small models, the setting moves the score by 0.2 to 0.3, more than the difference between most models. For o3, the setting moves it by 0.02.
Larkfield met the same effect in this course. In Change a mapping safely, chat-small, the model gpt-6-luna-2026-09-22, failed both of Larkfield’s consumers with reasoning_effort: none. Epoch AI lists the same model family only at max, with 0.905 on GPQA Diamond. Both are true. They measure different settings and different tasks.
The contracts
Section titled “The contracts”The catalog uses the course’s format. A capability has a contract (what it promises) and a mapping (what serves it). Two parts change, because the evidence is public:
settingslists the reasoning settings that a mapping may send. Public results cannot show latency, so this is Hollin’s rule for speed and cost:fast-smallallows onlynone,minimalandlow.quality_floornames a public benchmark, by its file name, and a minimum at the mapped setting.
Open config/catalog.yaml. The contract of fast-small is complete:
| Part | fast-small | strong-reasoning |
|---|---|---|
| For | short answers and drafts about bills | multi-step calculations, research answers |
| Features | chat, structured_output, tools | chat, structured_output, tools |
| Settings | none, minimal, low | low, medium, high, xhigh, max |
| Quality floor | GPQA Diamond ≥ 0.74 | OTIS Mock AIME 2024–2025 ≥ 0.75 |
| Data classes | public, internal, personal | public, internal, personal |
| Price band (in / out, US$ per 1M tokens) | 0.50 / 2.00 | 2.50 / 12.00 |
Why GPQA Diamond for the help assistant? No public benchmark asks bill questions. GPQA Diamond has the most models at low settings. So it is the widest screen for “reasons well enough when it answers fast”, not a test of the task. Why OTIS Mock AIME for strong-reasoning? Its problems need several exact steps in a row, as a corrected bill does. The two price bands are Larkfield’s, from A capability catalog.
Task 1: complete the strong-reasoning contract
Section titled “Task 1: complete the strong-reasoning contract”-
In
config/catalog.yaml, understrong-reasoning:andcontract:, changeminimum: nulltominimum: 0.75. The floor is the billing agent’s minimum, the capability’s main consumer, which you meet in the next part. -
Change
price_band_usd_per_1m: {input: null, output: null}toprice_band_usd_per_1m: {input: 2.50, output: 12.00}. -
Run the task’s test:
python -m pytest -q -k strong_reasoning_contract. It prints1 passed, 23 deselected. -
Check today’s mapping against the contracts:
Terminal window python -m pubcat checkfast-small -> openai/o3-2025-04-16: the input price 2.0 is above the contract's band 0.5fast-small -> openai/o3-2025-04-16: the output price 8.0 is above the contract's band 2.0fast-small -> openai/o3-2025-04-16: retires on 2026-12-11 (50 days), inside the 60-day review windowstrong-reasoning -> openai/o3-2025-04-16: retires on 2026-12-11 (50 days), inside the 60-day review window
What this tells us. The starter’s catalog maps both capabilities to o3, as the apps call it today. Each line names the capability, the target and the rule it breaks, as estate catalog check does in the course. Two problems are new. The retirement window: a target that retires within 60 days fails the check, because a remap takes time. That is the review window of The agent lifecycle. And the price band: the help assistant pays a strong model’s price for short drafts. That was true before the retirement too, but nobody had written the band down.
Each app’s own minimum
Section titled “Each app’s own minimum”The course’s gate runs each consumer’s own evaluation against its own minimum. Hollin has no evaluations yet, so each app names a public benchmark and a minimum in its file in config/consumers/. Each app also has a workload: the tokens it sends and receives in a month. These are the owners’ estimates, an assumption.
What minimum? Each owner chose the same rule: no worse than today. Today’s model is o3 at the app’s setting. A score has an uncertainty, so “no worse” means “not lower than today’s score minus two standard errors”. A smaller drop could be noise.
| App (owner) | Capability | Benchmark | Today (o3) | Minimum | Workload a month (assumption) |
|---|---|---|---|---|---|
| Help assistant (Joel) | fast-small | GPQA Diamond | 0.798 ± 0.029 at low | 0.798 − 2 × 0.029 = 0.74 | 40 M tokens in, 6 M out |
| Billing agent (Ada) | strong-reasoning | OTIS Mock AIME | 0.839 ± 0.044 at high | 0.839 − 2 × 0.044 = 0.75 | 8 M in, 2 M out |
| Research assistant (Lena) | strong-reasoning | SimpleQA Verified | 0.494 ± 0.016 at high | your task | 3 M in, 1 M out |
Task 2: the research assistant’s minimum
Section titled “Task 2: the research assistant’s minimum”- Find today’s score and standard error of o3 on SimpleQA Verified in the output of
python -m pubcat show o3-2025-04-16above. - Compute today’s score minus two standard errors, and round it down to two decimals.
- In
config/consumers/research-assistant.yaml, replaceminimum: nulland its comment with your number. Write the sum inwhy:. - Run
python -m pytest -q -k research_assistants_minimum. It prints1 passed, 23 deselected.
Check your answer
0.494 − 2 × 0.016 = 0.462, rounded down to 0.46. The line becomes minimum: 0.46, and why: can say "no worse than today: 0.494 - 2 x 0.016 = 0.462, rounded down to 0.46".
Lena’s minimum is low in absolute terms: o3 answers fewer than half of the questions. That is the point of “no worse than today”. The research assistant is not good at facts from memory today, and the remap must not make it worse. A better assistant needs a search tool, not a new model.
The gate on public evidence
Section titled “The gate on public evidence”Task 3: write the gate’s decision
Section titled “Task 3: write the gate’s decision”Try the gate before you write it, on one of the cheapest models in the file, gpt-6-luna:
python -m pubcat gate strong-reasoning --to openai/gpt-6-luna --param reasoning_effort=maxIt stops with NotImplementedError: Task 3: write the gate's decision in pubcat/gate.py.
- Open
pubcat/gate.pyand finddecide(). It gets one row per consumer, each with a"verdict":"pass","fail"or"no evidence". - Replace the three Task 3 lines with one line. It returns
"pass"if there is a row and every verdict is"pass". Otherwise it returns"block". - Run
python -m pytest -q -k every_consumer. It prints4 passed, 20 deselected. The four cases are: both pass, one fails, one without evidence, and no consumer at all.
Check your code
def decide(rows: list[dict]) -> str: """The gate's decision: 'pass' only if there is at least one consumer and EVERY consumer passes its own minimum. A failed consumer or a consumer without evidence blocks the remap. The average does not count.""" return "pass" if rows and all(r["verdict"] == "pass" for r in rows) else "block"rows and makes a capability without consumers block: with nobody to pass, nothing is proven.
The gate on three candidates
Section titled “The gate on three candidates”Predict first: gpt-6-luna at max costs 20 times less per token than o3. Does it pass both consumers of strong-reasoning? Does the average of their scores pass the average of their minimums?
python -m pubcat gate strong-reasoning --to openai/gpt-6-luna --param reasoning_effort=maxRemap gate: strong-reasoning from openai/o3-2025-04-16 {'reasoning_effort': 'high'} to openai/gpt-6-luna {'reasoning_effort': 'max'}
billing-agent PASS otis_mock_aime_2024_2025 0.989 (minimum 0.75); today 0.839; US$1.80 vs US$32.00 a month today research-assistant FAIL simpleqa_verified 0.414 (minimum 0.46); today 0.494; US$0.80 vs US$14.00 a month today
Average of the consumers' scores: 0.701 (average of their minimums: 0.605)Decision: BLOCK - every consumer must pass its own evaluationWhat this tells us. This is the case of Change a mapping safely, now with real public data and no constructed consumer. The average of the two scores, 0.701, is well above the average minimum, 0.605. A leaderboard that averages benchmarks would rank this model high. But the research assistant would lose 8 points of factual accuracy, and only Lena’s own minimum shows it.
Two more candidates show the other outcomes:
python -m pubcat gate strong-reasoning --to anthropic/claude-haiku-5-5 --param reasoning_effort=highpython -m pubcat gate strong-reasoning --to vertex/gemini-3.8-flash --param reasoning_effort=high billing-agent PASS otis_mock_aime_2024_2025 0.972 (minimum 0.75); today 0.839; US$1.80 vs US$32.00 a month today research-assistant NO EVIDENCE no public simpleqa_verified score for claude-haiku-5-5 at high: no evidence, no remapDecision: BLOCK - every consumer must pass its own evaluation
billing-agent PASS otis_mock_aime_2024_2025 0.989 (minimum 0.75); today 0.839; US$13.50 vs US$32.00 a month today research-assistant PASS simpleqa_verified 0.697 (minimum 0.46); today 0.494; US$6.00 vs US$14.00 a month todayDecision: PASS(The first and the blank lines of each run are left out here.) Epoch AI ran claude-haiku-5-5 on SimpleQA Verified only at max. At high, Lena has no evidence, so the gate blocks: “no evidence, no remap”, as in the course. gemini-3.8-flash at high passes both.
Explore what the decision needs
Section titled “Explore what the decision needs”Ben’s question is about money, and Noor’s is about rules. One table answers both. List every target that keeps the strong-reasoning contract, cheapest first, with each consumer’s verdict:
python -m pubcat candidates strong-reasoning --limit 16strong-reasoning: 227 targets, 43 keep the contract, 13 also pass every consumer. Cheapest first: target US$/month billing-agent research-assistant azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'} 1.75 pass 0.811 fail 0.117 openai/gpt-6-luna {'reasoning_effort': 'max'} 2.60 pass 0.989 fail 0.414 azure/gpt-6-luna {'reasoning_effort': 'max'} 2.60 pass 0.989 fail 0.414 anthropic/claude-haiku-5-5 {'reasoning_effort': 'high'} 2.60 pass 0.972 no evidence anthropic/claude-haiku-5-5 {'reasoning_effort': 'xhigh'} 2.60 pass 0.989 no evidence vertex/claude-haiku-5-5 {'reasoning_effort': 'high'} 2.60 pass 0.972 no evidence vertex/claude-haiku-5-5 {'reasoning_effort': 'xhigh'} 2.60 pass 0.989 no evidence openai/gpt-5.6-luna {'reasoning_effort': 'max'} 5.80 pass 0.983 fail 0.410 azure/gpt-5.6-luna {'reasoning_effort': 'max'} 5.80 pass 0.983 fail 0.410 openai/gpt-5.4-nano-2026-03-17 {'reasoning_effort': 'high'} 5.95 pass 0.878 fail 0.117 azure/gpt-5.4-nano-2026-03-17 {'reasoning_effort': 'high'} 5.95 pass 0.878 fail 0.117 vertex/gemini-3.1-flash-lite {'reasoning_effort': 'high'} 7.25 pass 0.800 no evidence azure/gpt-5-mini-2025-08-07 {'reasoning_effort': 'high'} 8.75 pass 0.867 fail 0.216 azure/gpt-5-mini-2025-08-07 {'reasoning_effort': 'medium'} 8.75 pass 0.783 no evidence vertex/gemini-3.8-flash {'reasoning_effort': 'high'} 19.50 pass 0.989 pass 0.697 vertex/gemini-3.7-flash {'reasoning_effort': 'high'} 19.50 pass 0.972 pass 0.692What this tells us. A target is a model at a provider with a setting, so one model gives several rows. The billing agent passes on every row: the contract’s floor is its minimum. The research assistant decides. Mathematics and facts from memory are different skills. The small models solve the competition problems and miss most of the facts, as the gpt-5-nano row shows, with 0.811 and 0.117. Note also azure/gpt-5-nano-2025-08-07: the OpenAI offer of the same model is not in the list, because it retires on 11 December 2026. Azure’s retires on 9 February 2027, outside the window.
- passes every rule and consumer
- blocked by the research assistant
Text description of the chart
Eighteen dots. The cheapest five, from US$1.75 to US$8.75 a month, all lie below Lena’s minimum of 0.46: gpt-5-nano at high scores 0.117, gpt-6-luna at max 0.414. The first dots above the line are gemini-3.8-flash and gemini-3.7-flash at high, at US$19.50, with 0.697 and 0.692. Further right, from US$43.50 to US$58, are larger models: some above the line, such as gpt-6.1-sol at max with 0.739 at US$52, and some below. Price and factual accuracy are only loosely related.
The funnel shows how many targets each rule removes. choose prints it for each capability:
strong-reasoning: all targets (model x provider x setting) 227 is not a preview model 219 has the contract's features 201 uses an allowed setting 141 inside the price band 80 the provider may receive the data 80 does not retire inside the review window 56 reaches the contract's floor 43 every consumer passes its own minimum 13Each rule is one line of check_target() in pubcat/catalog.py. Preview models are out because providers change or remove a preview at short notice. The data step removes none here: every one of Hollin’s providers may receive personal data, an assumption. The retirement window removes 24 targets. Ten of them are at OpenAI and retire on 11 December 2026, the same day as o3. Six more retire at OpenAI on 23 October 2026.
A baseline and three alternatives, under one measure
Section titled “A baseline and three alternatives, under one measure”Ben asked for the top of a leaderboard. That is a fair alternative, so the project compares four ways to choose, with one measure. Each way must keep every rule: the contract and every consumer’s minimum. Then the lowest monthly cost wins.
python -m pubcat compareapproach broken rules US$/month1. keep today's model 4 174.00 fast-small: openai/o3-2025-04-16 {'reasoning_effort': 'low'} - price; retire strong-reasoning: openai/o3-2025-04-16 {'reasoning_effort': 'high'} - retire2. top of a leaderboard 3 192.00 fast-small: vertex/gemini-4-argon {'reasoning_effort': 'high'} - price; settings strong-reasoning: vertex/gemini-4-argon {'reasoning_effort': 'high'} - keeps every rule3. cheapest that keeps the contract 1 16.95 fast-small: openai/gpt-5.6-luna {'reasoning_effort': 'low'} - keeps every rule strong-reasoning: azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'} - research-assistant below its minimum4. cheapest that passes every consumer 0 34.70 fast-small: openai/gpt-5.6-luna {'reasoning_effort': 'low'} - keeps every rule strong-reasoning: vertex/gemini-3.8-flash {'reasoning_effort': 'high'} - keeps every rule
US$/month: the consumers' assumed workloads at list prices per token (more output tokens at a higher setting are not counted).What this tells us.
- Keep today’s model is the baseline. It costs US$174 a month and stops on 11 December.
- The top of a leaderboard is the model with the highest average of the three public scores at one setting: gemini-4-argon at
high. Forstrong-reasoningit keeps every rule. But it costs more than today. For the help assistant, it breaks the price band and the setting rule. - The cheapest that keeps the contract looks like a 90% saving. It sends Lena’s questions to a model that gets 12% of the facts right.
- The cheapest that passes every consumer breaks no rule and costs US$34.70 a month, 80% less than today.
Only the fourth keeps every promise. It costs twice as much as the third, and the difference, US$17.75 a month, is the price of not breaking the research assistant.
The recommendation
Section titled “The recommendation”Task 4: let the rules choose, and write the mapping
Section titled “Task 4: let the rules choose, and write the mapping”-
Propose a mapping for every capability:
Terminal window python -m pubcat chooseFor each capability, it prints the funnel and its proposal. For
strong-reasoning:cheapest that keeps the contract: azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'}, US$1.75 a month - blocked by the gate (research-assistant)proposed primary: vertex/gemini-3.8-flash {'reasoning_effort': 'high'}, US$19.50 a month, smallest margin +0.237proposed fallback: azure/gpt-5-2025-08-07 {'reasoning_effort': 'high'} (a different model: every consumer passed its gate)For
fast-small:openai/gpt-5.6-lunaatlow, US$15.20 a month, withazure/gpt-5.6-lunaatlowas the fallback: the same model and setting at another provider. At the end, a diff shows the mapping today and the proposal. -
Write the proposal into the catalog:
python -m pubcat choose --write. The old mapping stays in the file asprevious_mapping, the record of the change. -
Check the catalog:
python -m pubcat checkprintsThe catalog is consistent: every mapping keeps its contract. -
Run all the tests:
python -m pytest -q. The last line says24 passed. -
Open
docs/RECOMMENDATION.md. Fill in its five sections for Hollin’s app owners and Ben, with the outputs above. The finished project has one version of it.
How does the chooser decide between equal prices? gemini-3.8-flash and gemini-3.7-flash cost the same. A tie goes to the target with the larger smallest margin: its lowest score minus minimum over all consumers, the one furthest from failing. Here that is 0.237 against 0.222. Then it goes to the provider that the capability uses today, the smallest change. That is why fast-small stays at OpenAI and Azure is the fallback.
The fallbacks follow the course’s rule. For fast-small, the fallback is the same model and setting at another provider. An outage changes nothing for the help assistant. For strong-reasoning, no other of Hollin’s providers offers gemini-3.8-flash. So the fallback is a different model: gpt-5-2025-08-07 at high, the cheapest that passes at another provider. Map capabilities to models calls that “a remap in disguise”. It is allowed only because every consumer’s gate passed on it too.
Text description of the diagram
Before the remap, all three apps call o3-2025-04-16 at OpenAI, which retires in 50 days. After it, the help assistant asks for fast-small: gpt-5.6-luna at low, with the same model at Azure as the fallback. The billing agent and the research assistant ask for strong-reasoning: gemini-3.8-flash at high on Vertex AI, with gpt-5 at high on Azure as the fallback.
Noor’s recommendation, in short: fast-small on gpt-5.6-luna at low, and strong-reasoning on gemini-3.8-flash at high. That is about US$34.70 a month at list prices, instead of US$174. Each app changes once, from a model name to a capability name. Before the change goes live, each owner checks the new mapping on their own work. The limits below say why.
Run it on new input
Section titled “Run it on new input”A catalog is not done when it is written. Retirements, new models and new prices arrive every month. Here are three new inputs.
1. A second retirement (a what-if). Suppose the provider announced today that gpt-5.6-luna retires, which it has not. The --retire option removes a model’s offers for one run:
python -m pubcat choose --retire gpt-5.6-lunaWhat-if: gpt-5.6-luna retired today (2026-10-22).
fast-small: reaches the contract's floor 1 every consumer passes its own minimum 1 proposed primary: vertex/gemini-3.1-flash-lite {'reasoning_effort': 'low'}, US$19.00 a month, smallest margin +0.002 proposed fallback: none - no other provider passes(The first lines of the funnel, and the part for strong-reasoning, whose proposal does not change, are left out here.)
What this tells us. The mapping moves, and nobody touches an app. But look at what is left. One target passes, by 0.002 over Joel’s minimum. Its standard error is 0.031, so the public evidence cannot tell it apart from a model that fails. It has no fallback, and show gemini-3.1-flash-lite says that it retires on 7 May 2027. The rules found an answer, but the evidence is weak. Noor adds a next step: before such a day, Joel runs his own evaluation on two or three candidates.
2. A later day. The default day is the day of the frozen data. Check the written catalog on the day o3 retires:
python -m pubcat check --today 2026-12-11strong-reasoning -> azure/gpt-5-2025-08-07: retires on 2027-02-09 (60 days), inside the 60-day review windowThe fallback enters its own review window that day. Run check on a schedule, and the next remap starts 60 days early, not on the day an app fails.
3. The latest public data. data/fetch.py downloads the current versions of both sources into data/latest/, with their checksums:
python data/fetch.pypython -m pubcat --data data/latest datapython -m pubcat --data data/latest --models config/models-latest.yaml modelspython -m pubcat --models config/models-latest.yaml choosepython data\fetch.pypython -m pubcat --data data\latest datapython -m pubcat --data data\latest --models config\models-latest.yaml modelspython -m pubcat --models config\models-latest.yaml choosepython data/fetch.pypython -m pubcat --data data/latest datapython -m pubcat --data data/latest --models config/models-latest.yaml modelspython -m pubcat --models config/models-latest.yaml chooseWhen the course froze the data, the latest files were the same. The last line said The proposed mapping is the same as today's. Your results will differ: new models, scores, prices and dates arrive. Compare your proposal with the one above, and read every line of the diff before you write it.
Common mistake
A data file that changed
Symptom: python -m pubcat data prints this and exits with code 1:
epoch/simpleqa_verified.csv: wrong checksum (the file changed)Some data files are missing or changed. Download the project again, or run python data/fetch.py.Cause: a file in data/ is not the file in SHA256SUMS. The case study caused it on purpose: one score, o3’s 0.494, was “corrected” to 0.594 in an editor. The catalog would then accept models for the research assistant that the real data refuses. The same happens if a spreadsheet program saves a CSV file again.
Diagnose: the line names the file. Compare it with the copy in the downloaded zip, for example with git diff --no-index.
Recover: unzip the file from the starter again, then run python -m pubcat data until it prints Checksums: 6 files match SHA256SUMS. Never change public evidence by hand. If you disagree with a score, write it in the recommendation, with your reason.
Insights and limits
Section titled “Insights and limits”- Public results make a shortlist; they do not decide. No benchmark here is a bill question or an engineer’s question. The rules and the gate removed 214 of 227 targets, and they can still pass a model that fails Hollin’s real work. Each owner runs a sample of their own tasks first, as the course’s gate does.
- A cost belongs to a setting too. A model at
maxwrites many more output tokens than atlow. Neither file has token counts per task. The cost ofstrong-reasoningis a floor, not a forecast. Measure the tokens of the first week. - Features need one real call. The price file says every candidate can call functions. In A capability catalog, one model needed
reasoning_effort: nonefor tools, and another could not call tools at all. Ada makes one real tool call on each new target. - A missing date is not “never”. 41 of the 102 offers have no retirement date. Treat them as unknown.
- The prices are a third party’s record, and they can be late. Finance checks each one on the provider’s page.
- Benchmark questions can leak into a model’s training data, which raises its score. Your own tasks do not have that problem.
Responsible use
Section titled “Responsible use”- Credit the sources. CC BY 4.0 asks for credit to Epoch AI wherever you use its data. The MIT licence asks you to keep LiteLLM’s licence note with its file. Keep both in your version.
- A public score is not a verdict on a vendor. This ranking follows Hollin’s made-up rules and workloads. Another organisation would choose differently. Say “for these contracts”, not “the best model”.
- Label every assumption where it is used. The contracts, the workloads and the prices are assumptions; the second retirement is a what-if.
- The owners decide. Noor runs the gate and proposes the mapping. Each owner signs off on their own result. Ben’s saving is a reason to look, not an approval.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer two things. You can turn “which model should we use?” into rules that anyone can check. And you know where public evidence stops. Make it yours with one change: other apps, other benchmarks from Epoch AI’s archive, or another provider list.
What to put in the repository:
README.md(template below).- The project, with your
config/files, the tests passing, anddata/with its licences and checksums. docs/RECOMMENDATION.md: the decision, the evidence, the cost and the limits.- The output of
python -m pubcat compareandchoose, and the diff of the mapping. - The two credits from the sources box above.
A README template:
# A capability catalog from public evaluation results
## ProblemThree apps called one model straight; the provider retires it in 50 days.Goal: apps ask for a capability; rules choose the model from public evidence.
## Data and licencesEpoch AI benchmark results (GPQA Diamond, OTIS Mock AIME 2024-2025, SimpleQAVerified), CC BY 4.0, frozen copy. LiteLLM price file, commit 159b5b2, MIT:prices, features and retirement dates (assumptions; check provider pages).
## ApproachTwo contracts (features, allowed settings, a public floor at the mapped setting,price band, data classes, a 60-day retirement window). Each app's own minimum:today's score minus two standard errors. A gate: every app must pass. Thecheapest target that passes is the mapping; same-model fallback if possible.
## ResultsToday (o3): 4 broken rules, US$174/month. Top of a leaderboard: 3 broken rules,US$192. Cheapest that keeps the contract: 1 broken rule (research app 0.117 vs0.46), US$16.95. Cheapest that passes every app: 0 broken rules, US$34.70.
## LimitsPublic benchmarks are proxies; no token counts per setting; features need onereal call; prices and dates are a third party's record.How to describe it in an interview, in two sentences: “When three apps’ shared model was about to retire, I built a capability catalog whose rules choose the model from public benchmarks and prices. The cheapest option broke one app’s own minimum, so the gate chose one that kept every promise, at 80% less cost.”
Go further:
- Add a third capability,
embed, with an openly licensed embedding benchmark, and check its licence first. - Add the LMArena leaderboard (CC BY 4.0) as a second opinion for one consumer, and write down how you matched its model names.
- Replace one consumer’s public benchmark with 30 of its own recorded tasks, and compare the two gates.
- Make
checkrun every week in CI, and open an issue when a target enters the review window.
Knowledge check
Section titled “Knowledge check”Check your understanding
gpt-6-luna at max passes the billing agent with 0.989 and fails the research assistant with 0.414. The average of the two scores, 0.701, is above the average of the minimums, 0.605. Ben says the average shows it is good enough. What do you answer?
Show the answer
The average mixes two different skills and two owners’ promises. The billing agent’s 0.989 cannot make up for the research assistant’s loss of 8 points. Lena’s users would get more wrong answers about equipment. Each minimum is its owner’s promise, and the average is nobody’s promise. So the gate blocks, as in the course: every consumer must pass its own evaluation. The cheapest target that passes both, gemini-3.8-flash at high, still costs 80% less than o3.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Epoch AI lists claude-haiku-5-5 with 0.972 on OTIS Mock AIME at high, and 0.238 on SimpleQA Verified at max. Why does the gate say 'no evidence' for the research assistant at high, and what would make it pass or fail?
Show the answer
A public score belongs to the setting it was measured at. The mapping would send reasoning_effort high, and nobody published a SimpleQA Verified score at high. The 0.238 at max is a different setting; it is even below Lena’s minimum. A missing score is not a zero and not a pass, so the gate blocks: no evidence, no remap. To decide, Lena needs a score at high: a public one, or her own evaluation at that setting.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In the what-if, gpt-5.6-luna retires and fast-small moves to gemini-3.1-flash-lite at low, with a margin of 0.002 and no fallback. The check says the catalog is consistent. Should Noor accept it?
Show the answer
Not on the public data alone. The margin is much smaller than the score’s standard error of 0.031. So the evidence cannot tell this model apart from one that fails. There is no fallback at another provider, and the model retires on 7 May 2027. The rules found the only target that passes, which is useful: the help assistant would not stop. But Noor should ask Joel to run his own evaluation on two or three candidates before that day. Widening the price band or the allowed settings is another option, with its cost written down.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You built a catalog from two public files: Epoch AI’s benchmark results (CC BY 4.0) and LiteLLM’s price file (MIT), frozen with checksums.
- A public score belongs to one setting: gpt-5.6-luna scored 0.636, 0.823 and 0.916 on GPQA Diamond at none, low and max. The mapping carries the setting, and a missing score is no evidence.
- The contracts added a retirement window: o3 retires in 50 days at OpenAI (and earlier at Azure), so today’s mapping failed the check.
- Each app’s minimum was “no worse than today”: today’s score minus two standard errors.
- The gate blocked gpt-6-luna although the average passed, because the research assistant fell below its own minimum.
- Under one measure, only “the cheapest that passes every consumer” kept every rule: US$34.70 a month against US$174 today. A what-if retirement moved the mapping again, to a model with weak evidence and no fallback.
Next, the second case study: replay open agent trajectories through the gateway, and measure what budgets, step limits and tool policies would have stopped.
Next: Contain a runaway agent