Skip to content
Français

A capability catalog from public evaluation results

  • Case study 1
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In this course, we wrote Larkfield's capability contracts from its own recorded evaluations, and the remap gate blocked a cheaper model because one consumer failed. Most organisations do not have evaluations of their own on the first day. In this case study, a company whose only model is about to retire builds its first catalog from public benchmark results and a public price file. The contracts, the gate and the rules are the course's; only the evidence is new.

What you will be able to do

Build a company’s first capability catalog from public evidence. You write the contracts and set each app’s own minimum. Then rules choose the mapping from a public price file and openly licensed benchmark results. You see how the mapping changes when the company’s only model retires, and when a second one would. You finish with a recommendation that says what the public data cannot tell you.

Case studyPortfolio piece

Time
About 2.5 hours
Runs in
VS Code, Python 3.12+
Skills used
  • Capability contracts
  • Public evidence at a setting
  • Consumer minimums
  • Remap gate
  • Rule-based mapping
  • Retirement window
  • Fallbacks
  • Recommendation with limits

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

This case study is a small project folder, public-catalog, in the course’s own style: Python, YAML and one command line. The starter has the data, the loaders and the checks. You write a contract, a minimum and the gate’s decision, then let the project choose and write the mapping. The finished project is the result, for when you are stuck or want to compare. The code is under the MIT licence, so you can publish your version.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the catalog and the gate of the course. These lessons matter most:

The page teaches one new idea: a public score belongs to one setting of the model.

What you need on your computer

  • Python 3.12 or newer, VS Code and a terminal, as in the course.
  • About 40 MB of free disk space for the virtual environment.
  • No account, no key and no money. Everything runs on your computer from files in the project. One optional step downloads the latest public data (about 5 MB).

Tested on macOS (Apple silicon) with Python 3.12.13, PyYAML 6.0.3 and pytest 9.1.1, from a fresh copy of the starter. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them.

Hollin Energy supplies electricity and gas to about 300,000 homes. Three teams built an AI app each, and each team copied the first team’s code:

  • The help assistant (Joel, customer care) drafts answers to customers’ questions about their bills. An agent reads every draft before it goes out.
  • The billing agent (Ada, billing) works out corrected bills and payment plans with tools. A person approves every change.
  • The research assistant (Lena, network engineering) answers engineers’ questions about standards and equipment from memory, without a web search.

All three call the same model straight, with the same key: o3-2025-04-16 on the OpenAI API. Noor, the platform engineer, reads a public price file and finds a line next to it: deprecation_date: 2026-12-11. The provider retires the model in 50 days. On that day, all three apps stop at once.

Ben, from finance, has a quick answer: “Take the top model of a leaderboard and move everyone to it.” Noor has another plan. She wants the apps to ask for a capability, as the course taught, so that the next retirement changes one line instead of three apps. But Hollin has no evaluations of its own yet. So the first catalog must stand on public evidence, and it must say clearly what that evidence cannot show.

Hollin Energy, its people, apps, contracts and workloads are made up for this case study. The models, providers, scores, prices and dates are real public data, from the two sources below.

File What it holds Part
config/models.yaml the public evidence for 61 models: scores by setting, offers by provider The data
config/catalog.yaml two contracts, fast-small and strong-reasoning, and their mapping The contracts
config/consumers/*.yaml the three apps, with their own benchmark and minimum Each app’s own minimum
pubcat/gate.py the gate’s decision The gate on public evidence
docs/RECOMMENDATION.md the decision, the evidence, the cost and the limits The recommendation
A flow, top to bottom. Two public sources: Epoch AI's benchmark results, a score per model and setting, CC BY 4.0; and LiteLLM's price file, with the price, the features and the retirement date per provider, MIT. Both go into config/models.yaml: 61 models, 102 offers. With config/providers.yaml, Hollin's four providers, they give the targets: every model at every provider with every setting, 227 of them. The targets go through the contract check: preview, features, setting, price band, data, retirement window and floor. Then the gate: each consumer's own minimum, at that setting. The cheapest target that passes is the mapping, with a fallback. A retirement date, or a what-if, removes a target before the contract check.
How the project turns two public files into a mapping. A retirement removes a target, and the same rules choose again.
Text description of the diagram

Two public files feed the project. Epoch AI’s results give a score per model and setting. LiteLLM’s price file gives the price, features and retirement date of each model at each provider. They become config/models.yaml, 61 models with 102 offers. With Hollin’s four providers, that makes 227 targets: a model, at a provider, with a setting. Each target goes through the contract check and then the gate, where every app must reach its own minimum. The cheapest target that passes becomes the mapping, with a fallback. A retirement date removes targets, and the same rules choose again.

  1. Download the starter project from the box at the top of this page. Unzip it into your projects folder, for example ~/projects. You get the folder public-catalog. Open it in VS Code.

  2. In the VS Code terminal, make a virtual environment and install the two packages:

    Terminal window
    python3 -m venv .venv
    source .venv/bin/activate
    python -m pip install -r requirements.txt

    The packages are PyYAML 6.0.3 and pytest 9.1.1, the versions of the course project. Your course virtual environment works too.

  3. Run the tests: python -m pytest -q --tb=no. The last line says 13 failed, 11 passed. That is expected: each failing test belongs to one of the four tasks, and its message names the task. At the end, all 24 pass.

The project joins two public sources. Both are in data/ as a frozen copy, unchanged, with a file SHA256SUMS.

Epoch AI’s benchmark results. Epoch AI, a research organisation, runs public benchmarks on many models and publishes every result. The project uses three of its benchmark files. GPQA Diamond has 198 hard, graduate-level science questions. OTIS Mock AIME 2024–2025 has competition mathematics problems with exact answers. SimpleQA Verified has short factual questions, answered from memory without a search. One row is one model at one setting on one benchmark: the share of questions it answered correctly (mean_score) and its standard error (stderr).

LiteLLM’s price file. LiteLLM is an open-source gateway. Its project keeps one large file with an entry for each model at each provider. An entry has the list price per token and the features that LiteLLM records, such as function calling. It sometimes has a deprecation_date, the day the provider retires the model.

Noor compared other sources first. The LMArena leaderboard (CC BY 4.0) gives relative ratings from people’s votes, so a fixed minimum would drift as new models join. The Open LLM Leaderboard results have no licence that we could find, and cover only open-weight models. The providers’ own price pages have no open licence, so the case study links to them.

Check that the files are the ones the case study used, and count what they hold:

Terminal window
python -m pubcat data
Checksums: 6 files match SHA256SUMS
gpqa_diamond 325 results, 225 models
otis_mock_aime_2024_2025 305 results, 204 models
simpleqa_verified 92 results, 83 models
Models with a public score: 234; with an offer at one of our providers: 61; without: 173

What this tells us. The files are unchanged. Epoch AI tested many more models on GPQA Diamond than on SimpleQA Verified: a missing score will be common, and it is not a zero. Only 61 of the 234 models are offered by one of Hollin’s four providers. The other 173 are models of other providers, older models, or open-weight models without a hosted price. Hollin’s providers are in config/providers.yaml: OpenAI, Azure OpenAI, Google Cloud Vertex AI and Anthropic. Each has a contract with a data processing agreement, which is an assumption of the case study. A provider that is not in the file is never called, whatever its scores, as in Map capabilities to models.

Build config/models.yaml, the evidence that the catalog reads, and look at today’s model:

Terminal window
python -m pubcat models
python -m pubcat show o3-2025-04-16
Wrote config/models.yaml: 61 models, 102 offers
Left out: 173 models with a public score but no offer at our providers
Model: o3-2025-04-16 (OpenAI)
gpqa_diamond low 0.798 ±0.029, medium 0.808 ±0.028, high 0.818 ±0.021
otis_mock_aime_2024_2025 low 0.600 ±0.074, medium 0.844 ±0.055, high 0.839 ±0.044
simpleqa_verified high 0.494 ±0.016
offer azure US$2.0 in / US$8.0 out per 1M tokens; chat, structured_output, tools, images; retires 2026-11-19
offer openai US$2.0 in / US$8.0 out per 1M tokens; chat, structured_output, tools, images; retires 2026-12-11

What this tells us. Three things. First, one model has several scores: o3 answered 60% of the mathematics problems at low and 84% at high. Second, the same model retires on different days at different providers: 11 December 2026 at OpenAI, 19 November 2026 at Azure. A retirement date belongs to an offer, not to a model. Third, each score has a standard error. On 198 questions, ±0.029 is about six questions, so a difference of 0.01 between two models means nothing.

models.yaml is the course’s models.yaml with two additions: the public scores, and a price and retirement date per provider. Do not edit it by hand. Change the data or the providers, and build it again.

Think of a car’s fuel use. The brochure says “4.2 litres per 100 km”. The small print says “at a steady 50 km/h”. In the city, with stops and starts, the same car uses 7 litres. The number is true, but only for the condition it was measured in.

Model scores work the same way. Many models take a reasoning setting, such as reasoning_effort: none, low, high or max. A higher setting lets the model reason longer before it answers. It answers better on hard questions, more slowly, and with more output tokens, which you pay for. Epoch AI writes the setting after the model’s name: gpt-5.6-luna_low is the model gpt-5.6-luna at low.

Terminal window
python -m pubcat show gpt-5.6-luna
Model: gpt-5.6-luna (OpenAI)
gpqa_diamond none 0.636 ±0.034, low 0.823 ±0.027, max 0.916 ±0.017
otis_mock_aime_2024_2025 none 0.400 ±0.074, low 0.667 ±0.071, max 0.983 ±0.012
simpleqa_verified max 0.410 ±0.016
offer azure US$0.2 in / US$1.2 out per 1M tokens; chat, structured_output, tools, images; retires 2028-01-11
offer openai US$0.2 in / US$1.2 out per 1M tokens; chat, structured_output, tools, images; retires no date in the price file

So the rule is: a score counts only for a mapping that sends the same setting. In the course, a mapping already carries its settings (params, from Map capabilities to models). The project looks up the score at the mapping’s setting. A missing score at that setting is no evidence.

  • gpt-5.6-luna
  • gpt-5.4-nano
  • o3
Public scores (Epoch AI, mean_score on GPQA Diamond's 198 questions), by the setting each model was run with. The fast-small contract needs 0.74 at none, minimal or low. The model is the same within each colour; only the setting changes.
Text description of the chart

Nine columns in three colours, one colour per model. gpt-5.6-luna: none 0.64, low 0.82, max 0.92. gpt-5.4-nano: none 0.56, low 0.72, high 0.78. o3: low 0.80, medium 0.81, high 0.82. The axis starts at 0.5. For the two small models, the setting moves the score by 0.2 to 0.3, more than the difference between most models. For o3, the setting moves it by 0.02.

Larkfield met the same effect in this course. In Change a mapping safely, chat-small, the model gpt-6-luna-2026-09-22, failed both of Larkfield’s consumers with reasoning_effort: none. Epoch AI lists the same model family only at max, with 0.905 on GPQA Diamond. Both are true. They measure different settings and different tasks.

The catalog uses the course’s format. A capability has a contract (what it promises) and a mapping (what serves it). Two parts change, because the evidence is public:

  • settings lists the reasoning settings that a mapping may send. Public results cannot show latency, so this is Hollin’s rule for speed and cost: fast-small allows only none, minimal and low.
  • quality_floor names a public benchmark, by its file name, and a minimum at the mapped setting.

Open config/catalog.yaml. The contract of fast-small is complete:

Part fast-small strong-reasoning
For short answers and drafts about bills multi-step calculations, research answers
Features chat, structured_output, tools chat, structured_output, tools
Settings none, minimal, low low, medium, high, xhigh, max
Quality floor GPQA Diamond ≥ 0.74 OTIS Mock AIME 2024–2025 ≥ 0.75
Data classes public, internal, personal public, internal, personal
Price band (in / out, US$ per 1M tokens) 0.50 / 2.00 2.50 / 12.00

Why GPQA Diamond for the help assistant? No public benchmark asks bill questions. GPQA Diamond has the most models at low settings. So it is the widest screen for “reasons well enough when it answers fast”, not a test of the task. Why OTIS Mock AIME for strong-reasoning? Its problems need several exact steps in a row, as a corrected bill does. The two price bands are Larkfield’s, from A capability catalog.

Task 1: complete the strong-reasoning contract

Section titled “Task 1: complete the strong-reasoning contract”
  1. In config/catalog.yaml, under strong-reasoning: and contract:, change minimum: null to minimum: 0.75. The floor is the billing agent’s minimum, the capability’s main consumer, which you meet in the next part.

  2. Change price_band_usd_per_1m: {input: null, output: null} to price_band_usd_per_1m: {input: 2.50, output: 12.00}.

  3. Run the task’s test: python -m pytest -q -k strong_reasoning_contract. It prints 1 passed, 23 deselected.

  4. Check today’s mapping against the contracts:

    Terminal window
    python -m pubcat check
    fast-small -> openai/o3-2025-04-16: the input price 2.0 is above the contract's band 0.5
    fast-small -> openai/o3-2025-04-16: the output price 8.0 is above the contract's band 2.0
    fast-small -> openai/o3-2025-04-16: retires on 2026-12-11 (50 days), inside the 60-day review window
    strong-reasoning -> openai/o3-2025-04-16: retires on 2026-12-11 (50 days), inside the 60-day review window

What this tells us. The starter’s catalog maps both capabilities to o3, as the apps call it today. Each line names the capability, the target and the rule it breaks, as estate catalog check does in the course. Two problems are new. The retirement window: a target that retires within 60 days fails the check, because a remap takes time. That is the review window of The agent lifecycle. And the price band: the help assistant pays a strong model’s price for short drafts. That was true before the retirement too, but nobody had written the band down.

The course’s gate runs each consumer’s own evaluation against its own minimum. Hollin has no evaluations yet, so each app names a public benchmark and a minimum in its file in config/consumers/. Each app also has a workload: the tokens it sends and receives in a month. These are the owners’ estimates, an assumption.

What minimum? Each owner chose the same rule: no worse than today. Today’s model is o3 at the app’s setting. A score has an uncertainty, so “no worse” means “not lower than today’s score minus two standard errors”. A smaller drop could be noise.

App (owner) Capability Benchmark Today (o3) Minimum Workload a month (assumption)
Help assistant (Joel) fast-small GPQA Diamond 0.798 ± 0.029 at low 0.798 − 2 × 0.029 = 0.74 40 M tokens in, 6 M out
Billing agent (Ada) strong-reasoning OTIS Mock AIME 0.839 ± 0.044 at high 0.839 − 2 × 0.044 = 0.75 8 M in, 2 M out
Research assistant (Lena) strong-reasoning SimpleQA Verified 0.494 ± 0.016 at high your task 3 M in, 1 M out

Task 2: the research assistant’s minimum

Section titled “Task 2: the research assistant’s minimum”
  1. Find today’s score and standard error of o3 on SimpleQA Verified in the output of python -m pubcat show o3-2025-04-16 above.
  2. Compute today’s score minus two standard errors, and round it down to two decimals.
  3. In config/consumers/research-assistant.yaml, replace minimum: null and its comment with your number. Write the sum in why:.
  4. Run python -m pytest -q -k research_assistants_minimum. It prints 1 passed, 23 deselected.
Check your answer

0.494 − 2 × 0.016 = 0.462, rounded down to 0.46. The line becomes minimum: 0.46, and why: can say "no worse than today: 0.494 - 2 x 0.016 = 0.462, rounded down to 0.46".

Lena’s minimum is low in absolute terms: o3 answers fewer than half of the questions. That is the point of “no worse than today”. The research assistant is not good at facts from memory today, and the remap must not make it worse. A better assistant needs a search tool, not a new model.

Try the gate before you write it, on one of the cheapest models in the file, gpt-6-luna:

Terminal window
python -m pubcat gate strong-reasoning --to openai/gpt-6-luna --param reasoning_effort=max

It stops with NotImplementedError: Task 3: write the gate's decision in pubcat/gate.py.

  1. Open pubcat/gate.py and find decide(). It gets one row per consumer, each with a "verdict": "pass", "fail" or "no evidence".
  2. Replace the three Task 3 lines with one line. It returns "pass" if there is a row and every verdict is "pass". Otherwise it returns "block".
  3. Run python -m pytest -q -k every_consumer. It prints 4 passed, 20 deselected. The four cases are: both pass, one fails, one without evidence, and no consumer at all.
Check your code
def decide(rows: list[dict]) -> str:
"""The gate's decision: 'pass' only if there is at least one consumer and EVERY consumer passes its own
minimum. A failed consumer or a consumer without evidence blocks the remap. The average does not count."""
return "pass" if rows and all(r["verdict"] == "pass" for r in rows) else "block"

rows and makes a capability without consumers block: with nobody to pass, nothing is proven.

Predict first: gpt-6-luna at max costs 20 times less per token than o3. Does it pass both consumers of strong-reasoning? Does the average of their scores pass the average of their minimums?

Terminal window
python -m pubcat gate strong-reasoning --to openai/gpt-6-luna --param reasoning_effort=max
Remap gate: strong-reasoning from openai/o3-2025-04-16 {'reasoning_effort': 'high'} to openai/gpt-6-luna {'reasoning_effort': 'max'}
billing-agent PASS otis_mock_aime_2024_2025 0.989 (minimum 0.75); today 0.839; US$1.80 vs US$32.00 a month today
research-assistant FAIL simpleqa_verified 0.414 (minimum 0.46); today 0.494; US$0.80 vs US$14.00 a month today
Average of the consumers' scores: 0.701 (average of their minimums: 0.605)
Decision: BLOCK - every consumer must pass its own evaluation

What this tells us. This is the case of Change a mapping safely, now with real public data and no constructed consumer. The average of the two scores, 0.701, is well above the average minimum, 0.605. A leaderboard that averages benchmarks would rank this model high. But the research assistant would lose 8 points of factual accuracy, and only Lena’s own minimum shows it.

Two more candidates show the other outcomes:

Terminal window
python -m pubcat gate strong-reasoning --to anthropic/claude-haiku-5-5 --param reasoning_effort=high
python -m pubcat gate strong-reasoning --to vertex/gemini-3.8-flash --param reasoning_effort=high
billing-agent PASS otis_mock_aime_2024_2025 0.972 (minimum 0.75); today 0.839; US$1.80 vs US$32.00 a month today
research-assistant NO EVIDENCE no public simpleqa_verified score for claude-haiku-5-5 at high: no evidence, no remap
Decision: BLOCK - every consumer must pass its own evaluation
billing-agent PASS otis_mock_aime_2024_2025 0.989 (minimum 0.75); today 0.839; US$13.50 vs US$32.00 a month today
research-assistant PASS simpleqa_verified 0.697 (minimum 0.46); today 0.494; US$6.00 vs US$14.00 a month today
Decision: PASS

(The first and the blank lines of each run are left out here.) Epoch AI ran claude-haiku-5-5 on SimpleQA Verified only at max. At high, Lena has no evidence, so the gate blocks: “no evidence, no remap”, as in the course. gemini-3.8-flash at high passes both.

Ben’s question is about money, and Noor’s is about rules. One table answers both. List every target that keeps the strong-reasoning contract, cheapest first, with each consumer’s verdict:

Terminal window
python -m pubcat candidates strong-reasoning --limit 16
strong-reasoning: 227 targets, 43 keep the contract, 13 also pass every consumer. Cheapest first:
target US$/month billing-agent research-assistant
azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'} 1.75 pass 0.811 fail 0.117
openai/gpt-6-luna {'reasoning_effort': 'max'} 2.60 pass 0.989 fail 0.414
azure/gpt-6-luna {'reasoning_effort': 'max'} 2.60 pass 0.989 fail 0.414
anthropic/claude-haiku-5-5 {'reasoning_effort': 'high'} 2.60 pass 0.972 no evidence
anthropic/claude-haiku-5-5 {'reasoning_effort': 'xhigh'} 2.60 pass 0.989 no evidence
vertex/claude-haiku-5-5 {'reasoning_effort': 'high'} 2.60 pass 0.972 no evidence
vertex/claude-haiku-5-5 {'reasoning_effort': 'xhigh'} 2.60 pass 0.989 no evidence
openai/gpt-5.6-luna {'reasoning_effort': 'max'} 5.80 pass 0.983 fail 0.410
azure/gpt-5.6-luna {'reasoning_effort': 'max'} 5.80 pass 0.983 fail 0.410
openai/gpt-5.4-nano-2026-03-17 {'reasoning_effort': 'high'} 5.95 pass 0.878 fail 0.117
azure/gpt-5.4-nano-2026-03-17 {'reasoning_effort': 'high'} 5.95 pass 0.878 fail 0.117
vertex/gemini-3.1-flash-lite {'reasoning_effort': 'high'} 7.25 pass 0.800 no evidence
azure/gpt-5-mini-2025-08-07 {'reasoning_effort': 'high'} 8.75 pass 0.867 fail 0.216
azure/gpt-5-mini-2025-08-07 {'reasoning_effort': 'medium'} 8.75 pass 0.783 no evidence
vertex/gemini-3.8-flash {'reasoning_effort': 'high'} 19.50 pass 0.989 pass 0.697
vertex/gemini-3.7-flash {'reasoning_effort': 'high'} 19.50 pass 0.972 pass 0.692

What this tells us. A target is a model at a provider with a setting, so one model gives several rows. The billing agent passes on every row: the contract’s floor is its minimum. The research assistant decides. Mathematics and facts from memory are different skills. The small models solve the competition problems and miss most of the facts, as the gpt-5-nano row shows, with 0.811 and 0.117. Note also azure/gpt-5-nano-2025-08-07: the OpenAI offer of the same model is not in the list, because it retires on 11 December 2026. Azure’s retires on 9 February 2027, outside the window.

  • passes every rule and consumer
  • blocked by the research assistant
Each dot is a model and setting that keeps the strong-reasoning contract and has a public SimpleQA Verified score (Epoch AI), at its cheapest provider. The cost is the two consumers' assumed workloads at list prices (LiteLLM's price file, an assumption). The line is Lena's minimum. Higher settings use more output tokens, which the cost does not count.
Text description of the chart

Eighteen dots. The cheapest five, from US$1.75 to US$8.75 a month, all lie below Lena’s minimum of 0.46: gpt-5-nano at high scores 0.117, gpt-6-luna at max 0.414. The first dots above the line are gemini-3.8-flash and gemini-3.7-flash at high, at US$19.50, with 0.697 and 0.692. Further right, from US$43.50 to US$58, are larger models: some above the line, such as gpt-6.1-sol at max with 0.739 at US$52, and some below. Price and factual accuracy are only loosely related.

The funnel shows how many targets each rule removes. choose prints it for each capability:

strong-reasoning:
all targets (model x provider x setting) 227
is not a preview model 219
has the contract's features 201
uses an allowed setting 141
inside the price band 80
the provider may receive the data 80
does not retire inside the review window 56
reaches the contract's floor 43
every consumer passes its own minimum 13

Each rule is one line of check_target() in pubcat/catalog.py. Preview models are out because providers change or remove a preview at short notice. The data step removes none here: every one of Hollin’s providers may receive personal data, an assumption. The retirement window removes 24 targets. Ten of them are at OpenAI and retire on 11 December 2026, the same day as o3. Six more retire at OpenAI on 23 October 2026.

A baseline and three alternatives, under one measure

Section titled “A baseline and three alternatives, under one measure”

Ben asked for the top of a leaderboard. That is a fair alternative, so the project compares four ways to choose, with one measure. Each way must keep every rule: the contract and every consumer’s minimum. Then the lowest monthly cost wins.

Terminal window
python -m pubcat compare
approach broken rules US$/month
1. keep today's model 4 174.00
fast-small: openai/o3-2025-04-16 {'reasoning_effort': 'low'} - price; retire
strong-reasoning: openai/o3-2025-04-16 {'reasoning_effort': 'high'} - retire
2. top of a leaderboard 3 192.00
fast-small: vertex/gemini-4-argon {'reasoning_effort': 'high'} - price; settings
strong-reasoning: vertex/gemini-4-argon {'reasoning_effort': 'high'} - keeps every rule
3. cheapest that keeps the contract 1 16.95
fast-small: openai/gpt-5.6-luna {'reasoning_effort': 'low'} - keeps every rule
strong-reasoning: azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'} - research-assistant below its minimum
4. cheapest that passes every consumer 0 34.70
fast-small: openai/gpt-5.6-luna {'reasoning_effort': 'low'} - keeps every rule
strong-reasoning: vertex/gemini-3.8-flash {'reasoning_effort': 'high'} - keeps every rule
US$/month: the consumers' assumed workloads at list prices per token (more output tokens at a higher setting are not counted).

What this tells us.

  1. Keep today’s model is the baseline. It costs US$174 a month and stops on 11 December.
  2. The top of a leaderboard is the model with the highest average of the three public scores at one setting: gemini-4-argon at high. For strong-reasoning it keeps every rule. But it costs more than today. For the help assistant, it breaks the price band and the setting rule.
  3. The cheapest that keeps the contract looks like a 90% saving. It sends Lena’s questions to a model that gets 12% of the facts right.
  4. The cheapest that passes every consumer breaks no rule and costs US$34.70 a month, 80% less than today.

Only the fourth keeps every promise. It costs twice as much as the third, and the difference, US$17.75 a month, is the price of not breaking the research assistant.

Task 4: let the rules choose, and write the mapping

Section titled “Task 4: let the rules choose, and write the mapping”
  1. Propose a mapping for every capability:

    Terminal window
    python -m pubcat choose

    For each capability, it prints the funnel and its proposal. For strong-reasoning:

    cheapest that keeps the contract: azure/gpt-5-nano-2025-08-07 {'reasoning_effort': 'high'}, US$1.75 a month - blocked by the gate (research-assistant)
    proposed primary: vertex/gemini-3.8-flash {'reasoning_effort': 'high'}, US$19.50 a month, smallest margin +0.237
    proposed fallback: azure/gpt-5-2025-08-07 {'reasoning_effort': 'high'} (a different model: every consumer passed its gate)

    For fast-small: openai/gpt-5.6-luna at low, US$15.20 a month, with azure/gpt-5.6-luna at low as the fallback: the same model and setting at another provider. At the end, a diff shows the mapping today and the proposal.

  2. Write the proposal into the catalog: python -m pubcat choose --write. The old mapping stays in the file as previous_mapping, the record of the change.

  3. Check the catalog: python -m pubcat check prints The catalog is consistent: every mapping keeps its contract.

  4. Run all the tests: python -m pytest -q. The last line says 24 passed.

  5. Open docs/RECOMMENDATION.md. Fill in its five sections for Hollin’s app owners and Ben, with the outputs above. The finished project has one version of it.

How does the chooser decide between equal prices? gemini-3.8-flash and gemini-3.7-flash cost the same. A tie goes to the target with the larger smallest margin: its lowest score minus minimum over all consumers, the one furthest from failing. Here that is 0.237 against 0.222. Then it goes to the provider that the capability uses today, the smallest change. That is why fast-small stays at OpenAI and Azure is the fallback.

The fallbacks follow the course’s rule. For fast-small, the fallback is the same model and setting at another provider. An outage changes nothing for the help assistant. For strong-reasoning, no other of Hollin’s providers offers gemini-3.8-flash. So the fallback is a different model: gpt-5-2025-08-07 at high, the cheapest that passes at another provider. Map capabilities to models calls that “a remap in disguise”. It is allowed only because every consumer’s gate passed on it too.

Two boxes, top to bottom. Today: the help assistant of Joel, the billing agent of Ada and the research assistant of Lena all call openai o3-2025-04-16, which retires on 2026-12-11, 50 days left. A thick arrow labelled remap: contracts, then the gate. After: each app asks for a capability. The help assistant asks for fast-small, served by openai gpt-5.6-luna at low, with the same model at azure as the fallback. The billing agent and the research assistant ask for strong-reasoning, served by vertex gemini-3.8-flash at high, with azure gpt-5 at high as the fallback.
One retirement, before and after. The apps now name a capability, so the next retirement changes the catalog, not the apps.
Text description of the diagram

Before the remap, all three apps call o3-2025-04-16 at OpenAI, which retires in 50 days. After it, the help assistant asks for fast-small: gpt-5.6-luna at low, with the same model at Azure as the fallback. The billing agent and the research assistant ask for strong-reasoning: gemini-3.8-flash at high on Vertex AI, with gpt-5 at high on Azure as the fallback.

Noor’s recommendation, in short: fast-small on gpt-5.6-luna at low, and strong-reasoning on gemini-3.8-flash at high. That is about US$34.70 a month at list prices, instead of US$174. Each app changes once, from a model name to a capability name. Before the change goes live, each owner checks the new mapping on their own work. The limits below say why.

A catalog is not done when it is written. Retirements, new models and new prices arrive every month. Here are three new inputs.

1. A second retirement (a what-if). Suppose the provider announced today that gpt-5.6-luna retires, which it has not. The --retire option removes a model’s offers for one run:

Terminal window
python -m pubcat choose --retire gpt-5.6-luna
What-if: gpt-5.6-luna retired today (2026-10-22).
fast-small:
reaches the contract's floor 1
every consumer passes its own minimum 1
proposed primary: vertex/gemini-3.1-flash-lite {'reasoning_effort': 'low'}, US$19.00 a month, smallest margin +0.002
proposed fallback: none - no other provider passes

(The first lines of the funnel, and the part for strong-reasoning, whose proposal does not change, are left out here.)

What this tells us. The mapping moves, and nobody touches an app. But look at what is left. One target passes, by 0.002 over Joel’s minimum. Its standard error is 0.031, so the public evidence cannot tell it apart from a model that fails. It has no fallback, and show gemini-3.1-flash-lite says that it retires on 7 May 2027. The rules found an answer, but the evidence is weak. Noor adds a next step: before such a day, Joel runs his own evaluation on two or three candidates.

2. A later day. The default day is the day of the frozen data. Check the written catalog on the day o3 retires:

Terminal window
python -m pubcat check --today 2026-12-11
strong-reasoning -> azure/gpt-5-2025-08-07: retires on 2027-02-09 (60 days), inside the 60-day review window

The fallback enters its own review window that day. Run check on a schedule, and the next remap starts 60 days early, not on the day an app fails.

3. The latest public data. data/fetch.py downloads the current versions of both sources into data/latest/, with their checksums:

Terminal window
python data/fetch.py
python -m pubcat --data data/latest data
python -m pubcat --data data/latest --models config/models-latest.yaml models
python -m pubcat --models config/models-latest.yaml choose

When the course froze the data, the latest files were the same. The last line said The proposed mapping is the same as today's. Your results will differ: new models, scores, prices and dates arrive. Compare your proposal with the one above, and read every line of the diff before you write it.

Common mistake

A data file that changed

Symptom: python -m pubcat data prints this and exits with code 1:

epoch/simpleqa_verified.csv: wrong checksum (the file changed)
Some data files are missing or changed. Download the project again, or run python data/fetch.py.

Cause: a file in data/ is not the file in SHA256SUMS. The case study caused it on purpose: one score, o3’s 0.494, was “corrected” to 0.594 in an editor. The catalog would then accept models for the research assistant that the real data refuses. The same happens if a spreadsheet program saves a CSV file again.

Diagnose: the line names the file. Compare it with the copy in the downloaded zip, for example with git diff --no-index.

Recover: unzip the file from the starter again, then run python -m pubcat data until it prints Checksums: 6 files match SHA256SUMS. Never change public evidence by hand. If you disagree with a score, write it in the recommendation, with your reason.

  • Public results make a shortlist; they do not decide. No benchmark here is a bill question or an engineer’s question. The rules and the gate removed 214 of 227 targets, and they can still pass a model that fails Hollin’s real work. Each owner runs a sample of their own tasks first, as the course’s gate does.
  • A cost belongs to a setting too. A model at max writes many more output tokens than at low. Neither file has token counts per task. The cost of strong-reasoning is a floor, not a forecast. Measure the tokens of the first week.
  • Features need one real call. The price file says every candidate can call functions. In A capability catalog, one model needed reasoning_effort: none for tools, and another could not call tools at all. Ada makes one real tool call on each new target.
  • A missing date is not “never”. 41 of the 102 offers have no retirement date. Treat them as unknown.
  • The prices are a third party’s record, and they can be late. Finance checks each one on the provider’s page.
  • Benchmark questions can leak into a model’s training data, which raises its score. Your own tasks do not have that problem.
  • Credit the sources. CC BY 4.0 asks for credit to Epoch AI wherever you use its data. The MIT licence asks you to keep LiteLLM’s licence note with its file. Keep both in your version.
  • A public score is not a verdict on a vendor. This ranking follows Hollin’s made-up rules and workloads. Another organisation would choose differently. Say “for these contracts”, not “the best model”.
  • Label every assumption where it is used. The contracts, the workloads and the prices are assumptions; the second retirement is a what-if.
  • The owners decide. Noor runs the gate and proposes the mapping. Each owner signs off on their own result. Ben’s saving is a reason to look, not an approval.

This case study shows an employer two things. You can turn “which model should we use?” into rules that anyone can check. And you know where public evidence stops. Make it yours with one change: other apps, other benchmarks from Epoch AI’s archive, or another provider list.

What to put in the repository:

  • README.md (template below).
  • The project, with your config/ files, the tests passing, and data/ with its licences and checksums.
  • docs/RECOMMENDATION.md: the decision, the evidence, the cost and the limits.
  • The output of python -m pubcat compare and choose, and the diff of the mapping.
  • The two credits from the sources box above.

A README template:

# A capability catalog from public evaluation results
## Problem
Three apps called one model straight; the provider retires it in 50 days.
Goal: apps ask for a capability; rules choose the model from public evidence.
## Data and licences
Epoch AI benchmark results (GPQA Diamond, OTIS Mock AIME 2024-2025, SimpleQA
Verified), CC BY 4.0, frozen copy. LiteLLM price file, commit 159b5b2, MIT:
prices, features and retirement dates (assumptions; check provider pages).
## Approach
Two contracts (features, allowed settings, a public floor at the mapped setting,
price band, data classes, a 60-day retirement window). Each app's own minimum:
today's score minus two standard errors. A gate: every app must pass. The
cheapest target that passes is the mapping; same-model fallback if possible.
## Results
Today (o3): 4 broken rules, US$174/month. Top of a leaderboard: 3 broken rules,
US$192. Cheapest that keeps the contract: 1 broken rule (research app 0.117 vs
0.46), US$16.95. Cheapest that passes every app: 0 broken rules, US$34.70.
## Limits
Public benchmarks are proxies; no token counts per setting; features need one
real call; prices and dates are a third party's record.

How to describe it in an interview, in two sentences: “When three apps’ shared model was about to retire, I built a capability catalog whose rules choose the model from public benchmarks and prices. The cheapest option broke one app’s own minimum, so the gate chose one that kept every promise, at 80% less cost.”

Go further:

  • Add a third capability, embed, with an openly licensed embedding benchmark, and check its licence first.
  • Add the LMArena leaderboard (CC BY 4.0) as a second opinion for one consumer, and write down how you matched its model names.
  • Replace one consumer’s public benchmark with 30 of its own recorded tasks, and compare the two gates.
  • Make check run every week in CI, and open an issue when a target enters the review window.

Check your understanding

gpt-6-luna at max passes the billing agent with 0.989 and fails the research assistant with 0.414. The average of the two scores, 0.701, is above the average of the minimums, 0.605. Ben says the average shows it is good enough. What do you answer?

Show the answer

The average mixes two different skills and two owners’ promises. The billing agent’s 0.989 cannot make up for the research assistant’s loss of 8 points. Lena’s users would get more wrong answers about equipment. Each minimum is its owner’s promise, and the average is nobody’s promise. So the gate blocks, as in the course: every consumer must pass its own evaluation. The cheapest target that passes both, gemini-3.8-flash at high, still costs 80% less than o3.

Check your understanding

Epoch AI lists claude-haiku-5-5 with 0.972 on OTIS Mock AIME at high, and 0.238 on SimpleQA Verified at max. Why does the gate say 'no evidence' for the research assistant at high, and what would make it pass or fail?

Show the answer

A public score belongs to the setting it was measured at. The mapping would send reasoning_effort high, and nobody published a SimpleQA Verified score at high. The 0.238 at max is a different setting; it is even below Lena’s minimum. A missing score is not a zero and not a pass, so the gate blocks: no evidence, no remap. To decide, Lena needs a score at high: a public one, or her own evaluation at that setting.

Check your understanding

In the what-if, gpt-5.6-luna retires and fast-small moves to gemini-3.1-flash-lite at low, with a margin of 0.002 and no fallback. The check says the catalog is consistent. Should Noor accept it?

Show the answer

Not on the public data alone. The margin is much smaller than the score’s standard error of 0.031. So the evidence cannot tell this model apart from one that fails. There is no fallback at another provider, and the model retires on 7 May 2027. The rules found the only target that passes, which is useful: the help assistant would not stop. But Noor should ask Joel to run his own evaluation on two or three candidates before that day. Widening the price band or the allowed settings is another option, with its cost written down.

Recap

  • You built a catalog from two public files: Epoch AI’s benchmark results (CC BY 4.0) and LiteLLM’s price file (MIT), frozen with checksums.
  • A public score belongs to one setting: gpt-5.6-luna scored 0.636, 0.823 and 0.916 on GPQA Diamond at none, low and max. The mapping carries the setting, and a missing score is no evidence.
  • The contracts added a retirement window: o3 retires in 50 days at OpenAI (and earlier at Azure), so today’s mapping failed the check.
  • Each app’s minimum was “no worse than today”: today’s score minus two standard errors.
  • The gate blocked gpt-6-luna although the average passed, because the research assistant fell below its own minimum.
  • Under one measure, only “the cheapest that passes every consumer” kept every rule: US$34.70 a month against US$174 today. A what-if retirement moved the mapping again, to a model with weak evidence and no fallback.

Next, the second case study: replay open agent trajectories through the gateway, and measure what budgets, step limits and tool policies would have stopped.

Next: Contain a runaway agent