Skip to content
Français

Change a mapping safely

  • Lesson 6
  • 45 min
  • Module 2: Ask for capabilities, not models

Where we are

In the last two lessons, we gave each capability a contract and mapped it to a model, with fallbacks, data rules and pins. But your practice remap passed the catalog check without one question about quality, and one change to a mapping changes every agent that uses the capability at once. In this lesson, Omar asks for a cheaper model behind strong-reasoning, and we decide it as a release, with each consumer's own evaluation as the gate. The rental company changes its cars first.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Decide a change of the model behind a capability as a release. Every consumer’s own evaluation must pass before any traffic moves. Then a share of the traffic moves first, someone watches it, and the change can go back. You also decide who may approve.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

What you need on your computer

  • Your project after the last lesson: python -m pytest -q prints 100 passed, 1 deselected, and try_route.py is deleted.
  • A terminal with the virtual environment active. The estate does not need to run: the gate starts its own provider simulator inside the command.

Tested on macOS (Apple silicon) with Python 3.12.13; the Windows and Linux commands are not tested. Every model answer is replayed from the agents course and the evaluation course. One consumer in this lesson is constructed, and the lesson says so each time.

The rental company wants cheaper mid-size cars. It lends the new model to 100 customers for a weekend, and asks each for a score from 1 to 5. The average is 4.4. The manager wants to replace every mid-size car next Monday.

Then she reads the scores one by one. Most customers gave 5. The family with the trailer gave 1: the new car has no tow bar strong enough, and their trip failed. In an average of 100 scores, one failed trip is almost invisible. For that family, it is the whole answer.

So she changes the plan. Each big customer tests the car on its own trip, with its own rule for “good enough”. If one of them fails, the change waits. If all pass, two branches change first. Someone watches the complaints for a week, and the old cars stay in the car park, so the branches can switch back. And the salesperson who asked for cheaper cars does not decide alone. Each customer signs off on its own test.

The problem: one remap changes every consumer at once

Section titled “The problem: one remap changes every consumer at once”

Omar, the product manager, looks at the bill. In Module 1, you saw that one workflow task costs about 50 times one triage ticket, because it makes several chat-strong calls. He asks Kwame: “Move strong-reasoning to chat-small. It costs 20 times less per token.”

The numbers back him. On the recorded evaluations, the support workflow’s 70 tasks cost US$0.78 on chat-strong and US$0.04 on chat-small. The release judge’s 101 replies cost US$0.42 and US$0.03 (replayed, at the course’s prices). And in the last lesson, the catalog check accepted chat-small for strong-reasoning.

But a mapping is shared. One line in catalog.yaml changes the model for every consumer of the capability, at the same moment. Two of Larkfield’s apps ask for strong-reasoning: Amira’s support workflow and Priya’s release judge. Their prompts and their evidence were made on chat-strong.

There is also a trap in how the change is usually judged. Look at a small, made-up case first:

Consumer Items Passed Rate Its own minimum Result
A 10 6 0.60 0.90 fails
B 90 88 0.98 0.85 passes
All items together 100 94 0.94

The average over every item is 0.94. A platform that checks one average against 0.85 says yes. But consumer A fails badly, and the average hides it because B has nine times more items. Like the family with the trailer, A’s result is the only one that matters to A.

The idea: a gate per consumer, then a release

Section titled “The idea: a gate per consumer, then a release”

A remap is a change of the mapping: another model or provider behind a capability. Treat it as a release, like a new version of code, with these steps:

  1. Proposal. Someone writes the candidate mapping, with the settings the new model needs.
  2. The remap gate. The platform finds every consumer: every registration that lists the capability. It runs each consumer’s own evaluation, the one named in its registration, on the current model and on the candidate. The remap passes only if every consumer reaches its own minimum. A consumer without an evaluation that the gate can run is “no evidence”, and no evidence means no remap.
  3. Canary. Only then, a share of the real traffic moves first. At Larkfield, that is one of three gateway replicas, which you build in Module 4.
  4. Watch. Errors, refusals, the model that answered, and each consumer’s own signals, such as the share of tasks that go to a person.
  5. Promote or roll back. If all is well, the change goes to every replica. If not, the previous configuration version comes back.

The gate comes before the canary for a reason. A canary sends real customers’ work to the new model. For an agent that writes, such as the workflow that sends reships, a bad canary leaves real changes behind. A rollback changes the mapping back; it cannot take a parcel back.

A remap as a release, left to right. 1, Proposal: Omar asks, Kwame writes the candidate mapping strong-reasoning to chat-small. 2, Gate: each consumer's own evaluation; release-judge 70 of 101 = 0.693, minimum 0.9, FAIL; support-workflow 64 of 70 = 0.914, minimum 0.95, FAIL. If one fails: BLOCK, back to the proposal with the evidence, which is today's real result. If all pass: 3, Canary, the new mapping on 1 of 3 replicas. 4, Watch: errors, the model that answered, each consumer's own signals. If healthy: 5a, promote to every replica. If a problem: 5b, roll back to the last configuration version. Below: who approves. Each consumer's owner signs off on its own result, Amira for the workflow and Priya for the judge. Kwame, the platform owner, runs the release. Omar may ask, but he does not approve. A rollback changes the mapping back; it does not undo what an agent already did: a reship stays sent.
A remap is a release: a gate per consumer, a canary, a watch, and a way back. Omar's request stops at the gate.
Text description of the diagram

The release has five steps. First, the proposal: Omar asks, and Kwame writes the candidate mapping, strong-reasoning to chat-small. Second, the gate runs each consumer’s own evaluation. Here both fail. The release judge reaches 70 of 101, 0.693, below its minimum of 0.9. The support workflow reaches 64 of 70, 0.914, below 0.95. So the remap is blocked and goes back to the proposal, with the evidence. If every consumer had passed, the third step would be a canary on one of three replicas. Then someone would watch the errors, the model that answered and each consumer’s own signals. Then the change would be promoted to every replica, or rolled back to the last configuration version. Each consumer’s owner signs off on its own result, Kwame runs the release, and Omar may ask but does not approve. A rollback does not undo what an agent already did.

Choose the model behind strong-reasoning, and which consumers count. The explorer shows the gate’s real runs, not a model of them:

Explore

One remap, every consumer's own evaluation

Gate decision
BLOCK
Average over every item
0.784: fails an average check of 0.85
Evaluation cost
US$0.07 vs US$1.19 today

The average over every item is 0.784, below 0.85 too. Here the average and the gate agree: block. Failing: release-judge and support-workflow. A consumer’s own minimum is the promise that its owner made; the average is nobody’s promise.

  • reaches its own minimum
  • below its own minimum
  • line: the consumer’s minimum
strong-reasoning from chat-strong to chat-small: each consumer's evaluation
ConsumerCandidate against its minimumVerdict
release-judge
Priya
agreement with the code checks 70/101 = 0.693 · minimum 0.90 · today 0.980102 answers replayed from recordings with another settingFAIL
support-workflow
Amira
task success 64/70 = 0.914 · minimum 0.95 · today 1.0002 unexpected changes in the databaseFAIL
Real runs of python -m estate remap gate on replayed answers. The what-if triage consumer is constructed; its answers are real recordings. gemma3:4b: US$0 per token is an assumption (own hardware).
Text description of the explorer

The explorer has two controls. The first chooses the model behind strong-reasoning: chat-strong, which serves it today, chat-small, or gemma3:4b on Larkfield’s own hardware. The second chooses the consumers that count: the two real consumers, or the support workflow with a constructed what-if consumer. That consumer is the triage service as if it asked for strong-reasoning. The readout shows the gate’s decision and the cost of the evaluation runs. It also checks the average over every item against a floor of 0.85, as a single average check would. A table shows one row per consumer: a bar for its rate, a line at its own minimum, today’s rate and the verdict.

It starts on chat-small with the two real consumers. The release judge reaches 70 of 101, 0.693, below its minimum of 0.9; today 0.980. The support workflow reaches 64 of 70, 0.914, below 0.95, with 2 unexpected changes in the database; today 1.000. The average over every item is 0.784, and the decision is BLOCK. The evaluation runs cost US$0.07 on chat-small against US$1.19 today.

With chat-strong, both consumers pass and nothing changes. With gemma3:4b, the workflow reaches 28 of 70, 0.400, and the judge has no evidence: nobody recorded its requests on gemma3:4b. The decision is BLOCK.

With the constructed set and chat-small, the what-if triage consumer reaches 90 of 102 (0.882). This passes its minimum of 0.85 and beats chat-strong’s 87 of 102. The workflow still fails. The average is 0.895, so a single average check of 0.85 would pass. The gate blocks the remap anyway.

Larkfield: run the gate on Omar’s request

Section titled “Larkfield: run the gate on Omar’s request”
  1. Copy this lesson’s files from the snapshot end-of-m02: the gate, its command, the evaluations, the draft registrations, the test and the new README.md:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m02
    cp $S/estate/remap.py estate/
    cp $S/estate/cli/remap.py estate/cli/
    cp -R $S/evaluations $S/registrations .
    cp $S/tests/test_remap.py tests/
    cp $S/README.md .
  2. Run the tests. The last line says 104 passed, 1 deselected. Your project is now the same as the snapshot end-of-m02.

  3. Open registrations/release-judge.yaml. It is a draft: the next module completes it. Find the two parts that the gate reads: capabilities: [strong-reasoning], and under evaluations: the evaluation, its measure agreement and its minimum 0.9. Priya wrote that minimum; it is her promise, not the platform’s.

  4. Run the gate for one consumer first, the release judge:

    Terminal window
    python -m estate remap gate strong-reasoning --to azure/chat-small --param reasoning_effort=none --consumer release-judge

    It takes about a second:

    Remap gate: strong-reasoning from azure/chat-strong to azure/chat-small with {'reasoning_effort': 'none'}
    release-judge FAIL agreement 70/101 = 0.693 (minimum 0.9); today 99/101 = 0.980; cost US$0.0319 vs US$0.4165 today
    warning: 102 answers come from recordings made with another reasoning_effort: the real model may answer differently
    items that changed: T-64720, T-64740, T-64746, T-64800, T-64876, T-64944, T-65003, T-65104, T-65126, T-65270, T-65353, T-65552 ...
    Average over every item: 0.693 with the candidate, 0.980 today (mean of the consumers' rates: 0.693)
    Decision: BLOCK - every consumer must pass its own evaluation

Read the output step by step. The gate ran the judge’s evaluation twice: on today’s mapping and on the candidate. The judge reads each support reply and decides whether it is bad. The evaluation course compared that verdict with the code checks of the same replies. On chat-strong, the judge agreed on 99 of 101 replies. On chat-small, it agreed on 70. One reply has no verdict on either model: the provider’s content filter stopped it, so 101 count, not 102.

items that changed lists the 31 replies whose result differs between the two models (the first 12 shown). Priya can read them and see where chat-small judges differently.

The warning is the gate being honest about its own limit. The evaluation course recorded the judge on chat-small without reasoning_effort. The candidate sets reasoning_effort: none, which chat-small needs for tools. The simulator’s fingerprint leaves that field out, so it replays the answers that were recorded without it. The judge’s numbers are the same in a run without the setting, which is how the evaluation course recorded them. Still, a real gate calls the candidate model for real.

Note also what the judge is. It is the instrument that checks the support replies before they ship. If the remap made the judge weaker, every later release of the replies would be checked by a weaker judge. A cheaper judge is not a saving if it lets bad replies through.

Now run the gate on both real consumers, as Kwame must before any change:

Terminal window
python -m estate remap gate strong-reasoning --to azure/chat-small --param reasoning_effort=none
Remap gate: strong-reasoning from azure/chat-strong to azure/chat-small with {'reasoning_effort': 'none'}
release-judge FAIL agreement 70/101 = 0.693 (minimum 0.9); today 99/101 = 0.980; cost US$0.0319 vs US$0.4165 today
warning: 102 answers come from recordings made with another reasoning_effort: the real model may answer differently
items that changed: T-64720, T-64740, T-64746, T-64800, T-64876, T-64944, T-65003, T-65104, T-65126, T-65270, T-65353, T-65552 ...
support-workflow FAIL task_success 64/70 = 0.914 (minimum 0.95); today 70/70 = 1.000; cost US$0.0399 vs US$0.7771 today
unexpected changes in the database: 2 (today 0)
items that changed: T-90103, T-90106, T-90201, T-90305, T-90403, T-90407
Average over every item: 0.784 with the candidate, 0.988 today (mean of the consumers' rates: 0.804)
Decision: BLOCK - every consumer must pass its own evaluation

The workflow’s evaluation is the agents course’s 70 resolution tasks, each on a fresh practice database. Every proposal is approved as it is: the worst case. chat-small solved 64. The line unexpected changes in the database: 2 is the most serious one. Both changes are on T-90305, a ticket written in German. The workflow’s policy hands that ticket to a person. On chat-small, the agent sent a return label and a reship instead. A canary would have sent those to a real customer.

So Omar’s request fails on both real consumers, and the average fails too: 0.784, against 0.988 today. This is the real result of the course’s data, and the decision is clear: BLOCK, with the evidence above. Kwame sends Omar the output, not an opinion.

Tomás draws the opposite conclusion: “So the strongest model is always the safe choice. Let us serve fast-small from chat-strong too.” The gate can test that. Move fast-small up to chat-strong:

Terminal window
python -m estate remap gate fast-small --to azure/chat-strong
Remap gate: fast-small from azure/chat-small to azure/chat-strong
doc-assistant NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer
helpdesk-assistant NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer
support-workflow NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer
ticket-triage PASS decisions 87/102 = 0.853 (minimum 0.85); today 90/102 = 0.882; cost US$0.5252 vs US$0.0353 today
items that changed: T-64746, T-64800, T-65317, T-65388, T-65449, T-65552, T-65615, T-65853, T-81018, T-81046, T-81048, T-81056 ...
Average over every item: 0.853 with the candidate, 0.882 today (mean of the consumers' rates: 0.853)
Decision: BLOCK - every consumer must pass its own evaluation

For the triage service, the stronger model is worse: 87 of 102 right decisions, against 90 on chat-small, for 15 times the cost. The evaluation course found the same. “Always the newest or strongest model” is wrong for some consumers. Measure per consumer; do not guess.

Common mistake

NO EVIDENCE: a consumer without an evaluation

Symptom: the gate prints NO EVIDENCE for one or more consumers, and Decision: BLOCK, even though no consumer failed. Above: the document assistant, the help-desk assistant and the support workflow, for fast-small.

Cause: the gate found a registration that lists the capability, but no evaluation that it can run. The help-desk assistant’s draft names no evaluation at all. The document assistant names the capstone course’s held-out questions, and the support workflow names none for fast-small; this project has no runner for either. The gate cannot know whether the new model is good enough for them, so it does not guess.

Diagnose: open the registration of each consumer that says NO EVIDENCE, for example registrations/helpdesk-assistant.yaml, and look under evaluations: for the capability.

Recover: the consumer’s owner adds an evaluation that the gate can run, with a measure and a minimum, before the remap. Or the consumer stops asking for that capability. Never remove a consumer from the gate to make a remap pass: that consumer’s calls would still move.

Today the gate blocks, so nothing moves. When a remap does pass, Larkfield’s next steps use the gateway that you build in Module 4:

  • Canary. python -m estate config publish --stage canary sends a new configuration to one of the three gateway replicas: about a third of the calls.
  • Watch. Every call records its consumer, its outcome and the model that answered. Watch the errors per consumer and each consumer’s own signal, such as tickets that go to a person.
  • Promote or roll back. promote sends the configuration to every replica; rollback brings back the previous version, in seconds, because the mapping is configuration, not code.

Who approves. The capability belongs to the platform, but the evidence belongs to the consumers. So each consumer’s owner signs off on its own result: Amira for the workflow, Priya for the judge. Kwame, the platform owner, runs the release and can stop it. Omar may ask for a remap, and his reason is a good one, but he does not approve it. A consumer whose evaluation fails can also choose a pin: it keeps chat-strong until its retire_by date, and the others move. This project’s gate counts every consumer of the capability, pinned or not; Module 3 adds the pins to the registrations.

Make your own proposal. The cheapest model in models.yaml is gemma3:4b, on Larkfield’s own hardware: US$0 per token (an assumption: the hardware and power are not counted). Its data rule is already decided: the data stays at Larkfield.

Predict first: which consumers of strong-reasoning will pass, which will fail, and is there any consumer that the gate cannot judge? Then run the gate on the two real consumers, and write your decision in two sentences, with the evidence.

Terminal window
python -m estate remap gate strong-reasoning --to local/gemma3:4b
Check your prediction
Remap gate: strong-reasoning from azure/chat-strong to local/gemma3:4b
release-judge NO EVIDENCE no answer of local/gemma3:4b to score: no evidence, no remap for this consumer
support-workflow FAIL task_success 28/70 = 0.400 (minimum 0.95); today 70/70 = 1.000; cost US$0.0000 vs US$0.7771 today
unexpected changes in the database: 1 (today 0)
items that changed: T-80005, T-90102, T-90103, T-64872, T-65273, T-90104, T-80009, T-65070, T-90105, T-90106, T-90107, T-90201 ...
Average over every item: 0.400 with the candidate, 1.000 today (mean of the consumers' rates: 0.400)
Decision: BLOCK - every consumer must pass its own evaluation

A decision with evidence: “Block. The support workflow solved 28 of 70 tasks on gemma3:4b, far below its minimum of 0.95. It also made one unexpected change in the database. The release judge has no evidence, because nobody ran its evaluation on gemma3:4b.” The agents course recorded its 70 tasks on gemma3:4b; no course recorded the judge there, so there is no answer to score. Free tokens do not help a model that fails the work.

Completion check: the average passes, one consumer fails

Section titled “Completion check: the average passes, one consumer fails”

This check uses a constructed consumer. In Larkfield’s real data, no remap passes the average while one consumer fails: both real consumers failed together. So the course made a what-if registration, registrations/what-if/ticket-triage-on-strong.yaml: the triage service as if it asked for strong-reasoning. It is not part of the estate. Its answers are real: the evaluation course recorded the same triage requests on both models.

Run the gate with the support workflow and the what-if consumer:

Terminal window
python -m estate remap gate strong-reasoning --to azure/chat-small --param reasoning_effort=none \
--consumer support-workflow --consumer ticket-triage-on-strong \
--also registrations/what-if/ticket-triage-on-strong.yaml
Remap gate: strong-reasoning from azure/chat-strong to azure/chat-small with {'reasoning_effort': 'none'}
support-workflow FAIL task_success 64/70 = 0.914 (minimum 0.95); today 70/70 = 1.000; cost US$0.0399 vs US$0.7771 today
unexpected changes in the database: 2 (today 0)
items that changed: T-90103, T-90106, T-90201, T-90305, T-90403, T-90407
ticket-triage-on-strong PASS decisions 90/102 = 0.882 (minimum 0.85); today 87/102 = 0.853; cost US$0.0353 vs US$0.5252 today
warning: 102 answers come from recordings made with another reasoning_effort: the real model may answer differently
items that changed: T-64746, T-64800, T-65317, T-65388, T-65449, T-65552, T-65615, T-65853, T-81018, T-81046, T-81048, T-81056 ...
Average over every item: 0.895 with the candidate, 0.913 today (mean of the consumers' rates: 0.898)
Decision: BLOCK - every consumer must pass its own evaluation

Now write the decision as Kwame would send it to Omar, in the knowledge check below.

Use it for every remap, also a “small” one, such as a new version of the same model family. Use it for any fallback that changes the model, too. The name of the capability stays; the answers change.

It needs an evaluation per consumer. That is the real cost. Each team keeps a set of cases, a measure and a minimum, in a form that the gate can run. A consumer without one blocks every remap of its capability, which is the right pressure.

It replays here, and calls for real elsewhere. This course’s gate replays recorded answers in about a second, at no cost. A real gate calls the candidate model on every case. At the course’s prices, the two real evaluations above cost US$0.07 on chat-small and US$1.19 on chat-strong. Prices change: check the current page.

It does not replace the canary and the watch. An evaluation is a fixed set of cases. Real traffic has cases that the set does not, so a share of traffic and a person watching come next.

Check your understanding

Completion check: in the what-if run, the average over every item is 0.895, above a floor of 0.85. Write the decision for Omar, with the evidence, and explain why the average is the wrong check.

Show the answer

Block the remap. The support workflow reaches task success of 64 of 70, 0.914, below its own minimum of 0.95; today it solves 70 of 70. Its run also made 2 unexpected changes in the practice database, both on T-90305, where the policy hands the ticket to a person. The what-if triage consumer passes: 90 of 102 against a minimum of 0.85, better than chat-strong’s 87. The average of 0.895 mixes 102 triage items with 70 workflow tasks, so the larger consumer hides the one that fails. Each consumer’s minimum is its owner’s promise; the average is nobody’s promise. The what-if consumer is constructed, but its answers are real recordings.

Check your understanding

The gate printed a warning: 102 answers come from recordings made with another reasoning_effort. Does that make the judge's FAIL unreliable? What would a real gate do?

Show the answer

The FAIL holds for what was recorded. The evaluation course recorded the judge on chat-small without the setting, and those answers give 70 of 101 in a run without it too. But the candidate mapping adds reasoning_effort none, and the real model might answer a little differently with it. The warning says so instead of hiding it. A real gate calls the candidate model, with the candidate’s settings, on every case of the evaluation. That costs money, about US$0.03 for the judge’s 101 replies at the course’s prices, and gives an exact answer.

Check your understanding

Omar says: 'The remap saves money on every task. I am the product manager, so I approve it.' Who should approve a remap, and what can Amira do if her workflow fails but Priya's judge passes?

Show the answer

Each consumer’s owner signs off on its own evaluation: Amira for the workflow, Priya for the judge. Kwame, the platform owner, runs the release and can stop it. Omar’s request is a good reason to look, but the evidence belongs to the consumers, so he does not approve. If only the workflow fails, Amira can pin strong-reasoning to chat-strong with a retire_by date, and the other consumers move. Or she improves the workflow’s prompts for chat-small and runs the gate again.

Recap

  • A remap changes the model for every consumer of a capability at once. So it is a release: proposal, gate, canary, watch, promote or roll back.
  • The remap gate runs each consumer’s own evaluation against its own minimum. No evidence, no remap.
  • Omar’s request, strong-reasoning to chat-small: the judge 70/101 = 0.693 (minimum 0.9). The workflow 64/70 = 0.914 (minimum 0.95), with 2 unexpected changes on T-90305. Average 0.784. BLOCK, although it would cost about 20 times less.
  • An average can pass while one consumer fails: the constructed what-if reaches 0.895 and is still blocked.
  • The cheaper model is not always worse: for triage, chat-small beats chat-strong, 90 against 87 of 102.
  • Each consumer’s owner approves its own result; the platform owner runs the release; the person who asks does not approve.

Your project now matches the snapshot end-of-m02: 104 passed. In this module, you gave apps capabilities instead of models, mapped them to models and changed a mapping with evidence. But the gate found its consumers in draft forms, and any script with a key can still call a provider without being in them. Next, in Module 3, every agent gets a registration that says who owns it, what it may do and what it may cost.

Next: What a registration holds