Final assignment
- After the case studies
- about 3 hr
Where we are
In this course, we put controls around every agent: capabilities, registrations and identities, a model gateway and a tool gateway, budgets, policies, kill switches and evidence. Each module added one control and tried it alone. Now the whole estate must come under governance at once, during a month with an outage, a remap request and a runaway agent. In this assignment, you do it on your own, and you hand in the evidence.
What you will be able to do
Use every control of the course on the whole estate at once. You register every app with its own identity and decide the weekly digest. Then you live through a practice month: a remap request, a new approval rule, a drill, an outage, a stopped control plane and a runaway loop. You end with the month’s estate report and a build-or-buy recommendation. Every claim in your pack points to a command and its output.
Before you startWhat to know and to have ready
What you need to know
- Modules 1 to 7. Each part names the lessons that it uses. The certificate also asks for one case study.
What you need on your computer
- The course’s setup: Python 3.12, VS Code, two terminals, and the labs repository in
~/projects/nextia-learning-labs, as in Five agents and nobody knows. - The snapshot
end-of-m07, which you copy into a new folder. Your course project stays as it is. - Git, so that
git diffshows every change. No account, key or money.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
The situation: Larkfield’s next month
Section titled “The situation: Larkfield’s next month”The board asks Kwame one question: “Is every AI system that Larkfield runs under control?” He asks you to build the pack of evidence that answers it. The month brings four events: Tomás asks for “the strongest model for everyone”, Grace changes an approval rule, the provider has an outage, and a bad deployment of the workflow starts a loop.
Your month is a practice month, not the replayed September. In Module 6, the course team replayed 30 days of September with a replayed clock. The project does not include that replayer. So your month is the calendar month of your run, in one or two sessions on the real clock. Every model answer is replayed: a recording of an earlier course. The outage, the egress rule and the stopped control plane are simulated. The loop is constructed: a practice incident that you cause on purpose. Your volumes are small, so your costs are cents. Three requests are new, and no lesson answered them: Tomás’s remap, Grace’s rule and the workflow’s quota.
Text description of the diagram
Part 1 proves item 2 of the pack, Part 2 item 1, Part 3 item 4, Part 4 item 3 and Part 5 item 5. The parts are not in the order of the items: the runaway loop uses up the workflow’s quota for the rest of the day, so it comes last.
Label every number in your pack: replayed, measured (your own run), simulated, constructed, price or assumption. Your times, IDs and some counts differ. The outputs below are targets for their shape, not for every digit.
Set up a fresh copy
Section titled “Set up a fresh copy”-
Copy the snapshot into a new folder, and save its first state in Git:
Terminal window cd ~/projectscp -R nextia-learning-labs/C32/snapshots/end-of-m07 ai-estate-finalcd ai-estate-finalgit init -q && git add -A && git commit -qm "end-of-m07"Terminal window cd ~\projectsCopy-Item -Recurse nextia-learning-labs\C32\snapshots\end-of-m07 ai-estate-finalcd ai-estate-finalgit init -q; git add -A; git commit -qm "end-of-m07"Terminal window cd ~/projectscp -R nextia-learning-labs/C32/snapshots/end-of-m07 ai-estate-finalcd ai-estate-finalgit init -q && git add -A && git commit -qm "end-of-m07" -
Make a virtual environment, install the requirements and run the tests, as in Meet Larkfield’s AI estate. The last line says
195 passed, 1 deselected. -
Make an empty
REPORT.mdwith the sections of What to hand in. After each part, paste your commands, outputs and decisions into it.
Part 1: every app registered, one identity each
Section titled “Part 1: every app registered, one identity each”Uses What a registration holds, One identity per agent, The agent lifecycle and Find unregistered AI.
-
Create the registry, register the seven forms, review the three high-risk ones and issue credentials, with the commands of the project’s
README.md(“Start the estate”, withoutestate up). Then runpython -m estate list. It ends with7 registrations. -
Read each form in
registrations/. Check that its budget, limits, tools and data classes fit the app, and write one line per app in your report. -
Decide the weekly digest. Lee left, so the form needs an owner who is here. Write
registrations/weekly-digest.yamlwith a new owner, from the failure case “A form for the weekly digest” in What a registration holds. Check it and register it:Terminal window python -m estate check registrations/weekly-digest.yamlpython -m estate register registrations/weekly-digest.yamlThe form is complete.Registered weekly-digest (version 1): active -
Now choose one path, and write the reason in your report:
- Retire it. Run
python -m estate retire weekly-digest --reason "…". Move the form toregistrations/retired/, so that the setup loop does not register it again in a fresh state. (The remap gate already skips an agent that the registry has retired.) Deleteapps/digest/,tests/test_digest.pyand the last test oftests/test_evidence.py, which reads the script. Say who closes the Fernway account and asks Fernway to delete the messages. - Keep it. The new owner moves it to
fast-smallthrough the gateway, with its own credentials and no Fernway key. The simulator has no recording of the digest’s prompt on chat-small, so the call ends with422 no_recordingafter every gateway check has passed.
- Retire it. Run
-
Close the side doors. In
config/provider-keys.yaml, setaz-key-1andaz-key-2tostatus: revoked, and add Priya’saz-key-3asrevoked. Onlyaz-key-9, the gateway’s key, stays. -
In
apps/replay.py, replace the four old"key"values with"". Deletetest_direct_replay_uses_the_apps_own_keyintests/test_replay.py: it checks that an old key still works. -
In
tests/test_evidence.py, change the expectedaz-key-3finding to the new text, as in the practical check of Find unregistered AI. Then scan the code and run the tests:Terminal window python -m estate discover --scan appsruff check --fix .python -m pytest -q0 finding(s).Not found by any of these: AI features inside bought tools, and AI used on personal devices outside Larkfield's network. They enter the inventory through the buying process and by asking people.…191 passed, 1 deselectedruff --fixremoves two imports that the deleted tests used. The numbers are for the “retire” path. -
Start the estate with the egress rule on, and try both doors:
Terminal window SIM_EGRESS=gateway-only python -m estate uppython -m apps.replay ticket-triage --units 2python -m apps.replay ticket-triage --units 5 --via gatewayTerminal window $env:SIM_EGRESS = "gateway-only"python -m estate uppython -m apps.replay ticket-triage --units 2python -m apps.replay ticket-triage --units 5 --via gatewayTerminal window SIM_EGRESS=gateway-only python -m estate uppython -m apps.replay ticket-triage --units 2python -m apps.replay ticket-triage --units 5 --via gatewayThe straight call gets
egress_blocked x4; the gateway answersstatus {200: 10}. Replay the other apps with--via gateway, runpython -m apps.helpdesk T-30002, and send one call without a token, as in One identity per agent (401 missing_token).
Part 2: the catalog and Tomás’s remap request
Section titled “Part 2: the catalog and Tomás’s remap request”Uses A capability catalog, Map capabilities to models and Change a mapping safely.
Tomás writes: “chat-strong is our best model. Map fast-small to it, so that every app gets the best answers.” Decide with evidence, without changing config/catalog.yaml.
-
Run
python -m estate catalog checkon the real catalog. It saysThe catalog is consistent. -
Copy
config/catalog.yamltostate/tomas-catalog.yaml. In the copy, change both targets offast-smalltochat-strong, and remove theirparams_if_tools. Predict what the check says. Then run it on the copy:Terminal window python -m estate catalog check --file state/tomas-catalog.yaml -
Predict which consumers pass. Then run the gate:
Terminal window python -m estate remap gate fast-small --to azure/chat-strong -
Write the decision as Kwame would send it to Tomás, each sentence with its evidence. Name what must exist before the request could pass.
Check your prediction
fast-small -> azure/chat-strong: the contract promises tools, the model does not have itfast-small -> azure/chat-strong: the input price 2.0 is above the contract's band 0.5fast-small -> azure/chat-strong: the output price 10.0 is above the contract's band 2.0fast-small -> azure-canada/chat-strong: the contract promises tools, the model does not have itfast-small -> azure-canada/chat-strong: the input price 2.0 is above the contract's band 0.5fast-small -> azure-canada/chat-strong: the output price 10.0 is above the contract's band 2.0Remap gate: fast-small from azure/chat-small to azure/chat-strong
doc-assistant NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer helpdesk-assistant NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer support-workflow NO EVIDENCE no evaluation that the gate can run: no evidence, no remap for this consumer ticket-triage PASS decisions 87/102 = 0.853 (minimum 0.85); today 90/102 = 0.882; cost US$0.5252 vs US$0.0353 today items that changed: T-64746, T-64800, T-65317, T-65388, T-65449, T-65552, T-65615, T-65853, T-81018, T-81046, T-81048, T-81056 ...
Average over every item: 0.853 with the candidate, 0.882 today (mean of the consumers' rates: 0.853)Decision: BLOCK - every consumer must pass its own evaluationBoth checks block it. The contract promises tools, which chat-strong refuses on this API, and the price is four to five times the band. In the gate, three consumers have no evaluation, and triage gets worse for about 15 times the cost. A kept digest appears as a fifth consumer with no evidence; a retired one is skipped.
Part 3: tools, approvals and a drill
Section titled “Part 3: tools, approvals and a drill”Uses A gateway for tools, Autonomy and approvals as policy and Kill switches and containment.
Grace writes: “A reship sends goods a second time. From now on, a team lead approves every reship. Labels stay with the support agents.”
-
Show the tool permissions:
python -m estate tools helpdesk-assistant, thenpython -m apps.helpdesk T-30201 --obedient(403 tool_not_allowed). -
In
config/policy/approvals.yaml, split the rulelabels-and-reshipsinto two rules:return-labels(support agent) andreships(team lead). Runpython -m estate policy test. Predict first: do the 12 cases still pass? -
Add at least three cases to
config/policy/approvals-tests.yaml, on both sides of the new line: Sam may approve a label, Sam may not approve a reship, Camille may approve a reship. Run the tests until they pass, then publish withpython -m estate config publish --note "…". -
Run the project’s tests. One workflow test fails: it still expects Sam to approve the reship of T-90103. Change it to the new rule: Sam is refused, Camille approves
--as team_lead. -
Plan the drill before you run it: which switch, after which task, what you expect, and who lifts it. Use at least two levels of the containment ladder, in a plan that differs from the lesson’s. Reset the practice database with
python -m resolver reset, and run the day in parts withpython -m apps.workday --tasks …. -
When a run waits for a team lead, approve it as Camille:
Terminal window RESOLVER_PROVIDER=gateway RESOLVER_TOOLS=gateway python -m resolver approve RUN_ID --as team_lead --approver usr-camilleIn PowerShell, set the two variables with
$env:first, as in the module. -
Record the drill: an incident of kind
drill(python -m estate incident add, thenend), andkill_switch_testedinregistrations/support-workflow.yamlset to the day of the drill. Register the form again.
One drill, and what it showed
The plan: stop every order write after task 3 (python -m estate kill writes orders --reason …), lift it after task 5, suspend the workflow for task 9, and resume it. The day, shortened:
task 1 T-90103 resolve (rule W5) | approval by usr-sam refused: Not approved by the platform: Sam may not approve this: it needs a team lead or above (rule reships)task 2 T-90104 resolve (rule W2) | approved by usr-sam: request_refund LK-793996: done | donetask 3 T-90603 resolve (rule W7) | approved by usr-sam: request_refund LK-645223: done | donetask 4 T-65273 resolve (rule W7) | approved by usr-camille: request_refund LK-507862: failed request_refund: refused by the tool gateway: Stopped: writes are stopped for orders (kill switch 1: drill: wrong goods sent, stop every order write). | handed_to_persontask 5 T-64872 resolve (rule W7) | approved by usr-sam: request_refund LK-739156: failed request_refund: refused by the tool gateway: Stopped: writes are stopped for orders (kill switch 1: drill: wrong goods sent, stop every order write). | handed_to_persontask 9 T-90102 Stopped: provider_error. No proposal: the ticket goes to a person. | Last error: The platform refused a token: {"error":{"code":"invalid_client","message":"support-workflow cannot get a token: its registration is suspended."}}task 11 T-90106 resolve (rule W5) | approval by usr-sam refused: Not approved by the platform: Sam may not approve this: it needs a team lead or above (rule reships)task 13 T-90203 resolve (rule W4) | approval by usr-sam refused: Not approved by the platform: Sam may not approve this: it needs a team lead or above (rule reships)In tasks 1, 11 and 13, the workday tool asks the person that the app’s own check names, Sam, and the platform refuses them: the app’s copy of the rules is out of date, and the policy wins. After Camille approves the three reships, python -m resolver changes ends with 13 change(s) in the practice database. Three tasks went to people: two refunds during the stop, and the new run during the suspension.
Part 4: the gateway on four scenarios
Section titled “Part 4: the gateway on four scenarios”Uses One door for model calls, Quotas, budgets and rate limits, Guardrails at the gateway and When the gateway fails.
First write the gateway’s settings in your report, with a reason for each: the budgets and limits of each app and team, the failure mode of each risk tier in config/gateway.yaml, and the guardrail. Then show them working.
- One guardrail. Send the triage request with a card number, as in Guardrails at the gateway. Look for
x-gateway-redacted: card-numbers: 1 card number(s) redacted, and for the guardrail inpython -m estate usage --last 1. Explain the422in one sentence. - A provider outage. Open an incident. Take down East US 2 (
python -m provsim control azure down), replay triage and one workflow unit, and readpython -m estate usage --last 1:provider azure-canadaandfallback 1. Then take down both regions ('*'), and replay triage, the workflow and the document assistant: each getsno_provider_available. Write what each app does then. Set both controls back tonormal, wait 10 seconds for the breaker, replay triage, and end the incident. - The control plane stops. In terminal 1, run
python -m apps.traffic --seconds 120. At about 20 seconds, runpython -m estate down controlin terminal 2. At about 90 seconds, ask replica a withcurl -s http://127.0.0.1:8323/ready. At about 110 seconds, runpython -m estate up control. Find the moment when each risk tier’s failure mode starts. - The workflow’s quota from its real use. The workflow has 2,000,000 tokens a day. Compute what it really needs: 45 runs a day (an assumption) × the tokens of one run, as
python -m resolver runprints them (T-90103: 4,647 in and 266 out). Add room, and show the arithmetic. Changetokens_per_day, register the form again and review it. - The runaway loop, last. Run
python -m apps.runaway support-workflow --attempts 300three times; each run waits for the next minute. Then run one real task, T-90104, through both gateways. Say which limit stopped the loop, after how many answers, at what cost, and what happened to the real task. The quota counts the UTC day.
Check your readings
The control plane stopped at 20 s. The traffic, terminal 1, shortened:
70 s | triage ok 17 | workflow - | documents ok 3 | replicas a 6, b 7, c 7 80 s | triage ok 15 | workflow - | documents degraded 1, ok 3 | replicas a 7, b 6, c 6 90 s | triage degraded 17 | workflow control_unavailable 1 | documents degraded 2 | replicas a 6, b 7, c 7 100 s | triage degraded 19 | workflow - | documents - | replicas a 7, b 6, c 6 110 s | triage degraded 17 | workflow - | documents degraded 3 | replicas a 6, b 7, c 7 120 s | triage degraded 2, ok 14 | workflow control_unavailable 1, ok 2 | documents ok 1 | replicas a 7, b 7, c 6For 60 seconds, the cached copies serve every app. Then triage (medium) and documents (low) fail open, marked degraded, and the workflow (high) fails closed.
The loop, with a quota of 450,000 tokens (45 runs × about 5,000 tokens, twice):
answered 120 | cost US$0.5897 | 0.7 srefused rate_limited 180…answered 26 | cost US$0.1278 | 0.5 srefused quota_exhausted 94, rate_limited 180first quota_exhausted: after 26 answers and US$0.1278 (0.2 s): 429 support-workflow has used its 450,000 tokens for today.…answered 0 | cost US$0.0000 | 0.5 srefused quota_exhausted 120, rate_limited 180The rate limit slowed the loop, and the quota stopped it in its second minute: 146 answers, US$0.72. Your count differs, because the drill and the replays used part of the day’s tokens. With 2,000,000 tokens, the course’s simulated loop in Quotas, budgets and rate limits cost US$4.40. The price of the tighter quota is the real task: T-90104 stops, and its ticket goes to a person.
Part 5: the month’s evidence and build or buy
Section titled “Part 5: the month’s evidence and build or buy”Uses Usage and cost per agent, Evidence for governance, Find unregistered AI, Gateways compared and Roll it out to an organisation.
-
Write the month’s report, for the month of your run, as in Evidence for governance:
python -m estate report --month $(date -u +%Y-%m) --out state/report.md. -
Save the simulator’s bill and egress log of your month, and run discovery and adoption on them:
Terminal window python -m provsim bill > state/bill.csvpython -m provsim egress > state/egress.csvpython -m estate discover --bill state/bill.csv --egress state/egress.csv --scan appspython -m estate adoption --month $(date -u +%Y-%m) --bill state/bill.csvOn Windows, get the month as in the lesson, and write the files with
| Out-File -Encoding utf8. -
Check that the report agrees with your incidents, drill and egress test. For every finding and overdue review, say who acts, and by when.
-
Write the build-or-buy recommendation for Larkfield: about one page, with the four questions of the module’s independent task, the must-haves from
docs/gateway-comparison.md, and the numbers of your month. Add the cost of a change of product in two years, and what in your design keeps it low. -
Stop the estate with
python -m estate down.
What the course’s month showed
Provider calls through the gateway: 456 of 456 (100.0%); around it: 0.Friction: the share of each agent's calls that the gateway refused. support-workflow 757 of 955 refused (79.3%): rate_limit 541, quota 214, failure_mode 2 ticket-triage 0 of 240 refused (0.0%)Every billed call used az-key-9, the gateway’s key. Discovery still lists two egress lines, from triage-svc-01 and priya-laptop: your own test of the egress rule in Part 1, every connection blocked. The report lists the release judge’s overdue review: Priya reviews it, or the judge stops. The workflow’s friction is the loop, not normal work.
Fresh-environment run
Section titled “Fresh-environment run”Copy the project without state/ and .venv, make a new virtual environment and install the requirements. Run ruff check ., the tests, python -m estate init, the registration of every form, python -m estate discover --scan apps and python -m estate policy test. Paste the last lines into your report. If a step needed something that the README does not say, fix the README.
What to hand in
Section titled “What to hand in”Your project folder without .venv and state/, with your changes in Git, and REPORT.md:
# Larkfield's AI estate under governance: the practice month## 1. The catalog and the remap (Tomás's request: the check, the gate, the decision and what it needs)## 2. Registrations and identities (the list, one line per app, the digest's decision, the key inventory, the scan with 0 findings, the egress test, the call without a token)## 3. Gateway configuration (budgets, quotas, failure modes with reasons, the guardrail; the four scenarios with their outputs; the quota arithmetic; the loop's cost)## 4. Tools, approvals and the drill (Grace's rule, the policy tests, the changed project test, the drill plan, the day, the incident and kill_switch_tested)## 5. The month's evidence (the report, discovery, adoption and friction, each finding explained)## 6. Build or buy (the recommendation, the must-haves, the cost of a change of product)## 7. Fresh-environment run## 8. Labels and limits (which numbers are replayed, measured, simulated, constructed, assumed; what this practice month cannot say about production)Include git diff --stat against your first commit. A reviewer reads the diff of the forms, the catalog copy, the policy, the key inventory and the tests.
Check your work
Section titled “Check your work”A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. To complete the course, all five must be demonstrated; you can improve an item and send it again. Without a reviewer, use the table on your own pack.
| Required item | Demonstrated when | Partially demonstrated, for example | Not yet demonstrated, for example |
|---|---|---|---|
| 1. A capability catalog with contracts and mappings, and one remap decided with evaluation evidence (Module 2) | The catalog passes its check. The decision quotes the check and each consumer’s result, and names the missing evidence. | The decision quotes only the average, or ignores the consumers with no evidence. | The mapping changed without the gate, or “it is the better model”. |
| 2. Registrations, identities and permissions for every app, the digest registered or retired (Module 3, Find unregistered AI) | Every app has an owner who is here, its own credentials and only its tools. The digest is retired with its code, or on the gateway with a new owner. No team or personal key is left. | One key is still to_revoke. Or the digest is retired, but its script with the Fernway key is still there. |
The digest still calls Fernway, or two apps share a credential. |
| 3. Gateway configuration with budgets, quotas, a failure mode per tier and one guardrail, shown working (Module 4) | Each setting has a reason. The outputs show the guardrail, the fallback, both failure modes and the limit that stopped the loop, with its cost. | A failure mode has no reason, or the loop’s cost is missing. | Only the happy path ran, or the high tier fails open. |
| 4. Tool policies and an approval policy with tests, and a recorded kill-switch drill (Module 5) | Grace’s rule has tests on both sides, and the project’s tests pass. The drill had a written plan with two levels or more, and is recorded. | The drill is not recorded, or the tests cover one side of the rule. | A reship passes with Sam’s approval, or no drill ran. |
| 5. The month’s estate report and a build-or-buy recommendation (Module 6, Module 7) | Every finding and overdue review has an owner and an action. The recommendation answers the four questions with must-haves and numbers. | A finding is not explained, or a product is named with no must-haves. | No report, or “buy LiteLLM” with no reasons. |
Not accepted, whatever the rest shows (the item is “not yet demonstrated”):
- An agent can call a model or a tool without a valid registration. Show a call without a token, a retired or suspended agent and an unlisted tool, each refused. (Items 2 and 4.)
- A shared or personal key remains. Every key but the gateway’s is revoked, and the scan of
apps/finds no key. (Item 2.) - A write can bypass the approval policy. Every write tool in
config/tools.yamlis named by a rule or refused bydefault: deny, and a test shows it. (Item 4.)
Common mistake
A pack that passes on paper
Symptom: the report says “every key is revoked”, and config/provider-keys.yaml agrees. But python -m estate discover --scan apps still lists four keys in apps/replay.py.
Cause: the inventory records a decision; it does not change the code. The simulator cannot revoke a key as a real provider does, so here the old keys still work wherever no egress rule stops them. In a real estate, a key left in code is ready for the next copy of the file.
Fix: remove the keys from the code, scan again until it finds nothing, and keep the egress rule on. The inventory, the scan and the egress log must agree.
Common mistake
A drill that stops the day
Symptom: the drill’s tasks all end with Stopped: provider_error and HTTP 429: rate limited. python -m resolver changes says 0 change(s).
Cause: the runaway loop ran before the drill. It used up the workflow’s quota for the day, so the drill’s real tasks were refused. A limit per agent cannot tell a loop from real work.
Fix: run the drill before the loop, or the next day (UTC). Say in your report what this shows: the quota protects the money, and it stops the real work too.
Recap
In this course, you put every AI system of an organisation under controls that work in code:
- Ask for a capability, not a model, and remap only when every consumer passes its own evaluation.
- Register every agent before its first call, with an owner who is here and its own short-lived credentials.
- Send every call through a door that checks, limits and records it, and fails open or closed by risk.
- Keep people in charge of actions with tested approval policies and kill switches that you have tried.
- Keep the evidence, and roll the platform out from must-haves, one app at a time.
A gateway does not make an estate governed. The registrations, the policies, the drills and the people who read the evidence do.