Evidence for governance
- Lesson 18
- 40 min
- Module 6: Usage, evidence and shadow AI
Where we are
In the last lesson, we turned the gateway's records into usage and cost reports per agent, team and capability, and joined them with the app's trace for the cost of a task. But the governance board asks other questions: is every agent registered, which ones are high risk, what went wrong this month? In this lesson, we answer the board from evidence, the way a restaurant answers a health inspector, and we say which question the evidence cannot answer.
Lesson 18 of 21 · Governing AI Agents: Gateways, Registries and ControlsEvidence for governance
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Answer a governance board’s questions about an organisation’s AI from the platform’s records: the registry, the gateway’s records and the audit trail. You read Larkfield’s September report and write your own. Then you name what the evidence cannot answer.
Before you startWhat to know and to have ready
What you need to know
- The usage reports and the join by run ID, from Usage and cost per agent.
- What a registration holds, its risk tier and its review date, from What a registration holds, and the audit trail of
estate events, from The agent lifecycle.
What you need on your computer
- Your project
~/projects/ai-estateafter the last lesson: the snapshotend-of-m05, with the last lesson’s traffic instate/, and its virtual environment active. If you start from a fresh copy ofend-of-m05, do the “Start a fresh estate” and “Group the rows” steps of the last lesson first. Your numbers then match this lesson’s. - The labs repository in
~/projects/nextia-learning-labs, for this lesson’s files from the snapshotend-of-m06.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, not run. The September report is replayed traffic; its outage, runaway and digest are simulated or constructed practice incidents, as the earlier modules said.
A restaurant inspection
Section titled “A restaurant inspection”A health inspector visits a restaurant and asks the same questions every time. Is the fridge cold enough? Is the kitchen cleaned every day? Who is in charge? Has anyone been ill after a meal?
The restaurant does not answer with promises. It shows records. The fridge has a thermometer that writes the temperature every hour, by itself. The cleaning list has a signature for each day, when the staff remember to sign it. The inspector trusts the thermometer more than the list, because nobody can forget to write it. And some questions have no record at all: “Is the food good?” is not in any log. For that, someone must taste it.
The problem: questions that take a week
Section titled “The problem: questions that take a week”Larkfield has an AI governance board: Grace, a lawyer, the head of security and the finance manager. Once a month, they ask Kwame five kinds of questions:
- Inventory coverage: is every AI system that we run registered, with an owner?
- Risk tiers: which systems are high risk, and are their controls in place and tested?
- Exceptions: who is allowed to skip a rule, why, and until when?
- Incidents: what went wrong this month, and is anything still open?
- Spending: what did AI cost, per team, against the budgets?
Before the platform, Kwame answered by sending messages to every team and waiting a week. Amira said three systems, Mei said two, and nobody mentioned the digest script. The answers were promises, not records. The board could not check them, and next month the same work started again.
The idea: evidence that the controls write
Section titled “The idea: evidence that the controls write”Evidence is a record that exists because a control did its work, not because someone was asked. Larkfield’s platform keeps four kinds, and each one answers different questions. Two are written by code, on every call or command, so they are the thermometer. Two are written by people, so they are the cleaning list.
Text description of the diagram
At the top are the records that code writes on every call or command. The gateway records have one row per model call and per tool call. The audit trail, which estate events prints, has a line for every registration, review, suspension, credential and approval. Below are the records that people write, only if they do it. The registration forms hold the risk tier, the owner, the review date and whether the kill switch was tested. Incidents and exceptions are written with estate incident and estate exception. All four feed estate report, the month’s estate report, which goes to the governance board. A dashed line from the forms to the board shows a gap: the form still says kill switch tested: not yet, because the drill ran but nobody updated the form.
The audit trail is the list of events in the registry: who registered, reviewed, suspended or approved what, and when. An exception is a written permission for one agent to skip one rule, with a reason, a person who approved it and an end date. An incident is a written record of something that went wrong: what, when it started and ended, and which agents it touched.
The monthly estate report puts the four together in the order of the board’s questions: python -m estate report --month YYYY-MM. It reads the registry and the usage store, and it says where each number comes from.
Add the lesson’s files
Section titled “Add the lesson’s files”-
Copy the report, its commands and the replayed month’s evidence from the snapshot
end-of-m06. In~/projects/ai-estate:Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m06cp $S/estate/report.py estate/cp $S/estate/cli/evidence.py estate/cli/cp -R $S/evidence .Terminal window $S = "..\nextia-learning-labs\C32\snapshots\end-of-m06"Copy-Item $S\estate\report.py estate\Copy-Item $S\estate\cli\evidence.py estate\cli\Copy-Item -Recurse $S\evidence .Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m06cp $S/estate/report.py estate/cp $S/estate/cli/evidence.py estate/cli/cp -R $S/evidence .estate/report.pywrites the report.estate/cli/evidence.pyadds four commands:report,incident,exceptionanddiscover(the next lesson’s). The folderevidence/holds the replayed month: the Azure bill, the Fernway expense line, a summary of the egress log and the September report. -
Run the tests:
Terminal window python -m pytest -qThe last line says
190 passed, 1 deselected. The tests of the report come with the next lesson’s files. If two tests of the workflow fail, read the failure case below. -
Check the new command:
Terminal window python -m estate report --helpusage: python -m estate report [-h] --month MONTH [--out OUT]options:-h, --help show this help message and exit--month MONTH--out OUT
Common mistake
Two workflow tests fail after the last lesson
Symptom: python -m pytest -q ends with:
FAILED tests/workflow/test_checkpoint.py::test_a_crashed_cli_run_resumes - re...FAILED tests/workflow/test_first_run.py::test_first_run_replays_a_recorded_decision2 failed, 188 passed, 1 deselected in 37.65sAbove it, the error says No credentials at …/support-workflow.json: run python -m estate credentials issue support-workflow, with a path in a temporary folder.
Cause: RESOLVER_PROVIDER and RESOLVER_TOOLS are still set to gateway in your terminal. The workflow’s own tests then try to reach the platform, in a test folder that has no credentials. On Windows, $env: variables stay until you close the terminal. On macOS and Linux, this happens if you used export.
Diagnose: echo $RESOLVER_TOOLS on macOS and Linux, or $env:RESOLVER_TOOLS in PowerShell. If it prints gateway, this is the cause.
Recover: remove the two variables, then run the tests again. On macOS and Linux: unset RESOLVER_PROVIDER RESOLVER_TOOLS. In PowerShell: Remove-Item Env:RESOLVER_PROVIDER, Env:RESOLVER_TOOLS. The tests then say 190 passed. Set the variables only for the command that needs them, as the lessons do.
Worked example: Larkfield’s September report
Section titled “Worked example: Larkfield’s September report”The board meets after the replayed month. Kwame brings one file, evidence/estate-report-2026-09.md, which estate report wrote at the end of September. Open it in VS Code, and press Ctrl+Shift+V (Cmd+Shift+V on macOS) to see it as a page. The diagram puts its seven sections beside the board’s questions.
Text description of the diagram
The report has seven sections, each beside the board’s question that it answers. Section 1, the inventory: 7 registrations, all active, but 17 findings outside the registry, and 40,754 model calls through the gateway. It answers “is every AI system registered?”, and the answer is no: az-key-3 and fw-7Q2K are on the bill. Section 2, the risk tiers: 3 high, 3 medium and 1 low, and the support workflow fails closed, but its kill switch is “not yet” tested. That is a gap: the source is the forms. Section 3, the spending: the workflow US$14.28, 57% of its 25, triage US$6.48, 43% of its 15, and the service team US$20.76 of its 40. The source is the gateway’s records, which do not include the US$7.36 paid straight. Section 4, what the controls stopped: 28,807 rate-limit refusals, 6,375 quota refusals, 40 calls with no provider and 15 provider errors, and 123 answers from a fallback. It answers “did the limits and fallbacks work?” from records written by code. Section 5: no exceptions, and three incidents, the shadow AI still open, the outage and the runaway of 837 calls for US$4.09. Section 6: the release judge’s review is due on 1 October. Section 7 says what the evidence cannot answer: whether the answers were right, what a whole task cost, and AI inside bought tools. It hands those over to the evaluations, the traces, the buying process and a later course on AI governance.
Now answer the board’s five questions from the file, one at a time. Each answer names its line.
Is every AI system registered? Section 1 says 7 registrations: 7 active, then Found outside the registry (17 findings, python -m estate discover). So the answer is no. The list names az-key-3 chat-strong: 202 requests, US$0.8330: a key that nobody told the platform about and the Fernway key fw-7Q2K. The report shows the first 12 findings; the next lesson prints all 17 and follows them.
Which systems are high risk, and are their controls tested? Section 2: 3 high, 3 medium, 1 low. The three high ones are the workflow and the two tool servers, and all fail closed. But each line ends with kill switch tested: not yet. You ran a kill-switch drill in Kill switches and containment. Nobody wrote it into the registration forms, so the evidence says it never happened. This is a cleaning list without a signature.
Who may skip a rule? Section 5: No exceptions. That is a true answer, and a surprising one. The document assistant embeds its documents on a local model, so those calls never pass the gateway. Nobody wrote that down as an exception, so the report cannot tell a known gap from an unknown one.
What went wrong? Section 5 lists three incidents. The shadow-AI incident of 15 September is still open. The outage of 17 September lasted from 10:00 to 11:30. The runaway of 22 September answered 837 calls for US$4.09 before the quota stopped it. Section 4 shows the same events in the gateway’s own rows. route 40 are the calls that had no provider during the outage: 35 triage calls, 2 workflow calls and 3 document questions. The runaway’s 28,800 rate-limit refusals and 6,363 quota refusals are in rate_limit 28,807 and quota 6,375. The other 19 refusals were normal workflow calls on the same day, so those tasks went to people.
What did it cost? Section 3: the service team spent 20.76 of 40.00, and no agent passed its budget. The capability table shows strong-reasoning 2,690 calls, 14.70, the number Omar wanted.
Four of the five answers came from records that code wrote. The two weak answers, the kill switch and the exceptions, came from records that people forgot to write.
Guided practice: write your own month’s evidence
Section titled “Guided practice: write your own month’s evidence”Your estate has the last lesson’s traffic. Now you cause a small practice outage, record it as an incident, approve an exception, and write this month’s report.
-
Make the simulated provider unavailable in both regions, and send two document questions:
Terminal window python -m provsim control '*' downpython -m apps.replay doc-assistant --units 2 --via gateway --seed 11Controls now: [{"provider": "*", "deployment": null, "mode": "down", "seconds": 5.0, "rate": 0.5, "requests_left": null}] (simulated)doc-assistant: 2 requests through the gateway | status {503: 2}refused or failed: no_provider_available x2tokens: 0 in, 0 out | cost: US$0.0000On Windows, write
"*"with double quotes.--seed 11picks two questions that the last lesson did not send. -
Open an incident while it lasts. A person writes this record; nothing writes it for you:
Terminal window python -m estate incident add --kind outage --agent doc-assistant --summary "practice outage: both Azure regions down, 2 document questions had no answer"Incident 1 opened. -
End the outage, check that the assistant answers again, and close the incident:
Terminal window python -m provsim control '*' normalpython -m apps.replay doc-assistant --units 2 --via gateway --seed 12python -m estate incident end 1python -m estate incident listControls now: [] (simulated)doc-assistant: 2 requests through the gateway | status {200: 2}answered by: chat-small (gpt-6-luna-2026-09-22) x2tokens: 2,219 in, 271 out | cost: US$0.0004Incident 1 ended.1 outage … -> …: practice outage: both Azure regions down, 2 document questions had no answerThe start and the end (left out here:
…) are in UTC, on the estate’s clock. In the course’s run, the steps took less than a second, so the start and the end were the same second. When you record an incident later, give its real times with--at, for example--at 2026-11-02T10:00:00+00:00. -
Approve the exception that the September report missed. First try it without an end date:
Terminal window python -m estate exception add doc-assistant --rule gateway-onlyAn exception needs --reason and --until (a date): exceptions end.This refusal is on purpose. An exception without an end date becomes a permanent rule that nobody decided. Give the reason and the date:
Terminal window python -m estate exception add doc-assistant --rule gateway-only --until 2027-03-31 --reason "its embeddings run on a local model on the CPU: no provider call to route"Exception 1: doc-assistant may skip gateway-only until 2027-03-31. -
Write this month’s report into a file. The month must be the month of your run, so let the shell give it:
Terminal window python -m estate report --month $(date -u +%Y-%m) --out state/report.mdTerminal window $m = (Get-Date).ToUniversalTime().ToString("yyyy-MM")python -m estate report --month $m --out state\report.mdTerminal window python -m estate report --month $(date -u +%Y-%m) --out state/report.mdSaved the report to state/report.md-uasks for the month in UTC, the estate’s clock.
Open state/report.md in VS Code. These are its sections 3 to 6 from the course’s run (the incident’s times left out: …):
| Agent | Team | Calls answered | Refused | Cost | Budget | Used ||---|---|---|---|---|---|---|| doc-assistant | data | 7 | 0 | 0.00 | 3.00 | 0% || helpdesk-assistant | service | 0 | 0 | 0.00 | 2.00 | 0% || orders-mcp | data | 0 | 0 | 0.00 | - | - || release-judge | evaluation | 5 | 0 | 0.02 | 8.00 | 0% || support-mcp | platform | 0 | 0 | 0.00 | - | - || support-workflow | service | 5 | 0 | 0.03 | 25.00 | 0% || ticket-triage | service | 40 | 0 | 0.01 | 15.00 | 0% |
## 4. What the controls stopped
| Check | Model calls stopped or failed ||---|---|| route | 2 |
Answers from a fallback: 0. Calls served in fail-open mode: 0. Calls changed by a guardrail: 0.
| Agent | Tool calls | Outcome | Calls | With an approval ||---|---|---|---|---|| support-workflow | read | ok | 2 | 0 || support-workflow | write | ok | 5 | 3 |
## 5. Exceptions and incidents
- Exception 1: doc-assistant may skip gateway-only until 2027-03-31 (its embeddings run on a local model on the CPU: no provider call to route; approved by usr-kwame)- Incident 1 (outage), … to …: practice outage: both Azure regions down, 2 document questions had no answer
## 6. Reviews and owners
- release-judge: review overdue since 2026-10-01 (22 days)Check that the records agree with each other. Your incident says that 2 document questions had no answer. Section 4, written by the gateway, says route 2: two calls had no provider. The two records were written by different hands, and they agree. Section 3 counts them as failed, not refused: the gateway did not refuse them, the provider was not there. So the document assistant shows 7 answered: 5 from the last lesson and 2 after the outage.
Section 4 also shows the workflow’s tool calls: 5 writes, 3 with an approval. The other 2 are add_ticket_note writes. The approval policy says that a note needs no approval, because it changes nothing for the customer. The evidence lets the board check that this is a rule, not a hole.
Section 6 is new since September: the release judge’s review is overdue, because the estate’s clock is now in October. Your number of days differs.
Finally, look at the audit trail of the document assistant:
python -m estate events doc-assistantThe events, with their times left out (…):
… doc-assistant register usr-kwame {"status": "active", "version": 1}… doc-assistant credentials_issued usr-kwame {}… doc-assistant exception usr-kwame {"id": 1, "rule": "gateway-only", "until": "2027-03-31"}The exception is in the audit trail with the person who approved it, usr-kwame, the actor of your terminal. --by names another person, for example --by usr-grace. The incident is not in this list: it is not about one registration. estate incident list shows it.
What the evidence cannot answer
Section titled “What the evidence cannot answer”Section 7 of every report has three lines, and they are the most honest part of it:
- Whether the answers were right: that is each consumer's evaluation (the remap gate) and its reviews.- What a whole task cost or why it failed: join these records with the app's traces by run ID.- AI inside bought tools, and AI used outside Larkfield's network: the buying process and asking people.The gateway sees that the workflow made 2,589 calls in September. It does not see whether the refunds were right. That is the evaluation of Change a mapping safely, and the people who approve. The cost of a whole task is the join of the last lesson. And a help-desk product that Larkfield buys with an AI feature inside sends nothing through the gateway at all. The next lesson looks for what the records miss.
The board also asks questions that no technical record can answer. Is this use of AI acceptable for our customers? Who is accountable when the workflow is wrong? Which risk tier should a new kind of agent get? Which laws apply to the digest’s data? These need policies, risk decisions and people with that duty. A later course on AI governance teaches that side. This course gives it the evidence; that course decides what the evidence must show.
When to use the report, and what it costs
Section titled “When to use the report, and what it costs”Use it every month, before the board meets, and after every incident. It is one command, and every number in it has a source. Keep each month’s file: next year, an auditor asks for the history, not today’s state.
Do not use it as a statement that the estate is safe. A report with no incidents may mean that nothing happened, or that nobody wrote anything down. A report is complete only for what passed the platform.
What it costs is mostly people’s time. The records that code writes cost nothing extra. The records that people write cost a habit: an incident opened while it happens, an exception with an end date, a form updated after a drill. Without that habit, the report is precise about cost and silent about risk, as Larkfield’s September report was.
Your turn: three questions for the board
Section titled “Your turn: three questions for the board”This is the module’s completion check. Use only evidence/estate-report-2026-09.md.
- The board asks: “Did any team go over its AI budget in September?” Answer with the numbers from the report.
- The board asks: “Did customers feel the outage or the runaway?” Answer from sections 4 and 5.
- The board asks: “Is the support workflow’s kill switch tested?” Answer, and say what Kwame must do so that next month’s report answers it.
- Name one question that the evidence cannot answer, and say where its answer must come from.
Check your answers
- No. The teams: service US$20.76 of US$40, data US$0.13 of US$10, evaluation US$0.42 of US$10, platform US$0.00 of US$5. The highest agent is the workflow, at 57% of its US$25. But the answer covers only the gateway’s records: US$7.36 went straight to the provider before the apps moved, and only the provider’s bill shows it.
- Yes, a little. The outage left 40 calls without a provider (
route 40): 35 triage calls, 2 workflow calls and 3 document questions. Those tickets and tasks went to people, and the document assistant said “try later”. During the runaway, 19 normal workflow calls were refused for the rest of the day. They are inrate_limit 28,807andquota 6,375, beside the loop’s own 28,800 and 6,363. 123 calls answered from the second region, so customers did not feel them. - The evidence says no (
kill switch tested: not yet), although a drill ran. Kwame must record it where the report reads it: setkill_switch_testedto the drill’s date inregistrations/support-workflow.yamland register the new version, which a review then approves. - For example: “Were the workflow’s 2,589 answers right?” The gateway records calls, not correctness. The answer comes from the workflow’s evaluation, its reviews and the people who approved. Other good answers: “Does the help-desk product we bought use AI?” (the buying process); “Is it acceptable to send customer messages to a provider?” (the governance policy and the board’s own decision).
Knowledge check
Section titled “Knowledge check”Check your understanding
The report says 'No incidents.' Tomás says: 'Good, so nothing went wrong this month.' What can the report really say, and how would you check it?
Show the answer
It can only say that nobody opened an incident. Incidents are written by people. Check the records that code writes: section 4 of the report (refusals by check, calls with no provider, fallbacks, fail-open calls) and estate usage --by outcome. If the gateway shows many route or quota refusals on one day and no incident covers that day, something went wrong and nobody recorded it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does estate exception add refuse an exception without --until, and what would go wrong without that rule?
Show the answer
An exception is a decision to accept a risk for a limited time, for a stated reason. Without an end date, nobody looks at it again, and it becomes a permanent rule that nobody decided. The date forces a new decision: end the exception, fix the cause, or approve it again with a new reason.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical: record the support workflow's kill-switch drill as an incident of kind drill, for the time you ran it in Module 5. Write the report again and find your incident in it.
Show the answer
Run, for example, python -m estate incident add --kind drill --agent support-workflow --at 2026-10-22T14:00:00+00:00 --summary "kill-switch drill: refund writes stopped, drained, credentials revoked", then python -m estate incident end 2 --at 2026-10-22T14:20:00+00:00. The --at times must be in the month that you report. Write the report again with the same --month and --out. Section 5 now lists Incident 2 (drill), 2026-10-22T14:00 to 2026-10-22T14:20: kill-switch drill: refund writes stopped, drained, credentials revoked, before the outage, because incidents are listed by their start time. Section 2 still says kill switch tested: not yet, because it reads the registration form, not the incidents.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Evidence is a record that exists because a control worked. The gateway’s records and the audit trail are written by code; the forms, the incidents and the exceptions are written by people.
python -m estate report --month YYYY-MManswers the board’s five questions in order: inventory, risk tiers, spending, what the controls stopped, exceptions and incidents, reviews.- In September, four answers came from code-written records. The two weak ones, the kill switch and the exceptions, came from records that people forgot.
estate incident add | end | listandestate exception add | listwrite the people’s part. An exception must have an end date.- Section 7 says what the evidence cannot answer: correctness, the cost of a whole task, and AI that never passes the gateway.
Your project now has the report, the incident and exception commands and the replayed month’s evidence. You can now answer the board from records. But the report’s first section already said it: 17 findings outside the registry. A report covers only what passed the platform. Next, we look for AI that nobody registered, in the bill, the egress log and the code.
Next: Find unregistered AI