Skip to content
Français

Evidence for governance

  • Lesson 18
  • 40 min
  • Module 6: Usage, evidence and shadow AI

Where we are

In the last lesson, we turned the gateway's records into usage and cost reports per agent, team and capability, and joined them with the app's trace for the cost of a task. But the governance board asks other questions: is every agent registered, which ones are high risk, what went wrong this month? In this lesson, we answer the board from evidence, the way a restaurant answers a health inspector, and we say which question the evidence cannot answer.

What you will be able to do

Answer a governance board’s questions about an organisation’s AI from the platform’s records: the registry, the gateway’s records and the audit trail. You read Larkfield’s September report and write your own. Then you name what the evidence cannot answer.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • Your project ~/projects/ai-estate after the last lesson: the snapshot end-of-m05, with the last lesson’s traffic in state/, and its virtual environment active. If you start from a fresh copy of end-of-m05, do the “Start a fresh estate” and “Group the rows” steps of the last lesson first. Your numbers then match this lesson’s.
  • The labs repository in ~/projects/nextia-learning-labs, for this lesson’s files from the snapshot end-of-m06.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written, not run. The September report is replayed traffic; its outage, runaway and digest are simulated or constructed practice incidents, as the earlier modules said.

A health inspector visits a restaurant and asks the same questions every time. Is the fridge cold enough? Is the kitchen cleaned every day? Who is in charge? Has anyone been ill after a meal?

The restaurant does not answer with promises. It shows records. The fridge has a thermometer that writes the temperature every hour, by itself. The cleaning list has a signature for each day, when the staff remember to sign it. The inspector trusts the thermometer more than the list, because nobody can forget to write it. And some questions have no record at all: “Is the food good?” is not in any log. For that, someone must taste it.

Larkfield has an AI governance board: Grace, a lawyer, the head of security and the finance manager. Once a month, they ask Kwame five kinds of questions:

  • Inventory coverage: is every AI system that we run registered, with an owner?
  • Risk tiers: which systems are high risk, and are their controls in place and tested?
  • Exceptions: who is allowed to skip a rule, why, and until when?
  • Incidents: what went wrong this month, and is anything still open?
  • Spending: what did AI cost, per team, against the budgets?

Before the platform, Kwame answered by sending messages to every team and waiting a week. Amira said three systems, Mei said two, and nobody mentioned the digest script. The answers were promises, not records. The board could not check them, and next month the same work started again.

The idea: evidence that the controls write

Section titled “The idea: evidence that the controls write”

Evidence is a record that exists because a control did its work, not because someone was asked. Larkfield’s platform keeps four kinds, and each one answers different questions. Two are written by code, on every call or command, so they are the thermometer. Two are written by people, so they are the cleaning list.

A flow from top to bottom. Written by code, on every call or command: the gateway records, one row per model call and tool call; the audit trail, estate events: register, review, suspend, credentials, approvals. Written by people, only if they do it: the registration forms, with the risk tier, owner, review date and kill switch tested; incidents and exceptions, with estate incident and estate exception. All four feed estate report, the month's estate report, which goes to the governance board. A dashed line from the registration forms to the board says: kill switch tested: not yet (the drill ran, nobody updated the form).
Who writes the evidence. Records that code writes on every call cannot be forgotten; records that people write are only as complete as the people.
Text description of the diagram

At the top are the records that code writes on every call or command. The gateway records have one row per model call and per tool call. The audit trail, which estate events prints, has a line for every registration, review, suspension, credential and approval. Below are the records that people write, only if they do it. The registration forms hold the risk tier, the owner, the review date and whether the kill switch was tested. Incidents and exceptions are written with estate incident and estate exception. All four feed estate report, the month’s estate report, which goes to the governance board. A dashed line from the forms to the board shows a gap: the form still says kill switch tested: not yet, because the drill ran but nobody updated the form.

The audit trail is the list of events in the registry: who registered, reviewed, suspended or approved what, and when. An exception is a written permission for one agent to skip one rule, with a reason, a person who approved it and an end date. An incident is a written record of something that went wrong: what, when it started and ended, and which agents it touched.

The monthly estate report puts the four together in the order of the board’s questions: python -m estate report --month YYYY-MM. It reads the registry and the usage store, and it says where each number comes from.

  1. Copy the report, its commands and the replayed month’s evidence from the snapshot end-of-m06. In ~/projects/ai-estate:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m06
    cp $S/estate/report.py estate/
    cp $S/estate/cli/evidence.py estate/cli/
    cp -R $S/evidence .

    estate/report.py writes the report. estate/cli/evidence.py adds four commands: report, incident, exception and discover (the next lesson’s). The folder evidence/ holds the replayed month: the Azure bill, the Fernway expense line, a summary of the egress log and the September report.

  2. Run the tests:

    Terminal window
    python -m pytest -q

    The last line says 190 passed, 1 deselected. The tests of the report come with the next lesson’s files. If two tests of the workflow fail, read the failure case below.

  3. Check the new command:

    Terminal window
    python -m estate report --help
    usage: python -m estate report [-h] --month MONTH [--out OUT]
    options:
    -h, --help show this help message and exit
    --month MONTH
    --out OUT

Common mistake

Two workflow tests fail after the last lesson

Symptom: python -m pytest -q ends with:

FAILED tests/workflow/test_checkpoint.py::test_a_crashed_cli_run_resumes - re...
FAILED tests/workflow/test_first_run.py::test_first_run_replays_a_recorded_decision
2 failed, 188 passed, 1 deselected in 37.65s

Above it, the error says No credentials at …/support-workflow.json: run python -m estate credentials issue support-workflow, with a path in a temporary folder.

Cause: RESOLVER_PROVIDER and RESOLVER_TOOLS are still set to gateway in your terminal. The workflow’s own tests then try to reach the platform, in a test folder that has no credentials. On Windows, $env: variables stay until you close the terminal. On macOS and Linux, this happens if you used export.

Diagnose: echo $RESOLVER_TOOLS on macOS and Linux, or $env:RESOLVER_TOOLS in PowerShell. If it prints gateway, this is the cause.

Recover: remove the two variables, then run the tests again. On macOS and Linux: unset RESOLVER_PROVIDER RESOLVER_TOOLS. In PowerShell: Remove-Item Env:RESOLVER_PROVIDER, Env:RESOLVER_TOOLS. The tests then say 190 passed. Set the variables only for the command that needs them, as the lessons do.

Worked example: Larkfield’s September report

Section titled “Worked example: Larkfield’s September report”

The board meets after the replayed month. Kwame brings one file, evidence/estate-report-2026-09.md, which estate report wrote at the end of September. Open it in VS Code, and press Ctrl+Shift+V (Cmd+Shift+V on macOS) to see it as a page. The diagram puts its seven sections beside the board’s questions.

The monthly estate report for September, replayed, as seven sections on the left, each linked to the board's question and the source on the right. 1. Inventory: 7 registrations, 7 active; found outside the registry: 17 findings; model calls through the gateway: 40,754. Question: is every AI system registered? No: az-key-3 and fw-7Q2K are on the bill. Source: the registry and estate discover. 2. Risk tiers: 3 high, 3 medium, 1 low; support-workflow fail_closed, kill switch tested: not yet. Question, marked as a gap: which systems are high risk, and is each one's kill switch tested? Source: the registration forms. 3. Spending: support-workflow US$14.28, 57% of 25; ticket-triage US$6.48, 43% of 15; service team US$20.76 of 40. Question: what did AI cost, per agent and team? Source: the gateway's records; not in it: US$7.36 paid straight. 4. What the controls stopped: rate_limit 28,807, quota 6,375, route 40, provider 15; answers from a fallback: 123. Question: did the limits and fallbacks work? Source: the gateway's records, one row per call, written by code. 5. Exceptions and incidents: no exceptions; incidents: shadow-ai (open), outage, runaway (837 calls, US$4.09). Question: what went wrong, and what is open? Source: incidents that people wrote down; none, if nobody did. 6. Reviews and owners: release-judge review due on 2026-10-01. Question: who owns each system, and is a review overdue? Source: the registry. 7. What this evidence cannot answer: were the answers right, what did a whole task cost, AI inside bought tools. Marked as a gap: hand over to the evaluations, the app's traces, the buying process, and to a later course on AI governance.
Larkfield's estate report for September, annotated. Each section answers one of the board's questions from one source; the red sections are where the evidence has a gap.
Text description of the diagram

The report has seven sections, each beside the board’s question that it answers. Section 1, the inventory: 7 registrations, all active, but 17 findings outside the registry, and 40,754 model calls through the gateway. It answers “is every AI system registered?”, and the answer is no: az-key-3 and fw-7Q2K are on the bill. Section 2, the risk tiers: 3 high, 3 medium and 1 low, and the support workflow fails closed, but its kill switch is “not yet” tested. That is a gap: the source is the forms. Section 3, the spending: the workflow US$14.28, 57% of its 25, triage US$6.48, 43% of its 15, and the service team US$20.76 of its 40. The source is the gateway’s records, which do not include the US$7.36 paid straight. Section 4, what the controls stopped: 28,807 rate-limit refusals, 6,375 quota refusals, 40 calls with no provider and 15 provider errors, and 123 answers from a fallback. It answers “did the limits and fallbacks work?” from records written by code. Section 5: no exceptions, and three incidents, the shadow AI still open, the outage and the runaway of 837 calls for US$4.09. Section 6: the release judge’s review is due on 1 October. Section 7 says what the evidence cannot answer: whether the answers were right, what a whole task cost, and AI inside bought tools. It hands those over to the evaluations, the traces, the buying process and a later course on AI governance.

Now answer the board’s five questions from the file, one at a time. Each answer names its line.

Is every AI system registered? Section 1 says 7 registrations: 7 active, then Found outside the registry (17 findings, python -m estate discover). So the answer is no. The list names az-key-3 chat-strong: 202 requests, US$0.8330: a key that nobody told the platform about and the Fernway key fw-7Q2K. The report shows the first 12 findings; the next lesson prints all 17 and follows them.

Which systems are high risk, and are their controls tested? Section 2: 3 high, 3 medium, 1 low. The three high ones are the workflow and the two tool servers, and all fail closed. But each line ends with kill switch tested: not yet. You ran a kill-switch drill in Kill switches and containment. Nobody wrote it into the registration forms, so the evidence says it never happened. This is a cleaning list without a signature.

Who may skip a rule? Section 5: No exceptions. That is a true answer, and a surprising one. The document assistant embeds its documents on a local model, so those calls never pass the gateway. Nobody wrote that down as an exception, so the report cannot tell a known gap from an unknown one.

What went wrong? Section 5 lists three incidents. The shadow-AI incident of 15 September is still open. The outage of 17 September lasted from 10:00 to 11:30. The runaway of 22 September answered 837 calls for US$4.09 before the quota stopped it. Section 4 shows the same events in the gateway’s own rows. route 40 are the calls that had no provider during the outage: 35 triage calls, 2 workflow calls and 3 document questions. The runaway’s 28,800 rate-limit refusals and 6,363 quota refusals are in rate_limit 28,807 and quota 6,375. The other 19 refusals were normal workflow calls on the same day, so those tasks went to people.

What did it cost? Section 3: the service team spent 20.76 of 40.00, and no agent passed its budget. The capability table shows strong-reasoning 2,690 calls, 14.70, the number Omar wanted.

Four of the five answers came from records that code wrote. The two weak answers, the kill switch and the exceptions, came from records that people forgot to write.

Guided practice: write your own month’s evidence

Section titled “Guided practice: write your own month’s evidence”

Your estate has the last lesson’s traffic. Now you cause a small practice outage, record it as an incident, approve an exception, and write this month’s report.

  1. Make the simulated provider unavailable in both regions, and send two document questions:

    Terminal window
    python -m provsim control '*' down
    python -m apps.replay doc-assistant --units 2 --via gateway --seed 11
    Controls now: [{"provider": "*", "deployment": null, "mode": "down", "seconds": 5.0, "rate": 0.5, "requests_left": null}] (simulated)
    doc-assistant: 2 requests through the gateway | status {503: 2}
    refused or failed: no_provider_available x2
    tokens: 0 in, 0 out | cost: US$0.0000

    On Windows, write "*" with double quotes. --seed 11 picks two questions that the last lesson did not send.

  2. Open an incident while it lasts. A person writes this record; nothing writes it for you:

    Terminal window
    python -m estate incident add --kind outage --agent doc-assistant --summary "practice outage: both Azure regions down, 2 document questions had no answer"
    Incident 1 opened.
  3. End the outage, check that the assistant answers again, and close the incident:

    Terminal window
    python -m provsim control '*' normal
    python -m apps.replay doc-assistant --units 2 --via gateway --seed 12
    python -m estate incident end 1
    python -m estate incident list
    Controls now: [] (simulated)
    doc-assistant: 2 requests through the gateway | status {200: 2}
    answered by: chat-small (gpt-6-luna-2026-09-22) x2
    tokens: 2,219 in, 271 out | cost: US$0.0004
    Incident 1 ended.
    1 outage … -> …: practice outage: both Azure regions down, 2 document questions had no answer

    The start and the end (left out here: …) are in UTC, on the estate’s clock. In the course’s run, the steps took less than a second, so the start and the end were the same second. When you record an incident later, give its real times with --at, for example --at 2026-11-02T10:00:00+00:00.

  4. Approve the exception that the September report missed. First try it without an end date:

    Terminal window
    python -m estate exception add doc-assistant --rule gateway-only
    An exception needs --reason and --until (a date): exceptions end.

    This refusal is on purpose. An exception without an end date becomes a permanent rule that nobody decided. Give the reason and the date:

    Terminal window
    python -m estate exception add doc-assistant --rule gateway-only --until 2027-03-31 --reason "its embeddings run on a local model on the CPU: no provider call to route"
    Exception 1: doc-assistant may skip gateway-only until 2027-03-31.
  5. Write this month’s report into a file. The month must be the month of your run, so let the shell give it:

    Terminal window
    python -m estate report --month $(date -u +%Y-%m) --out state/report.md
    Saved the report to state/report.md

    -u asks for the month in UTC, the estate’s clock.

Open state/report.md in VS Code. These are its sections 3 to 6 from the course’s run (the incident’s times left out: …):

| Agent | Team | Calls answered | Refused | Cost | Budget | Used |
|---|---|---|---|---|---|---|
| doc-assistant | data | 7 | 0 | 0.00 | 3.00 | 0% |
| helpdesk-assistant | service | 0 | 0 | 0.00 | 2.00 | 0% |
| orders-mcp | data | 0 | 0 | 0.00 | - | - |
| release-judge | evaluation | 5 | 0 | 0.02 | 8.00 | 0% |
| support-mcp | platform | 0 | 0 | 0.00 | - | - |
| support-workflow | service | 5 | 0 | 0.03 | 25.00 | 0% |
| ticket-triage | service | 40 | 0 | 0.01 | 15.00 | 0% |
## 4. What the controls stopped
| Check | Model calls stopped or failed |
|---|---|
| route | 2 |
Answers from a fallback: 0. Calls served in fail-open mode: 0. Calls changed by a guardrail: 0.
| Agent | Tool calls | Outcome | Calls | With an approval |
|---|---|---|---|---|
| support-workflow | read | ok | 2 | 0 |
| support-workflow | write | ok | 5 | 3 |
## 5. Exceptions and incidents
- Exception 1: doc-assistant may skip gateway-only until 2027-03-31 (its embeddings run on a local model on the CPU: no provider call to route; approved by usr-kwame)
- Incident 1 (outage), … to …: practice outage: both Azure regions down, 2 document questions had no answer
## 6. Reviews and owners
- release-judge: review overdue since 2026-10-01 (22 days)

Check that the records agree with each other. Your incident says that 2 document questions had no answer. Section 4, written by the gateway, says route 2: two calls had no provider. The two records were written by different hands, and they agree. Section 3 counts them as failed, not refused: the gateway did not refuse them, the provider was not there. So the document assistant shows 7 answered: 5 from the last lesson and 2 after the outage.

Section 4 also shows the workflow’s tool calls: 5 writes, 3 with an approval. The other 2 are add_ticket_note writes. The approval policy says that a note needs no approval, because it changes nothing for the customer. The evidence lets the board check that this is a rule, not a hole.

Section 6 is new since September: the release judge’s review is overdue, because the estate’s clock is now in October. Your number of days differs.

Finally, look at the audit trail of the document assistant:

Terminal window
python -m estate events doc-assistant

The events, with their times left out (…):

… doc-assistant register usr-kwame {"status": "active", "version": 1}
… doc-assistant credentials_issued usr-kwame {}
… doc-assistant exception usr-kwame {"id": 1, "rule": "gateway-only", "until": "2027-03-31"}

The exception is in the audit trail with the person who approved it, usr-kwame, the actor of your terminal. --by names another person, for example --by usr-grace. The incident is not in this list: it is not about one registration. estate incident list shows it.

Section 7 of every report has three lines, and they are the most honest part of it:

- Whether the answers were right: that is each consumer's evaluation (the remap gate) and its reviews.
- What a whole task cost or why it failed: join these records with the app's traces by run ID.
- AI inside bought tools, and AI used outside Larkfield's network: the buying process and asking people.

The gateway sees that the workflow made 2,589 calls in September. It does not see whether the refunds were right. That is the evaluation of Change a mapping safely, and the people who approve. The cost of a whole task is the join of the last lesson. And a help-desk product that Larkfield buys with an AI feature inside sends nothing through the gateway at all. The next lesson looks for what the records miss.

The board also asks questions that no technical record can answer. Is this use of AI acceptable for our customers? Who is accountable when the workflow is wrong? Which risk tier should a new kind of agent get? Which laws apply to the digest’s data? These need policies, risk decisions and people with that duty. A later course on AI governance teaches that side. This course gives it the evidence; that course decides what the evidence must show.

Use it every month, before the board meets, and after every incident. It is one command, and every number in it has a source. Keep each month’s file: next year, an auditor asks for the history, not today’s state.

Do not use it as a statement that the estate is safe. A report with no incidents may mean that nothing happened, or that nobody wrote anything down. A report is complete only for what passed the platform.

What it costs is mostly people’s time. The records that code writes cost nothing extra. The records that people write cost a habit: an incident opened while it happens, an exception with an end date, a form updated after a drill. Without that habit, the report is precise about cost and silent about risk, as Larkfield’s September report was.

This is the module’s completion check. Use only evidence/estate-report-2026-09.md.

  1. The board asks: “Did any team go over its AI budget in September?” Answer with the numbers from the report.
  2. The board asks: “Did customers feel the outage or the runaway?” Answer from sections 4 and 5.
  3. The board asks: “Is the support workflow’s kill switch tested?” Answer, and say what Kwame must do so that next month’s report answers it.
  4. Name one question that the evidence cannot answer, and say where its answer must come from.
Check your answers
  1. No. The teams: service US$20.76 of US$40, data US$0.13 of US$10, evaluation US$0.42 of US$10, platform US$0.00 of US$5. The highest agent is the workflow, at 57% of its US$25. But the answer covers only the gateway’s records: US$7.36 went straight to the provider before the apps moved, and only the provider’s bill shows it.
  2. Yes, a little. The outage left 40 calls without a provider (route 40): 35 triage calls, 2 workflow calls and 3 document questions. Those tickets and tasks went to people, and the document assistant said “try later”. During the runaway, 19 normal workflow calls were refused for the rest of the day. They are in rate_limit 28,807 and quota 6,375, beside the loop’s own 28,800 and 6,363. 123 calls answered from the second region, so customers did not feel them.
  3. The evidence says no (kill switch tested: not yet), although a drill ran. Kwame must record it where the report reads it: set kill_switch_tested to the drill’s date in registrations/support-workflow.yaml and register the new version, which a review then approves.
  4. For example: “Were the workflow’s 2,589 answers right?” The gateway records calls, not correctness. The answer comes from the workflow’s evaluation, its reviews and the people who approved. Other good answers: “Does the help-desk product we bought use AI?” (the buying process); “Is it acceptable to send customer messages to a provider?” (the governance policy and the board’s own decision).

Check your understanding

The report says 'No incidents.' Tomás says: 'Good, so nothing went wrong this month.' What can the report really say, and how would you check it?

Show the answer

It can only say that nobody opened an incident. Incidents are written by people. Check the records that code writes: section 4 of the report (refusals by check, calls with no provider, fallbacks, fail-open calls) and estate usage --by outcome. If the gateway shows many route or quota refusals on one day and no incident covers that day, something went wrong and nobody recorded it.

Check your understanding

Why does estate exception add refuse an exception without --until, and what would go wrong without that rule?

Show the answer

An exception is a decision to accept a risk for a limited time, for a stated reason. Without an end date, nobody looks at it again, and it becomes a permanent rule that nobody decided. The date forces a new decision: end the exception, fix the cause, or approve it again with a new reason.

Check your understanding

Practical: record the support workflow's kill-switch drill as an incident of kind drill, for the time you ran it in Module 5. Write the report again and find your incident in it.

Show the answer

Run, for example, python -m estate incident add --kind drill --agent support-workflow --at 2026-10-22T14:00:00+00:00 --summary "kill-switch drill: refund writes stopped, drained, credentials revoked", then python -m estate incident end 2 --at 2026-10-22T14:20:00+00:00. The --at times must be in the month that you report. Write the report again with the same --month and --out. Section 5 now lists Incident 2 (drill), 2026-10-22T14:00 to 2026-10-22T14:20: kill-switch drill: refund writes stopped, drained, credentials revoked, before the outage, because incidents are listed by their start time. Section 2 still says kill switch tested: not yet, because it reads the registration form, not the incidents.

Recap

  • Evidence is a record that exists because a control worked. The gateway’s records and the audit trail are written by code; the forms, the incidents and the exceptions are written by people.
  • python -m estate report --month YYYY-MM answers the board’s five questions in order: inventory, risk tiers, spending, what the controls stopped, exceptions and incidents, reviews.
  • In September, four answers came from code-written records. The two weak ones, the kill switch and the exceptions, came from records that people forgot.
  • estate incident add | end | list and estate exception add | list write the people’s part. An exception must have an end date.
  • Section 7 says what the evidence cannot answer: correctness, the cost of a whole task, and AI that never passes the gateway.

Your project now has the report, the incident and exception commands and the replayed month’s evidence. You can now answer the board from records. But the report’s first section already said it: 17 findings outside the registry. A report covers only what passed the platform. Next, we look for AI that nobody registered, in the bill, the egress log and the code.

Next: Find unregistered AI