Meet Larkfield's AI estate
- Lesson 3
- 40 min
- Module 1: Why agents need platform controls
Where we are
In the last lesson, we found the control points (identity, registry, model gateway, tool gateway and audit) and what each one can see. So far they are boxes on a picture, and the estate is a folder of files. In this lesson, we start Larkfield's estate on our own computer, send real recorded requests from each app, and read what the provider records, so that every later lesson changes something that we have already seen run. We start with a match replay.
Lesson 3 of 21 · Governing AI Agents: Gateways, Registries and ControlsMeet Larkfield's AI estate
1. Why agents need platform controls
2. Ask for capabilities, not models
3. Register every agent
4. The model gateway
5. Govern tools and actions
6. Usage, evidence and shadow AI
7. Build, buy and roll out
What you will be able to do
Start Larkfield’s AI estate on your computer and send each app’s real recorded requests to the provider. Then read the only records that exist before governance: the provider’s bill and the network’s egress log. You also learn which evidence in this course is replayed and which is simulated, and how to start again.
Before you startWhat to know and to have ready
What you need to know
- Larkfield’s apps, their owners and their keys, from Five agents and nobody knows.
- The control points and what each one sees, from Rules in documents, rules in code.
- How to use a terminal, a virtual environment and
pip, from Python for Practical AI Engineering. The agents course used the same steps.
What you need on your computer
- The project
~/projects/ai-estatefrom the first lesson of this module, and the labs repository in~/projects/nextia-learning-labs. - Python 3.12 or newer. Check with
python3.12 --versionon macOS and Linux, orpy -3.12 --versionon Windows. - Two terminals, for example two tabs in VS Code. Port 8320 must be free: the provider simulator uses it.
- An internet connection, once, for
pip. The packages and the virtual environment take about 85 MB.
You need no account, no key and no payment. Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them.
A match replay
Section titled “A match replay”Think of a football coach who studies last week’s match. She plays the recording, stops it, and plays one attack again. Every time she plays it, the same thing happens, in the same order. She can show it to the team and to another coach, and they all see the same match.
A replay has one more property that matters here. It never shows a goal that was not scored. If the coach asks, “What if our striker had shot with the left foot?”, the recording has no answer. It can only show what really happened.
The problem: practising governance on real providers
Section titled “The problem: practising governance on real providers”Kwame could build the platform against real Azure models. But every call costs money and needs a key. A real model may give a different answer to the same request tomorrow, so a lesson could not show you the output you will see. He cannot cause a provider outage on purpose to test a fallback. And a month of Larkfield’s traffic is tens of thousands of calls.
So the course does what the coach does. It plays back what really happened.
The idea: a provider simulator that replays recordings
Section titled “The idea: a provider simulator that replays recordings”The earlier courses called real models and saved every request and every answer. This course merges those recordings into a provider simulator: a small server on your computer that looks like a model provider to the apps. It speaks the same API as Azure’s OpenAI-compatible endpoint.
When an app sends a request, the simulator computes a fingerprint of the request body, a SHA-256 hash. If an earlier course recorded exactly that body, the simulator gives back the recorded answer, with its real tokens and its real time. If not, it answers with an error, 422 no_recording, and never invents an answer. A few fields that do not change the answer, such as stream, are left out of the fingerprint. Every other character counts: the course’s tests change max_completion_tokens of a triage request to 999, and the simulator answers 422.
Text description of the diagram
An app sends a request with its key. First, the simulator writes a line in its egress log: the host that sent it, the provider, the key ID and the status. Then it checks the key: an unknown key gets 401 invalid_key. Next, it checks the recorded feature rules. The real provider refused some requests, for example a tool call that one model cannot make. The simulator refuses them with the same error text. Then it computes the request’s fingerprint, a SHA-256 hash of the body, byte for byte, with five fields left out, such as stream. If no earlier course recorded that exact body, the answer is 422 no_recording: the simulator never invents an answer. If a course recorded it, the answer is 200 with the recorded answer, its tokens and its latency. Then the simulator adds a line to the bill: key ID, model, tokens and cost.
What is replayed and what is simulated
Section titled “What is replayed and what is simulated”Every piece of evidence in this course has a label. Use the same labels in your own notes.
| Label | What it is | Examples |
|---|---|---|
| replayed | a real answer of a real model, recorded by an earlier course, played back by the simulator | the answers of chat-small (gpt-6-luna-2026-09-22), chat-strong (gpt-6.1-sol-2026-09-29), embed-small and gemma3:4b |
| measured | a number from a run of the course’s code, such as the gateway’s added time | your own runs, and the course team’s runs on a laptop |
| simulated | behaviour that the course made up to practise a situation | the bill, the egress log, outages, slow answers, a second Azure region |
| constructed | a made-up case or data, written by the course author | the weekly digest script, Fernway and its one answer |
| assumption | a stated value, such as a price or a daily volume | 45 workflow runs a day; the prices of the recordings |
The simulator holds 5,253 recorded answers to 3,953 different requests. Some requests were recorded more than once, so a repeated request can get a different real answer, as a real model would give. The answers come from the agents course (2,363), the evaluation course (2,260), the capstone course (353), the LLM applications course (7) and this course (269, most of them the triage service’s embeddings). One answer is constructed: Fernway’s. Each app in the estate is a replayer: it sends exactly the request bodies that its course sent, in order.
Set up the project
Section titled “Set up the project”You downloaded the snapshot start in the first lesson. Now you make its Python environment, check it, and add this lesson’s files: the estate’s command line.
-
Move into the project:
Terminal window cd ~/projects/ai-estate -
Make and activate a virtual environment with Python 3.12:
Terminal window python3.12 -m venv .venvsource .venv/bin/activateTerminal window py -3.12 -m venv .venv.venv\Scripts\Activate.ps1Terminal window python3.12 -m venv .venvsource .venv/bin/activateThe prompt now starts with
(.venv). From here, typepythonon every system. Activate the environment again in every new terminal. -
Install the pinned packages:
Terminal window python -m pip install -r requirements-dev.txtThis takes about a minute. The last lines may suggest a newer
pip. You can ignore that. -
Run the tests of the snapshot:
Terminal window python -m pytest -qThe last line:
88 passed, 1 deselected in 4.12sYour time differs. 69 of the 88 tests are the agents course’s own tests of the support workflow. The deselected test calls a real provider, and only runs if you ask for it.
-
Copy this lesson’s files from the snapshot
end-of-m01: the folderestate, its test and the newREADME.md. In~/projects/ai-estate:Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m01cp -R $S/estate .cp $S/tests/test_estate_basics.py tests/cp $S/README.md .Terminal window $S = "..\nextia-learning-labs\C32\snapshots\end-of-m01"Copy-Item -Recurse $S\estate .Copy-Item $S\tests\test_estate_basics.py tests\Copy-Item -Force $S\README.md .Terminal window S=../nextia-learning-labs/C32/snapshots/end-of-m01cp -R $S/estate .cp $S/tests/test_estate_basics.py tests/cp $S/README.md . -
Run the tests again:
Terminal window python -m pytest -qThe last line now says
91 passed, 1 deselected. Your project is now the same as the snapshotend-of-m01.
Start the estate
Section titled “Start the estate”The command python -m estate starts and stops the estate’s services. At this stage there is only one service, the provider simulator. Later modules add the control plane, the gateways and the tool servers, and the same command starts them all.
-
Create the estate’s state folder:
Terminal window python -m estate initThe output (your path differs):
State folder: /Users/you/projects/ai-estate/stateThe estate has no platform yet: each app calls the provider with its own key.The folder
state/holds the practice state: today, the simulator’s bill and egress log; later, the registry, the keys and the usage records. Git ignores it. -
Start the services:
Terminal window python -m estate upThe output (your process number differs):
provsim up (process 44451) http://127.0.0.1:8320The simulator now runs in the background. You keep using this terminal.
-
Check that it answers:
Terminal window python -m estate statusprovsim running {"ok":true,"recordings":5253,"speed":0.0,"egress":"open"}speed 0.0means that the simulator answers at once, without the recorded wait.egress openmeans that the simulated network lets every machine reach every provider, as Larkfield’s network does today.
Worked example: five tickets through the triage service
Section titled “Worked example: five tickets through the triage service”The triage service reads each new ticket and sends two requests: one chat call that chooses the team, and one embedding for search. Send five tickets:
python -m apps.replay ticket-triage --units 5The output, replayed:
ticket-triage: 10 requests straight to the provider | status {200: 10}answered by: chat-small (gpt-6-luna-2026-09-22) x5, embed-small (text-embedding-3-small) x5tokens: 10,018 in, 2,025 out | cost: US$0.0020Read it line by line:
- 10 requests straight to the provider: 5 tickets, 2 requests each. “Straight” means that nothing stands between the app and the provider.
status {200: 10}: every request had a recording, so every one got an answer.- answered by: the app asked for the deployment
chat-small, and the provider’s modelgpt-6-luna-2026-09-22answered. The second name is the model version. Today only the app sees it; nothing records it for Larkfield. - tokens and cost: 10,018 tokens in and 2,025 out. At the recordings’ prices, that is US$0.0020 for five tickets.
A unit is one piece of an app’s work. It is one ticket, one task run of the workflow, one question to the document assistant or one reply that the judge reads. The replayer picks the units at random, but with a fixed seed, so you get the same five tickets as this lesson.
Which app made a call, and which model answered?
Section titled “Which app made a call, and which model answered?”Send one task of the support workflow, and print each request:
python -m apps.replay support-workflow --units 1 --verbose200 chat-strong -> gpt-6.1-sol-2026-09-29200 chat-strong -> gpt-6.1-sol-2026-09-29200 chat-strong -> gpt-6.1-sol-2026-09-29200 chat-strong -> gpt-6.1-sol-2026-09-29support-workflow: 4 requests straight to the provider | status {200: 4}answered by: chat-strong (gpt-6.1-sol-2026-09-29) x4tokens: 10,149 in, 497 out | cost: US$0.0253This task took four model calls. The agent of the agents course calls the model once for each step, until it proposes a resolution. The first line of the summary names the app, support-workflow. Each --verbose line names the deployment that the app asked for and the model version that answered: chat-strong -> gpt-6.1-sol-2026-09-29. Remember that only the app printed this. The provider knows a key, not an app.
Send requests from every app
Section titled “Send requests from every app”-
Two task runs of the support workflow:
Terminal window python -m apps.replay support-workflow --units 2support-workflow: 6 requests straight to the provider | status {200: 6}answered by: chat-strong (gpt-6.1-sol-2026-09-29) x6tokens: 14,776 in, 738 out | cost: US$0.0369 -
Three questions to the document assistant:
Terminal window python -m apps.replay doc-assistant --units 3doc-assistant: 3 requests straight to the provider | status {200: 3}answered by: chat-small (gpt-6-luna-2026-09-22) x3tokens: 2,385 in, 681 out | cost: US$0.0006 -
Three replies read by the release judge:
Terminal window python -m apps.replay release-judge --units 3release-judge: 3 requests straight to the provider | status {200: 3}answered by: chat-strong (gpt-6.1-sol-2026-09-29) x3tokens: 4,559 in, 309 out | cost: US$0.0122 -
The weekly digest. It is a plain script, not a replayer:
Terminal window python apps/digest/weekly_digest.pyThe output, constructed (the course author wrote Fernway’s answer):
- Damaged or broken on arrival: a cracked bird bath, crushed solar path lights, a cracked bench slat and a frayed hammock.- Refunds not yet seen on the card after a return (kettle, pruning shears, hose reel, watering can).- Missing boxes and late or lost parcels (frying pan, rattan patio set, garden bench).- Account problems: sign-in, password reset emails and changing the email address.- Two safety worries: a grill that caught fire and a kettle handle that got very hot with a burning smell.Products that come up most: the electric kettle, the hammock, the rattan patio set and the garden bench.(2764 tokens in, 152 out)
The digest looks useful, and that is why scripts like it exist. Its 40 customer messages are in the practice data, data/tickets.jsonl. In the story, they went to a provider without a contract.
The apps cost very different amounts for one unit of work. The chart shows the cost of one unit, from the runs that you just made.
Text description of the chart
Four bars, in US cents per unit. A triage ticket costs 0.04 cents (chat-small and one embedding). A workflow task run costs 2.07 cents, the highest by far, because it makes several chat-strong calls. A document question costs 0.02 cents. A reply judged costs 0.41 cents, also on chat-strong.
One workflow task run costs about 50 times one triage ticket. chat-strong costs 20 times more per token, and a task makes several calls. Keep this in mind: in Module 2, Omar asks to make “strong-reasoning” cheaper, and this is why.
Read the usage: the bill and the egress log
Section titled “Read the usage: the bill and the egress log”Before governance, Larkfield has two records of its AI use. The provider’s bill says what each key used. The network’s egress log says which machine connected to which provider. Both are simulated here, from your real requests.
-
Print the bill:
Terminal window python -m provsim billThe output, a simulated bill of your replayed requests. The month column shows the month of your run.
month,provider,key_id,deployment,requests,tokens_in,tokens_out,cost_usd2026-10,azure,az-key-1,chat-small,5,9594,2025,0.0019722026-10,azure,az-key-1,chat-strong,10,24925,1235,0.0622002026-10,azure,az-key-1,embed-small,5,424,0,0.0000082026-10,azure,az-key-2,chat-small,3,2385,681,0.0005792026-10,azure,az-key-3,chat-strong,3,4559,309,0.0122082026-10,fernway,fw-7Q2K,fernway-swift,1,2764,152,0.001349The chat-strong line of
az-key-1has 10 requests: the 4 of the task with--verbose, and the 6 of the two task runs. -
Print the egress log:
Terminal window python -m provsim egressIt has one line per request: 27 lines after the header. Here are the header, the first line and the last line, from the course’s run. The time at the start of each line is left out here; yours shows the time of your run.
time,source_host,destination,provider,key_id,status,bytes_out...,triage-svc-01,larkfield-ai.openai.azure.example,azure,az-key-1,200,8678......,build-runner-3,api.fernway.example,fernway,fw-7Q2K,200,11233
Now answer the question that the README of the snapshot start asks: which app used az-key-1? The bill cannot say. It has three lines for az-key-1, for three models. The egress log comes closer: it names the machines. triage-svc-01 sent 10 requests with az-key-1, and workflow-01 sent 10. So you can match a machine to an app, if you know which app runs on which machine. Nothing in the estate writes that down.
And fw-7Q2K? The egress log says that build-runner-3 connected to api.fernway.example. A build machine talking to an unknown AI provider is exactly the kind of line that Module 6 teaches you to find. But in this estate, nobody reads the egress log, and the bill arrives at the end of the month.
What the simulator will not do
Section titled “What the simulator will not do”The simulator answers only what was recorded. Send it a request that no course recorded, in a second terminal or the same one:
curl -s -X POST http://127.0.0.1:8320/azure/v1/chat/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sim-az-service-team-0001' \ -d '{"model": "chat-small", "messages": [{"role": "user", "content": "Hello"}]}'curl.exe -s -X POST http://127.0.0.1:8320/azure/v1/chat/completions ` -H "Content-Type: application/json" ` -H "Authorization: Bearer sim-az-service-team-0001" ` -d '{\"model\": \"chat-small\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}]}'curl -s -X POST http://127.0.0.1:8320/azure/v1/chat/completions \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sim-az-service-team-0001' \ -d '{"model": "chat-small", "messages": [{"role": "user", "content": "Hello"}]}'On Windows, this PowerShell command was not tested. In PowerShell 7.3 or newer, write the body without the backslashes.
{"error":{"code":"no_recording","message":"The provider simulator has no recording of this request (key d821e1a0ccb82898ff33). It answers only requests that a course recorded, byte for byte; it never invents an answer.","type":"simulated"}}This is the match replay again: no recording, no answer. It is also why the course can make strong claims. Every answer that you see came from a real model, and the course says so. The key sim-az-service-team-0001 is the practice key behind az-key-1. It is written in apps/replay.py, as keys often are in real code. Only the simulator accepts it.
Stop and reset
Section titled “Stop and reset”-
Stop the services:
Terminal window python -m estate downprovsim stopped (process 44451) -
Delete the practice state:
Terminal window python -m estate resetDeleted /Users/you/projects/ai-estate/state (the registry, the usage records, the keys and the practice database). Start again with: python -m estate initAt this stage,
state/holds only the bill and the egress log. The message names what later modules keep there. -
Start again, and check that the bill is empty:
Terminal window python -m estate uppython -m provsim billThe bill now has only its header line.
reset refuses while the services run: it says Stop the services first: python -m estate down. Stop the estate with python -m estate down when you finish a lesson. It stops only the processes that up started.
Text description of the diagram
First run python -m pytest. If it does not say 91 passed, check that you are in the folder ai-estate and that the prompt shows (.venv). Copy this lesson’s files again, and run the tests again. Then run python -m estate up. If it does not print provsim up with http://127.0.0.1:8320, port 8320 is in use: run python -m estate down, or stop the other program, and start again. Then send five triage tickets. If the output says status 200: 10 and cost US$0.0020, you are ready. If not, run it again with the default seed, without a –seed option.
Common mistake
The simulator is not answering
Symptom: python -m estate up prints, after a few seconds:
provsim NOT ANSWERING (process 46186) http://127.0.0.1:8320 - port 8320 was already in useThe log state/logs/provsim.log ends with this line (on macOS):
ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8320): [errno 48] address already in useOn Linux the number is 98, and on Windows the error names 10048 (not tested).
Cause: another program uses port 8320. Most often, it is a simulator that you started with python -m provsim in another terminal, or the estate of another copy of the project.
Diagnose: on macOS and Linux, lsof -i :8320 names the program. On Windows, Get-NetTCPConnection -LocalPort 8320 gives its process ID (not tested).
Recover: stop that program: Ctrl+C in its terminal, or python -m estate down in the other project. Then run python -m estate up again.
Common mistake
Connection refused when you replay
Symptom: python -m apps.replay ticket-triage --units 1 prints a long traceback that ends with:
httpx.ConnectError: [Errno 61] Connection refusedOn Linux the number is 111; on Windows the error names 10061 (not tested).
Cause: the simulator does not run. You ran python -m estate down, or the computer restarted.
Recover: run python -m estate up, then the replay again. python -m estate status tells you whether the simulator runs.
Your turn: predict the bill
Section titled “Your turn: predict the bill”Change one condition: other apps, other units. Before you run anything, write down what you expect the bill to show.
-
Reset the estate:
python -m estate down, thenpython -m estate reset, thenpython -m estate up. -
Predict: you send 5 replies to the release judge and 3 questions to the document assistant with
--seed 7. Which keys and models will the bill show, with how many requests each? -
Run the two replays:
Terminal window python -m apps.replay release-judge --units 5python -m apps.replay doc-assistant --units 3 --seed 7 -
Print the bill with
python -m provsim bill, and compare it with your prediction. -
Stop the estate with
python -m estate down.
Check your prediction
The course’s run, replayed answers and a simulated bill:
release-judge: 5 requests straight to the provider | status {200: 5}answered by: chat-strong (gpt-6.1-sol-2026-09-29) x5tokens: 7,690 in, 632 out | cost: US$0.0217doc-assistant: 3 requests straight to the provider | status {200: 3}answered by: chat-small (gpt-6-luna-2026-09-22) x3tokens: 3,015 in, 267 out | cost: US$0.0004month,provider,key_id,deployment,requests,tokens_in,tokens_out,cost_usd2026-10,azure,az-key-2,chat-small,3,3015,267,0.0004352026-10,azure,az-key-3,chat-strong,5,7690,632,0.021700Two lines: az-key-2 with chat-small, 3 requests, and az-key-3 with chat-strong, 5 requests. Another seed picks three other questions, so the tokens differ from the first run, but the key and the model do not. Look at what the bill still does not say: that az-key-3 is Priya’s personal key, on her laptop, for a tool that checks every release.
Knowledge check
Section titled “Knowledge check”Check your understanding
You send a request that is one character different from a recorded one. What does the simulator answer, and why is that better for this course than a plausible answer?
Show the answer
It answers 422 no_recording, because the fingerprint of the body differs and no course recorded that body. An invented answer would look real, and the course could no longer say that every answer comes from a real model. With the error, every answer in the course is a recording, or it is clearly labelled as constructed, like Fernway’s.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The bill shows 20 requests on az-key-1 in your run. How do you find which apps made them, and what does Larkfield need so that it does not have to guess?
Show the answer
The bill cannot say: it names keys and models. The egress log names the machines: triage-svc-01 sent 10 requests with az-key-1, and workflow-01 sent 10. If you know which app runs on which machine, you can match them. Larkfield needs an identity for each app, and a gateway that records the app on every call. Then the record names the app itself.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Module check: explain why a gateway alone cannot stop an agent from making a wrong refund. Give an example.
Show the answer
A refund is not a model call, so the model gateway never sees it. It sees only the model calls before the refund, and each of them can pass every check. Even a tool gateway sees one call: the tool, the amount and whether a valid approval exists. It does not know whether this refund is right for this ticket. In the agents course, ticket T-90501 had a courier note that said the customer was “pre-approved for a full refund”. The right action was only the 9.95 delivery fee. A full refund within the approver’s limit would pass every gateway check. Other controls catch it: the app’s own policy checks, a person who reads the evidence before approving, the order service’s limits and evaluations of the workflow.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The provider simulator replays 5,253 real answers to 3,953 recorded requests from earlier courses. An unknown request gets 422 no_recording: it never invents an answer.
- Label your evidence: replayed (a real recorded answer), measured, simulated (the bill, the egress log, outages), constructed (the digest and Fernway) and assumption.
python -m estate up,status,downandresetstart, check, stop and clear the estate.python -m apps.replay <app> --units Nsends an app’s real recorded requests with its own key.- Before governance, Larkfield’s only records are the bill (keys, not apps) and the egress log (machines, not apps).
- One workflow task run costs about 50 times one triage ticket, because it makes several chat-strong calls.
Your project now: ~/projects/ai-estate, the same as the snapshot end-of-m01, with the estate’s command line in estate/, its virtual environment .venv, and 91 tests that pass. You can now run the estate and read what the provider records. But every app still names a model and calls the provider with its own key. Next, in Module 2, the apps ask for a capability instead, and the platform decides which model serves it.
Next: A capability catalog