Skip to content
Français

Meet Larkfield's AI estate

  • Lesson 3
  • 40 min
  • Module 1: Why agents need platform controls

Where we are

In the last lesson, we found the control points (identity, registry, model gateway, tool gateway and audit) and what each one can see. So far they are boxes on a picture, and the estate is a folder of files. In this lesson, we start Larkfield's estate on our own computer, send real recorded requests from each app, and read what the provider records, so that every later lesson changes something that we have already seen run. We start with a match replay.

What you will be able to do

Start Larkfield’s AI estate on your computer and send each app’s real recorded requests to the provider. Then read the only records that exist before governance: the provider’s bill and the network’s egress log. You also learn which evidence in this course is replayed and which is simulated, and how to start again.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project ~/projects/ai-estate from the first lesson of this module, and the labs repository in ~/projects/nextia-learning-labs.
  • Python 3.12 or newer. Check with python3.12 --version on macOS and Linux, or py -3.12 --version on Windows.
  • Two terminals, for example two tabs in VS Code. Port 8320 must be free: the provider simulator uses it.
  • An internet connection, once, for pip. The packages and the virtual environment take about 85 MB.

You need no account, no key and no payment. Tested on macOS (Apple silicon) with Python 3.12.13. The Windows (PowerShell) and Linux commands are written for you, but the course did not run them.

Think of a football coach who studies last week’s match. She plays the recording, stops it, and plays one attack again. Every time she plays it, the same thing happens, in the same order. She can show it to the team and to another coach, and they all see the same match.

A replay has one more property that matters here. It never shows a goal that was not scored. If the coach asks, “What if our striker had shot with the left foot?”, the recording has no answer. It can only show what really happened.

The problem: practising governance on real providers

Section titled “The problem: practising governance on real providers”

Kwame could build the platform against real Azure models. But every call costs money and needs a key. A real model may give a different answer to the same request tomorrow, so a lesson could not show you the output you will see. He cannot cause a provider outage on purpose to test a fallback. And a month of Larkfield’s traffic is tens of thousands of calls.

So the course does what the coach does. It plays back what really happened.

The idea: a provider simulator that replays recordings

Section titled “The idea: a provider simulator that replays recordings”

The earlier courses called real models and saved every request and every answer. This course merges those recordings into a provider simulator: a small server on your computer that looks like a model provider to the apps. It speaks the same API as Azure’s OpenAI-compatible endpoint.

When an app sends a request, the simulator computes a fingerprint of the request body, a SHA-256 hash. If an earlier course recorded exactly that body, the simulator gives back the recorded answer, with its real tokens and its real time. If not, it answers with an error, 422 no_recording, and never invents an answer. A few fields that do not change the answer, such as stream, are left out of the fingerprint. Every other character counts: the course’s tests change max_completion_tokens of a triage request to 999, and the simulator answers 422.

A flow, top to bottom. An app sends a request to the simulator with its key. The simulator writes a line in the egress log: host, provider, key ID, status. Is the key a practice key for this provider? No: 401 invalid_key. Yes: does a recorded feature rule refuse it? Yes: 400 with the error text that the real provider gave. No: the request's key is a SHA-256 hash of the body, byte for byte, with 5 fields left out, such as stream. Did an earlier course record this exact body? No: 422 no_recording, it never invents an answer. Yes: 200, the recorded answer with its tokens and latency, and a line on the bill: key ID, model, tokens, cost.
How the provider simulator answers one request. Only a request that an earlier course recorded gets a model's answer.
Text description of the diagram

An app sends a request with its key. First, the simulator writes a line in its egress log: the host that sent it, the provider, the key ID and the status. Then it checks the key: an unknown key gets 401 invalid_key. Next, it checks the recorded feature rules. The real provider refused some requests, for example a tool call that one model cannot make. The simulator refuses them with the same error text. Then it computes the request’s fingerprint, a SHA-256 hash of the body, byte for byte, with five fields left out, such as stream. If no earlier course recorded that exact body, the answer is 422 no_recording: the simulator never invents an answer. If a course recorded it, the answer is 200 with the recorded answer, its tokens and its latency. Then the simulator adds a line to the bill: key ID, model, tokens and cost.

Every piece of evidence in this course has a label. Use the same labels in your own notes.

Label What it is Examples
replayed a real answer of a real model, recorded by an earlier course, played back by the simulator the answers of chat-small (gpt-6-luna-2026-09-22), chat-strong (gpt-6.1-sol-2026-09-29), embed-small and gemma3:4b
measured a number from a run of the course’s code, such as the gateway’s added time your own runs, and the course team’s runs on a laptop
simulated behaviour that the course made up to practise a situation the bill, the egress log, outages, slow answers, a second Azure region
constructed a made-up case or data, written by the course author the weekly digest script, Fernway and its one answer
assumption a stated value, such as a price or a daily volume 45 workflow runs a day; the prices of the recordings

The simulator holds 5,253 recorded answers to 3,953 different requests. Some requests were recorded more than once, so a repeated request can get a different real answer, as a real model would give. The answers come from the agents course (2,363), the evaluation course (2,260), the capstone course (353), the LLM applications course (7) and this course (269, most of them the triage service’s embeddings). One answer is constructed: Fernway’s. Each app in the estate is a replayer: it sends exactly the request bodies that its course sent, in order.

You downloaded the snapshot start in the first lesson. Now you make its Python environment, check it, and add this lesson’s files: the estate’s command line.

  1. Move into the project:

    Terminal window
    cd ~/projects/ai-estate
  2. Make and activate a virtual environment with Python 3.12:

    Terminal window
    python3.12 -m venv .venv
    source .venv/bin/activate

    The prompt now starts with (.venv). From here, type python on every system. Activate the environment again in every new terminal.

  3. Install the pinned packages:

    Terminal window
    python -m pip install -r requirements-dev.txt

    This takes about a minute. The last lines may suggest a newer pip. You can ignore that.

  4. Run the tests of the snapshot:

    Terminal window
    python -m pytest -q

    The last line:

    88 passed, 1 deselected in 4.12s

    Your time differs. 69 of the 88 tests are the agents course’s own tests of the support workflow. The deselected test calls a real provider, and only runs if you ask for it.

  5. Copy this lesson’s files from the snapshot end-of-m01: the folder estate, its test and the new README.md. In ~/projects/ai-estate:

    Terminal window
    S=../nextia-learning-labs/C32/snapshots/end-of-m01
    cp -R $S/estate .
    cp $S/tests/test_estate_basics.py tests/
    cp $S/README.md .
  6. Run the tests again:

    Terminal window
    python -m pytest -q

    The last line now says 91 passed, 1 deselected. Your project is now the same as the snapshot end-of-m01.

The command python -m estate starts and stops the estate’s services. At this stage there is only one service, the provider simulator. Later modules add the control plane, the gateways and the tool servers, and the same command starts them all.

  1. Create the estate’s state folder:

    Terminal window
    python -m estate init

    The output (your path differs):

    State folder: /Users/you/projects/ai-estate/state
    The estate has no platform yet: each app calls the provider with its own key.

    The folder state/ holds the practice state: today, the simulator’s bill and egress log; later, the registry, the keys and the usage records. Git ignores it.

  2. Start the services:

    Terminal window
    python -m estate up

    The output (your process number differs):

    provsim up (process 44451) http://127.0.0.1:8320

    The simulator now runs in the background. You keep using this terminal.

  3. Check that it answers:

    Terminal window
    python -m estate status
    provsim running {"ok":true,"recordings":5253,"speed":0.0,"egress":"open"}

    speed 0.0 means that the simulator answers at once, without the recorded wait. egress open means that the simulated network lets every machine reach every provider, as Larkfield’s network does today.

Worked example: five tickets through the triage service

Section titled “Worked example: five tickets through the triage service”

The triage service reads each new ticket and sends two requests: one chat call that chooses the team, and one embedding for search. Send five tickets:

Terminal window
python -m apps.replay ticket-triage --units 5

The output, replayed:

ticket-triage: 10 requests straight to the provider | status {200: 10}
answered by: chat-small (gpt-6-luna-2026-09-22) x5, embed-small (text-embedding-3-small) x5
tokens: 10,018 in, 2,025 out | cost: US$0.0020

Read it line by line:

  • 10 requests straight to the provider: 5 tickets, 2 requests each. “Straight” means that nothing stands between the app and the provider.
  • status {200: 10}: every request had a recording, so every one got an answer.
  • answered by: the app asked for the deployment chat-small, and the provider’s model gpt-6-luna-2026-09-22 answered. The second name is the model version. Today only the app sees it; nothing records it for Larkfield.
  • tokens and cost: 10,018 tokens in and 2,025 out. At the recordings’ prices, that is US$0.0020 for five tickets.

A unit is one piece of an app’s work. It is one ticket, one task run of the workflow, one question to the document assistant or one reply that the judge reads. The replayer picks the units at random, but with a fixed seed, so you get the same five tickets as this lesson.

Which app made a call, and which model answered?

Section titled “Which app made a call, and which model answered?”

Send one task of the support workflow, and print each request:

Terminal window
python -m apps.replay support-workflow --units 1 --verbose
200 chat-strong -> gpt-6.1-sol-2026-09-29
200 chat-strong -> gpt-6.1-sol-2026-09-29
200 chat-strong -> gpt-6.1-sol-2026-09-29
200 chat-strong -> gpt-6.1-sol-2026-09-29
support-workflow: 4 requests straight to the provider | status {200: 4}
answered by: chat-strong (gpt-6.1-sol-2026-09-29) x4
tokens: 10,149 in, 497 out | cost: US$0.0253

This task took four model calls. The agent of the agents course calls the model once for each step, until it proposes a resolution. The first line of the summary names the app, support-workflow. Each --verbose line names the deployment that the app asked for and the model version that answered: chat-strong -> gpt-6.1-sol-2026-09-29. Remember that only the app printed this. The provider knows a key, not an app.

  1. Two task runs of the support workflow:

    Terminal window
    python -m apps.replay support-workflow --units 2
    support-workflow: 6 requests straight to the provider | status {200: 6}
    answered by: chat-strong (gpt-6.1-sol-2026-09-29) x6
    tokens: 14,776 in, 738 out | cost: US$0.0369
  2. Three questions to the document assistant:

    Terminal window
    python -m apps.replay doc-assistant --units 3
    doc-assistant: 3 requests straight to the provider | status {200: 3}
    answered by: chat-small (gpt-6-luna-2026-09-22) x3
    tokens: 2,385 in, 681 out | cost: US$0.0006
  3. Three replies read by the release judge:

    Terminal window
    python -m apps.replay release-judge --units 3
    release-judge: 3 requests straight to the provider | status {200: 3}
    answered by: chat-strong (gpt-6.1-sol-2026-09-29) x3
    tokens: 4,559 in, 309 out | cost: US$0.0122
  4. The weekly digest. It is a plain script, not a replayer:

    Terminal window
    python apps/digest/weekly_digest.py

    The output, constructed (the course author wrote Fernway’s answer):

    - Damaged or broken on arrival: a cracked bird bath, crushed solar path lights, a cracked bench slat and a frayed hammock.
    - Refunds not yet seen on the card after a return (kettle, pruning shears, hose reel, watering can).
    - Missing boxes and late or lost parcels (frying pan, rattan patio set, garden bench).
    - Account problems: sign-in, password reset emails and changing the email address.
    - Two safety worries: a grill that caught fire and a kettle handle that got very hot with a burning smell.
    Products that come up most: the electric kettle, the hammock, the rattan patio set and the garden bench.
    (2764 tokens in, 152 out)

The digest looks useful, and that is why scripts like it exist. Its 40 customer messages are in the practice data, data/tickets.jsonl. In the story, they went to a provider without a contract.

The apps cost very different amounts for one unit of work. The chart shows the cost of one unit, from the runs that you just made.

Your runs, replayed answers at the recordings' prices: 5 tickets, 3 workflow task runs, 3 document questions and 3 replies judged, divided per unit.
Text description of the chart

Four bars, in US cents per unit. A triage ticket costs 0.04 cents (chat-small and one embedding). A workflow task run costs 2.07 cents, the highest by far, because it makes several chat-strong calls. A document question costs 0.02 cents. A reply judged costs 0.41 cents, also on chat-strong.

One workflow task run costs about 50 times one triage ticket. chat-strong costs 20 times more per token, and a task makes several calls. Keep this in mind: in Module 2, Omar asks to make “strong-reasoning” cheaper, and this is why.

Read the usage: the bill and the egress log

Section titled “Read the usage: the bill and the egress log”

Before governance, Larkfield has two records of its AI use. The provider’s bill says what each key used. The network’s egress log says which machine connected to which provider. Both are simulated here, from your real requests.

  1. Print the bill:

    Terminal window
    python -m provsim bill

    The output, a simulated bill of your replayed requests. The month column shows the month of your run.

    month,provider,key_id,deployment,requests,tokens_in,tokens_out,cost_usd
    2026-10,azure,az-key-1,chat-small,5,9594,2025,0.001972
    2026-10,azure,az-key-1,chat-strong,10,24925,1235,0.062200
    2026-10,azure,az-key-1,embed-small,5,424,0,0.000008
    2026-10,azure,az-key-2,chat-small,3,2385,681,0.000579
    2026-10,azure,az-key-3,chat-strong,3,4559,309,0.012208
    2026-10,fernway,fw-7Q2K,fernway-swift,1,2764,152,0.001349

    The chat-strong line of az-key-1 has 10 requests: the 4 of the task with --verbose, and the 6 of the two task runs.

  2. Print the egress log:

    Terminal window
    python -m provsim egress

    It has one line per request: 27 lines after the header. Here are the header, the first line and the last line, from the course’s run. The time at the start of each line is left out here; yours shows the time of your run.

    time,source_host,destination,provider,key_id,status,bytes_out
    ...,triage-svc-01,larkfield-ai.openai.azure.example,azure,az-key-1,200,8678
    ...
    ...,build-runner-3,api.fernway.example,fernway,fw-7Q2K,200,11233

Now answer the question that the README of the snapshot start asks: which app used az-key-1? The bill cannot say. It has three lines for az-key-1, for three models. The egress log comes closer: it names the machines. triage-svc-01 sent 10 requests with az-key-1, and workflow-01 sent 10. So you can match a machine to an app, if you know which app runs on which machine. Nothing in the estate writes that down.

And fw-7Q2K? The egress log says that build-runner-3 connected to api.fernway.example. A build machine talking to an unknown AI provider is exactly the kind of line that Module 6 teaches you to find. But in this estate, nobody reads the egress log, and the bill arrives at the end of the month.

The simulator answers only what was recorded. Send it a request that no course recorded, in a second terminal or the same one:

Terminal window
curl -s -X POST http://127.0.0.1:8320/azure/v1/chat/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sim-az-service-team-0001' \
-d '{"model": "chat-small", "messages": [{"role": "user", "content": "Hello"}]}'

On Windows, this PowerShell command was not tested. In PowerShell 7.3 or newer, write the body without the backslashes.

{"error":{"code":"no_recording","message":"The provider simulator has no recording of this request (key d821e1a0ccb82898ff33). It answers only requests that a course recorded, byte for byte; it never invents an answer.","type":"simulated"}}

This is the match replay again: no recording, no answer. It is also why the course can make strong claims. Every answer that you see came from a real model, and the course says so. The key sim-az-service-team-0001 is the practice key behind az-key-1. It is written in apps/replay.py, as keys often are in real code. Only the simulator accepts it.

  1. Stop the services:

    Terminal window
    python -m estate down
    provsim stopped (process 44451)
  2. Delete the practice state:

    Terminal window
    python -m estate reset
    Deleted /Users/you/projects/ai-estate/state (the registry, the usage records, the keys and the practice database). Start again with: python -m estate init

    At this stage, state/ holds only the bill and the egress log. The message names what later modules keep there.

  3. Start again, and check that the bill is empty:

    Terminal window
    python -m estate up
    python -m provsim bill

    The bill now has only its header line.

reset refuses while the services run: it says Stop the services first: python -m estate down. Stop the estate with python -m estate down when you finish a lesson. It stops only the processes that up started.

A decision flow. Run python -m pytest. 91 passed? If no: check the folder ai-estate, check that the prompt shows (.venv), copy the lesson's files again, and run the tests again. If yes: python -m estate up. Does it print provsim up, http://127.0.0.1:8320? If no: port 8320 is in use; run python -m estate down, or stop the other program, and start again. If yes: python -m apps.replay ticket-triage --units 5. Does it print status 200: 10 and cost US$0.0020? If yes: ready, your estate gives the same answers as this course. If no: use the default seed, with no --seed option, and run it again.
The setup check. When five triage tickets cost US$0.0020 with ten answers, your estate gives the same answers as the lessons.
Text description of the diagram

First run python -m pytest. If it does not say 91 passed, check that you are in the folder ai-estate and that the prompt shows (.venv). Copy this lesson’s files again, and run the tests again. Then run python -m estate up. If it does not print provsim up with http://127.0.0.1:8320, port 8320 is in use: run python -m estate down, or stop the other program, and start again. Then send five triage tickets. If the output says status 200: 10 and cost US$0.0020, you are ready. If not, run it again with the default seed, without a –seed option.

Common mistake

The simulator is not answering

Symptom: python -m estate up prints, after a few seconds:

provsim NOT ANSWERING (process 46186) http://127.0.0.1:8320 - port 8320 was already in use

The log state/logs/provsim.log ends with this line (on macOS):

ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8320): [errno 48] address already in use

On Linux the number is 98, and on Windows the error names 10048 (not tested).

Cause: another program uses port 8320. Most often, it is a simulator that you started with python -m provsim in another terminal, or the estate of another copy of the project.

Diagnose: on macOS and Linux, lsof -i :8320 names the program. On Windows, Get-NetTCPConnection -LocalPort 8320 gives its process ID (not tested).

Recover: stop that program: Ctrl+C in its terminal, or python -m estate down in the other project. Then run python -m estate up again.

Common mistake

Connection refused when you replay

Symptom: python -m apps.replay ticket-triage --units 1 prints a long traceback that ends with:

httpx.ConnectError: [Errno 61] Connection refused

On Linux the number is 111; on Windows the error names 10061 (not tested).

Cause: the simulator does not run. You ran python -m estate down, or the computer restarted.

Recover: run python -m estate up, then the replay again. python -m estate status tells you whether the simulator runs.

Change one condition: other apps, other units. Before you run anything, write down what you expect the bill to show.

  1. Reset the estate: python -m estate down, then python -m estate reset, then python -m estate up.

  2. Predict: you send 5 replies to the release judge and 3 questions to the document assistant with --seed 7. Which keys and models will the bill show, with how many requests each?

  3. Run the two replays:

    Terminal window
    python -m apps.replay release-judge --units 5
    python -m apps.replay doc-assistant --units 3 --seed 7
  4. Print the bill with python -m provsim bill, and compare it with your prediction.

  5. Stop the estate with python -m estate down.

Check your prediction

The course’s run, replayed answers and a simulated bill:

release-judge: 5 requests straight to the provider | status {200: 5}
answered by: chat-strong (gpt-6.1-sol-2026-09-29) x5
tokens: 7,690 in, 632 out | cost: US$0.0217
doc-assistant: 3 requests straight to the provider | status {200: 3}
answered by: chat-small (gpt-6-luna-2026-09-22) x3
tokens: 3,015 in, 267 out | cost: US$0.0004
month,provider,key_id,deployment,requests,tokens_in,tokens_out,cost_usd
2026-10,azure,az-key-2,chat-small,3,3015,267,0.000435
2026-10,azure,az-key-3,chat-strong,5,7690,632,0.021700

Two lines: az-key-2 with chat-small, 3 requests, and az-key-3 with chat-strong, 5 requests. Another seed picks three other questions, so the tokens differ from the first run, but the key and the model do not. Look at what the bill still does not say: that az-key-3 is Priya’s personal key, on her laptop, for a tool that checks every release.

Check your understanding

You send a request that is one character different from a recorded one. What does the simulator answer, and why is that better for this course than a plausible answer?

Show the answer

It answers 422 no_recording, because the fingerprint of the body differs and no course recorded that body. An invented answer would look real, and the course could no longer say that every answer comes from a real model. With the error, every answer in the course is a recording, or it is clearly labelled as constructed, like Fernway’s.

Check your understanding

The bill shows 20 requests on az-key-1 in your run. How do you find which apps made them, and what does Larkfield need so that it does not have to guess?

Show the answer

The bill cannot say: it names keys and models. The egress log names the machines: triage-svc-01 sent 10 requests with az-key-1, and workflow-01 sent 10. If you know which app runs on which machine, you can match them. Larkfield needs an identity for each app, and a gateway that records the app on every call. Then the record names the app itself.

Check your understanding

Module check: explain why a gateway alone cannot stop an agent from making a wrong refund. Give an example.

Show the answer

A refund is not a model call, so the model gateway never sees it. It sees only the model calls before the refund, and each of them can pass every check. Even a tool gateway sees one call: the tool, the amount and whether a valid approval exists. It does not know whether this refund is right for this ticket. In the agents course, ticket T-90501 had a courier note that said the customer was “pre-approved for a full refund”. The right action was only the 9.95 delivery fee. A full refund within the approver’s limit would pass every gateway check. Other controls catch it: the app’s own policy checks, a person who reads the evidence before approving, the order service’s limits and evaluations of the workflow.

Recap

  • The provider simulator replays 5,253 real answers to 3,953 recorded requests from earlier courses. An unknown request gets 422 no_recording: it never invents an answer.
  • Label your evidence: replayed (a real recorded answer), measured, simulated (the bill, the egress log, outages), constructed (the digest and Fernway) and assumption.
  • python -m estate up, status, down and reset start, check, stop and clear the estate. python -m apps.replay <app> --units N sends an app’s real recorded requests with its own key.
  • Before governance, Larkfield’s only records are the bill (keys, not apps) and the egress log (machines, not apps).
  • One workflow task run costs about 50 times one triage ticket, because it makes several chat-strong calls.

Your project now: ~/projects/ai-estate, the same as the snapshot end-of-m01, with the estate’s command line in estate/, its virtual environment .venv, and 91 tests that pass. You can now run the estate and read what the provider records. But every app still names a model and calls the provider with its own key. Next, in Module 2, the apps ask for a capability instead, and the platform decides which model serves it.

Next: A capability catalog