Run the AI workload on your computer

What you will be able to do

Larkfield’s ticket API now asks an AI provider three questions for every new ticket. Run that work on your computer with a simulated provider, send your first load test, and read what it tells you.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The ticket API and its project folder ticket-api, from Databases and Storage for AI Applications: PostgreSQL in Docker, python -m scripts.load, transactions, and the connection pool from Connection management.
• fastapi dev and curl, from Build and Deploy Your First API.
• Docker Compose, from Containers and Docker.

What you need on your computer

• Python 3.12 or later, Docker Desktop, VS Code and a terminal.
• About 500 MB of free disk space for the new packages. No account, no key and no money.

You start from the snapshot start: the project at the end of the databases course, unchanged. If you have your own ticket-api from that course, keep it under another name. This course starts from a clean copy.

A clerk on the phone

Think of a post office counter. A customer brings a parcel for another country. The clerk weighs it in ten seconds. Then the clerk phones the customs office and waits on the line for two minutes until customs says yes.

On a normal day, this works. The customers wait a little, and the line stays short. But the clerk’s own work is ten seconds, and the phone call is two minutes. Almost all of the customer’s time goes to waiting for someone else. And customs has a rule: at most 100 calls per hour from this post office.

Now picture the week before a holiday, with ten times more customers. The line grows. The clerk cannot work faster, because the clerk is mostly waiting. And after 100 calls, customs stops answering for a while.

Larkfield’s ticket API is about to become that clerk.

Black Friday at Larkfield

You know Larkfield’s help desk from the databases course. Amira, the developer, owns the ticket API, and she guides you through this course. Mei, the data engineer, owns the database and watches its connections. Omar, the product manager, brings the numbers and asks what customers will see. Kwame, the solution architect, asks what each change costs. Tomás, an AI engineer, likes fast fixes. Sam asks the beginner’s questions, which may be yours too.

The API now does AI work for every new ticket. When a customer sends a ticket, the API asks an AI provider three things:

1. Classify: which team gets the ticket, and how urgent it is.
2. Draft a reply: a first answer that a person on the help desk checks before it is sent.
3. Embed: a vector of the ticket’s text, so the help desk can find similar tickets.

Black Friday is coming. Omar expects about ten times the usual number of tickets for a few hours. “Will the API cope?” he asks. Amira’s answer is honest: “I do not know yet. Nobody knows until we measure it.”

That is the whole idea of this module: measure before you change anything. In this lesson, you build the thing to measure. You run the API with its AI work on your computer, and you send it its first load.

The problem: you cannot test against the real provider

The AI work is slow, and the provider has limits. The course team called the real provider, a hosted chat model and an embedding model, one call at a time. The median times were 913 ms to classify, 1,097 ms to draft a reply and 198 ms to embed. The account allows 100 chat requests per minute. That limit is the provider’s quota.

So why not run your tests against the real provider? There are four reasons:

• Money and accounts. Every call costs a little, and you need an account and a key. This course needs neither.
• Someone else’s service. A load test sends many requests on purpose. You never load-test a service that you do not own.
• A shared quota. The quota belongs to the account. Your test would use up the quota that real users need.
• Unstable results. The provider’s speed changes from hour to hour, for reasons that you cannot see. You could not repeat a test.

You still need something that answers like the provider: slowly, with the same spread of times, and with the same quota.

The idea: a simulated provider

Pilots train in a flight simulator. It is not a plane, but the controls, the instruments and the timing behave like a real plane. A pilot can practise an engine failure without any danger.

This course does the same with the AI provider. A simulated provider is a small program on your computer that answers the same requests as the real provider. It does not think. It waits for a realistic time, and then it sends a simple answer in the right shape.

The AI provider in this course is simulated on your computer. Its times and its quota come from real calls that the course team recorded; its answers are simple. All the tickets, customers and messages are made up for the course, as in the databases course.

Where the simulated provider's behaviour comes from. The times and the quota are real; the answers are not.

Text description of the diagram

The course team made 763 real calls to the provider. A script turned the recorded times, the quota and the token counts into one file, calibration.json. The simulator reads that file. When the ticket API calls it, the simulator picks a recorded time, waits that long, and answers. Over the quota, it refuses with HTTP 429. It does not get slower under load, it has no real outages, and its answers are simple.

What the simulator copies from the recordings:

• Times. For each task, it draws a time from the 100 recorded times of that task. Its median and its slow calls are the real ones.
• The quota. 100 chat requests per minute. A burst gets about 100 calls through, and then the simulator refuses.
• A refusal. HTTP status 429 Too Many Requests, after about 0.13 s, with two hints about when to try again. Find the bottleneck looks at them closely.
• Token counts. So the cost of each ticket is realistic.

What it does not copy, and you must remember this when you read a result:

• The real provider got slower when many calls arrived at once. A draft reply took about 7% longer at 16 calls in flight, and about 30% longer at 40. The simulator always uses the times of one call at a time.
• Real outages and slow periods. The simulator’s slow and outage modes are made up for Module 5.
• The answers. A simulated draft reply is one fixed sentence. The timing is realistic; the text is not.

A tiny example: one ticket

Before you run anything, add up what one ticket costs in time. The three AI calls run one after the other, so their times add up:

Step	Who works	Time (median of the real calls)
Save the ticket	PostgreSQL	a few ms
Classify	the provider	913 ms
Draft a reply	the provider	1,097 ms
Embed	the provider	198 ms
Save the results	PostgreSQL	a few ms
Total		about 2,208 ms

So one ticket takes about 2.2 seconds. Almost all of it is waiting for the provider, like the clerk on the phone. The diagram shows the same request as a sequence.

One new ticket in this module's design: one transaction around everything, three provider calls one after the other.

Text description of the diagram

The API opens one transaction and saves the ticket. Then it calls the provider three times, and each call waits for its answer: about 0.9, 1.1 and 0.2 seconds. Then it saves the results and commits. The ticket is saved only if all the AI work worked. The database connection stays borrowed for all of the 2.2 seconds. That detail matters in Module 2.

Set up the project

You copy the snapshot start, check it, and then add this lesson’s files from the snapshot end-of-m01.

1. Get the files of the course. If you have Git, clone the labs repository once, in ~/projects. If you have the clone already, run git -C nextia-learning-labs pull instead.

   cd ~/projects
   git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

   Without Git, download the ZIP file of the repository, and unzip it in ~/projects. Its folder is then nextia-learning-labs-main. Use that name in the next steps.

2. If a folder ticket-api exists, rename it first. Then copy the snapshot start, and make your .env:

   macOS / Linux

   mv ticket-api ticket-api-databases     # only if the folder exists
   cp -R nextia-learning-labs/C21/snapshots/start ticket-api
   cd ticket-api
   cp .env.example .env

   Windows (PowerShell)

   Rename-Item ticket-api ticket-api-databases     # only if the folder exists
   Copy-Item -Recurse nextia-learning-labs\C21\snapshots\start ticket-api
   cd ticket-api
   Copy-Item .env.example .env

3. Open the folder in VS Code. In .env, replace change-me-local-only with a password of your own, in all three places.

4. Start Docker Desktop. Then start PostgreSQL and Azurite, the object storage of the databases course. The loader uploads the help desk’s files to Azurite.

   docker compose up -d db azurite

5. Make the virtual environment, install the packages, load the data and run the tests:

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt
   python -m scripts.load
   python -m pytest

   Windows (PowerShell)

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt
   python -m scripts.load
   python -m pytest

   The load applies 12 migrations and prints the small data: 200 tickets, 43 files and the rest. The tests end with 108 passed, 1 deselected. This is the databases course’s project, and it works.

6. Copy this lesson’s files from the snapshot end-of-m01. Ten files and two folders are new; sixteen files change.

   macOS / Linux

   S=../nextia-learning-labs/C21/snapshots/end-of-m01
   cp -R "$S/simulator" "$S/loadtest" .
   for f in migrations/013_ai_work.sql ticket_api/ai.py ticket_api/provider.py \
            ticket_api/intake.py ticket_api/prices.py ticket_api/timing.py \
            ticket_api/config.py ticket_api/errors.py ticket_api/main.py \
            ticket_api/models.py ticket_api/repository.py scripts/load.py \
            tests/fakes.py tests/test_ai.py tests/test_simulator.py tests/test_intake.py \
            tests/conftest.py contract/openapi.json requirements.txt requirements-lock.txt \
            pyproject.toml compose.yaml .env.example Dockerfile .dockerignore README.md; do
     cp "$S/$f" "$f"
   done

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C21\snapshots\end-of-m01"
   Copy-Item -Recurse "$S\simulator", "$S\loadtest" .
   $files = 'migrations\013_ai_work.sql', 'ticket_api\ai.py', 'ticket_api\provider.py',
            'ticket_api\intake.py', 'ticket_api\prices.py', 'ticket_api\timing.py',
            'ticket_api\config.py', 'ticket_api\errors.py', 'ticket_api\main.py',
            'ticket_api\models.py', 'ticket_api\repository.py', 'scripts\load.py',
            'tests\fakes.py', 'tests\test_ai.py', 'tests\test_simulator.py', 'tests\test_intake.py',
            'tests\conftest.py', 'contract\openapi.json', 'requirements.txt', 'requirements-lock.txt',
            'pyproject.toml', 'compose.yaml', '.env.example', 'Dockerfile', '.dockerignore', 'README.md'
   foreach ($f in $files) { Copy-Item "$S\$f" $f }

7. Install the new package, the load generator Locust:

   python -m pip install -r requirements.txt

8. Apply the new migration and load the data again. The tables already have data, so add --reset:

   python -m scripts.load --reset

   applied  013_ai_work
   Database is up to date (13 migrations).
   Loaded the small data in 1.0 s:
     customers                 40
     tickets                  200
     messages                 331
     attachments               43
     documents                 36
     ai_runs                  294
     ticket_embeddings        200
     43 files uploaded to the object storage.
     3 messages have no ticket: they are in orphaned_messages.

9. Run the tests: python -m pytest. Expect 123 passed, 1 deselected, 15 more than before. The new tests use a fake provider that answers at once, so they do not wait.

What the new files do

File	What it does
ticket_api/ai.py	The three requests to the provider, and the code that checks each answer. Classify asks for JSON with a fixed shape.
ticket_api/provider.py	The client that sends the requests. It turns every failure into one of four errors: a timeout, a refusal over the quota (429), no provider (5xx or no connection), or a wrong request.
ticket_api/intake.py	POST /v1/tickets: one transaction around the ticket and all its AI work, as in the diagram. It adds a Server-Timing header with the time of each part.
ticket_api/prices.py	Dated prices per million tokens, to estimate the cost of each call. Prices change: check the provider’s price page.
migrations/013_ai_work.sql	New tickets get numbers from the database: T-500001, T-500002, and so on. It also allows the new task embed.
simulator/	The simulated provider. calibration.json holds the recorded times and the quota.
loadtest/locustfile.py	The first workload for Locust.

The client works with any provider that has the same API: set PROVIDER_URL and PROVIDER_API_KEY in .env. You do not need to. Without these settings, the API calls the simulator at http://127.0.0.1:8300/v1.

Run it: three terminals

You now run two services, and you send requests from a third terminal. Activate the virtual environment in each terminal.

1. Terminal 1, the simulated provider. Leave it running.

   python -m simulator

   Simulated AI provider on http://127.0.0.1:8300/v1 (mode normal). Its times and quota come from real recorded calls; its answers are simple. Stop it with Ctrl+C.

2. Terminal 2, the API. Leave it running too.

   fastapi dev

3. Terminal 3, one ticket. On Windows, type curl.exe instead of curl, and put the command on one line without the \.

   curl -i -X POST http://127.0.0.1:8000/v1/tickets -H "Content-Type: application/json" \
     -d '{"customer_id": "C-0022", "subject": "Charged twice", "body": "Two payments for one order."}'

The author’s run printed this. Your times and dates will differ:

HTTP/1.1 201 Created
date: Sat, 10 Oct 2026 13:21:44 GMT
server: uvicorn
content-length: 758
content-type: application/json
server-timing: db;dur=10.3, classify;dur=787.0, draft_reply;dur=1167.7, embed;dur=222.6
x-request-id: ef8bf5f37ece

{"ticket_id":"T-500001","customer_id":"C-0022","subject":"Charged twice","team":"billing","priority":1,"status":"open","created_at":"2026-10-10T13:21:45.090248Z","draft_reply":"Thank you for your message about \"Charged twice\". A person on our help desk will look at it and reply to you soon. If you have an order number, please send it.","embedding_version":"embed-small-384-v1","ai_runs":[{"task":"classify","model":"chat-small (simulated)","latency_ms":787,"tokens_in":176,"tokens_out":15,"cost_usd":0.0000251},{"task":"draft_reply","model":"chat-small (simulated)","latency_ms":1168,"tokens_in":107,"tokens_out":37,"cost_usd":0.0000292},{"task":"embed","model":"embed-small (simulated)","latency_ms":223,"tokens_in":9,"tokens_out":0,"cost_usd":1.8e-7}]}

Read it from the top. The status is 201 Created, and the new ticket is T-500001. The server-timing header splits the request into parts: 10 ms in the database, then 787, 1,168 and 223 ms for the three AI calls. Each AI run says (simulated) in its model name, so a simulated answer can never pass for a real one. The runs also record the tokens and the cost: about US$0.00005 for this ticket. The draft reply is the simulator’s fixed sentence.

Your first load test

One ticket tells you how long one ticket takes. It does not tell you what happens when many tickets arrive. For that, you need a load test: a program sends many requests on purpose, in a planned way, and measures every answer.

The program is a load generator. This course uses Locust, which is free and open source (MIT licence). You describe the senders in Python, in loadtest/locustfile.py. Each simulated sender is a user. In this workload, a user sends a ticket, waits for the answer, and sends the next ticket at once. The tickets come from the practice data, picked with a fixed seed, so every run sends the same mix.

Two words describe the result:

• Latency is how long one request takes, from sending it to the end of the answer.
• Throughput is how many requests finish in each second.

Predict first. One user sends tickets for 60 seconds, and each ticket takes about 2.2 seconds. How many tickets per second will finish?

In terminal 3, run Locust against your own API. Never point a load generator at a service that you do not own.

locust -f loadtest/locustfile.py --host http://127.0.0.1:8000 --headless -u 1 -r 1 -t 60s

-u 1 is one user, and -r 1 starts one user per second. -t 60s stops the test after 60 seconds. --headless prints tables instead of opening Locust’s web page. Locust prints a table every few seconds. When it stops, read the last two tables. This is the author’s run, on a busy shared computer, with the name column made shorter. Your numbers will differ:

Type     Name                  # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|---------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
POST     POST /v1/tickets          24     0(0.00%) |   2388    1796    3619   2200 |    0.42        0.00
--------|---------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                24     0(0.00%) |   2388    1796    3619   2200 |    0.42        0.00

Response time percentiles (approximated)
Type     Name                      50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100% # reqs
--------|---------------------|--------|------|------|------|------|------|------|------|------|------|------|------
POST     POST /v1/tickets         2200   2400   2600   2600   3300   3400   3600   3600   3600   3600   3600     24

What the first table says:

• 24 requests, 0 failures. Every ticket got its 201.
• Avg 2,388 ms, Med 2,200 ms. The typical ticket took about 2.2 seconds, as the tiny example predicted.
• Min 1,796 ms, Max 3,619 ms. Some tickets were twice as slow as others. The provider’s times vary, and the simulator copies that.
• req/s 0.42. Throughput: 0.42 tickets per second. One user spends about 2.4 seconds per ticket, and 1 ÷ 2.4 is about 0.42.

The second table lists percentiles, such as 95%: 3,400 ms. Latency and capacity explains them, and why Locust rounds them.

The course measured the same test five times on a quiet computer: an Apple M4 with 10 cores and 16 GB of memory, with PostgreSQL in Docker. Every number of the course team in this module comes from that computer. The result was 0.43 tickets per second (0.41 to 0.45). Your result is a statement about your computer, this workload and the simulated provider. It says nothing yet about Black Friday.

Now ask the simulator what it counted:

curl http://127.0.0.1:8300/admin/stats

{"mode":"normal","in_flight":1,"max_in_flight":1,"calls":{"classify":{"200":25},"draft_reply":{"200":24},"embed":{"200":24}},"tokens":10153,"simulated":true}

There are three calls for each ticket, all with status 200, and never more than one at a time. Classify has one more call than the others: Locust stopped while one ticket was in the middle of its work. Stop the API and the simulator with Ctrl+C when you finish.

When to use a simulated provider

Sam asks, “If the simulator is not the real provider, why trust anything that it says?”

Amira answers: “Trust it for what it copies. It copies the real times and the real quota, so it shows how our service behaves when the provider is slow and limited. It cannot show how the real provider behaves on Black Friday.”

Use a simulated dependency when you test your own service’s behaviour. Load tests, timeouts, refusals and outages are all easier to cause on a simulator than on a real provider. Use it for repeatable tests, and when there is no budget or account.

Do not use it to judge answer quality: the simulated replies are one fixed sentence. Do not use it to predict the real provider’s speed under load, because the real provider gets slower and the simulator does not.

It has a cost. Someone must record real calls, keep the calibration current, and write “simulated” next to every number. A simulator that nobody checks against the real service slowly becomes fiction. Each measurement in this course says which provider answered.

Common mistakes

Common mistake

The simulator is not running: 503 ai_unavailable

Symptom: the curl answers in a few milliseconds, with an error:

HTTP/1.1 503 Service Unavailable
date: Sat, 10 Oct 2026 13:22:09 GMT
server: uvicorn
content-length: 138
content-type: application/json
x-request-id: 2c21e9863825
server-timing: db;dur=6.9, classify;dur=1.0

{"error":{"code":"ai_unavailable","message":"The AI provider is not available. Try again later.","request_id":"2c21e9863825","fields":[]}}

The API’s terminal logs AI provider: no connection to the provider: [Errno 61] Connection refused. The number after Errno can differ on Windows and Linux.

Cause: nothing answers on port 8300. Terminal 1 is closed, or the simulator stopped.

Diagnose: curl http://127.0.0.1:8300/admin/stats fails too. The server-timing header shows that classify ended after 1 ms, without an answer.

Fix: start python -m simulator in terminal 1, and send the ticket again. The failed ticket was not saved, because the transaction rolled back. But its number is used up: the next ticket gets the number after it. A database sequence never gives the same number twice.

Common mistake

The load stops with Connection refused on port 10000

Symptom: python -m scripts.load applies the migrations and then stops with a long error. Its last line is:

azure.core.exceptions.ServiceRequestError: HTTPConnection(host='127.0.0.1', port=10000): Failed to establish a new connection: [Errno 61] Connection refused

Cause: Azurite, the object storage, is not running. The loader uploads the help desk’s 43 files to it.

Fix: run docker compose up -d db azurite, and then python -m scripts.load --reset.

Reset. python -m scripts.load --reset loads the data again, and new tickets start at T-500001 again. The simulator forgets its counts when you stop it. If your project is broken, copy the snapshot end-of-m01 and keep your .env, as the snapshots README shows. That snapshot also has the files of the next three lessons.

Knowledge check

Check your understanding

Name two things that the simulated provider copies from the real one, and two that it does not. Why does this matter when you read a load-test result?

Show the answer

It copies the times (each one is drawn from the 100 recorded times of the task) and the quota (100 chat requests per minute, then 429). It also copies the shape of a refusal and the token counts. It does not copy the slowdown under load (the real provider was about 30% slower at 40 calls in flight), real outages, or the content of the answers. So a result shows how our service behaves with a slow, limited provider. It does not predict the real provider’s speed on Black Friday, and it says nothing about answer quality.

Check your understanding

With one user, Locust measured about 0.42 tickets per second. Predict the throughput with 2 users, if nothing else limits the service. Explain your number.

Show the answer

About 0.85 tickets per second, twice as much. Each user spends about 2.4 seconds per ticket, and two users wait at the same time, so twice as many tickets finish per second. The course measured 0.85 per second with 2 users (5 passes, quota on). The latency stays about the same, because each ticket still waits for its own three calls. The next lessons show where this simple rule stops.

Check your understanding

Practical check: show that your setup works. What do the tests, one curl and one 60-second Locust run print?

Show the answer

python -m pytest ends with 123 passed, 1 deselected. The curl prints 201 Created, a server-timing header with db, classify, draft_reply and embed, and three AI runs whose model says (simulated). Locust with -u 1 -t 60s shows 0 failures, a median of about 2,200 ms and about 0.4 requests per second. Your exact numbers differ.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Capacity plan from a public traffic traceModel arrivals and bursts from an open request log, and compute the queue that a given number of workers would build.
• Queue and cache for an open-data APIPut an open-data API under load, add a cache and a background export job, and measure what each change does.

Recap

• Every new ticket now gets three AI calls: classify, draft a reply, embed. In this module’s design, they run one after the other, inside one database transaction.
• The simulated provider runs on your computer. Its times and its quota of 100 chat requests per minute come from real recorded calls. Its answers are simple, and it does not slow down under load.
• One ticket takes about 2.2 seconds, almost all of it waiting for the provider. The Server-Timing header shows each part.
• A load test sends planned requests and measures each answer. Latency is the time of one request; throughput is how many finish per second. One Locust user gave about 0.43 tickets per second on a quiet computer.
• Your project has the files of this lesson: python -m pytest prints 123 passed.

Next, you turn Omar’s “ten times the usual” into numbers: arrivals per second, bursts and what customers expect.

Next: Workload model