CPU, GPU and managed endpoints

What you will be able to do

Decide where the escalation model runs. Choose a CPU or a GPU, and a place: a container platform, a managed endpoint or a job on a schedule. Decide from measured numbers, and write the decision down so that the team can check it later.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Start-up work and request work, and the memory of one worker, from Load and initialize: One copy for each worker.
• The service and its Server-Timing header, from Connect the API: routes.py.
• The serving patterns, from Batch inference and Online inference.
• What a container platform charges for, scale to zero and the free grant, from Cost awareness: What can charge in Docker and Cloud Essentials for AI Developers.

What you need on your computer

• The project ~/projects/escalation-service at the end of Module 3, with its .venv. If it is missing or broken, copy the snapshot end-of-m03, as in Use a snapshot.
• No cloud account. The prices come from the Azure Retail Prices API: region canadacentral, US dollars, pay-as-you-go, Linux. Prices change: check the current prices before you choose.

Tested with Python 3.14.6 and Docker Desktop (Docker Engine 29.6.1) on macOS, Apple silicon. The Windows and Linux steps are not tested.

A kitchen for one dish

Nobody rents a restaurant kitchen with twenty cooks to make one piece of toast. The size of the kitchen follows the dish and the number of guests. A large kitchen costs money every hour, even when nobody orders.

A model is the same. Before you choose where it runs, measure two things: how much work one prediction needs, and how many predictions arrive. Then choose the smallest place that does the job, and write down why.

The question: does AI need a GPU?

Tomás has read about AI servers. “Every AI service runs on GPUs. Let us rent one for the escalation model.”

Amira does not argue. She measures. The service already says where the time goes, in the Server-Timing header from Connect the API. She starts the service with fastapi dev, and in a second terminal she scores the example ticket three times. She shows only the headers (-D -) and keeps the third answer, because the first request after a start can be slower:

macOS and Linux

Terminal, in ~/projects/escalation-service

curl -s -D - -o /dev/null http://127.0.0.1:8000/v1/score -H "Content-Type: application/json" -d '{"ticket_id":"T-120001","channel":"email","team":"payment","segment":"home","region":"west","priority":1,"order_value":89.5,"word_count":112,"customer_tenure_days":400,"prior_tickets_90d":2,"created_hour":10,"prior_escalations_90d":1}'

Windows

Open http://127.0.0.1:8000/docs, choose POST /v1/score, Try it out, and Execute with the example ticket. The Response headers box shows server-timing.

The output (shortened):

HTTP/1.1 200 OK
date: Thu, 08 Oct 2026 15:05:35 GMT
server: uvicorn
content-length: 119
content-type: application/json
server-timing: frame;dur=0.66, prepare;dur=1.38, model;dur=0.05
x-request-id: 1da326860017

Read the three steps. frame builds a table from the ticket. prepare runs the pipeline’s preprocessing: one-hot encoding, filling missing numbers and scaling. model is the logistic regression itself: 0.05 ms. The whole request takes about 3 ms, so the model is less than 2% of it. Preprocessing takes 28 times as long as the model. The rest of the 3 ms is HTTP, JSON and the contract checks.

These values come from one run. In Connect the API, another run of the same request showed frame;dur=1.07, prepare;dur=1.81, model;dur=0.07. The values change a little from run to run, but the proportions stay the same: the model step is always the smallest.

CPU and GPU

A computer has a CPU (central processing unit). It has a few cores. Each core is fast and can do any kind of work: read a file, check a ticket, build a table. Think of a few skilled cooks.

A GPU (graphics processing unit) first existed to draw images. It has thousands of small cores that do one simple thing at the same time: multiply numbers and add them. Think of a thousand helpers who can only chop, but all at once. A GPU is fast only when the work is a very large number of the same multiplications.

How many does one escalation score need? The pipeline turns a ticket into 28 numbers, and the model has one weight for each. So the score is 28 multiplications, 28 additions and one last step that turns the sum into a number from 0 to 1. A CPU core does that in much less than a microsecond.

A neural network for images is different. A common image model, ResNet-50, does about 4 billion multiplications and additions for one photo, and large language models do more. For that work, thousands of helpers at once save real time. You meet these models in Deep Learning and Transformers Explained and Fine-Tuning and Adapting AI Models.

Now the worked example for Larkfield. Suppose that a GPU made the model step take no time at all. The request goes from about 3.0 ms to about 2.95 ms, and nobody at the help desk can see the difference. The GPU also adds work: the data must go from the CPU’s memory to the GPU’s memory and back. And the GPU costs more every hour, as the next sections show.

So the rule: use a GPU when one prediction needs millions or billions of the same multiplications. Neural networks for images, sound or text are examples. Use a CPU for small tabular models like this one, where the time goes to preprocessing and the web server.

The model is small, the program is not

The model file is small: model.joblib is 5,354 bytes, and the whole bundle is 45.1 kB. The program that runs it is not small. Python with scikit-learn, pandas, NumPy, SciPy and FastAPI uses about 190 MiB of memory in each worker process. The next lesson measures it in the container. Almost all of it is the libraries, not the model. In the next lesson, the container image shows the same on disk: 461 MB of libraries for a 45 kB bundle.

So for this model, the libraries set the size of the machine: about 200 MiB of memory for each worker, plus room. A large neural network is the other way round. The model itself can need many gigabytes, and the machine follows the model.

Three places to run a model

There are many products, but three kinds of place. They differ in operational effort: the work that your team does to keep the service running. Less effort for you means that a provider does the work, and you pay for it.

• A job on a schedule. A program starts, scores a file and stops, for example every night: the batch scoring of Batch inference. You pay only while it runs. Nobody waits for an answer, so a slow start does not matter.
• A container on a container platform. You build the image, and the platform runs it, as in the Docker course with Azure Container Apps. You choose the size, the number of copies and the probes. The platform can scale to zero: stop every copy when no requests come. The first request after that waits for a cold start: the time until a new copy is ready.
• A managed online endpoint. You give the provider the model and its environment, or your own container image. The provider makes the machines, updates their operating system, scales them, and adds monitoring and traffic rules. Azure Machine Learning managed online endpoints, Amazon SageMaker real-time endpoints and Google Vertex AI online prediction are examples. On Azure, you pay for each machine for every hour that the deployment exists, and an online endpoint cannot scale to zero.

Who does the work in each place. Filled layers are your team's work; light layers are the provider's. Less work for you usually costs more every hour.

Text description of the diagram

Three columns show the same five layers, from the top: the model bundle; the scoring code and the API; the image and its packages; scaling and traffic; the machines and their operating system.

A job on a schedule: your team owns the bundle, the code and the image. The platform starts the job at the time you choose and owns the machines. You pay only while the job runs.

A container on a container platform: your team owns the bundle, the code and the image. You set the size, the minimum and maximum number of replicas, and the probes; the platform does the scaling and owns the machines. You pay while a replica runs. It can scale to zero, with a cold start for the next request.

A managed online endpoint: your team owns the bundle and the scoring code. The provider can supply the image (a curated environment) or take yours, and it owns scaling, traffic and the machines. It adds monitoring, a traffic split between two deployments and traffic mirroring. You pay for every machine for every hour, and it cannot scale to zero.

Omar asks the question that decides most choices: “What does each one cost a month?”

What each place costs

Amira compares 30 days (720 hours) for each place. She uses the prices from the Azure Retail Prices API (canadacentral, US dollars, pay-as-you-go):

• Azure Container Apps: US$0.000034 per vCPU-second while active and US$0.000004 while idle, and US$0.000004 per GiB-second. Each month, the first 180,000 vCPU-seconds and 360,000 GiB-seconds are free (Cost awareness: What can charge in the Docker course).
• A managed online endpoint on Azure Machine Learning has no extra charge of its own: you pay for its virtual machines. Standard_D2as_v4 (2 vCPUs, 8 GiB, no GPU) costs US$0.107 an hour. Standard_NC4as_T4_v3 (4 vCPUs and one NVIDIA T4 GPU) costs US$0.584 an hour.

The worked example for the container app: one replica with 0.5 vCPU and 1 GiB runs all month, always active. This is the worst case.

Step	vCPU	Memory
Use: size × 2,592,000 seconds	0.5 × 2,592,000 = 1,296,000 vCPU-s	1 × 2,592,000 = 2,592,000 GiB-s
Minus the free grant	1,296,000 − 180,000 = 1,116,000	2,592,000 − 360,000 = 2,232,000
Times the price	1,116,000 × 0.000034 = US$37.94	2,232,000 × 0.000004 = US$8.93

Together: about US$46.87 a month. If the replica is idle most of the time, the vCPU price falls to the idle price, and the month costs about US$13.39. The endpoint is simpler: 720 hours × US$0.107 = US$77.04 for one CPU machine, and US$420.48 for one GPU machine. Microsoft recommends at least three instances for production, so that the others serve while Azure updates one. Three CPU machines cost US$231.12.

Compute for 30 days at pay-as-you-go prices (canadacentral, pay-as-you-go). The registry, storage and logs come on top. Check the current prices before you choose.

Text description of the chart

A column chart of the compute cost for 30 days, in US dollars. “Job”: a nightly job that runs for a few seconds, about 0, inside the free grant. “Container”: a container app with 0.5 vCPU and 1 GiB, one replica, always active, 47. “CPU ×1”: a managed online endpoint with one Standard_D2as_v4 machine, 77. “CPU ×3”: the same endpoint with three machines, as recommended for production, 231. “GPU ×1”: a managed endpoint with one Standard_NC4as_T4_v3 GPU machine, 420.

The GPU machine costs about 5.5 times the CPU machine. For Larkfield, it would save about 0.05 ms per request.

Larkfield’s choice

Now the load. Larkfield gets about 45 new tickets a day: 1,400 in July, mostly from 07:00 to 22:00. The service scores each ticket while an agent opens it, so somebody waits for the answer: online serving. In Online inference, Grace gave the service 50 ms of her 300 ms ticket page: p95 under 50 ms, network included. Forty-five tickets a day is about three in the busiest hour. One worker answers a request in about 3 ms.

Tomás: “So we need almost nothing. Why not scale to zero and pay nothing?” Because an agent waits for the first score after each quiet period, and most periods are quiet. In the Docker course, the first request after scale to zero waited 18 seconds for the 73 MB ticket-api image. The course team deployed this service’s image to Azure Container Apps, with 0.5 vCPU and 1 GiB. The image is 166 MB to download. The first request from zero replicas took 26.4 seconds (live). With 1 vCPU it still took 23.9 seconds. That is 500 times the budget. Without a score, the page opens without one, and the nightly batch scores the ticket later.

Omar: “And the managed endpoint? It does more for us.” It does, and the traffic split and the mirroring help with releases. But it costs at least US$77 a month for one machine, which is far larger than this model needs. And it never scales to zero. For a 5 kB model and 45 requests a day, that is the restaurant kitchen for one piece of toast. If Larkfield later serves several models, or a large one, the endpoint can become the better choice.

So Amira writes the decision: CPU only; one container on Azure Container Apps, as in the Docker course; 0.5 vCPU and 1 GiB; one worker; at least one replica in working hours. A short document that records a decision, its reasons and the numbers behind them is a decision record. It lets the team check the decision later, when a number changes.

Worked example: docs/infrastructure.md

Amira makes the folder docs in the project, and writes the file:

docs/infrastructure.md

# Infrastructure decision: escalation-service

Where Larkfield's escalation model runs, and why. Written by Amira
(C17 Module 4), with Priya (model), Grace (help desk) and Omar (product).
Change this file when one of its numbers changes.

## What the model needs

- Model: logistic regression with preprocessing, 28 weights. `model.joblib` is 5,354 bytes; the bundle is 45.1 kB.
- Time for one warm request on a laptop (Server-Timing): frame 0.66 ms, prepare 1.38 ms, model 0.05 ms. The whole request takes about 3 ms. The model is less than 2% of it.
- Memory: about 190 MiB for each worker (Python, scikit-learn, pandas, NumPy, SciPy, FastAPI). The libraries, not the model, set the size.
- A GPU would make only the model step faster (0.05 ms). Not needed.

## The load

- About 45 tickets a day (July: 1,400 tickets in 31 days), mostly from 07:00 to 22:00. The busiest hour had about 3 tickets on an average day.
- Online: an agent waits for the score when the ticket opens. Budget: p95 under 50 ms, network included (Online inference). Without an answer, the page opens without a score, and the nightly batch scores the ticket.

## The options (30 days, compute only)

Prices from the Azure Retail Prices API, canadacentral, USD, pay-as-you-go. Check again before deploying.

| Option | Cost for 30 days | Notes |
|---|---|---|
| Job on a schedule (batch) | about US$0 (free grant) | Nobody waits. Does not serve the help desk. Already used for the nightly file. |
| Azure Container Apps, 0.5 vCPU / 1 GiB, 1 replica | US$13.39 idle to US$46.87 always active | Scale to zero is possible, but the first request then waits for a cold start. |
| Azure ML managed online endpoint, 1 x Standard_D2as_v4 | US$77.04 | No scale to zero. Microsoft recommends 3 instances for production: US$231.12. |
| Managed endpoint with a GPU, 1 x Standard_NC4as_T4_v3 | US$420.48 | Saves about 0.05 ms per request. |

## The decision

- CPU only.
- One container on Azure Container Apps, as in C18: 0.5 vCPU and 1 GiB, one worker (`WORKERS=1`).
- At least one replica from 07:00 to 22:00, so that agents do not wait for a cold start.
- Revisit if: the load grows past what one worker serves inside the budget, the model becomes a neural network, or Larkfield serves several models (then compare a managed endpoint again).

## Check before we deploy

- [ ] Memory per worker in a container with a limit (Container resource settings).
- [ ] Cold start at 0.5 CPU (Container resource settings).
- [ ] p50, p95 and throughput with a fixed workload (Benchmark the service).
- [ ] The cold start and the latency in the cloud, with the network (to measure after the first deployment).

Every number has a source: a measurement, a price page with its date, or Grace’s load. The section Check before we deploy lists what is not measured yet. The next two lessons measure it, and the last one adds the results.

Guided practice: write the decision record

1. In ~/projects/escalation-service, with .venv active, start the service with fastapi dev.
2. In a second terminal, send the example ticket three times, as in The question. Write down the three server-timing values of the last answer.
3. Stop the service with Ctrl+C.
4. Make the folder docs and the file docs/infrastructure.md.
5. Write the sections What the model needs and The load, with your three times and the budget from Online inference.
6. Write The options, with a cost for 30 days for each. Use today’s prices from the official price pages, and write the date.
7. Write The decision and Check before we deploy.

Show the expected server-timing values of step 2

Your numbers are different, but their sizes are like these: frame below 1 ms, prepare 1 to 2 ms, model below 0.1 ms. On the course’s computer, prepare was 20 to 30 times model. If model is the largest of the three, check that you sent the example ticket. Then look at a later answer: the first request after a start can be slower.

Your turn: a model that needs a GPU

Priya has a new idea: read the text of each ticket with a neural network for text. The model is 400 MB. One ticket takes about 300 ms on one CPU and about 15 ms on a T4 GPU. These two times are assumptions for this exercise, not measurements. The load stays 45 tickets a day, and the agent still waits.

Change one section of your decision record: The decision. Is a GPU worth it now? What do you check first?

Show one answer

First compare both times with the budget from Online inference: p95 under 50 ms. 300 ms on a CPU does not fit, even before the network. 15 ms on a GPU fits. Now the GPU is what makes the online score possible, and its cost is the price of the feature. The other choice is to keep the CPU and score the text model only in the nightly batch. Nobody waits there, and 45 tickets take about 14 seconds. Before you decide, measure both times with the real model and real ticket texts. Also check memory: a 400 MB model in each worker is more than twice what a worker uses today.

Common mistake

Scale to zero for a service that people wait for

Symptom: the service passes every test. On Monday morning, the first agent opens a ticket and waits many seconds for the score. The agents after that get their scores at once.

Cause: the platform scaled the service to zero in the night, as its settings said. The first request then waited for a cold start. The platform finds a machine, downloads the image and starts the container. Then the service imports its libraries and loads the bundle. For this service on Azure Container Apps it took 26.4 seconds; in the Docker course it took 18 seconds for a smaller image.

Fix: choose the minimum number of replicas from who waits. For an online service in working hours, keep at least one replica, and accept the cost: here about US$13 to US$47 a month. For a nightly job, scale to zero, because nobody waits.

For a real AI service: the cold start grows with the image and the start-up work. The platform must download a large model and load it into memory before the first answer, which can take minutes. Scale to zero saves money only where a slow first answer is acceptable.

Check your understanding

Tomás says: the escalation service is slow, so let us move it to a GPU. Use the server-timing numbers to explain why a GPU would not help.

Show the answer

The model step is about 0.05 ms of a request of about 3 ms. The rest is HTTP, JSON, the contract checks and the preprocessing, and a GPU does not make them faster. Even if the GPU made the model step free, the request would go from about 3.0 to 2.95 ms. A GPU helps when one prediction needs millions or billions of multiplications, as in neural networks. This model needs 28.

Check your understanding

A team re-scores 2 million insurance policies once a month, and nobody waits for the result. Which of the three places fits best, and what does the team pay for?

Show the answer

A job on a schedule. It starts once a month, scores the file and stops, so the team pays only while it runs. A container that waits for requests, or a managed endpoint, would charge for many hours to serve one run a month. A slow start does not matter, because nobody waits.

Check your understanding

What does a managed online endpoint do for you that a container on a container platform does not, and how does Azure charge for it?

Show the answer

The provider makes and updates the machines, scales them, and adds monitoring, a traffic split between deployments and traffic mirroring. You can give it the model and a scoring script instead of a whole image. Azure charges for every virtual machine of the deployment for every hour. There is at least one machine (Microsoft recommends three for production), and it cannot scale to zero. One Standard_D2as_v4 machine cost about US$77 for 30 days at the prices in this lesson.

Recap

• Measure before you choose. For Larkfield, the model step is 0.05 ms of about 3 ms, and the libraries, not the 5 kB model, set the memory.
• A GPU does a very large number of the same multiplications at once. It pays off for neural networks, not for a logistic regression with 28 weights.
• Three places: a job on a schedule (pay while it runs), a container on a container platform (pay while a replica runs; scale to zero means a cold start) and a managed endpoint (less work for your team; on Azure, every machine charges every hour).
• docs/infrastructure.md records the decision, each number with its source, and what to check before you deploy.

Next, you put the service in a container and give it a size: CPU, memory and workers.

Next: Container resource settings