Hosted and self-hosted choices

What you will be able to do

Decide where Larkfield’s support model runs: on a provider’s servers, on a rented server, or on Larkfield’s own machine. Each choice gives the work of running the model to someone, and that work is part of the price.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• What a token is, from From text to tokens.
• Inference, using a trained model to produce an output, from Predicting continuations.
• The 24 test tickets: synthetic support tickets in English and French, written for this course, each with an answer key. Module 4 explains the choice of tickets. This lesson repeats the numbers it needs.

What you need

• A browser. The “Try it” box is optional and needs Ollama on a computer with at least 8 GB of memory.

Hosted models change often: your own results will differ.

Coffee at the office

An office wants coffee for its staff. It has three choices. It can buy each cup at the café next door: no machine, a price per cup, and the café changes its beans when it wants. It can rent a machine with a service contract: the company brings and repairs the machine, and the office chooses the beans. Or it can buy its own machine: no price per cup, but someone must clean it, fix it and buy the beans.

No choice is “the best”. The right one depends on how many cups the office drinks, how much the taste matters, and who has time to clean a machine.

The problem: where does the model run?

Grace wants an assistant that reads each ticket with the support policy and drafts a reply. Amira has sent the 24 test tickets to eight models: five hosted, and three on the office computer. (She ran one of the local models, Qwen3 4B, twice, with two output limits.) Two things in her log surprise Omar.

First, the hosted models answered in about 2 to 4 seconds. Gemma 3 4B ran on the office Mac mini (Apple M4, 16 GB of memory). A usual ticket took about 4 to 5 seconds, but the first one took longer. The Mac mini first had to load the model from the disk into memory: on a quiet computer, about 6.5 seconds before the work on the ticket could start.

Second, the hosted calls sent each ticket out of Larkfield, to another company’s servers. The Mac mini sent nothing anywhere.

So “which model?” is not one question. The same model can run in different places, and each place gives the work to a different team. Before Larkfield compares models, it must understand the places.

The idea: three ways to get a model

The coffee choices map to three ways to deliver a model:

• A hosted API (also called a managed API) is the café. A provider such as Anthropic, Google or OpenAI runs the model on its own servers. Your program sends the text over the internet and gets the answer back. (An API, an application programming interface, is the door through which one program asks another program for work.) You pay per token.
• Open weights are a model whose trained numbers, the weights, you can download. You can run them on your own machine: you bought the coffee machine. Gemma 3, Mistral 7B and Qwen3 are open-weight models; Ollama is a free program that runs them.
• A managed host for open weights is the rented machine. A company runs the open model that you choose, on its servers, and you pay per hour or per token.

The first question is where the ticket travels.

Where the ticket goes. With a hosted API (A) or a managed host (B), the ticket and the policy leave Larkfield. With open weights on Larkfield's own machine (C), they stay inside.

Text description of the diagram

Two areas side by side.

• Inside a dashed blue boundary, “Larkfield’s computers”: the support app, which holds the ticket and the policy, and a model server that runs open weights.
• Outside, “Outside Larkfield”: a hosted API on the provider’s servers, and a managed host that runs open weights.
• Arrow A goes from the support app to the hosted API. Arrow B goes from the support app to the managed host. Both cross the boundary.
• Arrow C goes from the support app to Larkfield’s own model server, and stays inside the boundary.

Option B surprises people: the weights are open, but the ticket still leaves. “Open weights” says what you may download. It does not say where the model runs.

Who owns what: the deployment matrix

The second question is who does the work. Grace draws one row for each job that someone must do while the assistant runs, and colours each cell by its owner.

The deployment decision matrix with operational ownership. Moving from left to right, Larkfield gains control of the model and the data, and takes on more of the work.

Text description of the diagram

A matrix with three columns and five rows. Each cell says who does the job; dark blue cells are Larkfield’s, dashed cells are shared, light grey cells are the provider’s or the host’s.

Job	Hosted API	Managed host for open weights	Open weights on your own machine
Servers and scaling	The provider’s servers; it scales for you (provider)	The host’s servers; you choose the size (shared)	Your machine; more load needs more machines (Larkfield)
Model version	The provider updates and retires it on its dates (provider)	You choose the weights; they change when you do (Larkfield)	You choose the weights; they change when you do (Larkfield)
Where the ticket goes	It leaves Larkfield; the provider’s terms apply (provider)	It leaves Larkfield; the host’s terms apply (provider)	It stays on Larkfield’s machine (Larkfield)
Security and uptime	The provider runs it; you protect the API key (shared)	The host runs servers; you set up access (shared)	You update, watch and restart it (Larkfield)
Cost	A price per token, set by the provider (provider)	A price per hour or token, set by the host (shared)	Hardware and staff time; no price per token (Larkfield)

Read the matrix one row at a time.

• Servers and scaling. If 300 tickets arrive on a sale day, the provider of a hosted API adds capacity; you notice nothing. A Mac mini answers one ticket at a time, so a queue builds.
• Model version. A provider changes and retires its models on its own dates. Your own weights change only when you change them. That is control, and also work: nobody updates them for you.
• Where the ticket goes. This row decides whether the provider’s data terms matter. The next lesson reads them.
• Security and uptime. With an API, your main job is to keep the API key secret. The API key is the password that lets your program use your account. With your own machine, you install updates, watch the disk, and restart the server after a power cut.
• Cost. A hosted API costs nothing when nobody writes in, and more when many people do. Your own machine costs the same every month, plus the time of the person who runs it.

One ticket, three ways

Follow ticket S01 (“My order L-20431 … is now 7 business days late”) through the options. The runs were.

A: hosted API, Claude Sonnet 5.5. The app sends the instructions, the policy and the ticket: 796 input tokens. About 1.8 seconds later, the reply comes back: 146 output tokens in a usual answer. Anthropic’s prices were $2 per million input tokens and $10 per million output tokens. So this usual ticket costs 796 × $2 ÷ 1,000,000 + 146 × $10 ÷ 1,000,000 = $0.0031. Some replies are longer, so the average over all 24 tickets is a little higher: about $0.0036 per ticket (Module 4 uses the average). Larkfield owns none of the servers.

C: open weights, Gemma 3 4B on the Mac mini. The model file is 3.3 GB on the disk. (The other two local models of the course are 2.5 GB for Qwen3 4B and 4.4 GB for Mistral 7B.) Ollama loads it into memory for the first request. On a quiet computer, that load took about 6.5 seconds. Then each ticket took about 4 to 5 seconds. (Resource trade-offs measures these times.) There is no price per ticket, but Larkfield bought the Mac mini, and Amira keeps it running.

B: a managed host for Gemma 3 would answer from the host’s servers, at the host’s price. The course did not run one, so it gives no numbers for it.

The running example: six of the models, two places

This chart shows six of Amira’s eight models. Gemini 3.1 Pro and Qwen3 4B, which think for a long time before they answer, are in Module 4’s full table.

Predict first: the hosted models run on large data-centre computers, and the local ones on one small desktop computer. How much slower will the local models be?

• Hosted API
• Local on a Mac mini M4

The median time per ticket on the 24 test tickets. Hosted calls went from Toronto to the provider and back; local calls ran on a Mac mini M4 with 16 GB. The first ticket on each local model was slower, because the model loaded first. One slow ticket hardly changes a median.

Text description of the chart

A column chart of the median seconds per ticket on the 24 test tickets. Hosted APIs (blue): Claude Haiku 5.5 2.2 s, Claude Sonnet 5.5 1.8 s, Claude Opus 5.5 3.9 s, Gemini 3.5 Flash 4.4 s. Local on a Mac mini M4 with Ollama (second colour): Gemma 3 4B 4.8 s, Mistral 7B 4.2 s. The first ticket on each local model took longer, because the model was loaded first.

These are the medians of the recorded runs on all 24 tickets. (On a quiet computer, with the model already loaded, one Gemma ticket took about 4.0 seconds.) The local models are slower, but not by much: 4 to 5 seconds against 2 to 4. For a reply that a person checks before it is sent, all of these times are acceptable.

The bigger difference is quality. In Module 4, a person reads and scores every reply. Claude Sonnet 5.5 wrote 22 replies of 24 that a person can send after a light check. Gemma 3 4B wrote 6, and Mistral 7B 12 (the full table). A small model that fits on a desktop does not do the same job as a large hosted one. Remember also that models change fast, and these numbers are a snapshot of one day; the method of comparing is what lasts.

“Why not just…”

Tomás: “Why not just run everything on the Mac mini? It is free.”

The matrix answers him. The Mac mini has no price per token, but it is not free. Somebody installs updates, restarts it, and plans for the day it breaks. On this task, the small models that fit in its 16 GB also made more mistakes. Larkfield gets about 45 tickets a day, about 1,350 a month. At about $0.0036 each, Sonnet costs about $4.90 a month for them. That is less than an hour of Amira’s time.

Omar asks the opposite: “Then why not just use the API for everything?” Because the ticket leaves Larkfield, under the provider’s terms. And the provider decides when the model changes. For some data (health notes, employee files), a company may not send it out at all. Then only option C remains.

When to use each option

• Hosted API: you want the strongest models, little operating work, and a cost that follows your volume. It does not fit when the data must not leave, or when you need the same model version for years.
• Managed host for open weights: you want to choose and keep a fixed open model, without running servers. The data still leaves, so the host’s terms matter.
• Open weights on your own machine: the data must stay inside. Or the volume is so high that a price per token costs more than hardware and staff. You need someone who can run a server, and a model small enough for your hardware. The last lesson of this module measures what “small enough” means.

Try it: run a model on your own computer (optional)

Only do this if you have Ollama installed and at least 8 GB of free disk space and memory. The commands were tested with Ollama 0.40.1 on macOS.

1. Open a terminal.
2. Download the model (about 3.3 GB): ollama pull gemma3:4b
3. Ask one question: ollama run gemma3:4b "Say hello in French."
4. While the model is still loaded, show what is in memory: ollama ps

The output of step 4 on the test computer:

NAME         ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
gemma3:4b    c30276f7dffe    2.9 GB    100% GPU     4096       llamacpp    9 minutes from now

SIZE is the memory that the loaded model uses: 2.9 GB. UNTIL says how long Ollama keeps the model in memory after the last request: 5 minutes by default, longer on the test computer. After that, the next request loads the model again, and is slow again. To remove the model, run ollama rm gemma3:4b.

Guided practice: who does the job?

Grace lists eight jobs that come with the assistant. For each job, decide who does it with a hosted API, and who does it with open weights on the Mac mini.

1. Restart the model server after a power cut.
2. Keep the API key secret.
3. Choose the day when the old model version stops working.
4. Handle three times more tickets during a sale.
5. Install security updates on the computer that runs the model.
6. Decide where the tickets are stored.
7. Pay for each ticket.
8. Test the assistant again when the model changes.

Show the answers

Job	Hosted API	Open weights on the Mac mini
1. Restart after a power cut	The provider	Larkfield
2. Keep the API key secret	Larkfield	No API key to protect (but Larkfield protects the machine)
3. Choose when the old version stops	The provider	Larkfield
4. Handle a sale day	The provider	Larkfield (more machines, or a queue)
5. Install security updates	The provider	Larkfield
6. Decide where the tickets are stored	The provider, under its terms	Larkfield
7. Pay for each ticket	Larkfield pays the provider	No price per ticket
8. Test again when the model changes	Larkfield, on the provider’s dates	Larkfield, on its own dates

Job 8 is Larkfield’s in both columns. Whoever runs the model, only Larkfield can check that the replies still follow its policy.

Your turn: move Gemma to a managed host

The Mac mini is old, and Amira is busy. Larkfield keeps Gemma 3 4B but moves it to a managed host. Use the matrix. Which of the eight jobs move away from Larkfield, which stay, and what new question does the move create?

Show a model answer

Jobs 1, 4 and 5 move to the host: it restarts the servers, adds capacity and installs updates. Job 3 stays with Larkfield, because Larkfield chooses the weights, so the model does not change on the host’s dates. Job 2 comes back: Larkfield now has an API key or an account to protect. Job 7 comes back too: the host charges per hour or per token. Job 8 stays with Larkfield. The new question is job 6: the tickets now leave Larkfield, so the host’s data terms must be read, as for a hosted API.

Common mistake

The provider changed the model, and the old request broke

Symptom: Amira reuses a request that worked with older Claude models. It asks for temperature 0, the setting from Predicting continuations that makes the answers as repeatable as possible. With Claude Sonnet 5.5, the API refuses it:

HTTP 400
{"type":"error","error":{"type":"invalid_request_error","message":"`temperature` is deprecated for this model."}, ...}

Cause: with a hosted API, the provider owns the model version (the second row of the matrix). Newer Claude models do not accept the temperature setting. The request did not change; the model behind it did.

Fix: remove the setting, and run the 24 test tickets again before the new model goes live. Write the exact model name in every record, such as “Claude Sonnet 5.5 on 2026-10-08”. Then you know which model a result belongs to.

Consequence in AI work: with a hosted API, you do not own the schedule of change. Plan for it: Module 5 covers model retirement and fallbacks. With open weights on your own machine, this failure does not happen by surprise, because nothing changes until you change it.

Knowledge check

Check your understanding

A clinic wants a model to summarise patient notes. The notes must not leave the clinic's network. Which of the three options remain, and which jobs does the clinic take on?

Show the answer

Only open weights on the clinic’s own machines remain. A hosted API and a managed host both send the notes out of the network (arrows A and B). With its own machines, the clinic owns every row of the matrix: the servers and their capacity, the model version and its updates, security and uptime, and the cost of hardware and staff. It also needs a model small enough for its hardware, and its own test that the model summarises its notes well.

Check your understanding

Predict: Larkfield's tickets grow from 45 a day to 45,000 a day. What happens to the monthly cost of the hosted API, and to the cost of the Mac mini option?

Show the answer

The hosted API cost grows with the volume: 1,000 times more tickets cost about 1,000 times more, about $4,900 a month for Sonnet at about $0.0036 per ticket. The Mac mini has no price per ticket, but one machine at about 5 seconds per ticket can answer only about 17,000 tickets a day, even without a break. Larkfield needs several machines, and more staff time to run them. At high volume, the comparison can change: then a fixed cost for hardware can be cheaper than a price per token. Larkfield must calculate both with real numbers.

Recap

• A model can reach you in three ways: a hosted API (the provider runs it), a managed host for open weights (a host runs the model you choose), or open weights on your own machine.
• Open weights are downloadable model weights. Open does not mean local: on a managed host, the ticket still leaves.
• The deployment matrix shows who owns each job: servers and scaling, the model version, where the ticket goes, security and uptime, and cost. Control and work move together.
• In the recorded runs, the hosted models took a median of 1.8 to 4.4 seconds per ticket. Gemma 3 4B and Mistral 7B on a Mac mini took 4.2 to 4.8 seconds. The first ticket took longer, while the model loaded.
• With a hosted API, the provider changes the model on its dates; one real example is the 400 error for temperature.

Next, the documents that decide what you may do: the model’s licence and the provider’s data terms.

Next: Licensing and data terms