Choose vendors and models as a leader

What you will be able to do

A vendor’s proposal shows what the vendor wants you to see. You learn to ask four questions at decision level: about data, evidence, cost and exit. Then you apply them when your team chooses a language model.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• Build, buy or partner, and vendor lock-in, from Build, buy or partner.

What you need

• The two vendor proposals for Larkfield’s support reply assistant. The vendors, their prices and their terms are constructed: made up for this course.
• The vendor comparison template. Its section 3 lists the questions of this lesson.

Buying a used car

A seller shows you a shiny car. “It uses only 5 litres of fuel per 100 kilometres,” she says. “Everybody who drives it loves it.”

You ask four questions before you pay. Whose numbers are these? The fuel figure may come from a flat test track, not from your hilly road. What does it cost each year? Insurance, repairs and fuel can cost more than the car. What do I sign? A loan with a long term ties you to the car. How do I get out? Can you sell it again, and with which papers?

A good buyer also asks for a test drive on their own road. The seller’s words describe the seller’s best day. Your own road shows your normal day.

The problem: a number that answers another question

Omar has two proposals for the support reply assistant. ReplyRight and DraftDesk are vendors made up for this course; their offers are constructed. ReplyRight’s proposal has one big number: “92% of drafts sent without edits.” It looks like strong evidence.

Read the line under it. The number comes from “a demo on 20 tickets that the vendor chose, from another shop”. That sentence has three problems:

• 20 tickets is a small test. Larkfield’s own test used 200 tickets (102 development and 98 holdout), and even a test on about 100 tickets has noise.
• The vendor chose the tickets. A seller shows the best roads.
• They came from another shop. Nobody knows if that shop had safety tickets, French tickets or questions that its policy does not answer.

And “sent without edits” is not the same as “right”. An agent who trusts a draft sends it unchanged, even when it is wrong. Larkfield’s own evidence has an example. Prompt v3 told a customer to use a “saved-card section” that does not exist. The reply was fluent and polite, so it was easy to send.

A vendor’s tool can also report its own certainty. That is not evidence either. You saw why in What AI does well and badly: 9 of the vision model’s 10 wrong values had the highest confidence, 1.0. One more recorded fact from the document AI course matters to a buyer. The lowest confidence on any value was 0.96, so a rule “check every value below 0.95” checked nothing. Three of 54 documents with a wrong or missing value went through.

The idea: four questions before you score

At decision level, a leader asks every vendor the same four questions. The template has them in its section 3, with two more.

1. Data. What do you keep from our data (inputs, outputs, logs), where, for how long? Do you use it to train or improve any model?
2. Evidence. Will you run a trial on our own examples, scored by us, before we sign?
3. Cost. What does the service cost at our volume, low, base and high, with every fee, for the whole contract?
4. Exit. How do we leave? What can we export, in which format, with how much notice?

Some answers are must-haves: a “no” removes the vendor, whatever its other scores. Write your must-haves before you read the proposals. Larkfield’s tickets have customers’ names, orders and addresses, so “no training on our tickets” is a must-have.

The four vendor questions. A failed must-have removes a vendor before any scoring. A vendor that passes is scored against the other options, including what you have today.

Text description of the diagram

Each proposal goes through four questions in order. First, data: what does the vendor keep, and does it train on it? A failed must-have removes the vendor. Second, evidence: are there results on our own examples, or a trial before we sign? A demo alone, with no trial, removes the vendor. Third, cost: what does it cost at our volume, with every fee, for the whole contract? Fourth, exit: what can we take with us when we leave? A failed must-have removes the vendor. A vendor that passes all four is scored against the other options, including Larkfield’s own assistant.

A tiny example: cost at our volume

Question 3 often changes the picture. A price “per 1,000 tickets” or “per agent” means nothing until you put in your own volume. Here is the arithmetic for Larkfield, with every value visible.

Larkfield gets about 822 tickets a day (an assumption, from the scaling course’s synthetic day). That is 822 × 365 ÷ 12 = about 25,003 tickets a month.

• ReplyRight (constructed): US$39 per agent per month × 12 agents = US$468 a month, plus US$2,500 to set up. The contract runs 24 months: 468 × 24 + 2,500 = US$13,732 that Larkfield must pay.
• DraftDesk (constructed): US$4.00 per 1,000 tickets × 25,003 = US$100.01 a month. But the minimum is US$300 a month, so Larkfield pays US$300. Only a third of the bill is for real use. The usage would reach the minimum at 75,000 tickets a month. Set-up: US$6,000. The contract runs 12 months: US$9,600.
• Our own assistant (prompt v2): the model fees are US$0.341 per 1,000 tickets, recorded in the MLOps course’s practice week. For 25,003 tickets, that is US$8.53 a month. The team’s upkeep adds 20 hours a month × US$60 = US$1,200 a month (both assumptions).

To compare like with like, take the first 12 months of each, with set-up:

The first 12 months at Larkfield's volume. ReplyRight US$8,116 and DraftDesk US$9,600 (constructed prices). Our own assistant US$14,502: US$102 of recorded model fees and US$14,400 of assumed upkeep. ReplyRight's contract runs 24 months, so Larkfield would owe US$13,732 in all.

Text description of the chart

A column chart with three columns, in US$ for the first 12 months, set-up included. ReplyRight: 8,116 (468 a month for 12 months, plus 2,500). DraftDesk: 9,600 (the 300 minimum for 12 months, plus 6,000). Our own assistant: 14,502 (8.53 a month of model fees and 1,200 a month of assumed upkeep). The vendors’ prices are constructed for the course; the model fees are recorded; the upkeep is an assumption.

Two things stand out. First, our own assistant has the smallest model fees, but the largest total, because of the team’s time. Second, DraftDesk’s price per ticket is 11.7 times our model fees, yet its minimum decides the bill at our volume. The chart also leaves out one cost: a bought product still needs some of Larkfield’s time, to connect it and to check its results. Count that time when you score.

The same questions for a model

Your team may ask you to approve a change of language model, for example “the strong model scores higher on a public ranking”. Ask the same four questions. Data and exit come from the model provider’s terms. Cost and evidence need your own numbers.

The system design and cost course measured this for Larkfield. It asked a small model and a strong model the same 67 of Grace’s questions, twice. This is recorded:

	Small model	Strong model
Correct answers, first and second run	58 and 58 of 67	57 and 58 of 67
Model cost per 1,000 questions	US$0.17	US$3.20

The strong model cost 19 times more and answered no more questions correctly. A ranking tests other people’s questions; this test used Larkfield’s own. As a leader, you do not need to run the comparison. You ask: which examples, how many, how many runs, and what does it cost at our volume? The LLMs course teaches the comparison itself, and its lesson on licensing and data terms explains the provider’s terms.

Ask one more question of every model and every vendor: what happens when the model changes? Providers update models. Ask how they tell you, and whether you can test the new version on your examples first.

Discuss: sign ReplyRight next month?

Omar wants a decision this week. Nadia has the two proposals and the team’s views.

• Omar: “ReplyRight is ready next month. It has the best result, 92%, and the lowest first-year cost.”
• Grace: “Their 20 tickets came from another shop. Did any of them need a person? Ours do, every day.”
• Mei, the data engineer: “They keep our tickets for 18 months and train on them unless we buy the Enterprise plan. And we cannot export the reply history if we leave.”
• Kwame: “The contract is 24 months. If it fails in month 3, we still pay.”

Before you read on, answer these:

1. Which of the four questions does ReplyRight’s proposal answer well, and which badly?
2. Which answers fail a must-have that Larkfield wrote before reading?
3. What would ReplyRight need to offer before Nadia could say yes?

The course’s answer. Do not sign ReplyRight as offered. It fails two must-haves. On data, it trains on customers’ tickets unless Larkfield buys a plan whose price is not in the proposal. On evidence, it offers no trial on Larkfield’s tickets, and its 92% answers a different question on another shop’s tickets. Its exit is also weak: no reply history to take away, after a 24-month contract. The lower first-year cost does not matter until the must-haves pass.

What would change the answer: ReplyRight could agree, in writing, to no training on Larkfield’s tickets, with the price of that plan. It could also run a trial on Larkfield’s own tickets, scored by Larkfield against today’s assistant. If it then passed the same tests as prompt v2, Nadia would score it with the other options. The next lesson shows how to run that trial fairly.

When to ask, and what it costs

Ask the four questions of every vendor and every model change, before anyone signs or switches. The cost is small: a few hours to write the questions, and a few weeks to wait for answers. A vendor that will not answer, or will not run a trial on your examples, has answered anyway. At this level, you do not compare models’ internals; you check that someone measured them on your examples.

Common mistake

The vendor's best number decides

Symptom: a team signs a product because its proposal says “92% sent without edits”. Some weeks later, agents send drafts with steps that the policy does not have.

Cause: the number came from a demo on examples that the vendor chose. It measured the agents’ trust in the drafts, not whether the drafts were right.

Fix: ask for evidence on your own examples, scored by your own rules, before you sign. Write the must-haves first, so a big number cannot make you forget them.

See it end to endBuild, buy or partner for invoice extractionCase study

In the case study, one vendor’s confidence threshold sends almost nothing to a clerk, and 3 of 54 documents reach the books with a wrong or missing value. Its claims and its data terms are read against results on Larkfield’s own documents.

Knowledge check

Check your understanding

Name the four questions to ask every vendor at decision level, and say which of them Larkfield made a must-have, and why.

Show the answer

Data: what the vendor keeps, where, for how long, and whether it trains on it. Evidence: results on our own examples, or a trial before we sign. Cost: the price at our volume, low, base and high, with every fee, for the whole contract. Exit: what we can export, in which format, with how much notice. Larkfield made “no training on our tickets” a must-have, because the tickets contain customers’ names, orders and addresses. A trial on its own tickets was the second must-have.

Check your understanding

DraftDesk charges US$4.00 per 1,000 tickets with a minimum of US$300 a month. Larkfield has about 25,003 tickets a month. What does it pay a month, and what share of the bill is real use?

Show the answer

Use: 25,003 × 4.00 ÷ 1,000 = US$100.01. That is below the minimum, so Larkfield pays US$300 a month. Real use is about a third of the bill. The usage would reach the minimum only at 75,000 tickets a month. A price per unit means nothing until you put in your own volume.

Check your understanding

Predict: a vendor's tool shows a confidence of 1.0 on every value it reads from your documents. Can you skip checking those values? Use the document AI course's recorded result.

Show the answer

No. In the document AI course, 9 of the vision model’s 10 wrong values had the highest possible confidence (1.0). Its lowest confidence on any value was 0.96, so a rule that checked values below 0.95 checked nothing. Three of 54 documents with a wrong or missing value went through. A tool’s confidence is its own opinion, not evidence. Only a test on your own documents shows how often it is wrong.

Recap

• Ask every vendor about data (kept? trained on?), evidence on your examples, cost at your volume and exit. Write the must-haves before you read.
• ReplyRight’s “92% sent without edits” came from 20 tickets that it chose, from another shop (constructed). It fails Larkfield’s two must-haves.
• At Larkfield’s volume, DraftDesk’s minimum decides its bill: US$300, of which a third is real use. Our own assistant’s model fees are US$8.53 a month, but its assumed upkeep is US$1,200.
• For models, the same questions apply. On 67 of Grace’s questions, the strong model cost 19 times more and was no more correct (recorded).
• A tool’s own confidence is not evidence: 9 of 10 wrong values had a confidence of 1.0 (recorded).

Next, you plan the trial that the four questions ask for, so that it is fair to every option.

Next: Run a fair pilot