Skip to content
Français

Vendor and model checks

  • Lesson 13
  • 35 min
  • Module 4: Policies people can follow

Where we are

In the last lesson, the review of a bought AI feature led to a vendor check. A check needs questions, and a vendor’s answers can sound complete while they leave out what matters. In this lesson, we ask ten questions, read two vendors’ answers and find the gaps, as you would before you give a cleaning company the key to your home.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Before Larkfield switches on an AI feature that it buys, ask the vendor ten questions. Mark every answer that is missing, vague, against Larkfield or not in the contract. Then decide: switch on, switch on with conditions, or keep off.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

You hire a cleaning company and give it a key to your home. Before you hand it over, you ask some questions. Will you use my home for anything except cleaning? Will you bring new staff here to train them? Will you keep a copy of my key, and for how long? Who else will come in? Where do you keep the key? What proof can you show that your staff are honest? Will you tell me at once if the key is lost? And when I stop, do I get the key back, and are the copies destroyed?

A good company answers with facts: “Only Ana and Lee, both checked; the key stays in a locked box; we call you within an hour if it is lost.” A company that answers every question with “we follow the highest standards” has answered nothing. And if the good answers are only in a phone call, they are gone when you need them.

TicketHarbor (made up for this course) sells the add-on in Larkfield’s help desk, with its AI reply suggester. Request R-04 asked the board to switch the suggester on, and the board sent it back because there were no vendor answers. Now TicketHarbor has answered all ten questions (constructed). Every row is filled in. A busy reader could write “questionnaire answered” and send R-04 back to the board.

Now read row 2, about training: “Yes, by default (opt-out by e-mail).” If the suggester is switched on, every customer message can train the vendor’s models, unless someone at Larkfield sends an e-mail. Those customers wrote to get help with an order. And sending the data to a vendor does not move the responsibility. Canada’s federal privacy law for companies says that an organisation stays responsible for personal information that it sends to another company for processing (PIPEDA, Schedule 1, principle 4.1.3, official source, checked 22 October 2026). An answered questionnaire is not a checked one.

The idea: a vendor questionnaire, read for gaps

Section titled “The idea: a vendor questionnaire, read for gaps”

A vendor questionnaire is a fixed list of questions that every vendor of an AI feature answers before Larkfield switches it on. The course’s version has ten. A good answer is specific: a number, a name, or a clause in the contract.

The ten questions follow the data, from the moment Larkfield sends it until Larkfield leaves.

The path of one customer message through a bought AI feature, with the questionnaire's question numbers. Larkfield sends a customer message. At the vendor: 1 data use, is it used for anything except the service? 2 training, does anyone train models on it, on or off by default? 3 retention, how many days are the message, the prompt and the output kept? Then it may go to other companies: 4 sub-processors, which companies, including the model provider, and notice before a new one; 8 model information, which model, what it does badly, notice before it changes. Wherever it is: 5 location, which countries; 6 security, what independent evidence. If something goes wrong: 7 incident notice, how many hours. The output comes back to Larkfield's staff: 9 human oversight, what the product lets staff see and correct. When Larkfield leaves: 10 exit, deletion in how many days, and what can be exported.
The ten questions follow one customer message, from Larkfield to the vendor, its sub-processors and back. The questions are the course's own.
Text description of the diagram

The diagram follows one customer message. Larkfield sends it to the vendor. There, question 1 asks whether it is used for anything except the service. Question 2 asks whether anyone trains models on it, and whether that is on or off by default. Question 3 asks how many days the message, the prompt and the output are kept. The message may then go to other companies. Question 4 asks which ones, including the model provider, and whether Larkfield hears before a new one is added. Question 8 asks which model is used, what it does badly, and whether the vendor gives notice before it changes. Wherever the data is, question 5 asks in which countries, and question 6 what independent evidence of security exists. If something goes wrong, question 7 asks how many hours pass before Larkfield is told. The output comes back to Larkfield’s staff: question 9 asks what the product lets them see and correct. When Larkfield leaves, question 10 asks how many days until the data is deleted, and what can be exported.

Question 8 is the model check. The vendor chooses the model, and it can change the model without changing anything that Larkfield sees. One changed prompt in the triage service failed 0 of 6 lines of its release gate (calculated by this course from recorded results). A changed model inside a bought tool is the same kind of change, made by someone else. So Larkfield asks which model it is, what it is known to do badly, and for notice before it changes. Then the owner can test again, as the course Evaluating and Testing AI Systems shows.

To read the answers, mark each one with one of five marks:

  • Specific: a number, a name or a clause. It answers the question.
  • Vague: words with no number or name: “without undue delay”, “leading models”, “industry standard”.
  • Missing: no answer, an answer to another question, or “see our website”.
  • Against us: clear, but the wrong answer for Larkfield. For example, training on by default.
  • Not in the contract: a good promise that lives only in an answer sheet or an e-mail.

A tiny example: three of TicketHarbor’s rows

Section titled “A tiny example: three of TicketHarbor’s rows”

Farida marks three rows first. For each one she writes the answer, what a good answer looks like, her mark, and what to ask for.

Row TicketHarbor’s answer A good answer Mark Ask for
2 Training “Yes, by default (opt-out by e-mail).” “No”, or “off by default”, in the contract against us, and not in the contract Training off for Larkfield’s data, as a contract term
3 Retention “As long as the account is active.” A number of days for each kind of data vague Days for messages, prompts and outputs
7 Incident notice “We notify customers without undue delay.” A number of hours, in the contract vague A number of hours, in the contract

The intermediate results matter. Row 2 has a clear answer, but clear is not good: it is against Larkfield. Rows 3 and 7 sound like answers but give no number. Then she marks the other seven rows in the same way.

TicketHarbor's ten answers, each with a mark. 1 Data use, 'we use customer content to improve our AI features': against us. 2 Training, 'yes, by default, opt-out by e-mail': against us, not in the contract. 3 Retention, 'as long as the account is active': vague, no days. 4 Sub-processors, 'listed on our website': missing, the model provider is not named, no notice of new ones. 5 Location, 'United States and European Union': specific, ask whether it can change. 6 Security, 'an independent audit report under a confidentiality agreement': specific, sign and read it. 7 Incident notice, 'without undue delay': vague, no hours. 8 Model information, 'we use leading AI models': vague, no name, no notice of changes. 9 Human oversight, 'agents always review suggestions': missing, it describes Larkfield's staff, not the product. 10 Exit, 'export of tickets in 30 days; AI settings not exported': missing, nothing about deletion. Count: 2 specific, 3 vague, 3 missing, 2 against us.
TicketHarbor's answers with the gaps marked: 2 specific, 3 vague, 3 missing, 2 against Larkfield. Constructed for this course; TicketHarbor is made up.
Text description of the diagram

Here are TicketHarbor’s ten answers, each with its mark. Row 1, data use, says the vendor uses customer content to improve its AI features: against us. Row 2, training, says yes by default, with an opt-out by e-mail: against us, and not in the contract. Row 3, retention, says as long as the account is active: vague, with no number of days. Row 4, sub-processors, says they are listed on the vendor’s website: missing, because the model provider is not named and there is no notice of new ones. Row 5, location, says the United States and the European Union: specific; ask whether it can change. Row 6, security, offers an independent audit report under a confidentiality agreement: specific; sign the agreement and read it. Row 7, incident notice, says without undue delay: vague, with no hours. Row 8, model information, says leading AI models: vague, with no name and no notice of changes. Row 9, human oversight, says agents always review suggestions: missing, because it describes Larkfield’s staff, not the product. Row 10, exit, offers export of tickets for 30 days, with no AI settings: missing, because it says nothing about deletion. The count: 2 specific, 3 vague, 3 missing and 2 against Larkfield.

Ten rows were filled in, and two answer the question well. Row 9 is the most common trick: “agents always review” sounds like a control, but it is a promise about Larkfield’s staff. The question asked what the product lets staff see and correct.

Farida brings the marked answers to the team.

  • Omar: “Agents love the suggester. Switch it on today and fix the contract later.”
  • Tomás: “TicketHarbor is a big company. If something goes wrong, it is their problem.”
  • Mei: “Every customer message would train their models unless someone sends an e-mail. And the data can sit in the United States.”
  • Grace: “I promise customers that a person answers them. Does this change that promise?”

Before you read on, answer these:

  1. Switch on, switch on with conditions, or keep off?
  2. Which gaps must close before it is switched on, and which can close later?
  3. Who checks the conditions?

The course’s answer. Keep it off until the contract changes; then switch it on with conditions. Four gaps must close first, because they cannot be fixed once the data has gone. Training on Larkfield’s data must be off by contract, not by e-mail. The vendor must name its sub-processors, including the model provider, and give notice of changes. And the contract must give a retention period in days and an incident notice in hours. Grace owns the conditions, with the finance lead for the contract.

Omar’s “fix it later” fails on row 2: data that trained a model cannot be taken back. Tomás is wrong, as the privacy principle above says: Larkfield stays responsible. Mei’s second point is a question for the lawyer, not for this course. Is storing Canadian and EU customer messages in the United States acceptable under the laws that apply? Grace’s promise holds if the agent reads and sends every reply, which line 4 of the acceptable-use policy now says. Farida adds it to the agents’ training.

What would change the answer: if TicketHarbor refuses to change the contract, look for another vendor, or keep the feature off. A better product does not repair a contract term.

Send the questionnaire before Larkfield buys a tool with an AI feature, before it switches one on, and when a vendor announces a major change. Reuse the answers at each review: ask only what changed.

The cost is mostly reading time: an hour to mark ten answers, more if the contract must change. The vendor does the writing. The questionnaire does not replace the contract or the lawyer who reads it. It shows which clauses to ask for.

Do not send it for a tool with no AI and no access to personal data: the normal buying checks are enough.

Common mistake

The answers are filed, not read

Symptom: the inventory says “vendor check: a questionnaire is answered”. A year later, the vendor starts training on customer data, and it was allowed all along.

Cause: the answers were received and filed. Nobody marked them, and nobody turned the good answers into contract terms.

Fix: a vendor check is done only when every row has a mark and every gap has a decision. The decision is a contract term, a condition with an owner, or a reason to accept the gap. Write it in the decision record.

  1. Open the vendor answers. Find MinuteMoss (made up for this course), the meeting-notes tool on card S7.

  2. Copy its ten answers into a table with the columns: row, answer, a good answer, mark, ask for.

  3. Mark row 2, training: “No. We do not train models on customer content.”

    Check: specific. Ask that it is also written in the contract.

  4. Mark row 4, sub-processors.

    Check: a cloud provider in Canada, and “a large model provider”, not named. Mark it missing for the model provider: Larkfield cannot read that company’s terms.

Mark MinuteMoss’s other eight rows. Then decide for TicketHarbor and MinuteMoss: switch on, switch on with conditions (which ones, and who checks them), or keep off, each with your reasons. Finally, say what Larkfield should do about LedgerFern (made up for this course), whose invoice reader has no answers at all.

Show one possible answer

MinuteMoss’s marks. Specific: rows 1, 2, 3, 5 and 10 (only for the service; no training; recordings deleted after 30 days; Canada; deletion within 30 days and export at any time), and row 7 (72 hours, but ask when the clock starts: “of confirming” can be late). Vague: row 6 (encryption and single sign-on are product features, not independent evidence such as an audit report) and row 8 (a large language model that “can contain mistakes”, with no name and no notice of changes). Missing: row 4 (the model provider is not named) and row 9 (“not stated”). And none of the answers says which promises are in the contract.

MinuteMoss: approve for internal meetings, with conditions for calls with people outside Larkfield. Its answers are mostly specific. Conditions: the vendor names its model provider; a standard sentence at the start of external calls, and the tool only with the other side’s agreement; notes checked by the meeting owner before anyone acts on them. One gap is Larkfield’s own, not the vendor’s: what must people outside Larkfield be told before a call is recorded? Ask the lawyer. The board approved customer calls in its first quarter (R-01) with no conditions, before anyone read these answers. Send it back with the conditions. Omar owns them, as the accountable owner.

TicketHarbor: as in the discussion. Keep off until the four contract terms are in; then switch on with conditions. Grace owns them, with the finance lead.

LedgerFern: no decision is possible without answers. Send the questionnaire now. The invoice reader is medium tier (7 points) and already on, so the check is overdue. The finance lead decides whether it stays on meanwhile; a clerk checks every invoice.

Check your understanding

A vendor answers question 7: 'We notify customers in line with industry best practice.' Mark it, and write what you ask for.

Show the answer

Vague: there is no number of hours, and “industry best practice” names nothing that anyone could check. Ask for a number of hours from the moment the vendor knows of an incident that touches Larkfield’s data, written in the contract. Then the owner can check it after a real incident.

Check your understanding

Predict: TicketHarbor accepts all four contract terms, but adds that its model provider 'may change from time to time'. Does the decision change?

Show the answer

The decision stays “switch on with conditions”, but it needs one more condition. A new model provider is a new sub-processor (row 4) and a model change (row 8). So the contract must give notice before the change. The owner then treats the change as a major change. They test the suggester again before the new model reaches agents, and note the new provider in the inventory.

Check your understanding

Module completion check: give your rewritten acceptable-use policy to a person who has never seen it. Can they answer the three 'may I…?' questions from the page alone? Write the answers they should give, with the line numbers.

Show the answer

With the policy from the first lesson of this module: (1) a customer’s complaint in a free chatbot: no. A complaint with a name and an order number is customer data, and a free chatbot is not on the list (lines 1 to 3). (2) The meeting-notes tool in a call with a supplier: not yet. It is approved for internal meetings only, so ask the risk lead with the form (lines 1 and 5). (3) Let the reply suggester send without reading when the queue is long: no. The person who sends reads it in full (line 4). If any answer needed “ask someone” with no line to point to, the policy failed that question. Then fix the line: a defined word, an example, or a place on the list.

Recap

  • A vendor questionnaire asks ten questions that follow the data: use, training, retention, sub-processors, location, security, incident notice, model, human oversight and exit.
  • A good answer is specific: a number, a name or a contract clause. Mark the others vague, missing, against us or not in the contract.
  • The model check asks which model, what it does badly, and for notice before it changes. A model change is a change to the system.
  • Decide switch on, on with conditions, or keep off, close before switch-on the gaps that cannot be undone, and give each condition an owner.

In this module, you wrote rules for staff, gates for new AI work and a check for vendors. Larkfield now has policies that people can read. But a policy line alone does not stop anything. Next, you turn each line into a control with an owner and a proof.

Next: From policy to control