Building and buying AI
- Lesson 12
- 30 min
- Module 4: Policies people can follow
Where we are
In the last lesson, we wrote the rules for what staff do with AI tools. But staff choices are not the only way AI reaches Larkfield. Teams build new systems, and a tool that Larkfield already pays for can add an AI feature in an update. In this lesson, we set gates that each new piece of AI work passes, in proportion to its risk, as a school does before it takes children out of the building.
Lesson 12 of 19 · AI Governance: Policies, Risk and AccountabilityBuilding and buying AI
1. What governance is for
2. Laws and standards in plain words
3. Inventory, risk tiers and impact assessment
4. Policies people can follow
5. Controls, documentation and evidence
What you will be able to do
Decide what a new AI system, a change to one, or a new AI feature in a bought tool must show before it goes further. The checks grow with the risk, so low-risk work moves fast and high-risk work stops when the evidence is not there.
Before you startWhat to know and to have ready
What you need to know
- The tiers and their controls, from Risk tiers and proportionate controls, and when an impact assessment is needed, from Impact assessments.
- Stages and gates, from Milestones and roadmaps in the course Leading AI Products and Teams. This lesson does not teach them again: it sets them by tier.
What you need
- The build-and-buy policy template, copied into your document tool.
- The system cards S6, S7 and S8, and the evidence from earlier courses.
Three kinds of school trip
Section titled “Three kinds of school trip”A school takes children out of the building in three ways. A walk to the park needs the teacher and a line in the class diary. A day at a museum needs a letter to the parents and enough adults for the group. An overnight trip needs a written plan, more adults, and the head teacher’s signature before anyone books a bus.
The checks come before each step, and they grow with the risk. Nobody asks for a signed plan to visit the park, and nobody skips the plan for a night away.
And if the bus company changes, the school checks the new company, even when the trip is the same.
The problem: one quarter, both failures
Section titled “The problem: one quarter, both failures”Larkfield’s policy had no gates. It said only “new software, including AI tools, is approved by IT”. It failed in both directions that Governance that helps teams ship described: too little check for big changes, and too much for small ones. Three cases show it (constructed situations).
A built system went to everyone with no check. In the “last week” of the course MLOps and LLMOps: Operating AI in Production, version 3 of the triage prompt went to every customer. The numbers are recorded by that course: 1,798 tickets answered by version 3, 186 with no answer, and 10 tickets that needed a person reached nobody. A release gate existed. Applied to version 3, it passes 0 of 6 lines (calculated by this course from the recorded results). But no rule said that a change must pass it before customers see it.
A bought tool switched AI on by itself. TicketHarbor (made up for this course) sells an add-on that Larkfield plugs into its own help desk. An update added an AI reply suggester and switched it on for every agent, by default. It reads every customer message. The system card shows no accountable owner, no vendor check and no review. Nobody decided to use AI on customer messages: an update decided.
Small things waited for the board. In the board’s first quarter, 4 low-tier requests went to the full board. Their median wait was 17.5 working days, and all 4 were approved with no change. One “AI request” was a spreadsheet formula. A policy with no tiers lets the big things through and holds up the small ones.
The idea: gates in proportion to the risk
Section titled “The idea: gates in proportion to the risk”A stage gate is a point where the work stops until a named person checks named evidence. The leadership course uses gates to plan one product. Here, they become policy: every new AI system, and every major change to one, passes the same four gates. The tier decides the evidence at each gate, and who decides.
- Idea: an owner is named, the system has an inventory row, and the tier is set.
- Before real data: the checks that protect data and people before any real data is used.
- Before users: the evidence that it works, and the decision to let people use it.
- After launch: how it is watched, and when it is reviewed again.
Text description of the diagram
Every tier passes gate 1 in the same way. An owner is named, the system has an inventory row, the two questions before scoring are answered, and the tier is set. At gate 2, before real data, a low-tier system follows the acceptable-use rules. A medium-tier system needs a system card, and a privacy check if it uses personal data. A high-tier system needs a draft impact assessment and a privacy or legal review. At gate 3, before users, a low-tier system needs no extra check: the owner and the risk lead decide within 2 working days. A medium-tier system needs an evaluation result in a decision record and a designed approval for its actions. If it affects customers, it also needs an impact assessment signed by the owner, and the owner decides. A high-tier system needs the board’s review of the impact assessment and a release gate with written thresholds that it passes; the board decides. At gate 4, after launch, a low-tier system is reviewed in 12 months. A medium-tier system keeps a change log and is reviewed in 6 months. A high-tier system has monitoring with pause conditions and an incident plan, and is reviewed in 3 months.
The tiers come from the course’s own method, so the gates repeat its controls: 4 for a low-tier system, 9 for medium and 12 for high. The gates add one thing that the list of controls does not have: the order. A control that arrives after launch cannot stop a launch.
A tiny example: version 3 at gate 3
Section titled “A tiny example: version 3 at gate 3”The triage service is high tier (10 points). So a new prompt must pass gate 3 with a release gate that has written thresholds. Here is that gate applied to version 3, line by line (calculated from the recorded results of the course Evaluating and Testing AI Systems, against the release gate of the MLOps course):
| Line of the release gate | Threshold | Version 3 | Result |
|---|---|---|---|
| blockers | at most 0 | 1 (ticket T-65663) | fails |
| answers without text | at most 0 | 8 | fails |
| decisions: lower end of the difference from v2 | at least -0.03 | -0.184 | fails |
| needs-a-person tickets caught (share) | at least 1.0 | 0.958 | fails |
| cost per 1,000 tickets (US$) | at most 0.4 | 0.48 | fails |
| p95 latency (s) | at most 8.0 | 8.534 | fails |
The result at gate 3 is stop: 0 of 6 lines pass. Nobody needs to argue about whether the prompt “feels better”. The thresholds were written before the result existed.
The same change came back to the board as request R-06. This time, the gate was in the policy. The board rejected it in 7 working days (constructed). The difference from the “last week” was not a better test. The test was the same; the policy now said that it must pass.
A feature inside a bought tool
Section titled “A feature inside a bought tool”A bought AI feature needs the same gates, but Larkfield does not build it, so the order changes. The vendor’s update can arrive before anyone at Larkfield decides anything. The review therefore starts with finding it.
Text description of the diagram
The flow starts when a vendor adds or changes an AI feature, often in an update. Step 1 is to find it: the contract owner, IT or the team that uses the tool tells the risk lead. Switched on by default is not “approved”. Step 2 gives it an inventory row and a tier, as for a built system. Step 3 is the vendor check with the vendor questionnaire. For medium and high tier, the check comes before the feature is switched on; for low tier, within 30 days. If it is already on, the check is overdue, and the owner decides at once whether it stays on until the check is done. Step 4 is the decision: switch on, switch on with conditions, or keep off, written in a decision record. Beside the flow are Larkfield’s three bought features. The reply suggester is medium tier, was switched on by default with no check, and the board returned the request to switch it on. The meeting-notes tool is low tier, and its questionnaire is answered. The invoice reader is medium tier, is switched on, and nobody sent the questionnaire.
Look at the reply suggester. Its tier is medium (7 points), so the vendor check comes before it is switched on. The update switched it on first, so the policy was broken on the day of the update, and nobody at Larkfield did anything. That is why step 1 matters. Request R-04 then asked the board to switch it on. The board sent it back after 9 working days, because there were no vendor answers. Now there are: TicketHarbor’s answers are in the labs folder, and the next lesson reads them.
Discuss: who opens the gate?
Section titled “Discuss: who opens the gate?”Farida shows the gates to the team.
- Omar: “The release gate slows every prompt change. We built triage, we know it. Let the team decide.”
- Tomás: “Send every AI change to the board. Then nothing slips through.”
- Kwame: “A gate in a document is a wish. The deploy should refuse a change that has not passed.”
- Grace: “I never see the add-on’s updates. Finance owns that contract.”
Before you read on, answer these:
- Is Omar right for the triage service? For the document assistant?
- What did Tomás’s rule already cost in the first quarter?
- Where does Kwame’s point fit, and where does Grace’s?
The course’s answer. Omar is right for low-tier work, and that is what the fast path is for. The document assistant’s new manuals need the owner and the risk lead, not the board. He is wrong for the triage service. It is high tier, and its last unchecked change sent 10 tickets that needed a person to nobody. Tomás’s rule is the one that made 4 low-tier requests wait a median of 17.5 working days. A board full of small requests reads the big ones in a hurry.
Kwame is right that the strongest gate is one that a system enforces. The policy says which evidence each gate needs; a course on governing AI agents shows how a platform can refuse a change without it. Grace’s point belongs in step 1. Whoever owns a contract (here, the finance lead) must tell the risk lead when the vendor adds or changes an AI feature. And the vendor must say so before the change: question 8 of the vendor questionnaire asks for that.
What would change the answer: suppose the platform already blocked every change that fails the release gate. Then the policy could point to that control instead of a meeting. The gate stays; only who checks it changes.
When to use it, and what it costs
Section titled “When to use it, and what it costs”Use the gates for every new AI system and every new AI feature in a bought tool. Use them also for every major change: new users, new data, more autonomy or a new model. Do not use them for work with no AI in it: send it back within a day.
The cost sits at the start. A low-tier idea costs a form and up to 2 working days. A high-tier change costs an impact assessment, a release gate and a board meeting: days to weeks. That is the point. The cost of the check is paid before customers see the change, not by the customers afterwards.
Common mistake
Approved because it was already on
Symptom: the inventory shows a bought AI feature as “approved”. Nobody can find the decision, and nobody remembers making it.
Cause: the vendor’s update switched it on, and someone copied “on” into the status field. “On” became “approved” without a decision.
Fix: a status of “on, not reviewed” until a decision record exists. The policy says that switched on by default is not approved. A feature with no decision record after its due date goes to the owner at once: keep it on with a reason, or switch it off.
Guided practice: the invoice reader
Section titled “Guided practice: the invoice reader”-
Open the system card S8, the invoice reader in the accounting tool of LedgerFern (made up for this course).
-
Step 1, find it: write who should have told the risk lead. The card says the finance lead owns it.
-
Step 2, tier: write 7 points, medium (harm 2, people 2, autonomy 2, data 1).
-
Step 3, vendor check: write when it was due.
Check: before it was switched on. It is on, and the card says “vendor check: none”. So the check is overdue.
-
Write the decision for now: who decides whether it stays on until the answers arrive.
Check: the finance lead decides. A clerk checks and posts every invoice, which limits the harm meanwhile.
-
Write one more gap from the card: the owner is a role, not a named person.
Your turn: test the policy with three “may I…?” questions
Section titled “Your turn: test the policy with three “may I…?” questions”Write three questions that a team at Larkfield would really ask about building or buying AI. Answer each one from the build-and-buy policy alone, and write the line or gate that answers it. Use this one as your first: “May we switch on the new AI feature of a tool we already pay for?”
Show one possible answer
- “May we switch on the new AI feature of a tool we already pay for?” Not yet. It counts as a new system (the review of a bought feature). Give it an inventory row and a tier. If it is medium or high, the vendor check comes first. Then the board decides, because Larkfield’s board decides on vendors for medium- and high-tier uses (as for R-04). The decision goes in a decision record.
- “May we give the document assistant the new returns policy without the board?” Yes. The document assistant is low tier (5 points), so the fast path applies: the owner and the risk lead decide within 2 working days (gate 3, low). In the first quarter, the same request (R-09) went to the board and waited 17 working days, to be approved with no change.
- “May we release triage prompt version 4 if it passes the release gate on our development tickets?” No. The triage service is high tier: the release gate with written thresholds must pass, and the board reviews the impact assessment (gate 3, high). Passing on development tickets is the feasibility stage, not the gate.
If any answer needs “ask Farida”, the policy is missing a line.
Knowledge check
Section titled “Knowledge check”Check your understanding
The team wants to add a new knowledge-base article to the support MCP server (medium tier). Who decides, and what must they see first?
Show the answer
The accountable owner decides, with no board (gate 3, medium). First they see a current system card, the evaluation result in a decision record, and the entry in the change log. If the change affects what customers are told, the owner also checks that the impact assessment still holds. In the board’s first quarter, a request like this took the owner path and was decided in a few days.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: TicketHarbor announces that the reply suggester will send its reply by itself when an agent has not answered for 10 minutes. Which tier, and what must happen before it is switched on?
Show the answer
Score it again: it now acts alone, so autonomy is 3, and a wrong reply costs time or money (harm 2). The rule “acts alone with a harm of 2 or more” makes it high, whatever the sum. So it is a major change to a bought feature, and a high-tier one. The vendor check comes before it is switched on. It needs a full impact assessment that the owner signs and the board reviews. Until then, the setting stays off. The vendor should have announced it before the update, as question 8 of the questionnaire asks.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A stage gate stops the work until a named person checks named evidence. The tier decides the evidence and who decides.
- Four gates: idea, before real data, before users, after launch. Low tier takes the fast path; medium tier, the owner after the checks; high tier, the board, with a release gate that must pass.
- The gates add order: a control that arrives after launch cannot stop a launch.
- An AI feature in a bought tool is a new system: find it, give it a row and a tier, check the vendor, decide in a record. Switched on by default is not approved.
You can now say what each new piece of AI work must show, and when. But for a bought feature, much of the evidence comes from the vendor. And a vendor’s answers can sound complete while they leave out what matters. Next, you read two vendors’ answers and find the gaps.
Next: Vendor and model checks