Risk at leadership level

What you will be able to do

Turn “what could go wrong?” into a short list that a leader can act on. Each line says who could be harmed and how, how bad and how likely it is, and who owns it.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• The four labels of every number (recorded, price, assumption, constructed), from Read the evidence pack.
• The error examples of prompt v2 and prompt v3, from What AI does well and badly.
• The autonomy ladder and its review cost, from Human review by design.

What you need

• A browser and a document tool.
• The risk register template (a spreadsheet version is next to it) and the pilot evidence pack.

A school trip

A teacher plans a day trip to a museum with 30 children. Before the trip, she writes a short list. A child could get lost in the crowd. A child with a nut allergy could eat the wrong lunch. For each line, she writes a name. Mr Ali counts the children at every stop. Ms Chen carries the allergy medicine and checks every lunch box.

The list is short, and it is not a list of every bad thing in the world. It names the harms that are serious or likely on this trip. And every line has one person who acts, and one sign that tells them that something is wrong: the count is one child short.

The problem: worries with no owner

Larkfield’s team talked about risk many times. “The AI could give bad replies” came up in every meeting. But nobody wrote down which bad reply, who it would hurt, or whose job it was to look.

Then came the week that the MLOps and LLMOps course calls “last week” (a constructed situation, with recorded answers). Someone switched prompt v3 on for every ticket. The service accepted every ticket, so the dashboard stayed green. But prompt v3 gave no answer on 186 of 1,798 tickets, and 10 tickets that needed a person reached nobody. Everyone knew that bad replies were a risk. Nobody was watching that number.

A worry that everyone shares is a worry that nobody owns.

The idea: a risk register

A risk is a harm that could happen: to a customer, to an employee or to the business. A risk register is a table with one line per risk. Each line answers the same questions, so you can compare the lines and act on the worst first.

The seven parts of one register line, filled in for Larkfield's worst customer harm. The owner is a person, not a team.

Text description of the diagram

The diagram shows one line of a risk register as seven boxes, read in order. First, who could be harmed and how: a customer with a dangerous product waits for an answer. Second, the cause: the assistant gives no answer, so it also gives no “send to a person” decision. Third, the rating: impact 5 times likelihood 2 gives a score of 10. Fourth, one owner, a person: Grace. This box has a heavy border. Fifth, the controls: a ticket with no answer goes to a person. Sixth, the signal and who looks: the tickets that reached nobody, checked every day. Last, the decision: reduce the risk.

Read the parts in order:

1. Who could be harmed, and how. Write a person and a harm, not a technology. “Model failure” is a cause. “A customer with a dangerous product waits for nobody” is a harm.
2. Impact, from 1 (small) to 5 (serious harm to a person or to the business). It asks: if it happens, how bad is it?
3. Likelihood, from 1 (rare) to 5 (expected). It asks: how often will it happen, with the controls that exist today?
4. Score = impact × likelihood. It only sorts the list. A score of 10 is not “twice as bad” as a score of 5.
5. Owner. One person who watches the risk and makes sure that the controls exist. The owner does not have to build the control.
6. Control and signal. A control makes the harm less likely or less bad. A signal tells you that the harm is happening, and a named person looks at it.
7. Decision. You accept a risk (live with it, in writing), reduce it (add a control), transfer it (a contract or insurance carries part of it) or avoid it (do not do that part).

A tiny example

Back to the school trip. The teacher rates her two risks on the same scales:

Risk	Impact	Likelihood	Score	Owner	Signal
A child with a nut allergy eats the wrong lunch	5 (the child could need hospital care)	2 (the parents pack the lunches)	10	Ms Chen	a lunch box without a name
A child gets lost in the crowd	3 (frightening, usually found soon)	3 (a busy museum)	9	Mr Ali	the count is one child short

The allergy risk comes first, although it is less likely. A rare, serious harm can matter more than a common, small one. And the scales are judgements: another teacher could give the lost child a 4. That is fine, as long as the ratings are written down and the team agrees on them.

Rate likelihood from evidence, not from a feeling

Likelihood is where leaders guess most. Larkfield has recorded evidence, so use it. The evaluation course ran both prompts five times on the same 102 development tickets. The chart counts the tickets with no answer at all in each run:

• Prompt v2 (today)
• Prompt v3 (proposed)

Five runs of the same 102 development tickets (recorded), numbered 1 to 5 along the bottom. Prompt v3 leaves 11 to 19 tickets without an answer in every run. A ticket that needs a person got no answer in runs 1 and 3 only.

Text description of the chart

A column chart with two columns for each of five runs. Prompt v2 has no answer on 1, 0, 0, 0, 0 tickets in runs 1 to 5. Prompt v3 has no answer on 11, 16, 19, 14, 14 tickets. In run 1, one of prompt v3’s tickets without an answer was a ticket that needs a person (T-64740). In run 3, another one was (T-81036). Runs 2, 4 and 5 had no such ticket.

Two lessons come from this chart. First, prompt v3’s failures are not bad luck: there are between 11 and 19 in every run. Second, the most serious failure, a ticket that needs a person with no answer, appeared in only 2 of the 5 runs. If the team had run the test once and looked at run 2, it would have rated that risk “rare”.

The same care applies to “it never happened”. The judge model found 0 harmful prompt v2 drafts in 199. Zero in 199 is not zero. By the rule of three, the true rate could still be up to 15.1 in 1,000. Models change fast, and these recorded numbers come from one set of runs. The method lasts: rate likelihood from repeated runs, and treat “none seen” as “rare, not impossible”.

Larkfield’s register

Nadia, Larkfield’s head of customer operations, asks the team to write the register for the assistant as it works today. That is prompt v2, at the draft level: an agent edits and sends every reply. The risks come from the evidence pack. The ratings and owners are the team’s judgement, constructed for this course. Sorted by score:

#	Who could be harmed, and how	Impact	Likelihood	Score	Owner
R5	A new prompt or model reaches every customer before anyone checks it	5	3	15 (high)	Kwame
R3	Agents approve drafts without reading them, so more wrong replies go out	3	4	12 (high)	Nadia
R1	A customer with a dangerous product waits: no answer, so nobody is told	5	2	10 (high)	Grace
R2	A customer follows a step or page that does not exist	3	3	9 (medium)	Omar
R4	Customers’ names, orders and card endings are kept or used beyond the purpose	4	2	8 (medium)	Kwame

Larkfield's five risks on a heat map, each with its owner. The top right is where a leader looks first.

Text description of the diagram

The heat map is a grid of five rows and five columns. Impact goes up, from 1 (small) at the bottom to 5 (serious harm) at the top. Likelihood goes across, from 1 (rare) on the left to 5 (expected) on the right. Each cell is coloured by its score: pale for 1 to 4, yellow for 5 to 9 and red for 10 to 25. In the top row, impact 5, are R1 (owner Grace, likelihood 2) and R5 (owner Kwame, likelihood 3). R4 (owner Kwame) is at impact 4, likelihood 2. At impact 3 are R2 (owner Omar, likelihood 3) and R3 (owner Nadia, likelihood 4). Three risks are in red cells: R5, R3 and R1.

Look at three choices that the team made:

• R1’s owner is Grace, not Kwame. Kwame’s team built the control: a ticket with no answer now goes to a person. But Grace owns the promise that a safety concern always reaches a person, so she checks every day that it does.
• R5 is the top risk, and it is not about the model. It is about how changes reach customers. “Last week” showed that one setting can undo every test. The next two lessons, on governance and on launch, answer it.
• R4 is medium, not low. Tickets carry personal data, for example “the card ending in 3302” in one error example. How the model provider keeps that data is a question for a specialist, in the next lesson.

The controls behind each line are technical work. The AI security and privacy course teaches them, for example in Prioritize threats and Data inventory. The register is the place where a leader sees them together.

Discuss: can Larkfield launch with a risk it cannot remove?

R2 and R3 never go to zero at the draft level. Each step up the autonomy ladder saves agent time and lets more wrong replies through. Per 1,000 drafted tickets, the base calculation gives these numbers for prompt v2. They are calculated from recorded judge scores and from assumed review minutes and misses:

Level	People cost	Wrong or harmful replies that reach customers
Suggest (the agent writes every reply)	US$2,667	0
Draft (today)	US$1,253	1.5

Nadia meets the team.

• Omar: “Agents review every draft, so the risk is gone. We can write ‘accept’ and move on.”
• Grace: “My agents are busy. When a draft looks fine, they send it. The 10% that they miss is a guess.”
• Kwame: “Go down to ‘suggest’. Then no wrong draft can reach a customer.”

Before you read on, answer these:

1. Is Omar right that review removes the risk?
2. What does Kwame’s proposal cost, per wrong reply that it avoids?
3. What must the register say if Nadia accepts the risk?

The course’s answer. Omar is wrong: review reduces the risk, and the numbers already include it. The draft level still lets about 1.5 wrong replies per 1,000 drafted tickets through. Kwame’s proposal works, but it costs US$1,414 more per 1,000 drafted tickets to avoid 1.5 wrong replies. That is about US$942 per wrong reply avoided (calculated). The value estimate assumes US$50 for one harmful reply, with a range of US$20 to US$150.

So accepting the risk is reasonable, but only in writing. The template’s table “Risks you cannot remove” names the risk, why it is acceptable, who agreed (Nadia) and when you stop. For example: “we stop if agents agree with the assistant on fewer than 90% of reviews”. The 90% is the course’s choice; the normal week was 94.3%. Grace’s point matters too: the 10% miss share is an assumption, so measure it with a weekly sample of sent replies.

What would change the answer: a measured miss share much higher than 10%. Or a harm that costs far more than US$150, such as a wrong reply on a safety ticket. Any harmful draft in a larger sample would also change it. Then “suggest” for that kind of ticket is worth its cost.

When to use a register, and what it costs

Write a register for every AI feature that reaches customers or employees, before the pilot, and review it at every stage gate. Keep it short: five to ten lines that a leader can read in two minutes. A list of fifty risks hides the five that matter.

You do not need one for a short experiment on made-up data, where nobody can be harmed. Writing the register takes one short team meeting, and each review takes a few minutes. The real cost is the controls that the register asks for. That is the point: it shows where to spend.

Common mistake

A risk owned by a team

Symptom: a register line says “Owner: the engineering team” or “Owner: everyone”. Every review marks it “in progress”. When the harm happens, each person thought that another one was watching.

Cause: you cannot ask a team “did you look this morning?”. You can only ask a person. “Last week” at Larkfield had this shape: everyone knew that bad replies were a risk, and nobody watched the 10 tickets that reached nobody.

Fix: write one name per line. If nobody accepts a high-impact risk, that is a finding: the feature does not launch until someone does. The template’s first check says this: no risk with impact 4 or 5 without an owner.

Knowledge check

Check your understanding

Rewrite this register line so that a leader can act on it: 'Risk: model hallucination. Owner: AI team. Impact: high.'

Show the answer

Name a person and a harm, not a technology: “A customer follows a step or page that does not exist”. An example is the invented “saved-card section” of ticket T-65350. Rate impact and likelihood on the 1 to 5 scales, for example 3 and 3, score 9. Give one owner, a person (Omar in Larkfield’s register). Add the control now (an agent edits every draft). Add a signal with the person who looks (agents’ agreement with the assistant, checked weekly). Add the decision (reduce).

Check your understanding

Prompt v3 left a ticket that needs a person without an answer in 2 of 5 runs. A colleague ran the test once, saw no such ticket and rated the likelihood 1 (rare). What is wrong, and what rating would you give?

Show the answer

One run cannot show a failure that appears in some runs only. Runs 2, 4 and 5 had none; runs 1 and 3 had one each. Two runs in five is not “rare”. Base the rating on repeated runs: 3 (possible) or higher. With an impact of 5, the score is at least 15, a high risk, and prompt v3 does not ship.

Check your understanding

Why is Grace, not Kwame, the owner of R1 ('a customer with a dangerous product waits'), although Kwame's team built the control?

Show the answer

The owner watches the risk and makes sure that the controls work; the owner does not have to build them. Grace owns the promise that every safety concern reaches a person. She has a signal that she can check each day: tickets that needed a person and reached nobody. Kwame’s team keeps the control working, and Grace sees it if the control fails.

Recap

• A risk register has one line per risk: who could be harmed and how, the cause, impact and likelihood (1 to 5), a score that sorts the list, one owner who is a person, controls, a signal with someone who looks, and a decision (accept, reduce, transfer or avoid).
• Rate likelihood from evidence. Prompt v3’s most serious failure appeared in only 2 of 5 runs. And “0 harmful in 199” still allows up to 15 in 1,000.
• Larkfield’s top risk, R5, is not the model. It is a change that reaches every customer before anyone checks it.
• Review reduces a risk; it does not remove it. Accepting a risk is a written decision, with who agreed and what makes you stop.

Next, you decide who may approve what, in proportion to these scores.

Next: Governance in practice