Change and adoption

What you will be able to do

Plan how people start to use the assistant, and measure whether their work got better. A product that nobody uses has no value, and a product that everybody uses can still make the work worse.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

• The autonomy level and the review cost, from Human review by design.
• Calibrated trust, the error loop and the signs to watch, from Earn user trust.
• The value estimate and its biggest assumption, minutes to write a reply by hand, from Estimate value and return.

What you need open

• The adoption plan template, copied into your document tool. You also use your human-review canvas in the module practice at the end.

A gym membership for everyone

Think of a company that pays a gym membership for every employee. After three months, the report counts card swipes at the gym door. Many people swiped in January. Is anyone fitter?

The swipes count use: people came in. They do not show whether people trained in a way that helps, or whether anyone got fitter, which was the reason to pay. Some people swiped and sat in the café. Some never came, because nobody showed them how the machines work. A good plan shows people the machines, changes their week so that there is time to train, and checks fitness, not swipes.

The problem: the dashboard was green

Larkfield has seen this. The course MLOps and LLMOps: Operating AI in Production has a practice scenario called “last week” (a constructed situation, with recorded answers). In it, someone switched prompt v3 on for every ticket. The service accepted 100.00% of tickets. 89.1% of tickets got a draft. By a usage count, the assistant was busy.

But 186 of 1,798 tickets got no answer, and 10 tickets that needed a person reached nobody. Customers with safety problems waited. Nothing on the usage dashboard showed it.

More use is not always more value, either. The value model from Estimate value and return shows it. The chart gives the yearly net value at every adoption share: the share of drafts that agents use.

• High
• Base
• Low

Calculated net value a year against adoption, for the low, base and high assumptions. The base and high lines rise with use; the low line falls.

Text description of the chart

A line chart with three lines, from 0% to 100% adoption. The high line rises from about −US$5,000 to about US$1,188,000 a year. The base line rises from about −US$15,000 to about US$417,000, and passes zero at 3.4% adoption. At 70%, the base assumption, it is US$287,661. The low line falls slowly from about −US$48,000 to about −US$60,000: in the low case, every used draft loses money.

In the low case, each used draft saves only 0.91 minutes, worth less than the expected cost of a harmful reply that slips through. So more use loses more money. Use multiplies whatever each use is worth, good or bad.

The idea: usage, adoption and outcome

So a leader measures three things. The last one matters most:

• Usage: are people using it at all? For example, drafts opened.
• Adoption: do they use it where it helps, as the new way of working? For example, drafts used on the tickets that the assistant drafts.
• Outcome: did the thing we wanted get better? For example, agent minutes per ticket, and safety tickets that reach a person.

A fourth measure, quality, checks that errors are still caught. The adoption plan template says it in one line: “Usage is not success. Write the outcome first.”

Usage leads to adoption, and adoption leads to the outcome only if each use is good. Quality checks that it is.

Text description of the diagram

The diagram shows a chain of three boxes. Usage asks whether people use the assistant at all, measured by drafts opened per agent. Adoption asks whether they use it where it helps, measured by drafts used on tickets that the assistant drafts. Outcome has a thick border, because it is the reason for the product. It asks whether the work got better: minutes per ticket, and safety tickets that reach a person. A separate box, quality, points to the outcome with a dashed arrow. It asks whether errors are caught, measured by later reviews that agree with the assistant.

Getting from usage to the outcome is a change in how people work. Change management is the plan for that change: training, a written new workflow, champions, feedback, and support for people who do not want to change.

A tiny example: two teams with the same usage

Here are two teams in their first week with drafts (constructed numbers). Each team gets 450 drafts.

Measure	Team A	Team B
Usage: drafts opened	441 of 450 (98%)	441 of 450 (98%)
Adoption: drafts used, with or without edits	315 of 450 (70%)	45 of 450 (10%)
Outcome: minutes per drafted ticket (timing sample)	2.4	5.9
Agent hours saved in the week, against 6 minutes by hand	450 × (6 − 2.4) ÷ 60 = 27	450 × (6 − 5.9) ÷ 60 = 0.75

The usage is the same, and the outcome differs by a factor of 36. Team B opens every draft and then rewrites it. That is too little trust, from the previous lesson, and only the outcome shows it. The 6 minutes by hand is an assumption, so the first job of an adoption plan is to measure it.

Larkfield’s adoption plan

Nadia’s plan for the first agent team after the pilot is constructed for this course. Its numbers come from the evidence pack.

Part	What Larkfield does
Training	One hour on real drafts, with error examples 1 to 3. They show a silent safety failure, an invented step and a fluent wrong reply.
Changed workflow	Written on one page: edit the draft, then send. A ticket with no answer goes to a person. “Report this draft” for every bad draft.
Champions	Two respected agents from the pilot team answer questions for two weeks.
Feedback	The error loop from the previous lesson, with Grace as the owner.
People who do not want to use it	They may write by hand. Their reasons go to the six-week review.

The measures put the outcome first. Two numbers come from the practice week of the MLOps course (recorded replay). 320 of 320 safety tickets reached a person. Agents’ later reviews agreed with the assistant 94.3% of the time.

Kind	Measure	Target
Outcome	Minutes per ticket, from a timing sample of 50 tickets, before and after	Lower than by hand, by a margin the team agrees
Outcome	Safety tickets that reach a person	All of them
Quality	Later reviews that agree with the assistant	90% or more (the course’s choice)
Adoption	Drafts used	Tracked, no target
Usage	Drafts opened	Tracked, no target

The first team's rollout. The two reviews, with thick borders, are decisions, not reports.

Text description of the diagram

The rollout has five steps. Before launch, the team times 50 tickets written by hand, so the outcome has a baseline. Then comes one hour of training on real drafts, with the error examples. In weeks 1 and 2, the first team uses the drafts, and two champions help. At two weeks, the team times 50 tickets with drafts. At six weeks, the team looks at the outcome, the quality and the signs of trust, and decides: the next team, a change, or stop. The two reviews have thick borders, because each one ends with a decision.

Discuss: everyone on Monday, with a usage target?

• Omar: “Give the drafts to all agents on Monday. Target: 90% of agents use drafts every day.”
• Grace: “One team at a time. If something goes wrong, I want it to go wrong for one team.”
• Kwame: “The release gate already passed for prompt v2. The software is ready for everyone.”
• Priya (data scientist): “Nobody has timed an agent yet. We do not know whether drafts save time.”

Before you read on, answer these:

1. What would agents do to meet Omar’s target?
2. Which number decides most of the assistant’s value, and has it been measured?
3. What does “one team at a time” cost, and what does it buy?

The course’s answer. Roll out team by team, with no usage target. The first team goes through both reviews in the rollout picture. After a good six-week review, the other teams follow, one team a week. A usage target rewards opening drafts and clicking “send”. That pushes agents towards too much trust, the most expensive sign from the previous lesson.

The value model shows what matters. Adoption breaks even at 3.4% in the base case, so enough use is not the hard part here. The biggest assumption is minutes to write a reply by hand (a swing of US$291,624 a year), and nobody has measured it. Priya is right: time 50 tickets before launch. Kwame is right that the software is ready, but the people are not, and the gate measured the assistant, not the workflow.

One team at a time delays the full value by some weeks. It buys a measured outcome before the other agents change how they work. If something goes wrong, it goes wrong for one team.

What would change the answer: the timing. If the two-week timing shows that drafts do not save time, stop and find out why before the next team. If the first teams show a clear time saving and steady quality, larger steps are fair.

When to plan adoption, and what it costs

Plan it whenever an AI product changes how people work, which is almost always. A tool that nobody’s daily work depends on may need only a short note.

It costs time. The value estimate assumes 60 hours for training and the new workflow (30 to 120), part of the one-time cost. It also costs the champions’ time, and some weeks of delay. A plan with no outcome measure costs more: nobody can show the value, or see when it disappears.

Common mistake

Success measured by usage

Symptom: the monthly report says “98% of agents use the assistant”. At the same time, customers complain about wrong replies, and time per ticket has not changed.

Cause: the plan measured usage only. Nobody timed the work, and nobody looked at quality.

Fix: add the outcome measure (minutes per ticket from a timing sample) and the quality measure (later reviews that agree). Go back to the last decision point with these numbers.

See it end to endRecover a stalled AI projectCase study

The stalled project in this case study has no owner for adoption. You add one, with an adoption measure and an outcome measure, to its 90-day recovery plan.

See it end to endGo/no-go memo for a support reply assistantCase study

Omar asks to give the assistant to all agents at once. In the case study, the memo rolls it out one team a week, and measures the time to write a reply by hand before the second team starts.

See it end to endBuild, buy or partner for invoice extractionCase study

In the case study, the clerks’ work changes from typing every document to checking the ones a check sends them. Who does that work, and how it is counted, is part of the whole cost.

Module practice: a review design and an adoption plan

A worked example, small. Larkfield’s finance team wants an AI reader for receipts. The course Document and Multimodal AI recorded two designs on the same 54 receipts. They differ in how they read a receipt and in how they decide which documents a clerk checks. The volume and costs are assumptions from the cost ranges of the second case study.

	Design 1: the model’s confidence decides	Design 2: checks decide
What goes to a clerk	values below 0.95 confidence	totals that do not add up, missing values, unknown shops
Documents checked (recorded)	0 of 54	11 of 54 (20.4%)
Documents with wrong or missing values that reach the books (recorded)	3 of 54	0 of 54
Cost of review a month (calculated)	US$0	9,000 × 20.4% = 1,836 documents × 3 minutes = 91.8 hours × US$38 = US$3,488
Cost of mistakes a month (calculated)	9,000 × 3 ÷ 54 = 500 documents × US$25 = US$12,500	US$0 in the trial

The finance lead chooses design 2. It works at two levels at once. Documents that pass every check go to the books alone, and flagged documents wait for a clerk’s approval. Its review costs less than a third of design 1’s mistakes. The canvas then says who reviews: two finance clerks. They see the page with the value marked, and a corrected value keeps the original. It names one limit. The trial used receipts, not supplier invoices, so the first step is a trial on Larkfield’s own invoices.

The adoption plan’s outcome measure is minutes per document, timed before and after, and wrong values found at month-end. Usage, the share of documents that go through the reader, is tracked with no target.

Your task. For the support reply assistant, complete:

1. the human-review design canvas, all 11 boxes;
2. the adoption plan for its first team after the pilot, with the outcome measure first.

Show a worked solution

Canvas. Level: draft (prompt v2).

Box	Design
1. Who reviews	Every agent of the first team, after the one-hour training
2. What they review	Every draft: they edit, then send
3. Always to a person	Tickets that need a person (24 of 24 caught on holdout); tickets with no answer
4. What they see	The ticket, the draft, the policy section used, a mark when the policy has no answer
5. What they can do	Edit and send, reject and write by hand, send to a person, report
6. Errors	“Report this draft”: Grace’s team sorts it within one working day; the ticket joins the evaluation set; the agent hears what changed
7. What users are told	Agents: what it does, where it is weak (policy has no answer: 70%). Customers: an agent reads every reply before it is sent
8. Too much trust	Seconds per draft; no edits; known-bad drafts not caught
9. Too little trust	Every draft rewritten; time per ticket close to 6 minutes by hand
10. Review cost	About US$1,253 per 1,000 drafted tickets (calculated, base)
11. No answer or unsure	The ticket goes to a person

Adoption plan. First team: one agent team after the pilot. Owner: Grace.

Kind	Measure	Today	Target
Outcome	Minutes per ticket, 50-ticket timing sample	Not measured: time it before launch	Lower than by hand
Outcome	Safety tickets that reach a person	320 of 320 (practice week)	All
Quality	Later reviews that agree	94.3% (practice week)	90% or more
Adoption	Drafts used	—	Tracked
Usage	Drafts opened	—	Tracked

Review at two weeks (the timing) and at six weeks (next team, change or stop).

Your plan may differ. Check three things: the outcome comes first and has a baseline, every sign has a number, and every box names an owner or a rule.

Knowledge check

Check your understanding

Explain the difference between usage, adoption and outcome for Larkfield's assistant, with one measure for each.

Show the answer

Usage: are agents using it at all? Drafts opened. Adoption: do they use it where it helps, as their way of working? Drafts used on the tickets that the assistant drafts. Outcome: did the work get better? Minutes per ticket from a timing sample, and safety tickets that reach a person. Usage and adoption can be high while the outcome is poor, as with an agent who opens every draft and rewrites it.

Check your understanding

Predict: in the low case of the value model, what happens to the net value when adoption goes from 40% to 90%? Why?

Show the answer

It falls, from about −US$52,905 to about −US$58,944 a year (calculated). In the low case, each used draft saves only 0.91 minutes, worth less than the expected cost of a harmful reply that slips through. More use multiplies a loss. So a plan that pushes adoption without checking the outcome can make things worse. The cure is more evidence about harmful replies and about time saved, not more usage.

Check your understanding

Module completion check: justify the autonomy level for Larkfield's support reply assistant (prompt v2), with the cost of a mistake and the cost of review.

Show the answer

Choose the draft level. Cost of review: about US$1,253 per 1,000 drafted tickets (calculated: recorded judge shares × assumed minutes). Cost of a mistake: about 1.5 wrong replies per 1,000 reach a customer.

One level higher, approve, saves about US$640 per 1,000 but sends about 4.5 wrong replies, 3 more. That pays only if a wrong reply costs less than about US$213, which nobody knows. The approval minutes and the 30% miss share are assumptions too. Acting alone would send about 15 per 1,000.

One level lower, suggest, costs US$2,667 per 1,000 and gains little safety: the draft level already catches most errors. At every level, tickets that need a person, and tickets with no answer, go to a person. To move up: measured approval times, a measured miss share with known-bad drafts, and about 600 reviewed drafts with no harmful one.

Recap

• Measure in this order of importance: outcome (did the work get better?), quality (are errors caught?), then adoption and usage. Usage is not success.
• The “last week” dashboard showed 100.00% of tickets accepted while 10 tickets that needed a person reached nobody.
• More use multiplies what each use is worth: in the low case, the value falls as adoption rises.
• An adoption plan has training on real examples (with the errors), a written workflow, champions and a feedback loop. It leaves room for people who say no, and its review dates end in decisions.
• Larkfield rolls out team by team, times 50 tickets before launch, and sets no usage target.

Module 4 in one paragraph. You chose how much the assistant does alone with two costs. At the draft level, review costs US$1,253 per 1,000 drafted tickets, and 1.5 wrong replies per 1,000 get through. You saw why approving is not yet justified, and how reviewer fatigue turns a review into a click. You designed for calibrated trust: the policy next to the draft, a mark where it is weak, an error loop and signs with numbers. And you planned adoption with the outcome first, because usage can look perfect while customers wait.

Next, Module 5 builds the team that runs the assistant, and shows how to review work that you cannot write yourself.

Next: Team roles by stage