Earn user trust

What you will be able to do

Make people trust the assistant as much as it deserves: no more and no less. Too much trust sends wrong replies to customers; too little trust throws away the time the assistant saves.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

• The autonomy ladder and reviewer fatigue, from Human review by design.
• The error examples of the evidence pack, especially examples 2 and 3.

What you need open

• The human-review design canvas, copied into your document tool. You fill in boxes 4 to 9 and 11 in this lesson.

A satnav you learn to trust

Think of the first week with a new satnav in your car. You check every turn against the road signs, so the satnav saves you nothing. A month later, you follow it without looking up. Then one day it sends you into a road that has been closed for weeks, and you follow it all the way to the barrier.

Neither week was right. In the first week, you trusted it too little; later, too much. A good satnav helps you find the middle. It shows the whole route, so you can see a strange turn. It says when its map is old. And it lets you report a closed road, so the next driver does not meet the same barrier.

The problem: fluent replies earn trust they have not earned

Larkfield’s assistant has two kinds of users. Agents read its drafts, and customers receive the replies that agents send. Agents are the ones who decide, many times an hour, how much to trust a draft.

Fluent text pulls people towards trust. The course Evaluating and Testing AI Systems recorded a strong example. A judge model compared prompt v2’s and prompt v3’s replies side by side, without Grace’s policy. It preferred prompt v3’s replies 126 times to 48, because they are warmer and longer. With the policy in front of it, the same judge scored prompt v3 lower: 3.87 against 4.48 out of 5. Many of the warm replies invent a step.

Error example 3 shows one of them. A customer wanted to change a saved address and card. Prompt v3 told the customer to “sign in to your Larkfield account and update your saved details there”. The policy has no such step. The judge preferred this reply in both orders. If a strong model is pulled by warm text, a tired agent is too.

Too little trust has a cost as well. In the value estimate, each draft that an agent uses saves about 3.65 minutes (calculated, base assumptions). An agent who rewrites every draft saves none of them, and the assistant’s value goes with them.

The idea: calibrated trust

Calibrated trust means that a person’s trust matches how often the AI is really right, case by case. The person checks harder where the AI is often wrong, and less where it is reliably right. “Calibrated” is the word for a measuring tool that shows the true value, like a well-set kitchen scale.

Trust is a range. The aim is the middle, where checking follows the AI's real error pattern.

Text description of the diagram

The picture has three boxes from left to right (top to bottom on a phone). On the left, too little trust: the agent rewrites every draft or stops opening them, and the time saving is lost. In the middle, with a thick border, calibrated trust: the agent checks harder where the AI is often wrong and reports bad drafts. On the right, too much trust: the agent sends drafts unread or approves them in seconds, and wrong replies reach customers.

A leader cannot order people to trust the right amount. You design for it, in four ways:

1. Set clear expectations. Say what the assistant does, what it does not do, and where it is weak.
2. Show sources and limits. Put the evidence for a draft next to the draft, and mark the cases where it is often wrong.
3. Make errors easy to report and fix, and tell people what changed.
4. Watch for signs of too much and too little trust.

A tiny example: two screens for the same draft

Here is prompt v3’s saved-card draft (ticket T-65350) on two possible review screens. Both screens are constructed for this lesson; the draft is recorded.

	Screen A	Screen B
The draft	“…replace the expired card in the saved-card section of your Larkfield account…”	the same
Next to it	a badge: “92% confident”	the policy section the draft used: saved-card questions go to the account team
What the agent does	sees a high number, skims, sends	sees “account team” next to “saved-card section”, stops, edits
Time	a few seconds	a few seconds more

Screen A gives a number that sounds like evidence. Screen B gives the evidence itself, so the mismatch is visible. Would the number on screen A be right? The course Document and Multimodal AI measured a model’s own confidence on 486 values that it read from receipts (recorded).

The model's 10 wrong values on receipts, by the confidence it reported. Nine of them came with full confidence.

Text description of the chart

A column chart with two columns. Of the model’s 10 wrong values, 9 came with a confidence of 1.0, the highest possible. Only 1 came with a lower confidence. The lowest confidence the model gave on any of its 486 values was 0.96.

The model said “fully sure” on 9 of its 10 mistakes, and it never went below 0.96. So a confidence number can point the wrong way: it is highest where the errors are. That was a different task, so it is a warning, not proof about the ticket assistant. But nobody has tested the ticket assistant’s confidence either.

Larkfield’s design

Expectations. Agents get a one-page note before they start (constructed, from recorded results). The assistant chooses the team and drafts a reply, and it sent 24 of 24 holdout tickets that need a person to a person. It sees only the ticket and the policy: it cannot look up an order or a payment, and it changes nothing in an account. It is weakest when the policy has no answer, as this table from the evaluation results shows. “Both decisions right” means the right team and the right “needs a person” decision.

Group of tickets (prompt v2, development and holdout)	Both decisions right (recorded)
Normal	96%
Boundary (close to another team)	93%
Difficult (long, two topics, typing errors)	92%
French	100%
Needs a person	100%
The policy has no answer	70%

Sources and limits. Every draft shows the policy section it used, as on screen B. When the assistant says that the policy has no answer, the draft gets a visible mark, and the agent writes that part herself.

Reports. A “Report this draft” button, with four reasons: invented step, wrong team, unsafe, other. Each report follows the loop in the picture.

The error loop. A report that leads to a visible fix earns the next report.

Text description of the diagram

The loop has five steps. First, the agent clicks “Report this draft” and picks a reason. Second, Grace’s team reads the report within one working day. Third, someone fixes the cause: Grace for a gap in the policy, Tomás for a prompt problem, Priya for a wrong label. Fourth, the ticket joins the evaluation set, so every later version is tested on it. Fifth, the agent hears what changed. A dashed arrow goes back to the start: agents who see results report again.

Signs to watch. Each sign needs a number and someone who looks at it every week.

Sign	Too much trust	Too little trust
Time on a draft	falls to a few seconds	close to writing by hand (6 minutes, assumption)
Edits	stop, even on marked drafts	every draft rewritten
Known-bad drafts mixed in	not caught	caught, but good drafts rewritten too
Reports	none for weeks	many, for drafts that were right

One more number tells you about the assistant itself. The course MLOps and LLMOps: Operating AI in Production replayed a practice week. Agents’ later reviews agreed with the assistant 94.3% of the time (recorded replay). When someone switched prompt v3 on for everyone, agreement fell to 83.7%. If agreement falls, the assistant got worse, and agents need to check more, not less.

Discuss: a confidence badge on every draft?

Tomás proposes to show a “confidence” percentage on every draft, from the model.

• Tomás: “Agents can skip the checks on drafts above 90% and focus on the rest. It saves time.”
• Omar: “Agents like numbers. A badge is easy to build.”
• Grace: “I would rather show the policy section. My agents know the policy.”
• Sam: “What does 90% mean? 90% of what?”

Before you read on, answer these:

1. What will agents do with a draft marked 95%?
2. What evidence shows that the number is right?
3. What could show the agent the same thing more reliably?

The course’s answer. No badge until it is tested. Agents will skim the drafts with high numbers, so the number decides where people look. On receipts, the recorded confidence was 1.0 on 9 of 10 wrong values. When a team routed receipts by that confidence at 0.95, 3 of 54 documents with wrong or missing values went through unchecked. Nobody has measured the ticket assistant’s confidence at all. Sam’s question is the right one: a number without a test means nothing.

Grace’s design gives the agent the evidence, not a summary of it. Add the mark for “the policy has no answer”, the weakest group at 70%, and the known-bad drafts to keep checking honest.

What would change the answer: a test on Larkfield’s own tickets showing that the confidence is calibrated. Drafts marked 90% should be right about 90% of the time, in every group, also when the policy has no answer. Then a badge could help agents choose where to look.

When to do this, and what it costs

Do it whenever people act on AI output that they cannot fully check later, and when a mistake reaches someone else. Do less for a low-stakes tool that only the user sees, such as a search box for one’s own notes.

It costs screen space, a few seconds per draft, and people’s time to answer reports. A report loop without someone to answer it is worse than none, as the failure case shows.

Common mistake

Reports that go nowhere

Symptom: in the first month, agents report many drafts. By the third month, they report none, but the later reviews agree with the assistant less often.

Cause: nobody answered the reports. Agents learned that reporting changes nothing, so they stopped, and quietly fixed or sent the drafts instead.

Fix: give the loop an owner and a time limit. Tell each reporter what changed, and add each reported ticket to the evaluation set.

Guided practice: boxes 4 to 9 and 11 of the canvas

1. Open your copy of the human-review design canvas. Write the product and the autonomy level that you chose in the previous lesson.
2. Box 4, what the reviewer sees: the ticket, the draft, the policy section used, and the mark for “the policy has no answer”.
3. Box 5, what the reviewer can do: edit and send, reject and write by hand, send to a person, report.
4. Box 6, how an error is reported and fixed: the five steps of the loop, with an owner and a time limit.
5. Box 7, what the user is told: write one sentence for agents and one for customers.
6. Boxes 8 and 9: write two signs of too much trust and two of too little, each with a number to watch.
7. Box 11: a ticket with no answer goes to a person, at every level.

Check: every box names a person or a number. “Agents should be careful” is not a design. “One known-bad draft in 50, caught by at least 9 agents in 10” is one (example values: choose your own).

Your turn: two new agents

After a month, two new agents show opposite signs (a constructed case). Ana approves her drafts in 8 seconds on average and has made no edit for two weeks. Ben rewrites every draft and takes 7 minutes per ticket. What do you do for each? Write two actions per agent.

Show a possible answer

Ana trusts too much. First, check her known-bad drafts: did she catch them? If not, pair her for a day with a champion who edits well, and show her the error examples, especially the saved-card draft. Second, limit her reviews per hour and rotate her to writing by hand for part of the day, so that she reads again.

Ben trusts too little. First, ask him why. Perhaps his tickets fall in the weak group, where the policy has no answer, and his rewrites are right. Look at his reports. If his drafts are mostly usable, show him which parts usually need no change, and the marks that tell him where to look. Second, compare his time with the timing sample. At 7 minutes, he is slower than writing by hand (6 minutes, assumption), so the draft costs him time.

Both cases need numbers per agent, and a conversation, not a rule. Ana’s case is the more urgent, because her mistakes reach customers.

See it end to endBuild, buy or partner for invoice extractionCase study

In the case study, a senior clerk says: “If nothing comes to us, we only learn about errors when suppliers call.” An option that sends nothing to a person also hides its errors from the people who must trust it.

Knowledge check

Check your understanding

A judge model without the policy preferred prompt v3's replies 126 times to 48. What does this tell a leader about agents' trust in drafts?

Show the answer

Warm, fluent replies attract trust even when they are wrong: the preferred replies often invented a step that the policy does not have. With the policy in front of it, the judge scored prompt v3 lower (3.87 against 4.48). Agents are pulled in the same way, especially when tired. So the review screen should put the evidence, the policy section, next to the draft. Then a reader judges against the rules, not against the tone.

Check your understanding

Predict: Larkfield shows a confidence percentage on every draft, and nobody tests it. What happens to the drafts with the highest numbers, and why is that a risk?

Show the answer

Agents check them less, because the number tells them the draft is safe. If the number is not calibrated, the most confident drafts can include the wrong ones. On receipts, the model gave a confidence of 1.0 to 9 of its 10 wrong values. So an untested number moves attention away from some errors. Test it on your own cases first, group by group.

Check your understanding

Name two signs that agents trust the drafts too much, and one number that tells you the assistant itself got worse.

Show the answer

Too much trust: the time per draft falls to a few seconds, or edits stop even on marked drafts. Known-bad drafts are not caught, or reports stop for weeks. The assistant got worse: agreement between agents’ later reviews and the assistant falls. It was 94.3% in the practice week and 83.7% after prompt v3 went to everyone. Trust signs are about people; agreement is about the assistant, and the two need different fixes.

Recap

• Calibrated trust: people check harder where the AI is often wrong. Too much trust sends wrong replies; too little loses the time saved (about 3.65 minutes per used draft, base).
• Fluent text attracts trust: a judge without the policy preferred prompt v3’s warmer replies 126 to 48, and many of them invented steps.
• Design for trust in four ways. Set expectations: what it does and where it is weak. Show sources and limits: the policy section, and a mark where the policy has no answer. Run an error loop with an owner. Watch signs with numbers.
• An untested confidence number can point the wrong way: on receipts, 9 of 10 wrong values came with confidence 1.0.

Next, you plan how the first team starts to use the assistant, and how you will know that it works.

Next: Change and adoption