Estimate value and return

What you will be able to do

Estimate what a use case is worth in a year, as a low, base and high range. Subtract the costs of review and of mistakes. One number hides what nobody measured; a range shows it, and tells you what to measure next.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• The four labels of every number, from Read the evidence pack.
• The use case and the baseline, from Problems before tools.
• Why an AI answer is likely, not certain, and needs a check, from What AI does well and badly.

What you need

• Review time and cost, Part A, from the evidence pack.
• The value estimate template (or value-estimate.csv for a spreadsheet), copied into your document tool.

This lesson ends Module 1. Its last part is the module’s practice task and completion check, about 20 minutes.

A dishwasher at home

A friend says: “A dishwasher saves us an hour of washing up a day.” Does it? Only on the days when somebody loads it. Loading and unloading take time too. Some pans come out dirty and need washing by hand again. The machine costs money, water and repairs. And nobody ever timed the washing up by hand.

So the honest answer is a range: “between 20 minutes and an hour a day, if we use it every evening”. It is less exciting than “an hour a day”. It is also more useful: it says what to check.

The problem: a value that counts every ticket

Omar wants a quick number for Nadia. Larkfield gets about 822 tickets a day: 300,030 a year. Writing a reply by hand takes about 6 minutes, and an agent costs about US$32 an hour. Both are assumptions. So Omar multiplies:

300,030 tickets × 6 minutes ÷ 60 × US$32 = US$960,096 a year

This number is more than three times the careful estimate of the same pack, which is US$287,661. It makes five mistakes at once. It counts every ticket, but a quarter go straight to a person and get no draft. It assumes that agents use every draft. It forgets that reading and editing a draft takes time too. It ignores drafts that are not usable, and harmful replies that slip through. And it ignores the cost of running the assistant.

The idea: value is a chain of small numbers

The value of an AI tool is a chain. Start with the volume. Keep only the units where the tool can help and people use its output. Multiply by the time saved on each one, with review time subtracted. Then subtract the cost of mistakes and the cost of running it. Each link is one small number with a label, so anyone can check it and change it.

Two terms come with the chain:

• Adoption is the share of the possible uses where people really use the output. Here: the share of drafts that agents use. A tool that nobody uses saves nothing, however good it is.
• Return on investment (ROI) is what the work gives back compared with what it costs. In this course, the return is the net value a year, and the investment is the one-time cost to build, test and train. A simple form of it is the payback: the months until the net value has paid back the one-time cost.

A tiny example: one used draft

Start with a single draft that an agent uses. The evidence pack gives every input. Of prompt v2’s drafts, 86.9% were usable, recorded as 173 of 199 scored 4 or 5 by the judge model. The rest, 13.1%, were not usable. The minutes, the cost per hour and the harm are assumptions. All values below are calculated from them.

Step	Low	Base	High
Minutes by hand / to review a usable draft / lost on a draft that is not usable	4 / 2.5 / 3	6 / 1.5 / 2	8 / 1 / 1
Minutes saved = 86.9% × (by hand − review) − 13.1% × lost	0.91	3.65	5.95
Value of the time, at US$25 / US$32 / US$40 an hour	US$0.38	US$1.95	US$3.97
Expected harm = harmful share × share an agent misses × cost of one harmful reply	1.51% × 20% × US$150 = US$0.45	0.5% × 10% × US$50 = US$0.03	0% = US$0
Net for one used draft	−US$0.07	US$1.92	US$3.97

Look at the low column. Each used draft saves less than a minute, worth US$0.38. Its expected harm is US$0.45. So in the low case, every draft that an agent uses loses a few cents. Keep that in mind: it is the surprise of this lesson.

Larkfield’s assistant: from one draft to a year

Now scale the base case up to a year. The value tree shows every step with its number.

The base case of Larkfield's support reply assistant, from tickets to net value a year. Each box is one step of the evidence pack's calculation.

Text description of the diagram

The tree reads from left to right; on a phone, from top to bottom. It starts with 300,030 tickets a year: 822 a day for 365 days. Of these, 74.9% get a draft (recorded) and agents use 70% of the drafts (assumption): 157,306 drafts used. Each saves 3.65 minutes, so agents save 9,565 hours. At US$32 an hour, that time is worth US$306,096. Three red boxes are subtracted. Harm that slips through: 0.5% of drafts are harmful, agents miss 10% of those, and each costs US$50, which gives US$3,933. The model costs US$0.34 per 1,000 tickets, US$102 a year. Upkeep is 20 engineering hours a month at US$60, US$14,400 a year. The green box at the end is the net value: US$287,661 a year.

The one-time cost to build, evaluate and pilot, and to train the team, is (800 + 60) hours × US$60 = US$51,600 (assumptions). The net value pays it back in 2.2 months. Look also at the model cost: US$102 a year, about 0.03% of the value. People’s time, not the model, decides the return.

Now do the same with the low and the high values of every assumption. The pack gives the result:

	Low	Base	High
Value of the time saved	US$24,892	US$306,096	US$1,073,944
Cost of harmful replies	US$29,722	US$3,933	US$0
Model cost	US$75	US$102	US$137
Engineering upkeep	US$48,000	US$14,400	US$4,800
Net value a year	−US$52,905	US$287,661	US$1,069,007
Payback of the one-time cost	never	2.2 months	0.2 months

The range runs from a loss of US$52,905 to a gain of US$1,069,007. That is wide, and it is honest: most of its inputs were never measured.

What happens when more agents use the drafts?

Explore

Net value a year as adoption changes

Adoption

70%

Low: net a year

−US$56,528

Base: net a year

US$287,661

High: net a year

US$830,353

Break-even adoption (low / base / high)

never / 3.4% / 0.5%

When agents use 70% of the drafts, the net value a year is −US$56,528 in the low case, US$287,661 in the base case and US$830,353 in the high case. The low case falls as adoption rises: from −US$48,075 at 0% to −US$56,528 here. Each used draft saves less agent time than the harm it risks, so more use loses more money. This is the adoption that the base case itself assumes, so its value here is the one in the evidence pack.

• High case
• Base case
• Low case

One control: the share of drafts that agents use. Every other input stays at its low, base or high value. Computed from recorded results and stated assumptions.

Text description of the explorer

The explorer has one slider, the share of drafts that agents use, from 0% to 100% in steps of 5%. It starts at 70%, the base case’s own adoption. The readout gives the net value a year in the low, base and high cases at the chosen share. It also gives the adoption that each case needs to break even: never, 3.4% and 0.5%. The upper chart draws three lines on one scale. The high line rises from a loss of US$4,937 at 0% to a gain of US$1,188,334 at 100%. The base line rises from a loss of US$14,502 to a gain of US$417,160, and crosses zero at 3.4%. The low line falls, from a loss of US$48,075 at 0% to a loss of US$60,151 at 100%. The lower chart draws the low line alone on a larger scale, so its fall is easy to see. At 40%, the low case’s own adoption, it is a loss of US$52,905. At 70%, it is a loss of US$56,528.

The low line falls. More use loses more money, because each used draft loses a few cents, as the tiny example showed. In the base and high cases, the break-even comes very early: 3.4% and 0.5% of drafts used.

Which assumption matters most?

A wide range raises the next question: which input makes it wide? The pack answers it by changing one assumption at a time, from its low to its high value, with every other input at base. Each bar shows how far the net value moves.

One assumption at a time, from low to high, with the others at base (computed). The minutes to write a reply by hand move the result most: a swing of US$291,624.

Text description of the chart

A chart with six rows, one per assumption, on an axis from 100 to 450 thousand US dollars of net value a year. A dashed line marks the base, about 288 thousand. Each row is a bar from the net value with the assumption’s low value to the net value with its high value. Minutes to write a reply by hand, 4 to 8: from 142 to 433 thousand, a swing of 291,624 dollars. Adoption, 40% to 90%: from 158 to 374 thousand. Tickets a day, 600 to 1,100: from 206 to 390 thousand. Agent cost per hour, 25 to 40 dollars: from 221 to 364 thousand. Minutes to review a usable draft, 2.5 to 1: from 215 to 324 thousand. Minutes lost on a draft that is not usable, 3 to 1: from 277 to 299 thousand.

The longest bar is the minutes to write a reply by hand: a swing of US$291,624. It is ahead of adoption, US$215,831. And nobody has measured it. So before more is built, someone times a few agents writing replies today. That one measurement narrows the range more than any other.

Discuss: what does Nadia ask for next?

Nadia must decide what the team does next with the assistant.

• Omar says: “The base case pays back in 2.2 months. Let’s add the next features.”
• Priya, the data scientist, says: “The low case loses money at any adoption. We should stop until we know more.”
• Kwame says: “Let’s move to a cheaper model to improve the return.”

Before you read on, answer these:

1. Is the low case a reason to stop?
2. Which measurement buys the most certainty, for the least money?
3. Does Kwame’s idea change the decision?

The course’s answer. The low case is a warning, not a verdict. Its harm rate, 1.51%, comes from a simple rule, the “rule of three”. After 0 harmful drafts in 199, the true rate is probably below 3 ÷ 199. So zero in 199 is not zero, and more evidence about harm is worth more than a better guess at the value. Two cheap measurements come first: time a few agents writing replies, and have people review many more drafts for harm. Kwame’s idea does not help: the model is 0.03% of the value. And the evaluation results showed that a “cheaper” prompt made worse drafts. Omar’s features can wait for those two answers.

What would change the answer: timed agents who write a reply in less than 4 minutes, or harmful drafts found in a larger review. Either one moves the base case towards the low case, and then stopping is the right call.

See it end to endGo/no-go memo for a support reply assistantCase study

In this case study, you decide on prompt v3 with the whole evidence pack. The value range and the cost of review are part of your memo.

See it end to endBuild, buy or partner for invoice extractionCase study

The case study uses the same low, base and high method for invoices: the whole cost of each option over 36 months, at three volumes, with the cost of wrong values as its own line.

When to estimate value, and what it costs

Estimate value for every use case that passes questions 1 to 3 of the screen, before anyone builds. Use the value estimate template: inputs with labels and ranges, the seven steps, the low, base and high results, and the assumption that matters most. A first estimate takes an hour. Then update it as measurements replace assumptions.

Do not use the base case alone, and do not treat the range as a forecast. It shows what you believe and how unsure you are. And always count review time and mistakes: without them, any AI tool looks worth it.

Common mistake

The model price as the business case

Symptom: a slide says “the model costs only US$102 a year, so the assistant is almost free”. The team spends weeks choosing a cheaper model.

Cause: the slide counts only the model’s bill. The money is in people’s time. In the base case, agents save US$306,096 of time, against US$14,400 of upkeep and US$3,933 of harm. The model is about 0.03% of the value.

Fix: put every cost in the chain, people’s time first. Then ask which input moves the result most, and measure that one.

Consequence for a leader: effort goes to the smallest number. And a “cheaper” model that makes worse drafts can cost more in agent time than it saves.

Knowledge check

Check your understanding

In the low case, why does the net value fall when more agents use the drafts?

Show the answer

In the low case, each used draft saves 0.91 minutes, worth US$0.38 at US$25 an hour. Its expected harm is US$0.45: a harm rate of 1.51% × 20% missed × US$150. So each used draft loses a few cents, and more use loses more. The fixed costs (upkeep) do not change with adoption, so nothing makes up for the loss.

Check your understanding

The judge found 0 harmful drafts in 199. Why does the low case still use a harm rate of 1.51%, and what would you do about it?

Show the answer

Zero in 199 is not zero. A small sample cannot show that a rare harm never happens. The “rule of three” gives an upper limit. After 0 cases in n, the true rate is probably below 3 ÷ n: here, 3 ÷ 199 = 1.51%. To narrow it, review many more drafts for harm. More evidence about harm is worth more than a better guess at the value.

Check your understanding

Omar's quick estimate was US$960,096 a year: every ticket, 6 minutes each, at US$32 an hour. Name three things it leaves out.

Show the answer

Any three. Tickets that get no draft (a quarter go straight to a person). Adoption: agents do not use every draft. Review time: reading and editing a draft takes minutes too. Drafts that are not usable, which cost extra time. Harmful replies that slip through. The cost of running the assistant: model and upkeep. With all of them, the base case is US$287,661.

Module practice: screen four cards and value the best

You have seen the worked example: the value estimate for Larkfield’s support reply assistant. Now do the same with other cards.

Guided practice. Fill the value estimate template with Larkfield’s base values, to check that you can follow the chain.

1. Copy the inputs table from Part A of Review time and cost into the template, with each label.
2. Work out step 1, units used. You get 157,306 drafts used.
3. Work out step 2, minutes saved per used draft. You get 3.65 minutes.
4. Work out steps 3 to 6. Your net value a year is US$287,661, give or take a few dollars of rounding.
5. Work out step 7, the payback. You get 2.2 months.

Independent task. Change one thing: the cards. Take cards 3, 4, 6 and 7 from the use-case cards.

1. Screen each card with the use-case screen. Stop at the first “no” and write why.
2. Reject at least one card, with the simpler fix or the reason.
3. For the best card, write a value range with the template. Use the card’s volume, label every input, and give your own low, base and high values with a reason for each.
4. Name the assumption that changes your estimate most, and how you would measure it.

Show one good answer

Card	Problem clear?	Can AI do it?	Can a person check it?	Verdict
3. Fake return claims	yes	no: 4 known cases in two years; most claims never checked, so the labels are unknown	slowly	reject: a report of unusual return patterns, and a rule for staff to check
4. Late-delivery messages	a rule solves it	not needed	yes	reject AI: a rule on the carrier’s status and the promised date
6. Contract clauses	yes	yes: search and extract	yes, the quote and its page	continue, with the lawyer reviewing every quote
7. City requests	yes	yes: sorting, years of examples	yes, departments send wrong ones back	continue

The value range. Card 6 or card 7 can be “the best”; both are fair choices with a reason. A good range has these parts. The volume comes from the card and is labelled constructed. The time per unit today, the review time, adoption and the cost of an hour are labelled assumptions, each with a reason. The cost of a mistake is counted: for card 7, a day or two of delay; for card 6, a missed clause, the costliest error. Review time is subtracted. The low case is honest, even if it loses money.

The assumption that matters most is usually the time per unit today, as for Larkfield: nobody has timed it. Measure it by timing a few people on real work for a week.

Module 1 completion check

Answer both parts in a few sentences each. Then open the answer.

1. For Larkfield’s support reply assistant, name the assumption that changes the value estimate most, and say how you would measure it.
2. Pick one card from the eight. Give one sign that AI is the wrong tool for it.

Show the answer

1. The minutes to write a reply by hand. Moved from 4 to 8 minutes, with everything else at base, it changes the net value by US$291,624 a year. That is more than adoption (US$215,831). Nobody measured it. Time a few agents writing replies today, before more is built.
2. Any one of these. Card 3: only 4 examples in two years, and no fast way to check an answer. Card 4: an exact rule on the carrier’s status does the job. Card 8: no person checks the answer, and the result shows only months later.

Recap

This lesson:

• Value is a chain: volume, the share where the tool helps, adoption, minutes saved after review and the cost of an hour. Subtract harm, model and upkeep, and give a low, base and high range.
• Larkfield’s assistant is worth a loss of US$52,905 to a gain of US$1,069,007 a year. The base is US$287,661, with payback in 2.2 months. In the low case, every used draft loses a few cents, so more adoption loses more.
• The assumption that matters most is the unmeasured minutes to write a reply by hand. The model is 0.03% of the value: people’s time decides the return on investment.

Module 1, Decide where AI helps:

• Read every number with its label: recorded, price, assumption or constructed.
• Start from a use case: a user, a task, the cost of a mistake, and today’s baseline. Try a process change, a form, a report or a rule first.
• AI gives a likely answer. Use it where tasks repeat with many examples and someone can check the answer.
• Estimate the value as a range, and measure the assumption that moves it most before you build.

Next, Module 2: once a use case is worth doing, how do you get it? Build it, buy it, or work with a partner.

Next: Build, buy or partner