Problems before tools

What you will be able to do

Start every AI idea from a user, the task or decision that you want to improve and the cost of a mistake. Then check whether a rule, a report, a form or a process change solves it better. Many AI projects fail because they start from the tool and look for a problem later.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

• The four labels of every number (recorded, price, assumption, constructed), from Read the evidence pack.

What you need

• The use-case cards and the use-case screen template, open in your browser.
• Your document tool, for one copy of the screen.

A dirty floor and a robot vacuum

Your neighbour says: “Our hall floor is always dirty. I want to buy a robot vacuum.” It sounds reasonable. Then you look at the hall. The dirt is mud, and it comes in on the children’s boots. Nobody takes off their boots at the door.

A doormat and a shoe rack cost much less than a robot. They stop the mud before it reaches the floor. The robot would clean the same mud every day, and it would still miss the corners. Your neighbour started from a tool. You started from the problem: who makes the floor dirty, when, and why.

The problem: three ideas, one week

In her first week at Larkfield, Nadia gets three ideas for AI work. They are on the course’s use-case cards. The situations and their numbers are constructed for the course, except the lines marked as recorded.

• Tomás, an AI engineer, wants to “predict late deliveries with machine learning”. Customers write about 120 “Where is my order?” tickets a day (card 4).
• The shop’s owner read that “AI finds fraud”. He wants it for return claims. Larkfield has about 1,200 returns a month, and 4 confirmed frauds in the last two years (card 3).
• Omar, the product manager, wants to keep improving the assistant that drafts replies to support tickets (card 1).

Look closely at card 4. The carrier already sends a tracking status for every parcel, every hour. Larkfield already knows the promised date of every order. So the shop does not need to predict a late delivery: it can see one. A rule says it exactly. If a parcel is not delivered by its promised date, or its status is “held”, send the customer a message.

A model would do worse. Every AI answer is a likely answer, not a certain one, as the next lesson shows. So a model would sometimes warn about a parcel that arrives on time, and sometimes stay silent about one that is late. It would also need data, training, testing and upkeep. The rule needs none of these, and you can check it against the tracking page.

Card 3 fails in another way. With 4 frauds in two years, there is almost nothing to learn from. And most claims are never checked, so nobody knows how many frauds were missed. A model cannot learn a pattern from 4 examples and a pile of unknowns.

The idea: start from the use case

A use case is one specific user, one task or decision that they do, and the outcome that you want to improve. “AI for customer service” is not a use case. “Support agents write a reply to each ticket; we want good replies in less time, with no missed safety concern” is one. The name comes from software design: a case of how someone uses a system.

Every use case needs three facts before any tool:

1. The user: who does the work, or who suffers from the problem.
2. The task or decision: what exactly they do, how often, and how they do it today.
3. The cost of a mistake: what goes wrong when the work is wrong, and who pays for it.

The third fact is the one that people forget. A wrong team on a ticket costs a few minutes. A missed safety concern can hurt a child. The same accuracy can be fine for one task and unacceptable for another.

How the work is done today has a name too: the baseline. It is the result that any new tool must beat. For card 4, the baseline is “customers write a ticket, an agent looks up the parcel”. For the support assistant, the baseline of a new prompt is today’s prompt v2. The evidence pack’s error examples show why this matters. Prompt v2 also gave no answer on one of the 271 test tickets (recorded). Prompt v3 gave no answer on 25. So a new version is compared with today’s version, not with perfection: v2 is not perfect, and v3 is worse.

Try the cheapest fix first

Before AI, ask if a simpler fix solves the problem. The diagram puts the usual fixes in order, from the cheapest to the dearest.

Try the fixes from left to right. AI comes last because it is the only one that is never certain and always needs a person to check it.

Text description of the diagram

Five boxes form a chain from left to right; on a phone, from top to bottom. The first is “change the process”: who does what, and when. The second is “add a form”: ask the customer or the colleague for the facts that are missing. The third is “build a report”: show the pattern to a person who decides. The fourth is “write a rule”: if this, then that, exactly. The last box, in orange, is “use AI”: a likely answer, checked by a person. A note under the first box says these fixes are cheaper, exact and easy to check. A note under the last box says AI is dearer, never certain, and needs review and upkeep.

Each fix has a place. A process change fixes work that goes to the wrong person. A form fixes tickets that lack an order number. A report helps a person see a pattern, such as returns from one address. A rule fixes anything that you can write as “if this, then that”. AI helps where the task needs judgement over many varied cases, such as writing a reply to a free-text ticket. No rule can write that reply.

A tiny example: screen card 4

The use-case screen asks four groups of questions, in order. You stop at the first “no” and write why. Here is part 1, the problem, for card 4:

Question	Answer for card 4
Who is the user?	Customers who wait for an order
What task or decision do we want to improve?	Tell a customer when a delivery will miss its promised date
How is it done today?	The customer writes a ticket; an agent looks up the parcel. About 120 tickets a day (constructed)
What does a mistake cost, and who pays?	A wrong or late message, and one more ticket for an agent
Could a rule, a report, a form or a process change solve it?	Yes. A rule on the carrier’s hourly status and the promised date

The last row says “yes”, so the screen stops here. The result is a good one: a rule that the team can build, test and explain. The decision is “stop the AI idea, build the rule”, and the measure of success is fewer “Where is my order?” tickets.

The whole screen has four steps. The diagram shows them, with what each “no” means.

The use-case screen. Four questions in order; the first 'no' ends the screen with a decision and a reason.

Text description of the diagram

The flow starts with an idea for AI work. Question 1 asks whether there is a clear problem, a user and a cost of a mistake. Question 2 asks whether AI can do the task. Question 3 asks whether a person can check the answer. Question 4 asks whether it is worth it, as a range. If all four answers are yes, the result is “continue”, and the first step is to test the biggest assumption. Each “no” leads to its own red box. A no at question 1, or a simpler fix, means stop and use the rule, report, form or process change. A no at question 2 means stop: the events are rare, there are no examples, or every answer must be certain. A no at question 3 means change the idea: keep a person in charge, or stop. A no at question 4 means stop, or find a bigger problem.

This lesson covers question 1. What AI does well and badly covers questions 2 and 3, and Estimate value and return covers question 4.

Larkfield’s support assistant passes question 1

Card 1 is different from card 4. The user is clear: Grace’s support agents. The task is clear: write a reply to each ticket, and send safety concerns and special cases to a person. Today, agents write every reply by hand with the support policy open. That is the baseline. A mistake has a real cost: a wrong promise, an annoyed customer, or a missed safety concern.

Could a simpler fix do it? Partly. A form could ask for the order number, and a rule could send every ticket with the word “fire” to a person. But no rule can write a helpful reply to a free-text question about a cracked pan and a missing refund. The task needs judgement over many varied cases. So card 1 goes on to question 2.

Discuss: should Tomás build the late-delivery model?

Nadia meets Tomás, Grace and Mei, the data engineer, about card 4.

• Tomás says: “Machine learning can warn customers before the carrier knows. And the team learns a modern skill.”
• Grace says: “My agents answer about 120 ‘Where is my order?’ tickets a day. I want fewer of them, soon.”
• Mei says: “The carrier’s status arrives every hour. We already store it.”

Before you read on, answer these:

1. What is the user’s problem, in one sentence?
2. What does the rule give that a model cannot?
3. Is Tomás’s first point a reason to build the model now?

The course’s answer. The problem is the customer’s: they do not know that the parcel is late, so they write a ticket. A rule on the carrier’s status and the promised date solves it exactly, today, and anyone can check it on the tracking page. A model gives a likely answer, needs data and upkeep, and would be wrong sometimes. “Before the carrier knows” is a new problem, not this one, and nobody has shown that it exists. Build the rule first, then measure the “Where is my order?” tickets. Learning a skill is a good goal, but it is not a use case.

What would change the answer: evidence that many parcels arrive late without any warning in the carrier’s status. Then predicting lateness earlier could be a real use case. The rule would be its baseline, and the model would have to beat it.

See it end to endRecover a stalled AI projectCase study

In this case study, a late-delivery warning project ran for six months with no launch. You diagnose it, and a good plan compares the model with a simple rule before it spends more.

When to use the screen, and what it costs

Use the screen for every new idea, before anyone builds anything, and again when an idea changes. Part 1 takes a few minutes for one person who knows the work. It is the cheapest decision in the whole project.

Do not use it to stop every experiment. A small, time-limited experiment can be a fair way to answer an “unsure”, and Why AI plans are different shows how to plan one. The screen’s job is to make the problem and the baseline clear first. Then an experiment answers a real question.

The cost is honesty. A screen that ends with “build a rule” is a success, even if a person hoped to build AI. Write it down as a result, with the reason.

Common mistake

Starting from the technology

Symptom: a plan called “our AI strategy” lists tools and models, but no users. Its goals are “use AI in customer service” and “find fraud with AI”. Nobody can say what a mistake would cost, or how the work is done today.

Cause: the plan started from the tool. Without a user and a task, there is no baseline to beat and no way to measure success. So every result looks like progress.

Fix: rewrite each goal as a use case: the user, the task or decision, the cost of a mistake, and today’s baseline. Then run the use-case screen on each one. Card 3, “find fraud”, becomes “the returns team checks claims that look wrong”. The screen stops at question 2, because there are only 4 examples. A report of unusual return patterns for the team to check is a better first step.

Consequence for a leader: months of work with no launch, because nobody agreed what “working” means. The stalled project in the case study above started this way.

Knowledge check

Check your understanding

Name the three facts that every use case needs before any tool, and say which one people forget most often.

Show the answer

The user: who does the work or has the problem. The task or decision: what they do, how often, and how they do it today. The cost of a mistake: what goes wrong, and who pays. People forget the cost of a mistake most often. It decides how accurate the work must be, and whether a person must check every answer.

Check your understanding

A colleague proposes AI to send each website enquiry to the right sales team. Today, the website form has no field for the product. What simpler fix would you try first, and why?

Show the answer

Add a field to the form: “Which product is it about?”, with a short list. Then a rule sends each enquiry to the team for that product. The fix is cheap, exact and easy to check, and it removes most of the sorting work. AI may still help with enquiries that pick “other”, but only if those are many, and after the form change has been measured.

Check your understanding

What is a baseline, and what is the baseline for Larkfield's prompt v3?

Show the answer

A baseline is how the work is done today, and how well. It is the result that any new tool must beat. For prompt v3, the baseline is today’s prompt v2, not a perfect assistant. Prompt v2 also failed once, on 1 of 271 tickets (recorded). Prompt v3 failed on 25 of 271, so it does not beat its baseline.

Recap

• Start from a use case: a user, a task or decision, and the outcome to improve. Then add the cost of a mistake.
• The baseline is how the work is done today. Any new tool must beat it; prompt v3 is compared with prompt v2, not with perfection.
• Try the cheapest fix first: a process change, a form, a report or a rule. AI comes last, because it is never certain and always needs checking.
• Card 4 stops at question 1: a rule on the carrier’s status tells customers about late parcels exactly. Card 1, the support assistant, passes question 1: no rule can write the reply.

Next, questions 2 and 3 of the screen: which tasks AI does well, which it does badly, and why its answer is a likely one.

Next: What AI does well and badly