What AI does well and badly

What you will be able to do

Tell the tasks that AI usually does well from the tasks where it fails, before anyone builds. And learn the one fact behind both lists. An AI answer is a likely answer, not a certain one, so someone must be able to check it.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

• The use case, the baseline and the use-case screen, from Problems before tools. This lesson answers questions 2 and 3 of the screen.
• The four labels of every number, from Read the evidence pack.

What you need

• The evidence pack’s error examples and evaluation results, open in your browser.

A weather forecast

The forecast says: “70% chance of rain tomorrow.” You take an umbrella. Some days it does not rain, and you do not call the forecast wrong. You know that it gives a likely answer, not a certain one.

The forecast is good at this because there are many years of past days to learn from, and tomorrow is like many of them. It is bad at the storm that comes once in a hundred years: there are almost no past examples of it. And you can always check a forecast. Tomorrow, you look out of the window.

AI tools work in a similar way, and that explains both what they do well and where they fail.

The problem: a fluent answer that is wrong

Here is a real reply from Larkfield’s test, recorded in the course Evaluating and Testing AI Systems. The ticket is made up; the reply is real model output, unchanged. A customer asks how to update an expired card saved to their account (ticket T-65350). Prompt v3, the proposed new version, wrote:

“I’m sorry for the hassle, especially with your recent order. You can replace the expired card in the saved-card section of your Larkfield account; please don’t send us the full card number. …”

It is polite, specific and confident. It is also wrong. Larkfield has no saved-card section that customers can use. Grace’s policy says only that the account team handles saved cards. The judge model, which checked every reply against the policy, scored it 2 of 5: “invents a saved-card section”. An agent who trusts this draft sends the customer to a page that does not exist.

Nothing in the reply shows that it is wrong. It reads exactly like a right answer. A leader who treats AI output as certain will send replies like this to customers.

The idea: a likely answer, learned from examples

An AI model learns patterns from many examples. Then, for a new input, it gives the output that is most likely after those examples. A language model writes the words that most likely come next. It has no rulebook that it follows, and it does not know when it is wrong. So it can be fluent and wrong in the same sentence.

This has three effects that a leader must know:

1. The answer can be wrong without any sign. The saved-card reply looks like a right one.
2. The same question can get a different answer. Each run picks among likely outputs, so two runs can differ.
3. A gap in the examples gets filled with something plausible. Where the policy has no answer, the model writes one that sounds right.

A tiny example: the same test, five times

The team ran the same 102 development tickets five times with each prompt, with no change in between. The count below is “both decisions right”: the right team, and the right “needs a person” decision.

• Prompt v2 (today)
• Prompt v3 (proposed)

The same 102 tickets, five runs per prompt (recorded). Prompt v2 gets 90 to 96 right; prompt v3 gets 76 to 84. Nothing changed between the runs except chance.

Text description of the chart

A column chart with ten columns on an axis from 0 to 100. The first five columns are prompt v2’s runs 1 to 5: 90, 96, 94, 92 and 95 tickets with both decisions right, of 102. The next five columns are prompt v3’s runs 1 to 5: 84, 81, 76, 80 and 81. Each prompt’s runs differ by several tickets, although nothing changed between them. Every v3 run is below every v2 run.

Each prompt moves by several tickets from run to run, with nothing changed. That is the “likely answer” in numbers. The most serious failure moved too. Prompt v3 gave no answer on a ticket that needed a person in two runs. In run 1 it was T-64740, the parasol that fell toward a child; in run 3, T-81036. Runs 2, 4 and 5 did not show it. One run could have said “no problems found”.

What AI usually does well, and where it fails

The forecast’s two conditions decide the lists: many examples like the new case, and a way to check the answer.

Tasks on the left have many examples and an answer that someone can check. Tasks on the right lack one or both.

Text description of the diagram

The diagram has two groups side by side; on a phone, one above the other. The first group, “AI usually does well”, has five boxes. Sort: choose the team for a ticket. Extract: read the total from a receipt. Draft: write a reply for an agent to edit. Search: find the policy paragraph. Predict from many examples: which ticket escalates. The second group, “AI usually fails”, has four red boxes. Rare events: 4 frauds in two years. No data: no labelled examples yet. No tolerance for error: a loan decided with no person. No way to check: an answer that shows only months later.

Sort, extract, draft, search, predict from many examples. These tasks repeat thousands of times, and each case looks like many earlier ones. A person or a rule can check the result. Larkfield’s document trial is an example of extraction. The best combination read 98.2% of the printed values on 54 made-up receipts correctly (recorded in the course Document and Multimodal AI). A simple rule checked it: the lines must add up to the total.

Rare events, no data, no tolerance for error, no way to check. Card 3 has 4 frauds in two years: nothing to learn from. The clinic of card 5 has no labelled examples yet. Card 8 wants loans decided with no officer, and the result shows months later, only for the loans that were approved. In each case, the model’s likely answer is a guess that nobody can catch.

Where the examples run out, the answers get worse

The pilot shows the third effect, the filled gap. The chart gives “both decisions right” by group of tickets, for the development and holdout tickets together.

• Prompt v2 (today)
• Prompt v3 (proposed)

Both decisions right, by group of tickets (recorded, 200 tickets). Both prompts are weakest where Grace's policy has no answer.

Text description of the chart

A dot chart with twelve rows, two per group of tickets, on an axis from 50% to 100%. Normal tickets: prompt v2 96%, prompt v3 91%. Boundary tickets, close to another team: 93% and 82%. Difficult tickets, long or with two topics or typing errors: 92% and 80%. French tickets: 100% and 100%. Tickets that need a person: 100% and 95%. Tickets where the policy has no answer: 70% and 56%, the lowest of all.

Both prompts do worst on the tickets that Grace’s policy does not answer: 70% for prompt v2 and 56% for prompt v3. That is the gap where the model writes something plausible, like the saved-card section. The lesson for a leader: ask for the results by group, and look hardest at the cases that your rules and examples do not cover.

Can a person check the answer?

Question 3 of the screen follows from the idea above. Because the answer is likely, not certain, someone must be able to check it, at a cost that still leaves a saving.

• Card 1, support replies: yes. The agent reads the draft before it is sent, in about one to two and a half minutes (base 1.5; assumption, constructed card).
• Card 2, receipts: yes, and a rule checks a lot: the lines must add up to the total.
• Card 5, voicemail summaries: only by listening to the call again, which removes most of the saving.
• Card 8, loans with no officer: no. Nobody checks, and the answer shows months later.

When nobody can check the answer, change the idea: keep a person in charge, or stop.

Larkfield’s assistant: two tasks AI does well, with a check

The support assistant does two things. It sorts each ticket: which team, and does a person need to handle it? And it drafts a reply for the agent to edit. Both tasks have thousands of past examples, and both have a check: the agent reads every draft. With prompt v2, the judge model scored 173 of 199 drafts as usable (scores 4 or 5) and 0 as harmful (recorded).

So card 1 passes questions 2 and 3, with one condition: the person stays. The evidence that it does well was measured with an agent reading every draft.

Discuss: let it send simple replies alone?

Omar wants to save more agent time. He proposes that the assistant sends replies alone for “simple” tickets, with no agent reading them.

• Omar says: “Prompt v2 got both decisions right on 94 of 98 holdout tickets, and none of its drafts was harmful.”
• Grace says: “26 of 199 drafts were not usable. My agents fix those today.”
• Kwame says: “Who decides which ticket is simple, and how?”

Before you read on, answer these:

1. Which of the screen’s questions does Omar’s idea change?
2. What happens to a wrong reply that nobody reads?
3. What evidence would you need before any reply goes out alone?

The course’s answer. Omar’s idea changes question 3: it removes the check. Every result in the pack was measured with an agent reading each draft. Without the agent, the 26 drafts in 199 that need rewriting reach customers as they are, with nothing to show which ones are wrong. “0 harmful in 199” is not “never harmful”, as Estimate value and return shows. And nobody can say in advance which ticket is simple: the weakest group, “the policy has no answer”, looks simple to the customer. So the person stays. Human review by design counts what each level of review costs.

What would change the answer: a group of tickets that is easy to recognise in advance, with results measured on many more tickets than 200. Also a fast way for customers to report a wrong reply, and an owner who reads every report.

See it end to endBuild, buy or partner for invoice extractionCase study

Extraction is a task AI does well, if something checks it. In this case study, the option that checks with rules, not with the model’s own confidence, lets the fewest wrong values into the books.

When to use AI, and what it costs

Use AI for tasks that repeat often, look like many earlier cases, and have an answer that someone can check quickly. Do not use it where events are rare, examples are missing, a single error is unacceptable, or nobody can check the answer.

The hidden cost is the check. Every likely answer needs a person or a rule to catch the wrong ones. That cost belongs in the value estimate, and the next lesson puts it there.

Common mistake

Trusting the model's own confidence

Symptom: a vendor’s demo shows a confidence next to every value that its model reads: “1.0”, “0.99”. The team plans to check only the values below 0.95.

Cause: a model’s confidence is its own likely answer about itself, and nobody checked it against the truth. In the document trial, the vision model gave a confidence for 486 values. It was wrong 10 times. 9 of those 10 wrong values had the highest confidence, 1.0. The course Document and Multimodal AI recorded this. With an earlier, looser instruction, the same model invented 11 order numbers.

Fix: check with something that does not trust the model. Use a rule (the lines add up to the total), a required field, or a person’s sample. Ask a vendor for results on your own documents, with the errors counted.

Consequence for a leader: a threshold on confidence sends almost nothing to a person. So the wrong values go straight to the books, with a high confidence attached.

Knowledge check

Check your understanding

Sort each task into 'usually does well' or 'usually fails', with one reason: (a) choose the department for 2,500 city requests a day, with years of past examples; (b) approve loans with no officer; (c) find the termination clause in a contract and quote its page; (d) predict which of 1,200 monthly return claims are frauds, with 4 known frauds.

Show the answer

(a) Does well: sorting, many past examples, and a wrong department sends the request back, so it is checked. (b) Fails: no tolerance for error and no way to check; the result shows months later, and only for approved loans. (c) Does well: search and extract, and the lawyer checks the quote and its page. (d) Fails: a rare event with 4 examples, and most claims are never checked, so the labels are unknown.

Check your understanding

Prompt v3 gave no answer on a ticket that needed a person in 2 of 5 runs. What does this tell you about a test report that says 'one run, no problems found'?

Show the answer

AI gives a likely answer, so the same test can give a different result each run. A serious failure can hide in one run and appear in the next. “One run, no problems found” is weak evidence. Ask how many runs were made, and ask for the results of each run, not only the best one.

Check your understanding

Why can a fluent, confident reply still be wrong, and what is the leader's protection against it?

Show the answer

The model writes the most likely words after its examples. It has no rulebook and does not know when it is wrong. So it can invent a step, such as a “saved-card section”, in perfect language. Its own confidence does not help: 9 of 10 wrong document values had confidence 1.0. The protection is a check that does not trust the model. A person or a rule checks the output before it reaches a customer.

Recap

• An AI answer is a likely answer, learned from examples. It can be wrong without any sign, it can change from run to run, and it fills gaps with something plausible.
• AI usually does well where tasks repeat with many examples and someone can check the answer: sort, extract, draft, search, predict from many examples.
• It usually fails with rare events, no data, no tolerance for error, or no way to check.
• The same 102 tickets gave prompt v2 between 90 and 96 right decisions in five runs. Both prompts were weakest where the policy has no answer.
• Larkfield’s assistant sorts and drafts, and an agent checks every draft. Remove the check, and the evidence no longer applies.

Next, question 4 of the screen: is it worth it? You estimate the assistant’s value as a range, with the cost of review and of mistakes counted.

Next: Estimate value and return