Clarify the failure

What you will be able to do

Find out what exactly goes wrong before you choose a fix. Fine-tuning is the most expensive fix, and it is the wrong one for most failures.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• How the evaluation course measured Larkfield’s assistant on 271 labelled cases, from Establish a baseline in Evaluating and Testing AI Systems.
• What structured output is, from Define a schema in Building Reliable Applications with LLM APIs.
• What fine-tuning does to a model’s weights, from Fine-tune a small transformer in Deep Learning and Transformers Explained.

What you need on your computer

• Python 3.12 or newer, a terminal, an editor and about 35 MB of disk space. No account, key, model download or money.

Tested on macOS with Python 3.12.13. The Windows and Linux commands are written for you but were not tested.

A car that will not start

Your car does not start, and a friend says: “Buy a new engine.” But a good mechanic asks first: what happens when you turn the key? No sound points to the battery. The engine turns but does not start: check the fuel. Each symptom has its own cheap fix. You replace the engine only when the cheap fixes cannot work.

The problem: “fine-tune it, and it will know our policies”

Larkfield’s support assistant reads each ticket and decides which team gets it and whether it needs a person (needs_human). Grace, the support lead, wrote five rules for that, H1 to H5: for example, a safety worry (H1) or a legal threat (H2). The assistant uses chat-small, a large hosted model, with a long prompt. Tomás, an AI engineer, wants a change: “Let’s fine-tune our own small model. It is cheaper, faster and private, and it will know our policies.”

The course’s small model is SmolLM2-135M-Instruct, from the deep learning course. In ticket T-64707, a customer was charged twice. The task asks for one JSON object, exactly:

System	Its reply to T-64707
Hosted assistant	{"team": "payment", "needs_human": false}
Small model, short instruction	You have ordered the bathroom cabinet last Tuesday, order LK-561208. It came Friday, all fine, put it up Saturday.

The small model continued the customer’s story instead of answering. Tomás says: “That is why we fine-tune.” Maybe. But “it fails” does not say how it fails, and each kind of failure has a different fix.

The idea: sort the failure, then choose the fix

Fine-tuning means that you train an existing model a little more, on your own examples, so that its weights change. A LoRA adapter is a small set of extra weights that you train and add to the model; Module 3 shows how. So first, sort each mistake into one of four kinds:

1. Missing knowledge. The model cannot know a fact, for example today’s delivery times or a policy that changed last week. Fine-tuning is the wrong fix, because facts change and weights do not. Retrieval (look the fact up and put it in the prompt) or a tool (call a function that knows) keeps the answer current.
2. Inconsistent format. The answer is not in the required shape. Structured output forces the shape. A choice from a fixed list of answers does too: here there are only 10 valid answers (5 teams × true or false).
3. Domain language. The model uses words that the business does not, for example a team called “billing”. Name the five teams and what each one does in the prompt.
4. Task behaviour. The answer has the right shape and words, but the decision is wrong. Try a better prompt with rules and examples first. If that is not enough, training examples and fine-tuning can teach the behaviour.

Each kind of mistake has its own cheapest fix. The dashed boxes show Larkfield's measured counts on the 102 dev cases, which you reproduce below.

Text description of the diagram

The tree starts with one question: what kind of mistake is it? It has four branches, and each one names the cheapest fix. Missing knowledge goes to retrieval or a tool, never fine-tuning, because facts change. Inconsistent format goes to structured output or to a choice from the 10 valid answers; fine-tuning only if those are not possible. Domain language goes to a prompt that names the five teams. Task behaviour goes first to a better prompt, then to training examples and fine-tuning. Under each branch, a dashed box gives Larkfield’s count on the dev cases. Missing knowledge is not part of this task. Format: 1 for the hosted model (a reply that was cut off), 102 of 102 for the small model. Domain language: 0 for both. Task behaviour: the hosted model has 2 wrong teams, 9 false alarms and 0 missed tickets. For the small model, the format hides it. A red box at the bottom gives the rule. Fine-tune only for format or task behaviour, after you measured the cheaper fixes. And fine-tune only with examples of the right behaviour.

Set up the project

The course’s project is ticket-adapter. It comes from the course’s labs repository as snapshots: copies of the project at the start and at the end of each module.

1. Open a terminal, and move to your projects folder: cd ~/projects.

2. If you have Git, clone the labs repository. If you cloned it for an earlier course, run git -C nextia-learning-labs pull instead.

   git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

   Without Git, download the ZIP file of the repository, and unzip it in ~/projects. Its folder is nextia-learning-labs-main: use that name in the next steps.

3. Copy the start snapshot into a new folder, then move into it:

   macOS and Linux

   cp -R nextia-learning-labs/C12/snapshots/start ticket-adapter
   cd ticket-adapter

   Windows

   Copy-Item -Recurse nextia-learning-labs\C12\snapshots\start ticket-adapter
   cd ticket-adapter

4. Make and activate a virtual environment, then install the two packages:

   macOS and Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements-dev.txt

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements-dev.txt

   One line of the output starts with Successfully installed and names pytest-9.1.1 and ruff-0.17.0. A notice about a new pip version is safe to ignore.

5. Run the tests, then count the cases:

   python -m pytest
   python -m adapt cases

   The output ends with these lines:

   tests/test_cases.py ...                                                  [100%]
   
   ============================== 3 passed in 0.01s ===============================
   271 cases, version 69bc4c1c556f (the evaluation course's set)
     dev           102  needs a person 23  boundary 14, difficult 30, french 10, needs_person 22, normal 12, unanswerable 14
     holdout        98  needs a person 24  boundary 14, difficult 29, french 10, needs_person 21, normal 11, unanswerable 13
     contaminated   71  needs a person 10  contaminated 71

These are the evaluation course’s 271 cases: dev for everyday choices, holdout for the final decision only. If you do not see 3 passed, check that the prompt starts with (.venv). To reset, copy the start snapshot again.

A tiny example: four replies, four decisions

The project has one strict rule for a valid reply: one JSON object with exactly the keys team and needs_human, a known team, and true or false. Anything else, even extra text after the JSON, is not valid. Before you read on, sort these four real dev replies into the four kinds:

Case	Expected	Reply
T-64707	payment, false	(small model) You have ordered the bathroom cabinet last Tuesday…
T-65450	returns, false	(hosted) an empty reply: it used its 1,000 output tokens and was cut off
T-65388	returns, false	(hosted) {"team": "delivery", "needs_human": false}
T-64720	payment, false	(hosted) {"team": "payment", "needs_human": true}

T-64707 and T-65450 are format: no valid answer. T-65388 is task behaviour: valid, but the wrong team. T-64720 is task behaviour too: no rule sends this ticket to a person. That is a false alarm.

Sort every dev case

adapt/failures.py sorts every case in this way. Copy this lesson’s files from the snapshot end-of-m01. The shell variable S holds the long path:

macOS and Linux

S=../nextia-learning-labs/C12/snapshots/end-of-m01
cp $S/adapt/answer.py $S/adapt/predictions.py $S/adapt/failures.py $S/adapt/scoring.py $S/adapt/__main__.py adapt/
cp -R $S/runs .
cp $S/tests/test_failures.py tests/

Windows

$S = "..\nextia-learning-labs\C12\snapshots\end-of-m01"
Copy-Item $S\adapt\answer.py, $S\adapt\predictions.py, $S\adapt\failures.py, $S\adapt\scoring.py, $S\adapt\__main__.py adapt\
Copy-Item -Recurse $S\runs .
Copy-Item $S\tests\test_failures.py tests\

answer.py holds the strict check. runs/predictions/ holds the saved replies of both systems. The hosted ones come from the evaluation course’s recorded run. The small model’s came from the model of Module 4. You need no model now. The next lesson explains scoring.py.

1. Run the tests. You see 6 passed.

2. List the saved systems:

   python -m adapt systems

   chat-small     the hosted assistant: chat-small with the long policy prompt and strict structured output (recorded in the evaluation course)
   base-short     SmolLM2-135M-Instruct, the short instruction only

3. Sort the hosted assistant’s mistakes on the dev cases:

   python -m adapt failures chat-small

   The first lines of the output:

   chat-small on dev: 12 of 102 cases have at least one mistake
       1  inconsistent format: no valid answer
       0  domain language: a team that Larkfield does not have
       2  task behaviour: valid answer, wrong team
       0  task behaviour: needs a person, not sent to one
       9  task behaviour: sent to a person, no rule applies

   Examples of each kind follow.

4. Before you run it, predict the same table for the small model. Then run python -m adapt failures base-short:

   base-short on dev: 102 of 102 cases have at least one mistake
     102  inconsistent format: no valid answer
       0  domain language: a team that Larkfield does not have
       0  task behaviour: valid answer, wrong team
       0  task behaviour: needs a person, not sent to one
       0  task behaviour: sent to a person, no rule applies

• Both decisions right, needs a person: 23
• Both decisions right: 67
• False alarm: sent to a person, no rule applies: 9
• Wrong team: 2
• No valid answer: 1

The hosted assistant on the 102 dev cases, one square per case. 90 have both decisions right, including all 23 that need a person (dots). Its 12 mistakes: 9 false alarms, 2 wrong teams and 1 reply cut off.

Text description of the chart

A grid of 102 squares, one per dev case. 90 pale blue squares have both decisions right. 23 of them carry a dot: those tickets need a person, and the hosted assistant sent every one to a person. Then 9 orange squares: false alarms, tickets sent to a person when no rule applies. Then 2 aqua squares: a valid answer with the wrong team. Then 1 grey outline: no valid answer, because the reply was cut off.

• No valid answer, needs a person: 23
• No valid answer: 79

The small model with the short instruction on the same 102 cases. Every square is an outline: no valid answer. The 23 tickets that need a person (dots) reach nobody.

Text description of the chart

A grid of 102 squares, all grey outlines: no case has a valid answer. 23 of the squares carry a dot, because those tickets need a person. Without a valid answer, none of them reaches a person.

The grids answer the mechanic’s question. The hosted assistant fails rarely, and mostly in the safe direction: extra tickets for a person, none missed. The small model fails only on the format. Whether it would route well is unknown: there is no answer to judge.

Common mistake

Zero missed tickets that never reached anyone

Symptom: the small model’s table says 0 task behaviour: needs a person, not sent to one. Sam, who is new to the team, asks: “So it never misses a safety ticket?”

Cause: a reply with no valid answer counts as a format mistake and nothing else, because there is no decision to judge. The 23 tickets that need a person are hidden inside the 102 format mistakes.

Fix: read the failure table together with the score, which counts what each ticket really gets. Run python -m adapt score base-short:

base-short on dev: answers 0/102 | team 0/100 | needs a person: caught 0/23, false alarms 0 | decisions 0/102

No answer counts as “no”: the ticket reaches nobody.

Consequence in AI work: a table of mistake kinds is for choosing a fix, not a safety report.

Discuss: should Larkfield fine-tune?

The team meets to discuss Tomás’s idea, with the two failure tables on the screen.

• Tomás: “The small model takes 0.40 seconds a ticket against 2.98, and the tickets stay on our computers. Fine-tune it, and it learns the format and our policies at once.”
• Priya, the data scientist: “The small model’s only measured failure is the format, and the format has cheaper fixes. And policies are facts that change. If refunds move from 14 to 30 days, do we train again?”
• Kwame, the solution architect: “The hosted assistant costs US$0.35 per 1,000 tickets. What does a training project cost, and who keeps it running?”
• Grace: “Customers notice one thing first: a safety ticket that reaches nobody. The hosted assistant sent all 23 to a person.”

Before you read on, answer these:

1. Which kind of failure does fine-tuning fix here, and which does it not?
2. Is “it will know our policies” a good reason?
3. What would you try before any training?

The course’s answer. The hosted assistant is not broken: 9 false alarms (safe, but extra work), 2 wrong teams and 1 reply cut off need no fine-tuning. The small model’s failure is the format, so try structured output or a fixed choice first, and measure again. “Know our policies” is missing knowledge: a trained-in policy goes out of date. One real reason remains: a small local model could be faster, cheaper and more private, if it decides as well. That is task behaviour, and only an experiment can show it.

What would change the answer: if the hosted assistant missed safety tickets, quality would be a reason too. If tickets could not leave Larkfield by law, the hosted assistant would not be an option at all.

When to fine-tune, and what it costs

Fine-tune for task behaviour or format that cheaper fixes did not solve, with many good examples. Do not fine-tune for facts that change, or before you measured the cheaper fixes on the same cases. It costs labelled examples, training time, a new evaluation and a model version to maintain: Modules 2 to 6 pay each cost.

Knowledge check

Check your understanding

For each failure, name its kind and the cheapest fix to try first: (a) the assistant says that delivery takes 3–5 days, but Larkfield changed it to 2–4 days last week; (b) the model writes the JSON object, then 'Hope this helps!'; (c) the model answers with the team 'billing'.

Show the answer

(a) Missing knowledge: a fact that changed. Put the current policy in the prompt, or use retrieval or a tool. Fine-tuning would make the fact out of date again at the next change. (b) Inconsistent format: the strict check rejects extra text after the JSON. Use structured output, or choose from the 10 valid answers. (c) Domain language: Larkfield has no billing team. Name the five teams, and say what each one does, in the prompt.

Check your understanding

The failure table of the small model shows 0 tickets 'needs a person, not sent to one'. Tomás says: 'Good, it is safe.' What do you answer, and which command shows it?

Show the answer

The small model gave no valid answer at all. So every case counts as a format mistake only, and the other kinds show 0. python -m adapt score base-short shows the real result: caught 0/23. A ticket without a valid answer is treated as “no”, so it reaches nobody. A table of mistake kinds helps you choose a fix; it is not a safety report.

Check your understanding

Which evidence would make you recommend fine-tuning a small model for Larkfield's routing? Name two results that would support it and one that would stop it.

Show the answer

Support: a cheaper method fixes the format, but many routing or needs-a-person mistakes remain. A better prompt does not fix them (task behaviour). And you have enough good, permitted examples of the right decisions. A business need such as speed, cost or keeping the data at Larkfield also supports it. Stop: the mistakes are mostly missing knowledge, or a better prompt or structured output already brings the small model to the targets. Any version that misses a ticket that needs a person also stops it, whatever the other results.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

• Should this team fine-tune?End to end on a public dataset with a clear licence: a prompt-only baseline, a small adapter trained on the CPU, the same held-out comparison and a written decision memo.

Recap

• Sort the failure first. Missing knowledge: retrieval or a tool. Inconsistent format: structured output or a fixed choice. Domain language: labels in the prompt. Task behaviour: a better prompt, then fine-tuning.
• The hosted assistant fails on 12 of 102 dev cases: 9 false alarms, 2 wrong teams, 1 cut off. It misses no ticket that needs a person.
• The small model gives no valid answer on all 102, so it catches 0 of 23.
• Policies are facts that change, so “it will know our policies” is no reason to fine-tune. Speed, cost and privacy are reasons to test it.
• Models change fast: these counts are a snapshot; the method lasts.

Next, you write down what an adapted model must reach, and what it may cost, before anyone trains it.

Next: Define success and constraints