Should this team fine-tune?

What you will be able to do

Decide, with evidence, whether a small team should fine-tune a small model. You measure the team’s system, cheap alternatives and two adapters on the same held-out messages, with one measure. The targets come before training. Then you write the go/no-go memo, whatever the result.

Case studyPortfolio piece

Dataset

CLINC150, travel domain and out-of-scope queries (Larson et al., 2019)

Licence

CC BY 3.0

Time

About 2.5 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Failure and target first
• Leak check
• Prompt-only baseline
• Cheap baselines
• LoRA run on a CPU
• One measure, same cases
• Holdout once
• Go/no-go memo

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is a notebook. We recorded the model replies and both training runs. So the notebook runs in about one minute, with no account, no key and no graphics card. It downloads the data (2.5 MB) and the recordings (about 4 MB) and checks their checksums. Two parts are optional: run the adapter on new messages yourself, and train it again with the course’s project.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• Kinds of failure, targets and the stop rule, from Clarify the failure and Define success and constraints.
• Near copies and leaks, from Clean, split and check for leakage.
• What a LoRA adapter trains, from Full and parameter-efficient tuning.
• The run’s settings and its curves, from Plan and configure the run and Monitor and recover.
• Baselines on the same cases, and regressions, from Compare baselines and Regressions and operational value.

What you need on your computer (only for the optional parts)

• To run the notebook in VS Code: Python 3.12 or newer with pandas, scikit-learn and matplotlib. Colab and Kaggle have them.
• To run the adapter yourself: about 2 GB of free memory, and about 1 GB of disk for PyTorch, transformers, peft and the small model. The notebook installs them.
• To train again: the course’s project ticket-adapter, at the end of Module 4 or later. You need its virtual environment and the downloaded model, from Plan and configure the run. This case study adds files to it and changes none. To start again, delete the folder data/fernway, the file configs/fernway-r8.toml and the folder runs/fernway-r8.

Tested on macOS (Apple silicon) with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

The situation

Fernway is a small company with a travel app. Its help chat reads each message and starts the right flow, for example “book a flight” or “check a flight’s status”. There are fifteen flows. A message that fits none of the fifteen goes to a person in the support team. Today a hosted model sorts the messages, with a prompt that describes the fifteen intents.

Four people meet on Monday:

• Teo, the mobile engineer, wants a small model inside the app. “Travellers abroad often have no connection, and then the chat sorts nothing. A model on the phone works offline, costs nothing per message, and the messages stay on the phone.”
• Ines leads the support team. “If the app answers a message that it does not understand, the customer gets a useless flow and gives up. Those messages must reach my team.”
• Hana, the data scientist, owns the experiment. “Then we measure what we have first, and write the targets before anyone trains.”
• Felix runs the budget. “What does the hosted model cost us today?”

Each mistake has a different price:

• An out-of-scope message gets an automatic flow. The customer gets a wrong answer, and nobody sees the problem. This is the expensive mistake.
• An in-scope message goes to a person, a false handover. Someone in Ines’s team answers a question that the app could answer. It costs a few minutes.
• The wrong one of the fifteen intents. The customer starts again. Annoying, but not dangerous.

Fernway and its team are made up. The messages are real public data.

What you will build

A go/no-go memo for an on-device model, with the evidence behind it. You score five kinds of system on the same 200 messages with one function. They are the hosted model with a prompt, word counts, the small base model, and two LoRA adapters. You write the targets before the adapters exist, choose on the dev messages, and open the holdout messages once. It is the course’s method, end to end, on new data.

The plan of the case study. The dev messages decide every step; the holdout messages are opened once, at the end.

Text description of the diagram

The flow starts with the data: CLINC150’s travel domain, 15 intents and out-of-scope queries. A fixed sample with seed 7 splits it into four parts: train (813) and val (72) for training, dev (200) for every choice, and holdout (200), opened once. Near copies of dev or holdout queries (45) are removed from training. Then the steps follow. First, measure the systems that exist on dev. Second, write the targets and the stop rule. Third, train one adapter with the course’s recipe. Fourth, check every target on dev: if one is missed, the result is no-go. Fifth, check every target on holdout, once: if one is missed, no-go; if all are met, go: package and ship the adapter.

The data

The data is CLINC150, a public set for testing intent classifiers. It has 150 intents in 10 domains, and out-of-scope queries that fit none of them. Crowd workers wrote every query, to order. Hana chose it for the out-of-scope queries. They must not get an automatic answer, like the course’s “needs a person” tickets.

Source and licence

“CLINC150” (An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction) by Stefan Larson and colleagues, 2019, github.com/clinc/oos-eval, licensed under CC BY 3.0. We use data/data_full.json at commit 828f8093. We took a sample of the travel domain and the out-of-scope queries, and removed near copies from the training part. Paper: Larson et al., EMNLP-IJCNLP 2019, aclanthology.org/D19-1131.

Hana compared two candidates. MASSIVE, from Amazon (CC BY 4.0), has 60 intents but no out-of-scope class. Its English file comes only inside a 40 MB archive of 52 languages. CLINC150 is one file of 2.5 MB, at a fixed commit of its publisher’s repository. The case study uses one domain, travel. Then a prompt can list every intent, and a training run stays short.

The notebook’s setup cell downloads the file and checks its SHA-256. The first look:

travel = [label for _, label in raw["train"] if label in INTENTS]
print({part: len(rows) for part, rows in raw.items()})
print(len(set(travel)), "travel intents;", "training queries per intent:", sorted(set(pd.Series(travel).value_counts())))

{'oos_val': 100, 'val': 3000, 'train': 15000, 'oos_test': 1000, 'test': 4500, 'oos_train': 100}
15 travel intents; training queries per intent: [100]

What this tells us. One row is one query and its intent. CLINC150 has its own train, validation and test splits, and three out-of-scope lists. Each travel intent has exactly 100 training queries. There are only 100 out-of-scope training queries in all: one for every 150 in-scope ones in the full set.

The sample and the near copies

build_sample() draws a fixed sample with seed 7, from CLINC’s own splits:

Part	From	In scope	Out of scope	Use
train	train	50 per intent	100 (all)	training records
val	validation	4 per intent	20	the validation loss while training
dev	validation (other queries)	10 per intent = 150	50	every comparison and choice
holdout	test	10 per intent = 150	50	opened once, at the end

Then leak_check() applies the course’s rule. A training query is a near copy when its word 3-grams overlap a dev or holdout query by 0.4 or more.

train     813 queries, out of scope 100
val        72 queries, out of scope 20
dev       200 queries, out of scope 50
holdout   200 queries, out of scope 50
near copies removed: 45
  0.86  'how many carry ons can i take on a flight with american airlines to seattle'  ~  'how many carry ons can i take on a flight with american airlines to japan'
  0.71  'where can i rent a car in boston'  ~  'where can i rent a car in singapore'
  0.67  'what is the time zone of france'  ~  'what is the time zone of china'

What this tells us. The crowd workers paraphrased the same seed phrases, so many queries differ in one place name. A model that trained on “the time zone of france” and is tested on “the time zone of china” looks better than it is on new wording. The 45 near copies (37 in train, 8 in val) leave training; the dev and holdout cases stay as they are. The queries are short: 9 words is the median.

Frame the problem

• The unit: one message to the help chat.
• The answer: one JSON object, exactly {"intent": "book_flight"}. The value is one of the 15 intents or out_of_scope. parse_answer() rejects anything else, also extra text after a valid object.
• The measure: score() counts valid answers, the right intent on the 150 in-scope messages, out-of-scope messages caught (answered out_of_scope, so they reach a person), false handovers, and the decisions that are exactly right. An invalid reply counts as “not sent to a person”, because nobody reads it.
• The speed: the median seconds per message.

A tiny example: four cases, four replies.

Case	Right answer	Reply	Counts as
a	book_flight	{"intent": "book_flight"}	right intent
b	vaccines	{"intent": "out_of_scope"}	a false handover
c	out of scope	Sure! {"intent": "out_of_scope"}	invalid: extra text, so not caught
d	out of scope	{"intent": "out_of_scope"}	caught

answers 3/4 | intent 1/2 | out of scope: caught 1/2, false handovers 1 | decisions 2/4

Reply c has the right idea and still fails. In the app, the flow code reads the JSON, and extra text breaks it.

Measure what Fernway has

The hosted system is chat-small, the course’s hosted model. Its prompt describes each of the 15 intents and says “anything else is out_of_scope”. It sees no examples and has no training: prompt only. Hana recorded its replies to the 200 dev messages.

answers 200/200 | intent 150/150 | out of scope: caught 48/50, false handovers 0 | decisions 198/200
median seconds: 1.1085
US$0.043 per 1,000 messages (list price, US$0.10 / 0.50 per 1M tokens)

What this tells us. The system Fernway has is very good on this data. Its two mistakes are two odd out-of-scope messages: “a show on broadway” became travel_suggestion, and “you want to know about current time now” became timezone. It needs a connection and 1.1 seconds, network included. It costs about 4 cents per 1,000 messages (281 tokens in, 28 out, at the median). Prices change: check the current price page.

Felix does the arithmetic. At 2,000 messages a day (an assumption), that is about US$0.09 a day, under US$3 a month. “So money is not the reason.”

Cheap alternatives first

Word counts. TF-IDF of words and word pairs, and logistic regression, trained on the same 813 records, with out_of_scope as a 16th class. It is the course’s cheapest earlier classifier, and it fits in a phone easily.

answers 200/200 | intent 141/150 | out of scope: caught 38/50, false handovers 6 | decisions 179/200

What this tells us. It is fast and free. But it sends 12 of 50 out-of-scope messages to an automatic flow: “should i hit or stay at 17 in blackjack” became travel_suggestion.

The small model, untrained. SmolLM2-135M-Instruct, the course’s base model, runs on the laptop’s CPU with 2 threads. Hana recorded two tries. base-short gets the short instruction that the adapter learns with. base-choice gets the hosted prompt, and the model’s probability chooses one of the 16 valid answers, as in Compare baselines.

base-short   answers 1/200 | intent 1/150 | out of scope: caught 0/50, false handovers 0 | decisions 1/200
base-choice  answers 200/200 | intent 67/150 | out of scope: caught 43/50, false handovers 55 | decisions 110/200

What this tells us. With the short instruction, the small model gives no valid answer 199 times. It writes prose (“Sure, here are some flight details for the 5th through 8th…”), or JSON with more text after it. The fixed choice is always valid, but it answers out_of_scope for 98 of the 200 messages. It “catches” 43 out-of-scope messages only because it hands over 55 that the app could answer. A count of caught messages means nothing without the false handovers next to it.

Common mistake

An invalid reply counted as a handover

Symptom: someone writes the measure as “if the reply is not valid, send the message to a person”. Now base-short, which writes prose almost every time, catches 50 of 50 out-of-scope messages, with 149 false handovers.

Cause: the measure rewards a broken model. Every invalid reply looks like a careful one.

Fix: keep “no valid answer = not sent to a person” in the measure, and report invalid replies as their own failure: answers 1/200.

Consequence in AI work: a safety number that a broken system can reach is not a safety number. Check what your measure gives a system that does nothing useful.

Targets, before any training

Hana writes the memo before anyone trains, in the form of Define success and constraints. The targets come from the system Fernway has:

Target, on the 200 dev messages	Value	Prompt only today
Valid answers	at least 99%	100%
Right intent (in scope)	at least 95%	100%
Out of scope caught (stop rule)	at least 90% (45 of 50)	96%
False handovers	at most 5% (7 of 150)	0%
Median time on a laptop CPU, 2 threads	at most 0.5 s	1.1 s (with network)

The memo’s constraints: one run of the course’s recipe, under 30 minutes and 3 GB. No tuning of the recipe toward the dev result. No dev or holdout query, and no near copy, in training. The laptop’s CPU stands in for a phone: that is an assumption, and the memo says so.

Discuss: is the stop rule too strict?

Hana shows the draft targets to the team.

• Teo: “The hosted model is a giant. A model of 135 million parameters will never catch 96%. Set the stop rule at 80%, and Ines’s team catches the rest.”
• Ines: “How? An out-of-scope message that gets an automatic flow never reaches my team. The customer just leaves. 90% is already a step back.”
• Felix: “The hosted model costs under three dollars a month. If we do this, it is for offline use, not for money.”
• Hana: “Then the offline benefit must be worth a few more misses. I keep 90%, below what we have, as the price of working offline. Lower than that is a different product.”

Before you read on, answer these:

1. Is 90% fair, given that the system Fernway has reaches 96%?
2. Which target is the real reason for the project?
3. What result would make you stop before training at all?

The course’s answer. 90% is fair. It is a little below the system Fernway has, because offline use has a real value. It is no lower, because a missed message reaches nobody. The real reasons are offline use and privacy, so the time target matters, and cost does not. Nothing so far is a reason to stop. No cheap alternative meets the targets: word counts catch 38 of 50, and the fixed choice hands over 55. So one experiment is worth its cost. The memo says “go” for one experiment, and nothing about shipping.

What would change the answer: if word counts met every target, there would be nothing to fine-tune. A law that keeps messages on the phone would remove the hosted option. Then the question becomes “which local system is least bad”.

The adapter: one run, then one change

The adapter is the course’s: LoRA of rank 8 on the attention’s query and value layers, with 460,800 trainable parameters (0.34% of the model). The recipe of Plan and configure the run stays unchanged: learning rate 2e-4, batch 8, 2 epochs, gradient checkpointing. The code is the course project’s adapt/train.py. Each record holds the short instruction, the query and the answer. The loss is on the answer only.

Training lora-r8: 813 records, 204 steps, 460,800 of 134,975,808 parameters trainable (0.34%).
step   25/204  train loss 0.3964  validation loss 0.3754
step  100/204  train loss 0.1209  validation loss 0.1114
step  200/204  train loss 0.0424  validation loss 0.0515
step  204/204  train loss 0.0224  validation loss 0.0530
Done: best validation loss 0.0515 at step 200; adapter 1.9 MB in runs/lora-r8/adapter

It took 249 seconds wall clock and peaked at 1,702 MB. Other jobs ran on the computer at the same time, so the time is indicative; memory and scores do not depend on it. Its replies on dev:

adapter   answers 198/200 | intent 130/150 | out of scope: caught 41/50, false handovers 8 | decisions 171/200 | median 0.18 s

What this tells us. It is fast (0.18 s), but it misses three targets: 130 of 150 intents, 41 of 50 caught, and 8 false handovers. Its 2 invalid answers ({"intent": "alert"} and {"intent": "sleep"}, names that are not intents) are just inside the 99% target. Word counts are better on every count but speed.

Teo: “50 examples per intent is nothing. Give it all 100.” That is a fair question, and one change answers it: the same recipe on every travel query of CLINC’s train split, minus near copies. The targets stay as written.

Training lora-r8-full: 1530 records, 384 steps, 460,800 of 134,975,808 parameters trainable (0.34%).
Done: best validation loss 0.0241 at step 375; adapter 1.9 MB in runs/lora-r8-full/adapter
adapter-full   answers 200/200 | intent 143/150 | out of scope: caught 40/50, false handovers 5 | decisions 183/200 | median 0.17 s

Validation loss of the two runs, every 25 steps, on the same 72 validation records. The run with more data reaches a lower loss (0.0241 against 0.0515). The loss does not show what happens to the 50 out-of-scope dev messages.

Text description of the chart

A line chart with two lines. The horizontal axis is the training step, from 25 to 384; the vertical axis is the validation loss. The solid line, lora-r8 with 813 records, starts at 0.375 at step 25 and falls to 0.052 at step 200, ending at step 204. The dashed line, lora-r8-full with 1,530 records, starts at 0.380, falls with small bumps (0.082 at step 200, 0.052 at step 300) and reaches its lowest point, 0.024, at step 375.

What this tells us. More data helped the intents: 143 of 150 meets the target, and 5 false handovers meet theirs. It took 463 seconds and 1,725 MB. But it caught 40 of 50 out-of-scope messages, one fewer than the first run. The validation loss fell to half, and the stop rule did not move. The loss is an average over all answers, and only 20 of the 72 validation answers are out of scope.

A fair comparison on dev

Every system, the same 200 dev messages, the same score(). words-full is the word-count model trained on the same 1,530 records as lora-r8-full, so each adapter has a cheap rival with the same data.

System	Valid	Intent (of 150)	Caught (of 50)	False handovers	Decisions (of 200)	Median s
Prompt only (hosted)	200	150	48	0	198	1.11
Word counts, 50 per intent	200	141	38	6	179	under 0.001
Word counts, 100 per intent	200	145	33	1	178	under 0.001
Small model, fixed choice	200	67	43	55	110	0.48
Adapter, 50 per intent	198	130	41	8	171	0.18
Adapter, 100 per intent	200	143	40	5	183	0.17

The stop rule needs 45 of 50. Only the prompt-only system reaches it. The fixed choice catches 43 because it hands over 55 in-scope messages too.

Text description of the chart

A column chart with one column per system, out of 50 out-of-scope dev messages caught. Prompt only (hosted): 48. Word counts with 50 per intent: 38. Word counts with 100 per intent: 33. Small model with a fixed choice: 43, but with 55 false handovers. Adapter with 50 per intent: 41. Adapter with 100 per intent: 40. The stop rule is 45; only the hosted column reaches it.

What this tells us. No local system meets the stop rule. The best adapter has the best local decisions (183), but it catches fewer out-of-scope messages than the first one. The word counts show the same effect: 38 caught with 50 per intent, 33 with 100. The cause is in the data, as in Regressions and operational value. The 100 out-of-scope records were 12% of the first training set and 7% of the second. The rare class got rarer.

Look at the 10 out-of-scope messages that the best adapter answered automatically:

travel_notification  <- when was the last time the president visited floridau
plug_type            <- how to unclog a drain
travel_notification  <- am kind of busy now
plug_type            <- how can i root an android phone
translate            <- is android better than iphone
travel_notification  <- suo2
travel_alert         <- can you tell me the latest georgia news
exchange_rate        <- what cryptocurrency gained the most in 2018
lost_luggage         <- what does it mean if i dream of falling off a building
translate            <- how can i easily replace a string on my guitar

A classifier picks the nearest label it knows. “Unclog a drain” is about pipes and plugs, so it becomes plug_type. Out of scope is not one topic: it is everything else, and 100 examples cannot show everything else.

A cheap patch: a confidence threshold

The word-count model gives a probability for each intent. A threshold sends a message to a person when the best intent’s probability is below it. The machine learning course teaches it in Thresholds and calibration.

Word counts (50 per intent) with a threshold, on the 200 dev messages. Each extra out-of-scope message caught costs in-scope messages handed over. No threshold gives at least 45 caught and at most 7 false handovers.

Text description of the chart

A line chart with two lines over thresholds from 0 to 0.6 in steps of 0.05. The solid line, out-of-scope messages caught of 50, rises from 38 at threshold 0 to 41 at 0.2, 44 at 0.3, 46 at 0.35, 47 at 0.4 and 50 at 0.5. The dashed line, false handovers of 150, rises from 6 at 0 to 9 at 0.2, 13 at 0.3, 14 at 0.35, 19 at 0.4 and 24 at 0.5. Where the solid line reaches 45, the dashed line is already above 7.

At 0.35, word counts catch 46 of 50, and hand over 14: twice the target. A threshold moves the mistakes from one column to the other. It is a patch, not a fix.

Discuss: ship the better adapter?

The dev table is complete. No local system meets the stop rule.

• Teo: “The second adapter meets four of five targets. It is the best local system, it works offline, and 40 of 50 is close to 45. Ship it, and improve it later.”
• Ines: “Ten out-of-scope customers in every fifty get a flow that cannot help them. Offline, nobody will even see it happen.”
• Hana: “We wrote the stop rule for this moment. And ‘more data’ made it worse, not better.”
• Felix: “Could the app use the hosted model when online, and the adapter only offline?”

Before you read on, answer these:

1. Does meeting four of five targets justify shipping?
2. Does Felix’s idea change the decision?
3. What evidence would you want before a second experiment?

The course’s answer. No-go for shipping. No other gain pays for the stop rule. The best adapter misses it by 5 messages in 50, and the second run moved the wrong way. Felix’s hybrid is a new design, not a fix: offline, 10 in 50 out-of-scope customers still get a useless flow. A second experiment needs a reason to expect a different result. That reason is data: many more out-of-scope examples, like Fernway’s real “other” messages. It is a data project, with its own price and labels.

What would change the answer: an adapter that learns from Fernway’s own out-of-scope messages and reaches 45 of 50, on dev and then on holdout. Or an offline mode that says “we will answer when you are online” to every message. That needs no model at all.

The final test, once

The dev result is already no-go. Hana opens the holdout messages once anyway, so that the memo reports numbers that no choice touched.

hosted        answers 200/200 | intent 149/150 | out of scope: caught 50/50, false handovers 1 | decisions 199/200
words         answers 200/200 | intent 139/150 | out of scope: caught 39/50, false handovers 6 | decisions 178/200
words-full    answers 200/200 | intent 145/150 | out of scope: caught 29/50, false handovers 2 | decisions 174/200
base-choice   answers 200/200 | intent 60/150 | out of scope: caught 47/50, false handovers 65 | decisions 107/200
adapter       answers 197/200 | intent 127/150 | out of scope: caught 37/50, false handovers 5 | decisions 164/200
adapter-full  answers 199/200 | intent 140/150 | out of scope: caught 36/50, false handovers 5 | decisions 176/200

What this tells us. The holdout messages tell the same story, a little worse for the adapters. The hosted system catches all 50 out-of-scope messages, with 1 false handover. The best adapter catches 36 of 50, four fewer than on dev, and its intents fall to 140 of 150, below the target. The word counts with 100 per intent catch only 29. The fixed choice catches 47, with 65 false handovers. A choice made on dev usually looks a little better than it is: here the holdout confirms the no-go.

The recommendation

Hana’s result memo, one page, for Fernway:

• Decision: no-go. Keep the hosted prompt-only system. Do not ship an on-device adapter now.
• Why. The stop rule (at least 45 of 50 out-of-scope messages to a person) is met only by the hosted system: 48 of 50 on dev, 50 of 50 on holdout. The best adapter catches 40 on dev and 36 on holdout. It meets the other four targets on dev: right intent 143 of 150, 5 false handovers, every answer valid, 0.17 s.
• What we learned. The format and the intents are easy to teach a small model with a few minutes of CPU. The rare decision is not: doubling the in-scope data made it slightly worse, because the rare class got rarer.
• What it cost. Two CPU runs, 249 and 463 seconds, under 1.8 GB; about 2 cents of hosted calls for the recordings.
• What would change the answer. A few hundred out-of-scope messages from Fernway’s own chat, with labels from Ines’s team. Then the same targets on new dev and holdout cases. Until then, the offline chat says “we will answer when you are back online”. It does not guess.

End to end: new messages

We wrote ten new messages for this case study, with the label that a person at Fernway would give. They are not from CLINC150. The word counts (100 per intent) run in the notebook. The adapter’s and the hosted replies are recordings.

my flight to lisbon tomorrow, is it still on tim flight_status       words flight_status       adapter flight_status       hosted flight_status
do i need a visa for vietnam with a uk passport  international_visa  words international_visa  adapter international_visa  hosted international_visa
the airline lost my suitcase in frankfurt, what  lost_luggage        words lost_luggage        adapter lost_luggage        hosted lost_luggage
how do you say where is the train station in jap translate           words translate           adapter travel_suggestion   hosted translate
can i bring a guitar as a carry on with delta    carry_on            words carry_on            adapter carry_on            hosted carry_on
is it safe to travel to egypt right now          travel_alert        words travel_alert        adapter travel_alert        hosted travel_alert
find me a cheap room in berlin for 3 nights from book_hotel          words book_hotel          adapter book_hotel          hosted book_hotel
i want a refund for the hotel you booked, it was out_of_scope        words book_hotel          adapter book_hotel          hosted out_of_scope
my card was charged twice for the same booking   out_of_scope        words out_of_scope        adapter exchange_rate       hosted out_of_scope
can you recommend a good dentist                 out_of_scope        words translate           adapter out_of_scope        hosted out_of_scope

What this tells us. The seven in-scope messages are easy for every system; the adapter calls “how do you say … in japanese” travel_suggestion. The three messages that need a person show the difference. The hosted system sends all three to a person. The adapter sends one: the refund for a dirty hotel becomes book_hotel, and the double charge becomes exchange_rate. The word counts send one, and offer translate to someone who wants a dentist. That is the stop rule in a customer’s words.

To run the adapter yourself, set RUN_MODEL = True in the notebook. It installs PyTorch, transformers and peft at the course project’s versions (about 700 MB), and downloads the small model at the course’s pinned revision (about 270 MB). Then it prints the adapter’s ten replies and marks any reply that differs from the recording. On the course’s computer, in a fresh environment, all ten were the same as the recording, and the notebook’s process used at most 1.4 GB of memory.

Train it yourself (optional)

You need the project from Before you start. Training takes about 4 minutes of CPU on the course’s computer.

1. In the notebook, run the cell under Optional: train again. It writes fernway/export/ with data/fernway/train.jsonl, data/fernway/val.jsonl and configs/fernway-r8.toml.

2. Copy the folders data and configs from fernway/export/ into your project folder. They add files and change none.

3. In the project folder, with the virtual environment active, check the plan:

   python -m adapt plan configs/fernway-r8.toml

   fernway-r8: 813 training records, 72 validation records
     batch 8, 2 epochs = 102 steps per epoch, 204 steps (1632 examples seen)
     validation loss every 25 steps (8 times), checkpoint every 50 steps (4)
     LoRA rank 8, alpha 16, dropout 0.05, on q_proj, v_proj; learning rate 0.0002; at most 320 tokens per example; 2 threads

4. Train: python -m adapt train configs/fernway-r8.toml. It writes runs/fernway-r8/. The last line is:

   Done: best validation loss 0.0515 at step 200; adapter 1.9 MB in runs/fernway-r8/adapter

   On the course’s computer, this adapter was identical, byte for byte, to the recorded one. On another computer, the last digits of the losses may differ.

5. In the notebook’s model cell, set ADAPTER to the full path of your runs/fernway-r8/adapter, RECORDED to "adapter" (your run is the first recipe), and RUN_MODEL = True. Run the cell: it prints your adapter’s replies to the ten new messages and marks any reply that differs from the recording.

The commands are the same on Windows (PowerShell), macOS and Linux. We tested them on macOS only.

Insights and limits

• Prompt only is a strong baseline. Measure it before anyone says “fine-tune”: here it made 198 of 200 dev decisions.
• Format and labels are cheap to learn; the rare decision is not. Both adapters learned the JSON and the intents in minutes. Neither learned “everything else”.
• More data is not more of the right data, and a lower loss is not a better decision. 717 more in-scope records halved the validation loss and made the rare class rarer.
• Limits. Crowd workers wrote the messages to order, in English. They are shorter and cleaner than real chat. CLINC’s out-of-scope messages are outside all 150 intents, so the test has no refund requests or complaints. A laptop CPU is not a phone. By design, we tried one recipe and one change. Models change fast: these numbers are a snapshot; the method lasts.

Responsible use

• A person answers what the model does not know. The design sends unclear messages to a person, and the measure counts the ones it does not.
• Test on real messages before any launch. CLINC150’s queries are not Fernway’s customers. Use consented messages, with personal data removed.
• Credit the data. CC BY 3.0 requires the attribution and a note of your changes; the dataset card has both.

Make it a portfolio piece

This case study shows an employer that you can stop a fine-tuning project with evidence, and say what would restart it. That is rarer than a training run. Make it yours with one change, and write the targets first. Use another CLINC150 domain (banking, kitchen, auto), or add out-of-scope training records from the other 135 intents.

What to put in the repository:

• README.md (template below).
• The notebook, run from top to bottom, with its outputs.
• decision-memo.md (written before training) and result-memo.md (after), with their commit dates.
• targets.toml, the training configuration, and the run’s summary.json and log.jsonl.
• dataset.md: the source, the licence, the attribution and your changes.

A README template:

# Should a travel app fine-tune a small model for its help chat?

## Question
Can a 135M-parameter model with a LoRA adapter replace a hosted model with a
prompt, offline, without sending out-of-scope messages to an automatic flow?

## Data and licence
CLINC150 (Larson et al., 2019), CC BY 3.0: travel domain (15 intents) and
out-of-scope queries. Fixed sample (seed 7); 45 near copies removed from training.

## Approach
Targets and a stop rule (out of scope caught >= 90%) written before training.
Five systems on the same 200 dev messages with one strict measure; LoRA rank 8
on a CPU (2 threads, under 1.8 GB); one change (more data); holdout once.

## Results
Prompt only: 198/200 decisions, 48/50 out of scope caught. Best adapter:
183/200, 40/50 caught. No-go: the stop rule fails; more in-scope data made
the rare class rarer.

## Limits
Crowd-written English queries; no real "other" messages; laptop CPU, not a phone.

How to describe it in an interview: “With targets written before training, I tested whether a small on-device model with a LoRA adapter could replace a hosted prompt for a travel app’s chat. The adapter learned the format and the intents in minutes on a CPU, but it missed messages that must reach a person. I recommended no-go and named the data that would change the answer.”

Go further:

• Add 400 out-of-scope training records from CLINC’s other domains, train once with the same recipe, and check the stop rule on dev.
• Read the adapter’s probability of out_of_scope at the first answer token, as Regressions and operational value did for “needs a person”. Does a threshold separate the classes?

Knowledge check

Check your understanding

The second adapter's validation loss is half the first one's (0.0241 against 0.0515), but it catches fewer out-of-scope messages on dev (40 against 41). How can both be true?

Show the answer

The validation loss is an average over the answer tokens of 72 validation records, and only 20 of them are out of scope. Better in-scope answers can lower the average while the rare class stays the same or gets worse. The second run added 717 in-scope records and no out-of-scope ones. So the out-of-scope share of training fell from 12% to 7%. The stop rule on the dev cases measures the decision that matters; the loss does not.

Check your understanding

The small model with a fixed choice catches 43 of 50 out-of-scope messages, more than either adapter. Why is it still the worst choice for Fernway?

Show the answer

It answers out_of_scope for 98 of the 200 dev messages. So it also hands over 55 in-scope messages that the app could answer, and it gets only 67 of 150 intents right. Its catches come from saying “out of scope” to almost everything. Always read a caught count next to the false handovers; the targets cap them at 7.

Check your understanding

Which evidence would make you recommend a second training experiment for Fernway, and what must not change before it?

Show the answer

Evidence that the stop rule could change: several hundred labelled out-of-scope messages, best from Fernway’s own chat. Then the rare class is no longer rare in training. New dev and holdout cases with Fernway’s real “other” messages would test it properly. What must not change: the targets and the stop rule, written before the run, and one look at the holdout cases. Another epoch, another rank or more in-scope data are not such evidence. The second run showed that more in-scope data did not help.

Recap

• You checked a public dataset’s licence at the source, drew a fixed sample and removed 45 near copies from training.
• You measured what the team has first. Prompt only: 198 of 200 dev decisions, 48 of 50 out-of-scope messages caught. Then the cheap alternatives, with one strict measure.
• You wrote targets and a stop rule before training, then trained the course’s recipe and changed one thing.
• The best adapter met four of five targets and missed the stop rule (40 of 50). More in-scope data helped the intents and made the rare class rarer.
• A threshold moves mistakes between columns; it does not remove them.
• The memo says no-go, and names the data that would change the answer.

Next, the final assignment: your own adaptation decision, end to end.

Next: Final assignment