Fine-tune a small transformer

What you will be able to do

Find out what a pretrained model buys you, with real numbers. You fine-tune a small pretrained transformer to route real support messages, and compare it with word counts and a transformer trained from scratch. You measure the gap at 100, 500 and thousands of labelled messages, and what each router costs.

Case studyPortfolio piece

Dataset

BANKING77 (PolyAI, 2020)

Licence

CC BY 4.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Pretrained encoder
• Fine-tuning
• Frozen features
• Learning curve
• Fair comparison
• Serving cost
• Model card review

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs the steps of this page on the smaller sizes, with the same code, plus two tasks with checks. Its setup cells download the data (about 1 MB), the course’s code and DistilBERT (about 270 MB). To finish in under 10 minutes on a free CPU (about 2 minutes on the course’s computer), it fine-tunes on 500 messages for 8 epochs, not on all of them. The numbers for all messages on this page come from the course’s longer run, and the page says where.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• The training loop and evaluation mode, from Batches, epochs and validation, and early stopping, from Overfitting controls.
• Where word counts fail, from Why word order and context matter.
• The transformer block and the tiny transformer, from Transformer blocks, and masked-token prediction, from Training objectives.
• Pretraining, adaptation and inference, and loading a model with from_pretrained(), from Training versus inference; memory and speed, from Inference costs.
• TF-IDF with logistic regression, and the test set used once, from the machine learning course.

What you need on your computer (only for the local route; the notebook needs none of it)

• The virtual environment from Set up PyTorch, with pip install "transformers>=5,<6" scikit-learn. DistilBERT downloads about 270 MB on first use. Fine-tuning needs about 2 GB of free memory.

Tested with Python 3.12, PyTorch 2.14.1, transformers 5.19.0 and scikit-learn 1.9.1 on a Mac mini CPU, in Jupyter from a fresh start. We did not run the notebook in Colab or Kaggle.

The situation

Nadia leads the support team of an online bank. Customers write short messages in the app’s chat: “My card still hasn’t arrived”, “Why did my top-up fail?”. Each message must reach the right handler. The bank has 77 intents, one for each kind of request, and each intent has its own answer flow. Today a person reads each message and picks the intent. A wrong pick costs time: the customer gets the wrong answer, writes again, and an agent sorts it out by hand.

The bank will soon launch a new product, with new intents. For the first weeks, only a few hundred messages will have a label. Kofi, the bank’s machine learning engineer, has read about pretrained models. Nadia asks him two questions: “Is a pretrained model better than what we can build ourselves? And is it worth the extra cost when we have only a few labels?”

This is also the open question of this course. On Larkfield’s tickets, the tiny transformer lost to word counts: 94.2% against 98.8% on the validation tickets. Training versus inference gave the reason. 4,696 short tickets are far too little text to learn a language from nothing. A pretrained model has already read a lot of text. Does that change the result?

What you will build

You will train four routers on the same messages and compare them under one split and one metric. You repeat the comparison with 100, 500, 2,000 and all 9,003 training messages. That repeated comparison is a learning curve: accuracy as a function of the number of labelled examples. Then you test the best router once, look at its errors and measure its costs. You check what the pretrained model brings with it, and you run the same method on Larkfield’s tickets.

The data

The data is BANKING77, published by the company PolyAI with a research paper in 2020. It has 13,083 short messages to a bank’s online support, each with one of 77 intents. The paper calls them “customer service queries”; it does not say who wrote them. Treat them as realistic messages, not as a sample of one bank’s customers.

Source and licence

“BANKING77” by PolyAI (Casanueva, Temčinas, Gerz, Henderson and Vulić, 2020), from github.com/PolyAI-LDN/task-specific-datasets, licensed under CC BY 4.0. Paper: Efficient Intent Detection with Dual Sentence Encoders, arXiv 2003.04807. The notebook downloads the files from a fixed version (commit 57ec275) and checks their SHA-256. We changed nothing in them.

Why this dataset? Its task is Larkfield’s task with more teams, and its fine-grained labels give word counts a harder test. Its short messages make fine-tuning possible on an ordinary CPU, and its licence is clear. Many popular review datasets have unclear terms. The dataset card in the notebook folder lists the other candidates and why they lost.

Kofi loads the two files and looks at their size, the intents and three of them:

from collections import Counter
print(len(raw_train), "train |", len(raw_test), "test | columns:", list(raw_train[0].keys()))
counts = Counter(r["category"] for r in raw_train)
print("intents:", len(counts), "| messages per intent in train: min", min(counts.values()), "max", max(counts.values()))
print("test, per intent:", sorted(set(Counter(r["category"] for r in raw_test).values())))
for intent in ("top_up_failed", "top_up_reverted", "pending_top_up"):
    print(f"{intent:16}", [r["text"] for r in raw_train if r["category"] == intent][:2])

The output:

10003 train | 3080 test | columns: ['text', 'category']
intents: 77 | messages per intent in train: min 35 max 187
test, per intent: [40]
top_up_failed    ['I think my top-up has failed.', 'Top-up is not working']
top_up_reverted  ['My top up did not show up as shown and my money has disappeared, can you help me fix this glitch problem?', 'Has my top-up been cancelled?']
pending_top_up   ['How long does a top-up take to go through?', 'I am under the impression that my top up is still pending']

What this tells us. One row is one message and its intent; there is nothing else. The train file is uneven: contactless_not_working has 35 messages, card_payment_fee_charged 187. The test file has exactly 40 messages per intent. The three top-up intents share their key words, “top-up”, “money” and “not”. Only the rest of the message says whether a top-up failed, came back or is still waiting. That is the kind of message where word counts fail.

First checks

The numbers script checks the rest. There are no empty messages. 6 messages of the train file repeat another one. 10 test messages also appear in the train file, with the same words. Both are too few to change a result, so Kofi keeps the files as they are and notes it. The messages are short: a median of 10 words, the longest 79.

Frame the problem

• The unit: one message, as the customer sends it.
• The target: the intent, one of 77.
• The prediction time: when the message arrives, before anybody reads it. The input is the text alone.
• The metric: accuracy, the share of messages routed to the right intent. Every wrong intent costs about the same here: one more message from an annoyed customer. With 77 intents, always answering the most common one is right only 1.9% of the time.
• Sensitive content: there are no columns about people, but a message can contain a name or other personal details. The router must not depend on them. Section “Responsible use” checks it.

Split

The official test file stays sealed until the end. Kofi holds out 1,000 messages of the train file as the validation set, with the same share of each intent. Every model is compared on these same 1,000 messages. 9,003 messages remain for training. For the learning curve, subset() draws a training sample of 100, 500 or 2,000 messages, again with the same share of each intent:

from sklearn.model_selection import train_test_split

train, valid = train_test_split(raw_train, test_size=1000, stratify=[r["category"] for r in raw_train], random_state=0)

def subset(rows, n, seed=0):
    """n rows with the same share of each intent; all rows when n >= len(rows)."""
    if n >= len(rows):
        return list(rows)
    return train_test_split(rows, train_size=n, stratify=[r["category"] for r in rows], random_state=seed)[0]

In a sample of 100, most intents get one message, a few get two, and 2 get none. That imitates the new product’s first weeks. With so few messages, the result depends on which ones were drawn. So for 100 and 500, Kofi trains every model on three different samples (seed 0, 1 and 2). The page reports the median and the range.

The new idea: pretraining, then fine-tuning

Two new colleagues

Nadia hires two new agents for the chat. One has years of experience in another bank’s support. She reads the 77 intents and a few examples of each, and she routes well by lunch. The other, in this story, has never learned English. He must learn the language from the examples alone, and a few hundred examples are not enough.

The first colleague is a pretrained model. The second is a model trained from scratch.

The problem: a few labels hold too few words

A model that learns only from labelled messages knows only the words in them. With 100 training messages, the vocabulary has 364 words, and 12.7% of the words in the validation messages never appear in training. “arrived” may be there, but “come” may not. For TF-IDF and for the tiny transformer, an unseen word counts for nothing. With all 9,003 messages, only 0.7% of the validation words are unseen. A model that has read a lot of English knows that “my card hasn’t come” means “my card hasn’t arrived”. It knows this even if no labelled message says so.

The idea and its name

Pretraining is the long first training on a lot of unlabelled text, as Training objectives showed. Fine-tuning takes the pretrained network and adds a new output layer for your task. Then it trains everything a little more on your labelled examples, with a small learning rate. The name says what it does: the network is already good in general, and the extra training “tunes” it finely for one job. The new output layer is called the head, because it sits on top of the network.

This case study uses DistilBERT, a small pretrained encoder. An encoder is a transformer that reads the whole message in both directions. The tiny transformer of Module 6 does the same; the decoder SmolLM2 of Module 7 does not. Its makers pretrained it on books and English Wikipedia with masked-token prediction: hide 15% of the words, predict them. It has 6 blocks, 12 attention heads and 768 numbers per token: 66,362,880 parameters, about 320 times the tiny transformer. The name says “distilled”: it was trained to copy a larger model, BERT, and is 40% smaller. It is open, under the Apache-2.0 licence.

There are two ways to use it, and the case study measures both:

Three ways to build the router. The blue boxes are what training changes or what comes from pretraining. Only the fine-tuned model changes pretrained weights.

Text description of the diagram

Three rows, each from a message to an intent.

1. From scratch: the message goes into the course’s tiny transformer with random weights. It must learn English and the intents from the labelled messages only. A new head gives 77 scores, one per intent.
2. Frozen: the message goes into DistilBERT, pretrained on books and Wikipedia. Its weights do not change. It gives 768 numbers per message. Logistic regression, the only part trained, turns them into an intent.
3. Fine-tuned: the message goes into DistilBERT, which starts from the pretrained weights. All its weights change a little during training, with a small learning rate. A new head gives 77 scores.

When to use it, and what it costs: fine-tuning makes sense when your labelled data is too small to learn the language. The text should also be like the text of pretraining. The cost is a router about 380 times larger and much slower than a word-count router. Its licence and training data come from someone else, and every retrain takes fine-tuning time. The rest of this page measures each of these.

Four routers

Each router gets the same training samples and is judged on the same 1,000 validation messages. Each one also gets a small, equal amount of tuning on the validation set, so that no model loses only because of a bad setting.

1. Bag of words: TF-IDF and logistic regression

The baseline is the machine learning course’s router, which also routes Larkfield’s tickets best. It uses TF-IDF word weights and logistic regression, with the course’s word tokens (tokenize from ticketnet.py).

def fit_tfidf(texts_, y, min_df=1, C=100):
    return make_pipeline(TfidfVectorizer(token_pattern=None, tokenizer=tokenize, lowercase=False, min_df=min_df),
                         LogisticRegression(max_iter=3000, C=C)).fit(texts_, y)

Kofi tried six settings, each on samples of 100, 500 and all messages. min_df was 1 or 2: keep words seen at least once or twice. C was 1, 10 or 100: the regularization gets weaker as C grows. min_df=1, C=100 had the best mean validation accuracy (30.5%, 60.8% and 88.0%). The course’s Larkfield setting, min_df=2, C=1, was the worst (16.4%, 52.9% and 86.1%). With few messages per intent, the model needs every word and little regularization.

2. From scratch: the course’s tiny transformer

The course’s TinyTransformer comes from ticketnet.py. Only its last layer changes: 77 outputs instead of Larkfield’s 5. Training is the course’s loop with Adam at 0.001 and batches of 32. It keeps the epoch with the lowest validation loss (early stopping). Small samples get more epochs (150 for 100 messages, 20 for all), so that every run has enough steps.

def tiny_transformer(vocab_size, n_classes, seed=0):
    torch.manual_seed(seed)
    model = TinyTransformer(vocab_size)
    model.out = nn.Linear(64, n_classes)          # 77 intents instead of 5 teams
    return model

The vocabulary is built from each training sample, as in the course. With all 9,003 messages, it has 1,450 words (seen at least twice) and the network 205,581 parameters.

3. Frozen: DistilBERT as a feature maker

The cheapest way to use DistilBERT is to run each message through it once, with no training. The last layer gives 768 numbers per token; their average is one row per message. Then logistic regression learns the intents from these rows.

def features(texts_, batch_size=64):
    """The frozen encoder: the average of the last layer's token vectors (padding excluded), 768 per message."""
    out = []
    with torch.no_grad():
        for i in range(0, len(texts_), batch_size):
            b = encode(texts_[i:i + batch_size])
            h = encoder(**b).last_hidden_state                     # (batch, tokens, 768)
            m = b["attention_mask"].unsqueeze(-1).float()          # (batch, tokens, 1): 0 for padding
            out.append((h * m).sum(1) / m.sum(1))                  # (batch, 768)
    return torch.cat(out).numpy()

encode() calls DistilBERT’s tokenizer, which cuts each message into subword pieces and pads a batch to its longest message. attention_mask is 1 for a real token and 0 for padding, so the average skips the padding. The shapes in the comments are the course’s habit. The logistic regression on top standardizes the 768 numbers; C=1 won a grid of three.

4. Fine-tuned DistilBERT

AutoModelForSequenceClassification builds DistilBERT with a new head of 77 scores. The pretrained weights are loaded; only the head starts at random. The training loop has the course’s five steps, with batches of 16. labels= makes the model compute the cross-entropy loss itself:

def fine_tune(model, texts_, y, epochs, batch_size=16, lr=5e-05, seed=0):
    torch.manual_seed(seed)
    opt = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=0.01)
    y = torch.as_tensor(y)
    for epoch in range(1, epochs + 1):
        model.train()                              # training mode: dropout on
        order = torch.randperm(len(texts_)).tolist()
        for i in range(0, len(order), batch_size):
            idx = order[i:i + batch_size]
            opt.zero_grad()                                                    # 1
            out = model(**encode([texts_[j] for j in idx]), labels=y[idx])     # 2 and 3
            out.loss.backward()                                                # 4
            opt.step()                                                         # 5
        print(f"epoch {epoch}: last batch loss {out.loss.item():.3f}")
    return model

torch.manual_seed(0)
classifier = AutoModelForSequenceClassification.from_pretrained(MODEL, revision=REVISION, num_labels=len(INTENTS))
print("parameters with the new head:", count_parameters(classifier))

The output:

parameters with the new head: 67012685

The head adds 649,805 parameters: a layer of 768 × 768 and the output layer of 768 × 77, with their biases. AdamW is Adam with weight decay applied directly to the weights, the usual choice for fine-tuning. The learning rate is the key setting. It must be small, or the first steps destroy what pretraining learned. Kofi tried three rates for 10 epochs on 500 messages: 0.00005 reached 70.3%, 0.0001 reached 69.7%, and 0.0002 failed completely (see the failure case below). He kept 0.00005. The page’s runs then train longer, up to 80 epochs for 100 messages and 4 for all, and keep the best epoch. Each run keeps the epoch with the lowest validation loss, as for the tiny transformer.

The learning curve: one split, one metric

Every router, at every size, judged on the same 1,000 validation messages:

• TF-IDF + logistic regression
• Tiny transformer, from scratch
• Frozen DistilBERT (dashed)
• Fine-tuned DistilBERT

BANKING77: validation accuracy of the four routers with 100, 500, 2,000 and all 9,003 training messages (median of three samples for 100 and 500). Fine-tuning leads from 500 messages on; the tiny transformer trails at every size.

Text description of the chart

A line chart with four lines. The horizontal axis has four sizes: 100, 500, 2,000 and 9,003 training messages; the vertical axis is the validation accuracy from 0% to 100%. TF-IDF: 30.5%, 60.8%, 79.3%, 88.0%. The tiny transformer from scratch: 4.6%, 34.3%, 58.9%, 79.6%. Frozen DistilBERT with logistic regression, dashed: 29.5%, 60.6%, 77.5%, 87.0%, almost on top of TF-IDF. Fine-tuned DistilBERT: 32.2%, 72.2%, 85.5%, 91.6%. At 100 messages, the three lines other than the tiny transformer are close together near 30%. At 500, fine-tuning is 11 points above the others.

Training messages	100	500	2,000	9,003
Messages per intent, about	1.3	6.5	26	117
TF-IDF + logistic regression	30.5% (30.3–32.2)	60.8% (59.9–61.5)	79.3%	88.0%
Tiny transformer, from scratch	4.6% (4.5–7.0)	34.3% (32.7–35.0)	58.9%	79.6%
Frozen DistilBERT + logistic regression	29.5% (26.5–30.5)	60.6% (58.9–62.0)	77.5%	87.0%
Fine-tuned DistilBERT	32.2% (26.9–32.4)	72.2% (70.5–72.3)	85.5%	91.6%
Fine-tuned minus TF-IDF	+1.7	+11.4	+6.2	+3.6

What this tells us. Four findings, in the order of the chart:

• The tiny transformer from scratch is last at every size, as on Larkfield’s tickets. With 100 messages it is barely above chance (4.6%, against 1.9% for the most common intent). With all 9,003 it still trails word counts by 8 points.
• Fine-tuning is ahead from 500 messages on, but not by the same amount. With about 1 message per intent, nothing works. Three models are near 30%, and their ranges overlap. With 6 or 7 per intent (500 messages), fine-tuning is 11 points ahead: that is where pretraining helps most. With more messages, word counts catch up, and the gap shrinks to 3.6 points.
• Frozen features do not help. The frozen encoder with logistic regression is never better than TF-IDF. The 768 numbers were made for predicting masked words, and their average blurs what separates top_up_failed from top_up_reverted. Pretraining pays off here only when the encoder’s weights may change.
• The spread between samples is real. At 100 messages, one sample gave the fine-tuned model 26.9% and another 32.4%. With few labels, which messages you label matters as much as the model.

So the honest summary is not “pretraining always helps most with little data”. It helps most in the middle: enough examples to show each intent a few times, too few to learn its words from them alone.

The fine-tuning curve

With all messages, one epoch of fine-tuning already reaches 87.0% on validation, as much as the frozen features ever do. The validation loss is lowest after epoch 3 (91.6%); epoch 4 is a little worse, so early stopping keeps epoch 3. With 500 messages, the best epoch is between 22 and 24. With 100 messages, the validation loss was still creeping down after 80 epochs, at about 33% accuracy. Few examples give few steps, and the new head learns slowly.

What each router costs

Accuracy is half of Nadia’s question. The other half is what each router costs to train and to run. The numbers script measured it on the course’s computer, a Mac mini with 2 CPU threads. Other people were using it at the same time. Treat the times as orders of magnitude, not exact values.

	TF-IDF + logistic regression	Tiny transformer	Fine-tuned DistilBERT
Parameters	174,559	205,581	67,012,685
Size on disk	1.4 MB	0.8 MB	268 MB
Training, all 9,003 messages	about 1 second	about 2.5 minutes (20 epochs)	about 8 minutes (4 epochs)
Peak memory of the process, training	about 400 MB	about 650 MB	about 2,300 MB
Routing 1,000 messages	0.004 seconds	0.1 seconds	3.6 seconds
Routing one message	0.2 milliseconds	0.2 milliseconds	7 milliseconds

The training times include a validation pass after every epoch, as the course’s loops do. The script measured peak memory in a fresh Python process for each router. It ran a full fit for TF-IDF, one epoch for the tiny transformer and 30 fine-tuning steps for DistilBERT. Routing 1,000 messages with DistilBERT took about 1,160 MB. The frozen encoder costs the same as the fine-tuned one to run, because it runs the same 66 million parameters for every message.

What this tells us. The fine-tuned router is about 380 times larger than the word-count router, and about 900 times slower per batch. In absolute terms, 7 milliseconds per message is still fast for a chat, and 268 MB fits on any server. But each copy needs about a gigabyte of memory, and a cheap CPU will be slow at high traffic. Every retrain takes minutes, not seconds. Training needs about twice the memory of routing: weights, gradients and AdamW’s two running averages, as Inference costs explained. All of it ran on an ordinary computer, because the model is small. A model 100 times larger would not.

The final model, and the test, once

Every choice is made: the four routers trained on all 9,003 messages, with the settings above. Kofi opens the official test file once: 3,080 messages that no step has touched, 40 per intent.

Router, all 9,003 messages	Validation	Test, once
TF-IDF + logistic regression	88.0%	89.0%
Tiny transformer, from scratch	79.6%	80.1%
Frozen DistilBERT + logistic regression	87.0%	87.8%
Fine-tuned DistilBERT	91.6%	91.8%

What this tells us. The order is the same as on validation. The fine-tuned model routes 2,826 of the 3,080 test messages right, against 2,740 for TF-IDF: 86 more, 2.8 points. Its macro F1, the mean of the F1 scores of the 77 intents, is 0.918 against 0.890. So the gain is not limited to a few large intents. The test agrees with the validation set within 1.2 points for every router, so the small tuning did not fool us.

New messages, end to end

These six messages are made up for this case study; no customer wrote them. The two transfer and top-up messages use the same words in a different order. They are like the lamp tickets of Why word order and context matter. The code is the notebook’s; the output is from the model fine-tuned on all 9,003 messages.

p_new = predict_proba(classifier, NEW)
for text, p in zip(NEW, p_new):
    top = torch.topk(p, 2)
    print(text)
    print("   ", ", ".join(f"{INTENTS[int(i)]} {v:.3f}" for v, i in zip(top.values, top.indices)))

The output:

I topped up my account this morning but the money still isn't there.
    top_up_reverted 0.547, pending_top_up 0.227
My new card still hasn't arrived and I ordered it two weeks ago.
    card_arrival 0.992, lost_or_stolen_card 0.001
I was charged twice for the same coffee.
    transaction_charged_twice 0.992, extra_charge_on_statement 0.001
The transfer went through, but the top-up failed.
    top_up_failed 0.951, top_up_reverted 0.015
The top-up went through, but the transfer failed.
    top_up_failed 0.911, top_up_reverted 0.033
Can I get my money back for something I bought online? The seller never sent it.
    request_refund 0.878, reverted_card_payment? 0.071

What this tells us. The clear messages get clear answers, with 0.99. The first message is truly unclear: “the money still isn’t there” fits a reverted top-up and a pending one. The model splits its probability, 0.55 and 0.23. TF-IDF sends it to balance_not_updated_after_cheque_or_cash_deposit, which is wrong, and it routes the refund question to reverted_card_payment? (the dataset’s own label, with its question mark). The fine-tuned model gets both right.

The same-words pair is the honest surprise. TF-IDF gives both messages exactly the same probabilities, as it must: same words, same input. The fine-tuned model can tell them apart, and its probabilities do differ (0.951 and 0.911). But it sends both to top_up_failed; failed_transfer gets only 0.025 for the second one. It reads order, but on this pair the word “top-up” still wins. Reading order is a capacity, not a guarantee.

Where it goes wrong

The fine-tuned model routes 84 of the 1,000 validation messages wrongly, TF-IDF 120. They do not fail on the same messages:

	TF-IDF right	TF-IDF wrong
Fine-tuned right	848	68
Fine-tuned wrong	32	52

So fine-tuning fixes 68 of TF-IDF’s errors and adds 32 of its own. Its most common confusions are between intents that a person could also mix up. It takes card_delivery_estimate for card_arrival 3 times, pending_transfer for transfer_timing 3 times, and supported_cards_and_currencies for fiat_currency_support 3 times. Its weakest intent is card_delivery_estimate: 7 of 11 right, where TF-IDF gets 9. “Can you tell me when my card will arrive?” is labelled card_delivery_estimate; the model says card_arrival. Several labels sit on a boundary that the dataset does not define.

Two of the 68 fixes show what pretraining adds. “You have declined my payment.” is declined_card_payment; TF-IDF says declined_transfer. “Do you provide currency exchange to Euros?” is fiat_currency_support; TF-IDF says exchange_via_app, from the word “exchange”. The fine-tuned model reads the whole question.

How sure is it? On the validation set, the median probability of its top intent is 0.986 when it is right and 0.721 when it is wrong. That makes a fallback possible: send a message to a person when the top probability is below a threshold. At 0.5, 2.8% of the messages go to a person, and the router is right on 93.1% of the rest. It is not a cure: 22 wrong answers still have a probability above 0.9.

Common mistake

A learning rate that erases the pretraining

Symptom: Tomás wants faster training and fine-tunes with a learning rate of 0.0002 instead of 0.00005. On 500 messages, the best of 10 epochs reaches 1.8% validation accuracy: worse than always answering the most common intent. With 0.00005 and 0.0001, the same runs reach 70.3% and 69.7%.

Cause: fine-tuning starts from weights that already work. Large steps in the first batches, while the new head is still random, push those weights far from where pretraining left them. The network loses what it knew, and the few labels cannot teach it again.

Fix: use small learning rates for fine-tuning, usually between 0.00001 and 0.0001. Try two or three on the validation set, as Kofi did. Watch the first epochs: if the validation loss does not fall, stop and lower the rate.

What it means for a real AI service: a fine-tuning run can fail silently, with no error message. Always compare the fine-tuned model with a simple baseline on the same validation set before you use it.

Back to Larkfield

Priya runs the same method on Larkfield’s tickets, with the course’s split: 4,696 training tickets, 600 for validation, 600 for test. The tickets are longer than the bank’s messages, so DistilBERT reads the first 128 tokens of each. 226 training tickets are longer and lose their end; the course’s own networks cut at 120 word tokens. TF-IDF uses the course’s settings, so that its result on all tickets is the course’s 98.8%. For all tickets, the tiny transformer’s number is the course’s own run from Module 6. The numbers script trains it on the samples of 100 and 500.

• TF-IDF + logistic regression
• Tiny transformer, from scratch
• Frozen DistilBERT (dashed)
• Fine-tuned DistilBERT

Larkfield's tickets: validation accuracy with 100, 500 and all 4,696 training tickets (median of three samples for 100 and 500). Fine-tuning leads by far with 100 tickets; with all of them, it ties with word counts.

Text description of the chart

A line chart with four lines over three sizes: 100, 500 and all 4,696 training tickets. The vertical axis is the validation accuracy from 40% to 100%. TF-IDF: 71.8%, 95.3%, 98.8%. The tiny transformer from scratch: 44.8%, 75.5%, and 94.2% (the course’s run). Frozen DistilBERT, dashed: 74.2%, 87.7%, 94.3%. Fine-tuned DistilBERT: 85.2%, 97.2%, 98.7%. With 100 tickets, fine-tuning is 13 points above TF-IDF; with 500 the gap is 2 points; with all tickets the two lines meet.

Training tickets	100	500	4,696
Tickets per team, about	20	100	939
TF-IDF + logistic regression	71.8% (69.3–72.0)	95.3% (94.8–95.5)	98.8%
Tiny transformer, from scratch	44.8% (42.5–46.8)	75.5% (72.8–75.8)	94.2%
Frozen DistilBERT + logistic regression	74.2% (70.0–79.5)	87.7% (87.2–89.7)	94.3%
Fine-tuned DistilBERT	85.2% (82.8–92.8)	97.2% (96.5–97.5)	98.7%

What this tells us. The pattern of the bank’s data comes back, moved to the left. With 100 tickets, about 20 per team, fine-tuning is 13 points ahead of word counts. Five teams with distinct words need far fewer examples than 77 close intents, so word counts catch up early. With 500 tickets the gap is 2 points. With all tickets the fine-tuned model is one ticket behind, 98.7% against 98.8%.

Look at the tickets where order matters. On the validation set, the fine-tuned model routes all 52 two-topic tickets right. TF-IDF routes 88.5% of them, and the tiny transformer 84.6%. With the words of each validation ticket shuffled, it falls from 98.7% to 86.7%. The tiny transformer fell only 1 point, to 93.3%, and the bag of words cannot notice at all. So the fine-tuned encoder really reads the order. Its 8 wrong validation tickets are mostly on a policy boundary: 5 payment tickets about a refund that it sends to returns.

Then the test, once: fine-tuned DistilBERT 97.2%, TF-IDF 99.2%, the tiny transformer 91.2% (the course’s run). On new tickets, the pretrained model is worse than word counts by 2 points, 12 tickets out of 600.

So the course’s open question has its answer. Pretraining fixes what the tiny transformer lacked. With all tickets, the gap of 4.6 points to word counts closes; with 100 tickets, the pretrained model is far ahead. But on Larkfield’s tickets the words decide the team. There it does not beat the machine learning course’s linear model, and it costs hundreds of times more to run. Larkfield keeps TF-IDF and logistic regression. Try the fine-tuned model again if Larkfield adds a new team with only a few dozen labelled tickets. The same holds if two-topic tickets become common.

Recommendations

Kofi writes to Nadia in her words:

• Use the fine-tuned model for the chat. On 3,080 test messages it picks the right intent 91.8% of the time, against 89.0% for the word-count router. That is about 3 fewer wrong answers per 100 messages.
• For the new product, label a few examples of every intent before you switch it on. With one or two messages per intent, no model works. With six or seven, the pretrained model is 11 points ahead of word counts. On this data, those labels changed the result more than any model choice.
• Send unsure messages to a person. With a threshold of 0.5, about 3 messages in 100 go to an agent. The router is right on 93% of the rest. Choose the threshold from what an agent’s time and a wrong answer cost.
• Budget for it. One copy of the router needs about 1.2 GB of memory and about 7 milliseconds per message on a CPU. Retraining takes minutes. Word counts need a fraction of that.

And what not to use it for:

• Not for decisions about a customer, such as fraud checks, credit or closing an account. It was built to choose an answer flow, where a mistake costs a second message.
• Not for other banks, other languages or other channels without a new test on their messages.

Limits and next steps

• Unknown writers. The dataset does not say who wrote the messages. Real customers write longer, mix two requests, and make more typos. Test on the bank’s own messages before launch.
• Debatable labels. Several intents overlap, such as card_arrival and card_delivery_estimate. Part of the remaining error is in the labels, not the model.
• One run for the large sizes. Only 100 and 500 messages have three samples. For 2,000 and 9,003, a different seed could move the result by a point.
• A small tuning budget. Each router got a grid of three to six settings. A larger search might narrow or widen the gaps, so the gaps are estimates.
• Times from a shared computer. The costs are orders of magnitude. Measure on the real server.
• Next: freeze the lower blocks and fine-tune only the top ones, to cut training time. Try a smaller pretrained encoder. Check the fallback threshold with real agents.

Responsible use

A pretrained model brings more than its skills. Its makers chose its training text, and what it learned from that text comes with it.

What the pretrained model learned. DistilBERT was pretrained on BookCorpus, a collection of unpublished books, and English Wikipedia. Its model card has a section “Limitations and bias” with a test. The model fills in the hidden word of “The white man worked as a [MASK]” and “The Black woman worked as a [MASK]”. The numbers script repeated that test with the exact version used here. For the white man, the top words are blacksmith, carpenter, farmer, miner and butcher. For the Black woman, they are waitress, nurse, maid, a sexual stereotype, and housekeeper. The card says: “This bias will also affect all fine-tuned versions of this model.” Read the card of every model you use.

Does it reach the router? A bias in the pretrained weights matters only where the task can express it. Kofi tests the router directly with 8 messages that mention a person, such as “I sent money to James yesterday but it hasn’t arrived”. Each message gets 10 first names from different backgrounds, such as Emily, Mohammed, Aisha, Wei and Olga. In all 8 messages, every name gives the same intent. That is good evidence for this task and these templates, not proof for every message. Repeat such checks with the bank’s own messages, and with other personal details, such as a disability or a country.

Data and licences. Chat messages may contain personal details, so logs of the router need the same protection as the chat. BANKING77 (CC BY 4.0) must be credited to PolyAI in any copy. DistilBERT (Apache-2.0) allows commercial use; record its name and exact revision with the router.

People in the loop. The router chooses an answer flow; it decides nothing about the customer. Keep it that way, and let customers reach a person when the answer does not fit.

Make it a portfolio piece

A case study like this shows an employer that you can use a pretrained model with judgement. You measured it against simple baselines, at several data sizes, with its costs. Make it yours with one change. For example, try a different small encoder, a fallback to a person, or another openly licensed text dataset. Then make it public, and make it easy to read in five minutes.

What to put in the repository:

• README.md, the page that people read first (template below).
• The notebook, run from top to bottom, with its outputs.
• MODEL_CARD.md. Intended use: routing chat messages to an intent, with a person for unsure messages. Not for: decisions about customers. Then the data, the pretrained model and its revision, the evaluation, the limits, and what the pretrained model’s own card says about bias.
• EXPERIMENT.md, as in Save and inspect results. Record the model name and revision, the data commit and checksums, and the seeds. Add the library versions, the computer and the training times.
• The data source and licence, with the attribution text from the box above. Do not commit the fine-tuned weights (about 270 MB); a reader can rebuild them from the notebook.

A README template:

# Does pretraining pay off for routing support messages?

## Problem
Route short bank-support messages to one of 77 intents. For a new product,
only a few hundred messages have labels at first.

## Data and licence
BANKING77 (PolyAI, 2020), 13,083 messages, 77 intents, CC BY 4.0,
github.com/PolyAI-LDN/task-specific-datasets (commit 57ec275).
1,000 train messages held out for validation; official test used once.

## Approach
Four routers, one split, one metric (accuracy): TF-IDF + logistic regression,
a tiny transformer from scratch, frozen DistilBERT + logistic regression,
fine-tuned DistilBERT (distilbert-base-uncased, Apache-2.0). Learning curve
with 100 / 500 / 2,000 / 9,003 training messages, 3 samples for the small sizes.

## Results
Validation accuracy (1,000 messages), median of 3 samples for 100 and 500:
                     100     500   2,000   9,003
TF-IDF + LR        30.5%   60.8%   79.3%   88.0%
Tiny transformer    4.6%   34.3%   58.9%   79.6%
Frozen DistilBERT  29.5%   60.6%   77.5%   87.0%
Fine-tuned         32.2%   72.2%   85.5%   91.6%
Test (3,080 messages, once): fine-tuned 91.8%, TF-IDF 89.0%.

## Costs
Fine-tuned: 67.0 M parameters, 268 MB, about 7 ms per message on a CPU,
about 8 minutes to train on all messages. TF-IDF: 175 k parameters, 1.4 MB,
0.2 ms per message, about 1 second to train.

## Limits
Who wrote the messages is not documented; one kind of bank, English only;
the pretrained model inherits biases from its books and Wikipedia text.

How to describe it in an interview, in two sentences: “I fine-tuned DistilBERT to route bank-support messages to 77 intents, and compared it with word counts and a transformer from scratch, with 100 to 9,000 labels. Pretraining paid off most with a few examples per intent (11 points at 500 messages, 2.8 on the final test), so I documented when the 380-times-smaller word-count model is the better choice.”

Go further:

• Plot the router’s accuracy against the share of messages sent to a person, for thresholds from 0.3 to 0.9.
• Freeze all blocks but the last two, fine-tune only those and the head, and compare accuracy, time and memory.
• Run three seeds for the size 2,000 and for all messages too, and show the spread.

Knowledge check

Check your understanding

Frozen DistilBERT features with logistic regression were never better than TF-IDF, but fine-tuned DistilBERT was better from 500 messages on. Both use the same pretrained weights. Explain the difference.

Show the answer

The frozen route keeps every pretrained weight fixed and averages the last layer’s vectors. Those vectors were shaped for predicting masked words. Their average blurs the details that separate close intents, such as a failed and a reverted top-up. Logistic regression on top can only draw straight lines through what it is given. Fine-tuning lets every weight move a little toward the routing task. So the encoder learns which details matter for the 77 intents, and it keeps what it knows about English.

Check your understanding

With 100 labelled bank messages, three models were near 30%. With 100 Larkfield tickets, fine-tuning reached 85% and TF-IDF 72%. Both are 100 examples. Why do the results differ?

Show the answer

What matters is the number of examples per class, not the total. 100 bank messages over 77 intents give one or two examples per intent. That is too few for any model to learn what separates them. 100 Larkfield tickets over 5 teams give 20 per team. With a few examples per class, a pretrained model can use what it already knows about the words. That is where it helps most. With enough examples per class, word counts catch up.

Check your understanding

Tomás says: 'The fine-tuned model is better, so Larkfield should switch to it too.' Use the case study's numbers to answer him.

Show the answer

On Larkfield’s tickets, the fine-tuned model ties with TF-IDF on validation (98.7% against 98.8%) and is worse on the test set (97.2% against 99.2%). It routes the two-topic tickets better. But it is hundreds of times larger and slower, and it needs minutes to retrain. On the bank’s data it wins because the 77 intents are fine-grained and share words. Larkfield’s five teams are mostly decided by the words. So Larkfield should keep TF-IDF and logistic regression. It can try fine-tuning again if a new team starts with few labelled tickets.

Recap

• Fine-tuning starts from pretrained weights, adds a new head, and trains everything a little more. The learning rate must be small: 0.00005 here, while 0.0002 erased the pretraining.
• On BANKING77, with one split and one metric, fine-tuned DistilBERT beat word counts from 500 messages on. It led by 11.4 points with 500 messages and 3.6 points with all. On the test set, used once: 91.8% against 89.0%.
• Pretraining helped most with a few examples per intent. With about one per intent, nothing worked. Frozen features alone never beat TF-IDF, and the tiny transformer from scratch was last at every size.
• The price: 67 million parameters, 268 MB, 7 milliseconds per message and minutes to retrain. Word counts need 175,000 parameters and a fraction of a millisecond.
• On Larkfield’s tickets, fine-tuning closed the tiny transformer’s gap but did not beat word counts; Larkfield keeps its linear model.
• A pretrained model brings its makers’ training data and biases. Read its card, test the task for them, and keep a person for unsure messages.

Next, the last case study goes the other way. It trains a small decoder from scratch to write text, and shows how far that is from a real language model.

Next: Train a tiny language model