Classify banking requests with an LLM

What you will be able to do

Find out whether a prompt can replace a trained model. You route real bank-support messages to 77 intents with a hosted modelTermHosted modelA model that runs on a provider’s computers. Your code sends it requests over the internet, and you pay for each call.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One, a prompt and a strict schema, and no training examples. Then you compare it honestly with routers trained on 9,003 labelled messages: accuracy, errors, cost, speed, and what data leaves the bank.

Case studyPortfolio piece

Dataset

BANKING77 (PolyAI, 2020)

Licence

CC BY 4.0

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Prompt versions
• Structured output with a large enum
• Validation in code
• Fair comparison
• Confidence intervals
• Cost per 1,000 requests
• Fallback to a person
• Data minimisation

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is an analysis, so it comes as a notebook, not as a project folder. The application code is small: a prompt file, a schema and a check, as in Modules 2 and 3. The real work is measuring about 6,000 recorded answers and deciding from them. The notebook replays the real answers, so you need no account and no key, and it runs in under a minute. A live call to a local model or your own key is optional, in its last cell.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses Modules 2 and 3 most:

• Versioned prompts, a fixed set and the per-message view, from Compare prompts.
• Enums, strict structured output and “validate anyway”, from Define a schema. The model’s own confidence field, from Validate meaning.
• The 95% Wilson interval, from How sure can 24 tickets be? in the language models course, and the paired interval, from Compare alternatives in the mathematics course.
• Hosted and self-hosted models, from Hosted and self-hosted choices, and a provider’s data terms, from Licensing and data terms.

The deep learning course’s case study Fine-tune a small transformer trained the routers that you compare with here. It helps to know it, but you do not need it: this page quotes what it measured.

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code, use a virtual environment with pandas, scikit-learn and pydantic (version 2). We tested it in Jupyter on macOS, from a fresh start, with Python 3.12, pydantic 2.14.0 and scikit-learn 1.9.1. We did not run it in Colab or Kaggle.

Models and APIs change fast. The answers, prices and times on this page are a snapshot. The method is what lasts.

The situation

Nadia leads the support team of an online bank. Customers write short messages in the app’s chat: “My card still hasn’t arrived”, “Why did my top-up fail?”. Each message must reach one of 77 intents, and each intent has its own answer flow. In the deep learning course, Kofi, the bank’s machine learning engineer, trained a router for this. His fine-tunedTermFine-tuningMore training of a pretrained model on your own labelled examples, so that it does one task well. Its weights change a little.Learn more Training versus inferenceDeep Learning and Transformers Explained DistilBERT model chose the right intent for 91.8% of the test messages. It learned from 9,003 messages that people had labelled by hand.

Leila, a developer on the app team, has another idea. “Why train anything? A hosted language model can read the 77 intents and pick one. No labels, no training, live tomorrow.” Nadia asks the questions that decide it:

• Is it as accurate as Kofi’s router, on the same messages?
• What does it cost per message, and how fast is it?
• The messages contain card problems, amounts and sometimes names. May they leave the bank?
• What happens when it is not sure?

What you will build

You will build Leila’s classifier with the habits of this course. It has versioned prompt files, the customer’s text between tags, and a check in code. An enum of the 77 intents goes in strict structured outputTermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs. You compare two prompt versions on 1,000 fixed validation messages, then open the test file once. Then you compare with the trained routers on the same test messages. You measure accuracy with an intervalTermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI, the errors, cost and latency per 1,000 messages, and rules that send unsure messages to a person.

The data

The data is BANKING77, published by the company PolyAI in 2020. It has 13,083 short messages to a bank’s online support, each with one of 77 intents. The paper does not say who wrote them: treat them as realistic messages, not as one bank’s customers.

Source and licence

“BANKING77” by PolyAI (Casanueva, Temčinas, Gerz, Henderson and Vulić, 2020), from github.com/PolyAI-LDN/task-specific-datasets, licensed under CC BY 4.0. Paper: Efficient Intent Detection with Dual Sentence Encoders, arXiv 2003.04807. The notebook downloads the files from a fixed version (commit 57ec275) and checks their SHA-256. We changed nothing in them.

Why this dataset? The deep learning course’s case study used it. So you can score the language model on exactly the messages where the trained routers were scored. The dataset card in the notebook folder lists the other candidates, such as CLINC150, and why they lost.

The split is also the deep learning course’s. 1,000 messages of the official train file are the validation set, for comparing the prompts. The official test file, 3,080 messages with 40 per intent, stays sealed until the end. The notebook rebuilds the course’s TF-IDF router on the other 9,003 messages. It gets exactly the same numbers: 88.0% on validation and 89.0% on test.

Three intents show the difficulty:

train, valid = train_test_split(raw_train, test_size=1000, stratify=[r["category"] for r in raw_train], random_state=0)
test = raw_test
print(len(train), "train |", len(valid), "valid |", len(test), "test |", len({r["category"] for r in raw_train}), "intents")
for intent in ("card_arrival", "card_delivery_estimate", "get_physical_card"):
    print(f"{intent:24}", [r["text"] for r in train if r["category"] == intent][:2])

The output:

9003 train | 1000 valid | 3080 test | 77 intents
card_arrival             ['Is my card lost? I am still waiting for it to be delivered.', "It's been a week since you issued me a card and I still didn't get it. Should I keep waiting?"]
card_delivery_estimate   ['How long does it take for my physical card to be delivered.', 'how long does delivery take?']
get_physical_card        ["I didn't get a PIN yet. What gives?", "I haven't received my pin"]

What this tells us. card_arrival is a card that is late; card_delivery_estimate is a question about delivery times before anything is late. People could argue about some messages. And get_physical_card is mostly about the PIN of a new card, not about getting a card. The name does not say what the bank means by it. A trained model learns that from the examples. A model that sees only the name cannot.

Frame the problem

• The unit: one message, as the customer sends it. The target: the intent, one of 77.
• The metric: accuracy, the share of messages routed to the right intent, with a 95% interval. Each wrong intent costs about the same: one more message from an annoyed customer. Always choosing the most common intent is right for 1.9% of the validation messages and 1.3% of the test messages.
• Also measured: cost and latency per 1,000 messages, setup effort, and what data leaves the bank.

The new idea: a router without training examples

The temporary agent

On a busy day, Nadia brings in a temporary agent who has worked in other banks’ support. She gives him the list of 77 intents. He routes clear messages well, because he knows how customers write. But he does not know this bank’s habits. He cannot know that “get physical card” is the PIN question, or where a late card ends and a delivery question starts. A one-line note next to each intent would help him a lot.

The problem: a name is not a definition

The model sees what the temporary agent sees. With prompt v1, which lists the 77 names only, chat-small routed 0 of the 11 get_physical_card messages in the validation set right. “Where can I find my card PIN?” went to change_pin. That is a sensible reading of the name, and it is wrong for this bank.

The idea and its name

Classifying with only a description of the classes, and no labelled examples in the prompt, is called zero-shot classification. “Shot” means an example shown to the model: zero examples. With a few examples in the prompt, it is few-shot. The model brings what it learned in pretraining; the prompt brings the bank’s definitions. So the list of intents, and what each one means, is the specification. Write it with the care of a policy, as GraceTermGraceThe lead of Larkfield’s help desk. She writes the support policy and decides what a good answer to a customer is.’s policy was for LarkfieldTermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps in Specify the task.

A tiny worked example

Prompt v2 is v1 plus one line per intent, written by the course from the intent names and a few training messages for each. No validation or test message was used. Two of the 77 lines:

- card_arrival: An ordered card has not arrived yet, and the customer asks where it is.
- get_physical_card: Getting the PIN for a new physical card: where to find it or how to set it.

The same three validation messages, answered by chat-small (gpt-6-luna on Azure) with each prompt:

Message	True intent	v1 (names only)	v2 (names + one line)
Where can I find my card PIN?	get_physical_card	change_pin ✗	get_physical_card ✓
How do I track the card you sent me?	card_arrival	card_delivery_estimate ✗	card_arrival ✓
I ordered my card a couple weeks ago and haven’t received it yet. When can I expect it?	card_delivery_estimate	card_delivery_estimate ✓	card_arrival ✗

The line fixes the PIN question. It also moves the boundary between the two card intents, in both directions. A definition helps, but it cannot settle a label that people would argue about.

The trained router learns the bank's intents from labelled examples and runs on the bank's servers. The prompted router reads the bank's definitions in every request and runs at the provider, so every message leaves the bank.

Text description of the diagram

Two rows, each from a new message to an intent.

1. Trained router (the deep learning course): 9,003 labelled messages, each with its intent, go into training, which takes minutes on a CPU. The result is a model file on the bank’s servers. A new message goes into the model, which gives the intent and a probability.
2. Prompted router (this case study): a person writes the 77 intent names and one line for each. They become the prompt, and the schema with an enum of 77 intents. A new message travels with them to a hosted model: the message leaves the bank. Code checks every answer against the schema before the intent and the model’s own confidence are used.

When to use it, and what it costs. Zero-shot classification fits when you have no labelled data yet, or when the classes change often. Its costs: tokens for every message, and a provider that sees every message. And one cost hides behind “no training data”: you still need labelled messages to know whether it works. This case study used 4,080 of them.

The classifier

The classifier has the parts of the course project, with a different contract. The schema is a Pydantic model with two required fields, made from the same intent list as the prompt, so the two can never disagree:

INTENTS = [line.split("\t")[0] for line in Path("prompts/intents.tsv").read_text().splitlines()]

IntentAnswer = create_model(
    "IntentAnswer",
    __config__=ConfigDict(extra="forbid"),          # no other field allowed
    intent=(Literal[tuple(INTENTS)], ...),         # an enum of the 77 intents, required
    confidence=(Literal["low", "medium", "high"], ...),
)

def response_format():
    return {"type": "json_schema",
            "json_schema": {"name": "banking_intent", "strict": True, "schema": IntentAnswer.model_json_schema()}}

create_model builds the same kind of class as TicketAnalysis in Define a schema, but from a list: nobody should type 77 names twice. build_request() puts the prompt file in the system message and the customer’s text between <message> tags, as Larkfield’s v2 prompt did.

Every recorded answer then goes through gate 1, the Pydantic check of Handle incomplete results:

valid1000_chat-small_v1         1000 answers  {'valid': 1000}
valid1000_chat-small_v2         1000 answers  {'valid': 1000}
valid300_chat-small_v1_text      300 answers  {'valid': 300}
test_chat-small_v2              3080 answers  {'valid': 3080}
test770_chat-strong_v2           770 answers  {'valid': 770}

What this tells us. With strict structured output, all 5,850 answers were complete JSON with one of the 77 intents. None was cut off or refused.

The schema has a size limit

A 77-value enum is large, so Leila checked the provider’s limit for real. An enum with 1,000 values was accepted. With 1,001 values, chat-small refused the request before the model ran:

HTTP 400: Invalid schema for response_format 'many': Expected at most 1000 enum values in total within a single schema when using structured outputs, but received 1001. Consider reducing the number of enums, or use 'strict: false' to opt out of structured outputs.

The enum also costs tokens, because the schema travels with every request. For 77 intents, it adds about 460 input tokens per message. v1 has a medianTermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One of 1,069 input tokens with the schema, and 612 without.

Baselines

Router	Validation (1,000)	How it was made
Always the most common intent	1.9%	Counting
TF-IDF + logistic regression	88.0%	Trained on 9,003 labelled messages, about 1 second
Fine-tuned DistilBERT	91.6%	Trained on 9,003 labelled messages, about 8 minutes on a CPU

Both trained routers come from the deep learning course; the notebook rebuilds TF-IDF, and DistilBERT is quoted.

Compare two prompts on the same 1,000 messages

chat-small answered all 1,000 validation messages with each prompt: the same model, messages and scoring code, as in Compare prompts.

1,000 validation messages. chat-small is gpt-6-luna on Azure. TF-IDF is the deep learning course's router, trained on 9,003 labelled messages. Bottom: the paired difference v2 − v1 on the same messages.

Text description of the chart

Top panel, three rows: TF-IDF trained on 9,003 labelled messages, 88.0% (95% interval 85.8 to 89.9). chat-smallTermchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs with prompt v1, names only, 80.3% (77.7 to 82.7). chat-small with prompt v2, names and one line each, 84.2% (81.8 to 86.3). Bottom panel: the paired difference v2 minus v1 is +3.9 percentage points, with a 95% interval from +2.3 to +5.5, clear of zero.

The notebook counts the messages where only one version is right:

only v1 right 14 | only v2 right 53 | v2 - v1 = +3.9% (95% interval +2.3% to +5.5%)
v1: median 1069 input tokens per message
v2: median 1956 input tokens per message

What this tells us. 909 of the 1,000 messages got the same intent from both prompts. Of the rest, v2 fixed 53 and broke 14. The paired interval, +2.3 to +5.5 points, stays clear of zero, so this gain is not the sampling noise that Larkfield’s 69 tickets could not rule out. The fixes are the definitions at work. get_physical_card went from 0 of 11 to 10 of 11. “How can I tell where the money came from?” moved from receiving_money to verify_source_of_funds. The 14 losses are mostly on the boundaries that the lines moved, such as late card against delivery time.

The price: v2 sends 1,956 input tokens per message instead of 1,069. That is about US$0.23 instead of US$0.15 per 1,000 messages on chat-small. For 3.9 points, that is cheap. Leila chooses v2. She opens the test file once, for v2 only.

Even v2 is still 3.8 points behind TF-IDF on validation. That is the honest first result: 77 one-line definitions do not yet hold what 9,003 labelled messages hold.

The test, once

chat-small answered all 3,080 test messages with v2. chat-strong costs about 18 times more per message. So it answered a fixed sample of 770 test messages, 10 per intent, drawn before any answer was seen. For a fair comparison, the notebook also scores chat-small and TF-IDF on these 770 messages.

Top: all 3,080 test messages. Bottom: the fixed sample of 770 test messages that chat-strong answered. chat-small is gpt-6-luna and chat-strong is gpt-6.1-sol, both on Azure, prompt v2. DistilBERT is quoted from the deep learning course.

Text description of the chart

Top panel, all 3,080 test messages: TF-IDF with logistic regression 89.0% (95% interval 87.8 to 90.0). Fine-tuned DistilBERT 91.8% (90.7 to 92.7). chat-small with prompt v2 87.5% (86.3 to 88.6). Bottom panel, the 770 messages of the sample: TF-IDF 90.4% (88.1 to 92.3). chat-small 87.0% (84.5 to 89.2). chat-strongTermchat-strongThe name that Larkfield’s code gives to a larger hosted model: gpt-6.1-sol on Azure. It is often more accurate than chat-small, and it costs about 14 to 18 times more per call.Learn more Request anatomyBuilding Reliable Applications with LLM APIs with prompt v2 89.1% (86.7 to 91.1). On both panels, the prompted routers sit at or below TF-IDF, and DistilBERT is highest.

Router	Test messages	Right	Accuracy (95% interval)
TF-IDF + logistic regression	3,080	2,740	89.0% (87.8–90.0)
Fine-tuned DistilBERT	3,080	2,826	91.8% (90.7–92.7)
chat-small, prompt v2	3,080	2,695	87.5% (86.3–88.6)
chat-strong, prompt v2	770 (sample)	686	89.1% (86.7–91.1)
chat-small, the same 770	770	670	87.0% (84.5–89.2)

What this tells us. With no training examples and one line per intent, chat-small routes 87.5% of the test messages right. That is close to word counts trained on 9,003 labels, but not equal. On the same messages, TF-IDF is ahead by 1.5 points, with a paired interval from +0.1 to +2.8. Fine-tuned DistilBERT is ahead of both, by 4.3 points over chat-small. All the routers did a little better on test than on validation; the deep learning course saw the same.

chat-strong is better than chat-small on the same 770 messages: 22 messages only it got right, against 6, a gain of 2.1 points (+0.7 to +3.4). But it is not better than TF-IDF on those messages (90.4%): the difference cannot be told apart from zero. So the larger model buys about 2 points, for about 18 times the price per message.

The honest summary: a prompt reached about the level of a trained word-count router, without a single training example. It did not reach the fine-tuned model.

Where it goes wrong

The notebook counts the errors of chat-small on the test setTermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One by pair of intents, in either direction:

Intent A	Intent B	Errors
beneficiary_not_allowed	failed_transfer	18
declined_card_payment	declined_transfer	12
card_payment_not_recognised	direct_debit_payment_not_recognised	11
receiving_money	topping_up_by_card	10
exchange_via_app	fiat_currency_support	9
top_up_by_bank_transfer_charge	transfer_fee_charged	9
card_arrival	card_delivery_estimate	8

Several pairs troubled the deep learning course’s routers too. But the worst intent tells a new story. beneficiary_not_allowed is right for only 11 of its 40 test messages; 18 went to failed_transfer. Read them:

"I couldn't do a transfer to an account"                              → failed_transfer
"I want to buy cryptocurrency, but your app doesn't let me. Why?"     → exchange_via_app

TF-IDF routes both to beneficiary_not_allowed, because that is what the training labels say. In this dataset, crypto that the app refuses to buy counts as “beneficiary not allowed”. 11 of the 140 training messages of that intent mention crypto. And 22 of the 120 training messages of declined_transfer are about buying something or a card. The dataset labels “When I try and to buy something using my card it keeps getting declined” as declined_transfer. The model says declined_card_payment, and most people would agree with the model.

This is the real difference between the two routers. A trained router learns the bank’s conventions from its examples, even odd ones. A prompted router knows only what the definitions say. The fix is in the definitions, for example a v3 line that names crypto purchases. Then compare v3 with v2 on the validation set.

Do the two routers fail on the same messages?

	TF-IDF right	TF-IDF wrong
chat-small right	2,487	208
chat-small wrong	253	132

Only 132 messages beat both: the two routers make different mistakes. That is useful below.

Cost, speed and setup

Cost per 1,000 messages uses the course’s formula and the Azure prices that we recorded (check the current price page). Per million input and output tokens, chat-small costs US$0.10 and US$0.50, and chat-strong US$2 and US$10. It ignores the discount for cached inputTermCached tokensInput tokens that the provider kept from an earlier request that started with the same text, such as the same instructions and policy. Many providers charge less for them.Learn more Request anatomyBuilding Reliable Applications with LLM APIs, so it is an upper bound: 98.7% of the input tokens were cached.

chat-small   US$0.229 per 1,000 | latency median 1.28 s, p95 2.30 s
chat-strong  US$4.239 per 1,000 | latency median 1.78 s, p95 2.96 s

Most of the cost is the input: 1,956 tokens of prompt and schema for a message of about 10 words. The whole comparison, as Nadia needs it:

	TF-IDF + logistic regression	Fine-tuned DistilBERT	chat-small, prompt v2	chat-strong, prompt v2
Test accuracy	89.0%	91.8%	87.5%	89.1% (770 messages)
Cost per 1,000 messages	the bank’s server	the bank’s server	US$0.23	US$4.24
Time per message	0.2 ms on a CPU	7 ms on a CPU	1.3 s (median)	1.8 s (median)
Labelled messages to build it	9,003	9,003	none (77 one-line definitions)	none
Labelled messages to test it	yes	yes	yes	yes
A new intent	label examples, retrain (seconds)	label examples, retrain (minutes)	edit the prompt, test again	edit the prompt, test again
Where the message goes	stays in the bank	stays in the bank	to the provider	to the provider

The trained routers’ times were measured on a CPU in the deep learning course. The language models’ times were measured from Toronto to East US 2 and include the network.

What this tells us. The money is small either way: 100,000 messages a month cost about US$23 on chat-small. The larger differences are elsewhere. The language model is about 170 times slower than DistilBERT, though 1.3 seconds is fine for a chat. It needs no labels to start. And every message leaves the bank.

A person for unsure messages

Nadia wants unsure messages to go to a person. The model writes its own confidence, which is not a measured probability (Validate meaning). On the test set, its “high” answers were right 93.3% of the time, “medium” 68.0%, and “low” 48.9%. So it does sort the answers, roughly.

A second signal costs nothing: TF-IDF already runs on the bank’s server. When the two routers disagree, at least one is wrong. Four rules, on the same 3,080 test messages:

chat-small, prompt v2, the 3,080 test messages. Top: the share of messages sent to a person. Bottom: the accuracy of the messages routed automatically, with 95% intervals.

Text description of the chart

Four rules. No fallback: 0% to a person, 87.5% right among the rest (86.3 to 88.6). Confidence not high: 19.6% to a person, 93.3% right among the rest (92.3 to 94.3). Disagrees with TF-IDF: 17.2% to a person, 97.5% right among the rest (96.9 to 98.1). Either rule: 28.6% to a person, 98.4% right among the rest (97.7 to 98.8).

What this tells us. The model’s confidence sends 603 messages to a person and still lets 165 wrong ones through. Disagreement with TF-IDF sends fewer, 530, and lets only 63 wrong ones through. The routed messages are 97.5% right. The two routers’ different mistakes are worth more than the model’s opinion of itself. Choose the rule from what a person’s time and a wrong answer cost. For comparison, DistilBERT’s own probability with a threshold of 0.5 sent 2.8% of the validation messages to a person. It was right on 93.1% of the rest.

New messages, end to end

These messages are made up for this case study; no customer wrote them. They are the deep learning course’s new messages, so the answers can be compared. Each was sent three times to each deploymentTermDeploymentA model that an account sets up at a provider, with its own name and its own limits. Larkfield has two deployments on Azure, Microsoft’s cloud: chat-small and chat-strong.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs with prompt v2, because these deployments sample their answers:

Message	chat-small (3 runs)	chat-strong (3 runs)	Fine-tuned DistilBERT (deep learning course)
I topped up my account this morning but the money still isn’t there.	pending_top_up ×3	pending_top_up ×3	top_up_reverted 0.55, pending_top_up 0.23
My new card still hasn’t arrived and I ordered it two weeks ago.	card_arrival ×3	card_arrival ×3	card_arrival 0.99
I was charged twice for the same coffee.	transaction_charged_twice ×3	transaction_charged_twice ×3	transaction_charged_twice 0.99
The transfer went through, but the top-up failed.	top_up_failed ×3	top_up_failed ×3	top_up_failed 0.95
The top-up went through, but the transfer failed.	failed_transfer ×3	failed_transfer ×3	top_up_failed 0.91
Can I get my money back for something I bought online? The seller never sent it.	request_refund ×3	request_refund ×3	request_refund 0.88

What this tells us. Both deployments read the same-words pair right, every time: the failed part is the transfer. The fine-tuned model sent both messages to top_up_failed, and TF-IDF cannot tell them apart. A large model reads word order better than a small one. The confidence moved between runs (high or medium); the intent did not.

Common mistake

An enum that grows past the limit

Symptom: the product team wants each of the 77 intents per card product, 13 products in all. Leila builds the enum from the combinations. The first request fails before the model runs, with the real HTTP 400 above: “Expected at most 1000 enum values in total within a single schema … but received 1001.”

Cause: strict structured output builds a grammar from the schema, and the provider limits its size. 77 × 13 is 1,001 values. The limit is per schema, so two large enums in one schema share it.

Fix: keep the enum to what one decision needs. Ask for the intent (77 values), and let code look up the product from the customer’s account. Check the provider’s schema limits when you design. Never switch to strict: false to silence the error: the validation in code would then be your only guard.

And the opposite question: does the enum fix a problem you can see? Leila also sent v1 without the schema for 300 validation messages. All 300 answers were one of the 77 names, and 247 were right, against 246 with the schema. The schema buys a guarantee, not a better score, for about US$0.05 per 1,000 messages.

Recommendations

Leila and Kofi write to Nadia together:

• Keep the fine-tuned router for the chat. On the same 3,080 test messages it is right 91.8% of the time, against 87.5% for chat-small with the best prompt. It runs on the bank’s server, in milliseconds, and no message leaves the bank.
• Use the prompted router where there are no labels yet. For the new product’s intents, a prompt with clear definitions works from the first day. Keep the messages that people correct as labels, and train a router when each intent has a few examples.
• Write the intent definitions like a policy. One line per intent gained 3.9 points. Write down the bank’s conventions too; that also helps new agents.
• Combine the two for the fallback. When the trained router and the language model disagree, send the message to a person. That is about 17 in 100 messages, and 97.5% of the rest are right.
• Not chat-strong. It gained about 2 points over chat-small for about 18 times the price, and it did not beat word counts.

And what not to use it for: decisions about a customer, such as fraud checks, blocking a card or closing an account. The router chooses an answer flow; a person or a separate, tested process makes decisions.

Limits and next steps

• The model may have seen this data. BANKING77 has been public since 2020, and the hosted models’ training data is not published. Their scores may be a little better here than on the bank’s own messages. Test on those before launch.
• One recording per message. These deployments sample their answers (Request anatomy), so a second run would move a few messages.
• chat-strong on a sample. 770 messages give an interval about 4 points wide: enough to see that it does not beat TF-IDF, not to rank it exactly.
• Two prompt versions. A v3 with a few examples per intent (few-shot) might close part of the gap, at more tokens per message. Compare it on the same validation set, as here.
• Latency from Toronto, with 4 requests in parallel. Many requests met a real HTTP 429 and waited (Handle provider limits). The latency on this page is the time of the successful call only.
• Next: send to the LLM only the messages where TF-IDF is unsure. Try a self-hosted open model, so that no message leaves the bank.

Responsible use

What is in the messages. A bank’s chat messages contain personal and financial data: names, card and account numbers, amounts, and sometimes a person’s health or money problems. Privacy law and banking rules protect them. Sending them to a hosted model is a transfer to another company, and the bank must decide it on purpose.

What may be sent to a hosted model. Before the first real message, the bank’s data protection officer checks the provider’s terms with the questions of Licensing and data terms in the language models course. In which region is the data processed? How long are prompts kept, and who can read them? Does the provider train on them? Is there a signed data processing agreement? If any answer is unclear, use the trained router or a self-hosted model: then no message leaves the bank.

Send only what the task needs. The intent never needs a card number. Leila tested this with a made-up message:

Hello, this is Sarah Okafor (sarah.okafor@example.com, 07700 900123). My card 4929 1234 5678 9012 was used last night for a cash withdrawal of £200 that I never made. My account is 31926819, sort code 60-16-13.

Before the message leaves the bank, a redact() function replaces what patterns can find. The project’s redact() in The application boundary works the same way:

Hello, this is Sarah Okafor ([EMAIL], [PHONE]). My card [CARD NUMBER] was used last night for a cash withdrawal of £200 that I never made. My account is [NUMBER], sort code [SORT CODE].

Both versions got cash_withdrawal_not_recognised in all three runs on both deployments: redaction did not cost accuracy here. But the name is still there. Patterns find numbers and email addresses, not names or a story about an illness. Redaction reduces what leaves the bank; it does not make a message anonymous.

Logs. Log the intent, the model, the tokens, the latency and the request ID. Never log the message text; the usage log of Measure usage and cost does the same.

People in the loop. Send unsure messages to a person, and let every customer reach a person. Some intents cost more when they are wrong: a compromised_card message in the wrong queue can cost a customer money. Measure the errors of those intents separately.

The data. BANKING77 (CC BY 4.0) must be credited to PolyAI in any copy.

Make it a portfolio piece

This case study shows an employer that you can choose between a prompt and a trained model with evidence. Make it yours with one change, for example a prompt v3 with the bank’s conventions or a local open model. Then make it public, and easy to read in five minutes.

What to put in the repository:

• README.md, the page that people read first (template below).
• The notebook, run from top to bottom, with its outputs.
• prompts/ with every version you measured, never edited after you measured it.
• MODEL_CARD.md. Intended use: routing chat messages to an answer flow, with a person for unsure messages. Not for: decisions about customers. Then the models and the dates, the evaluation, the limits, and the data rules.
• The data source and licence, with the attribution text from the box above. Never commit an API key; read it from an environment variableTermEnvironment variableA named value that the system gives to a program when it starts, for example an API key. The value stays out of the code and out of Git.Learn more Configuration and secretsPython for Practical AI Engineering.

A README template:

# Can a prompt replace a trained intent router?

## Problem
Route short bank-support messages to one of 77 intents. A trained router needs
9,003 labelled messages; a prompted router needs none.

## Data and licence
BANKING77 (PolyAI, 2020), CC BY 4.0, github.com/PolyAI-LDN/task-specific-datasets
(commit 57ec275). The deep learning course's split: 1,000 validation messages,
the official 3,080 test messages used once.

## Approach
A hosted LLM with a prompt (77 intent names, then one line per intent) and strict
structured output (an enum of the 77 intents), validated in code. Two prompt
versions compared on the same 1,000 validation messages; the better one tested once.

## Results (test, 95% intervals)
Fine-tuned DistilBERT 91.8% (90.7-92.7) | TF-IDF 89.0% (87.8-90.0)
chat-small, prompt v2 87.5% (86.3-88.6) | chat-strong, 770 messages 89.1%
Prompt v2 vs v1 (validation): +3.9 points (+2.3 to +5.5).
Cost per 1,000: US$0.23 (chat-small), US$4.24 (chat-strong). Median 1.3 s.

## Limits
One recording per message; the hosted model may have seen this public dataset;
every message leaves the bank; the model's confidence is not a probability.

How to describe it in an interview, in two sentences: “I routed bank-support messages to 77 intents with an LLM, a strict enum schema and no training data, and compared it fairly with trained routers on the same 3,080 test messages. The prompt reached 87.5%, close to word counts but 4 points behind a fine-tuned model, so I recommended it for intents without labels, with disagreement between the two routers as the fallback to a person.”

Go further:

• Write prompt v3 with the conventions from “Where it goes wrong”, and compare it with v2 on the validation set.
• Run the same 1,000 validation messages through a local model with OllamaTermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One, and compare accuracy and time.
• Send 100 validation messages three times, and count how many change their intent between runs.

Knowledge check

Check your understanding

Prompt v2 was 3.9 points better than v1 on 1,000 validation messages. In Compare prompts, a difference of three tickets out of 69 was not trusted. Why is this difference trusted?

Show the answer

Two reasons. The set is much larger: 1,000 messages instead of 69. And the comparison is paired: both prompts answered the same messages. Only the 67 messages where exactly one version was right carry information. 53 of them favour v2 and 14 favour v1. The paired 95% interval of the difference, +2.3 to +5.5 points, does not include zero. On 69 tickets, the same kind of interval would have been wide and crossed zero. The content agrees too. The fixes are the ones the new lines were written for, such as the PIN messages of get_physical_card: 0 of 11 before, 10 of 11 after.

Check your understanding

Tomás says: 'The LLM needs no training data, so we can skip labelling altogether.' What does this case study say?

Show the answer

It needs no labels to build the router. But it still needs labelled messages to know whether it works, to choose between prompts and to set a fallback rule. This case study used 1,000 labelled validation messages and 3,080 labelled test messages. Without them, nobody would know that the prompt is at 87.5%. Nobody would know that one line per intent helps, or that crypto messages go against the bank’s convention. Labels also make the trained router possible later, which was more accurate here.

Check your understanding

Nadia asks: 'Which router should we use for a new product that launches next week with 12 new intents and no labelled messages?' Answer with the case study's numbers, and say what you would watch.

Show the answer

Start with the prompted router for the new intents: a trained router cannot learn intents that have no examples. Write one clear line per intent, because that gained 3.9 points here. Expect accuracy in the high 80s, not the 92% of the fine-tuned model. Expect about US$0.23 per 1,000 messages on chat-small. Check the provider’s data terms before any real message is sent, and redact what the task does not need. Send unsure messages to a person, and keep the corrected messages as labels. When each intent has a few examples, retrain the bank’s own router, and compare both on a fixed test set.

Recap

• Zero-shot classification routes with the class names and definitions only. The definitions are the specification. One line per intent took chat-small from 80.3% to 84.2% on 1,000 validation messages (+3.9 points, +2.3 to +5.5).
• Strict structured output with an enum of 77 intents kept every one of 5,850 answers valid. The provider’s limit is 1,000 enum values per schema, and the enum adds about 460 input tokens per message. Validate in code anyway.
• On the same 3,080 test messages: fine-tuned DistilBERT 91.8%, TF-IDF 89.0%, chat-small 87.5% with no training example. chat-strong gained about 2 points on a sample, for about 18 times the price.
• A trained router learns the bank’s conventions from its examples; a prompt knows only its definitions. Most errors sit there.
• Cost is small (US$0.23 per 1,000 messages on chat-small); latency is about 1.3 seconds; every message leaves the bank. Disagreement with the trained router is a better fallback signal than the model’s own confidence.
• Labels are still needed: to test, to choose and to set the fallback.

This was the last case study. In the final assignment, you build a support-ticket assistant of your own, with validated output, tests and the same kind of evidence.

Next: Final assignment