Skip to content
Français

Route banking questions

  • Case study 1
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we routed Larkfield's synthetic messages to six teams. In this case study, we use real open data: banking questions in 77 fine-grained intents. We group the intents into team routes, compare four kinds of method on one cost measure, and choose how many questions a person reads.

Case study 1 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceRoute banking questions

What you will be able to do

Take 13,083 banking questions with 77 fine-grained labels to a routing decision that you can defend with numbers. You will map the labels to 7 teams and compare four kinds of method on one cost measure. Then you choose how many questions a person reads, test once, and run the router on new questions.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • A written label mapping
  • TF-IDF baseline
  • Frozen embeddings
  • Fine-tuned small encoder
  • Recorded LLM, few-shot
  • Route confidence
  • Review threshold from costs
  • Fair comparison on a sample

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, plus three tasks with checks. Its setup cells install the packages and download the data (1 MB) and the course’s files (44 MB). The base model (about 470 MB) downloads once. It needs no account and no key: the LLM’s answers were recorded. After the install, it runs in about a minute on a laptop.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, sentence-transformers 6.2.0, Transformers 5.22.2 and PyTorch 2.15.0 (CPU). The setup cell makes a folder banking77. To start again, delete that folder. This case study does not change your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

Imagine a small online bank. Its customers write to the help desk in the app, and each question must reach the right team. Today, a person reads every question first and moves it to a team.

Leila leads customer service. She has seven teams, from cards to transfers. One of them is urgent, security: lost cards, stolen phones and payments that the customer does not recognise. A question for that team cannot wait in the wrong queue, because someone may be using the customer’s money.

Tunde is a data scientist. Leila asks him: “Can a model send most questions straight to the right team? How many must my people still read? And what does it cost?”

We invented Leila, Tunde and the bank. The questions come from a published dataset, described below.

The case study uses the course’s cost table. These are assumptions, the same values as for Larkfield:

  • A question in the wrong queue costs 4 dollars: the wrong team reads it and moves it, and the customer waits longer.
  • An urgent security question in the wrong queue costs 20 dollars.
  • A review by a person costs 1.50 dollars: about a minute of an agent’s time.

So “a person reads everything”, Leila’s process today, costs 1,500 dollars per 1,000 questions.

You will build a router that chooses one of 77 intents (what the customer wants), and from the intent, one of 7 routes (the team that answers). When it is unsure, it sends the question to a person. You will compare four kinds of method with one measure: the cost per 1,000 questions.

The data is Banking77, published by PolyAI. It has 13,083 short questions to an online bank, each labelled with one of 77 intents, such as card_arrival (“Where is my new card?”) or transfer_timing (“How long does a transfer take?”).

Crowd workers wrote the questions for an imagined bank, so no real customer is behind them. They are real in another sense: people wrote them, with their own words, typos and borderline cases. The Larkfield inbox was written by one model from 27 templates.

We compared it with four other open datasets; the dataset card says why it won.

The notebook downloads the two files from PolyAI’s repository at a fixed version and checks their checksums. Then Tunde loads them:

raw_train = pd.read_csv("data/train.csv").rename(columns={"category": "intent"})
test = pd.read_csv("data/test.csv").rename(columns={"category": "intent"})

The first checks print:

train file: 10,003 questions | test file: 3,080 | intents: 77
missing values: 0 | repeated questions: 0 | test questions also in train: 0
words per question: median 10, 90% have at most 22, longest 79
train questions per intent: from 35 (contactless_not_working) to 187 (card_payment_fee_charged) | test: 40 to 40
odd intent names: ['Refund_not_showing_up', 'reverted_card_payment?']

What this tells us. The files are clean, and the questions are short: half have 10 words or fewer. The test file has exactly 40 questions per intent. Two intent names break the pattern. We keep them as they are; renaming labels quietly is how two files stop agreeing.

Tunde also reads 100 random questions and the longest ones. A few of the longest rows are answers written by a bank agent, not questions. Real data has this kind of noise.

Think of a post office. Letters have exact addresses, but the sorting office has only a few bags, one per van. The office writes down once which streets go into which bag, and every sorter uses that list.

Banking77’s intents are the exact addresses. Leila’s teams are the bags. Tunde and Leila write down which team answers each intent. This is the written mapping: a file, intent_routes.csv, with one row per intent. It is a business decision, read by people and by code.

77 intents on the left, with arrows to 7 route boxes on the right. security: 6 intents, urgent, for example lost_or_stolen_card and card_payment_not_recognised. cards: 20 intents, for example card_arrival, card_delivery_estimate and change_pin. payments_fees: 14 intents, for example transaction_charged_twice, extra_charge_on_statement and exchange_rate. transfers: 11 intents, for example pending_transfer and failed_transfer. top_ups: 10 intents, for example pending_top_up and top_up_failed. cash_atm: 7 intents, for example atm_support and declined_cash_withdrawal. account: 9 intents, for example verify_my_identity and terminate_account.
The written mapping: each of the 77 intents goes to exactly one of 7 team routes. Near neighbours such as card_arrival and card_delivery_estimate go to the same team.
Text description of the diagram

On the left, one box: the 77 intents, Banking77’s labels, one per question. Seven arrows lead to seven route boxes. security has 6 intents and is urgent, for example lost_or_stolen_card and card_payment_not_recognised. cards has 20 intents, for example card_arrival, card_delivery_estimate and change_pin. payments_fees has 14 intents, for example transaction_charged_twice, extra_charge_on_statement and exchange_rate. transfers has 11, for example pending_transfer and failed_transfer. top_ups has 10, for example pending_top_up and top_up_failed. cash_atm has 7, for example atm_support and declined_cash_withdrawal. account has 9, for example verify_my_identity and terminate_account.

They followed three rules:

  • One team per kind of work. A team gets the intents that its people can answer without help. The cards team handles everything about the card itself: ordering, delivery, the PIN, virtual cards.
  • Urgency decides first. Every intent where someone else may be using the money goes to security, even when another team knows the topic. So card_payment_not_recognised goes to security, not to payments_fees.
  • Write the hard choices down. card_swallowed (a cash machine kept the card) could be cash_atm or cards. The customer needs a new card, so it goes to cards. supported_cards_and_currencies mixes two topics; it goes to payments_fees, with the other currency questions.

The notebook reads the file and checks it:

77 intents mapped to 7 routes; intents without a route: 0
intents priority train_questions share
route
security 6 urgent 799 0.080
cards 20 normal 2169 0.217
payments_fees 14 normal 2105 0.210
transfers 11 normal 1581 0.158
top_ups 10 normal 1232 0.123
cash_atm 7 normal 1104 0.110
account 9 normal 1013 0.101

What this tells us. The routes are not equal: cards and payments_fees get about a fifth of the questions each, and security 8%. This is the two-level shape of Hierarchical labels.

Tunde writes down his answers before he trains anything:

  • Unit: one question. It gets one route, or goes to a person.
  • Labels: the model learns the 77 intents; the route comes from the mapping. We test both levels.
  • One measure for the choice: the cost per 1,000 questions, with the course’s cost table. Route accuracy, the intent accuracy, the recall of security and the share sent to review stand next to it.
  • The split: 2,000 of the training questions become a validation set, stratified by intent. All choices are made on it. The test file stays closed until the end, and we use it once.
  • A fair comparison: the hosted LLM was recorded for a sample only: up to 10 questions per intent, 765 validation and 770 test questions. Every method is also scored on the same sample.
train 8,003 | validation 2,000 | test 3,080 (used once, at the end)

The course’s helper routing.py scores every method, with the bank’s 7 routes and every security question marked urgent.

A baseline: TF-IDF and logistic regression

Section titled “A baseline: TF-IDF and logistic regression”

Tunde starts with the course project’s baseline: TF-IDF of words, word pairs and letter groups, and logistic regression. It learns the 77 intents in 5 seconds. On the validation questions:

method intent_accuracy route_accuracy security_recall review_share total_per_1000
0 TF-IDF + LR 0.896 0.962 0.919 0.0 256.0
1 everything to a person NaN 0.000 0.000 1.0 1500.0

What this tells us. The baseline gets about 9 in 10 intents and 96 in 100 routes right. Many intent mistakes still land in the right team. The baseline costs 256 dollars per 1,000 questions, against 1,500 when a person reads everything.

On Larkfield’s synthetic inbox, the same pipeline also reached 0.962 on 7 routes. But the 77 intents are much harder (0.896). Fine-grained labels written by many people are where real data bites.

If the bank needs only 7 routes, why learn 77 intents? Tunde trains the same pipeline on the routes directly:

route accuracy, trained on the 7 routes: 0.944

What this tells us. Learning the 7 routes directly is worse: 0.944 against 0.962. Each route mixes very different questions, so one class per route is a harder job. Keep the fine labels, and map them afterwards.

Route confidence: add the intents of each route

Section titled “Route confidence: add the intents of each route”

To send unsure questions to a person, we need a confidence for the route. The model gives a probability for each of the 77 intents. Which number is the route’s confidence?

Think of a vote with 77 candidates from 7 parties. Two candidates of one party get 42% and 41%. No candidate has a majority, but the party has 83%, and the party is what you asked about.

The same holds for routes. The route confidence is the sum of the probabilities of all the intents of that route. The function route_proba() in the notebook does this sum for every question.

A question, 'I lost my wallet and all my cards were in it.', gets 77 intent probabilities from the fine-tuned encoder: lost_or_stolen_card 0.418, compromised_card 0.416, lost_or_stolen_phone 0.044, the 74 others 0.122 together. The old way takes the top intent only: 0.418 is below 0.75, so a person must read it. This case study adds the intents of each route: security 0.916, the other 6 routes 0.084 together. 0.916 is above 0.75, so the question goes to security.
One validation question. By its top intent alone, it would go to a person. Its route confidence, the sum of the security intents, is 0.916, so it goes straight to security.
Text description of the diagram

The question is “I lost my wallet and all my cards were in it.” The fine-tuned encoder gives 77 intent probabilities: lost_or_stolen_card 0.418, compromised_card 0.416, lost_or_stolen_phone 0.044, and the 74 others 0.122 together. One arrow, “old way”, takes the top intent only: 0.418 is below the threshold of 0.75, so a person must read it. The other arrow, “this case study”, adds the intents of each route: security 0.916, and the other 6 routes 0.084 together. 0.916 is above 0.75, so the question goes to security.

The model is not sure which security intent this is, but all three go to the same team. On the 2,000 validation questions, at a threshold of 0.75, the top intent is below the threshold for 138 questions, and the route confidence for only 57. The sum keeps 81 questions out of the review queue, and they still go to the right team.

The baseline makes 207 intent mistakes on the validation questions. Tunde counts them by pair:

intent mistakes: 207, of which inside the same route: 132 (64%); they cost nothing at the route level
intent predicted questions same_route
0 top_up_reverted top_up_failed 5 True
1 top_up_failed pending_top_up 4 True
2 wrong_exchange_rate_for_cash_withdrawal card_payment_wrong_exchange_rate 3 False
3 pending_transfer transfer_not_received_by_recipient 3 True
4 balance_not_updated_after_bank_transfer transfer_not_received_by_recipient 3 True
5 card_payment_wrong_exchange_rate wrong_exchange_rate_for_cash_withdrawal 3 False

What this tells us. Two of every three intent mistakes stay inside one team, so the mapping makes them free. The costly mistakes cross a team border: a wrong exchange rate on a cash withdrawal (cash_atm) against one on a card payment (payments_fees).

The best-known pair in Banking77 is card_arrival against card_delivery_estimate. Read two training questions:

  • “I ordered my card but it still isn’t here” is labelled card_arrival.
  • “I ordered my card a couple weeks ago and haven’t received it yet. When can I expect it?” is labelled card_delivery_estimate.

A person could label either one either way. On validation, the baseline confuses this pair only twice. On the test questions, the chosen model will confuse it 7 times. Both intents go to cards, so at the route level it costs nothing. If the bank later sends a different automatic answer for each intent, this border will matter again.

Common mistake

The labels that are both right

Symptom: Tunde reads the questions that the model gets “wrong” between card_arrival and card_delivery_estimate. For most of them, he agrees with the model.

Cause: the label border is not sharp. Many crowd workers labelled similar questions in different ways. A model cannot be more consistent than its labels, as Measure agreement showed.

Fix: do not chase these points with a bigger model. Group the two intents into one team, as the mapping does. If the difference matters to the business, write a labelling guideline for the border and relabel a sample, as in Labelling guidelines.

Consequence in AI work: part of every “error rate” on a public dataset is label noise. Read the mistakes before you buy more accuracy.

Each method learns on the same 8,003 training questions and is measured on the same 2,000 validation questions.

The course’s older embedding model turns each question into 384 numbers that capture its meaning. A logistic regression learns the 77 intents on them, as in Sentence embeddings for classification. Embedding all 13,083 questions takes 13 seconds on a laptop.

The result: intent accuracy 0.900, route accuracy 0.962, 232 dollars per 1,000. On Larkfield’s inbox, frozen embeddings lost clearly to TF-IDF (0.920 against 0.962). Here they tie on routes and cost a little less. Short questions written by many people use many words for one meaning, and that is where meaning-based numbers help.

Next, the same model, fine-tuned. Its 12 layers and a new classification head with 77 outputs learn from the training questions. The large word table stays frozen, as in Fine-tune a small encoder on CPU. 21.7 million of its 117.7 million parameters learn.

The first run used Larkfield’s recipe with 4 epochs. Its validation loss was still falling at the end (0.706), and it reached 0.890 on intents: it was under-trained. The second run used 6 epochs and a higher learning rate (0.0002). It trained for 484 seconds, about 8 minutes, on a laptop CPU (4 threads), and its validation loss stopped falling at about 0.37. That run is the ready checkpoint: 43.4 MB of changed weights in the labs repository. The notebook loads it, and an optional cell trains it again.

missing: ['bert.embeddings.word_embeddings.weight'] | unexpected: []
method intent_accuracy route_accuracy security_recall total_per_1000 ms_per_message
0 fine-tuned encoder 0.922 0.968 0.956 182.0 4.36

What this tells us. Only the frozen word table is missing, as expected. The fine-tuned encoder is the best so far on every column. It gets 0.922 on intents and 0.968 on routes, finds 95.6% of the security questions, and costs 182 dollars per 1,000. It needs about 4 milliseconds per question.

A hosted LLM (Azure chat-small) chose an intent for the sample of questions. The prompt lists the 77 intents grouped by route, and adds the 8 most similar training questions with their intents, as in Few-shot examples and label descriptions. The answer is JSON: one intent from the list and a confidence. We recorded each answer once, so you replay them with no key. Here is the prompt for one test question:

Labelled examples:
- "I was issued a new card a week ago but still haven't received it. How long will it be until I receive it?" -> card_arrival
- "I ordered my card a couple weeks ago and haven't received it yet. When can I expect it?" -> card_delivery_estimate
- "My card hasn't arrived in the mail yet. I ordered it two weeks ago. What can I do?" -> card_arrival
...
Question:
I still have not received my new card, I ordered over a week ago.
few-shot: 1535 answers, all valid: True | tokens in 1083, out 20 | 0.118 dollars per 1,000 | median 1378 ms

Each of the 1,535 answers costs about a tenth of a cent and takes about 1.4 seconds.

The LLM has answers only for the sample, so every method is scored on the same 765 validation questions:

method intent_accuracy route_accuracy security_recall total_per_1000
0 embeddings (frozen) + LR 0.901 0.962 0.967 193.5
1 LLM few-shot (recorded) 0.898 0.963 0.950 209.3
2 fine-tuned encoder 0.920 0.967 0.933 214.4
3 TF-IDF + LR 0.885 0.954 0.900 308.5

What this tells us. On this sample, the frozen embeddings are cheapest and the encoder is third. But the three are within 21 dollars, and one urgent security question in the wrong queue changes a row by 26 dollars per 1,000. On the full 2,000 questions, the encoder was clearly cheapest (182 against 232 and 256). Use a small sample to find large gaps, not small ones.

The LLM is as good as the trained models, with no training. But it is not better, it takes over a second per question, and every question leaves the bank.

Tunde chooses the fine-tuned encoder: the most accurate and cheapest on the full validation set, fast, and it runs inside the bank.

Now the second decision: which questions go to a person? A question whose route confidence is below a threshold goes to review, as in Decide, review or send to a person. Each review costs 1.50 dollars, but it removes the risk of a 4- or 20-dollar mistake. Tunde tries every threshold from 0.30 to 0.99 on the validation questions:

def with_review(P, threshold):
"""The routes, with "review" where the highest route probability is below the threshold."""
return np.where(P.max(axis=1) >= threshold, P.idxmax(axis=1), "review")
  • Total cost per 1,000 questions
  • Wrong routes
  • Reviews (1.50 dollars each)
Fine-tuned encoder, 2,000 validation questions. As the threshold rises, wrong routes cost less and reviews cost more. The total is lowest at 0.75: 138.8 dollars per 1,000, with 2.8% of the questions reviewed.
Text description of the chart

Three lines against the threshold, from 0.30 to 0.97. The cost of wrong routes falls from 182 dollars per 1,000 at 0.30 to 154 at 0.60, 110 at 0.70, 96.8 at 0.75, 56 at 0.90 and 23.2 at 0.95. The cost of reviews rises from 0 at 0.30 to 15 at 0.60, 33 at 0.70, 42 at 0.75, 91.5 at 0.90 and 141 at 0.95. The total is 182 at 0.30, 169 at 0.60, 143 at 0.70, lowest at 0.75 with 138.8, then 147.5 at 0.90, 164.2 at 0.95 and 256 at 0.97. At 0.99, not shown, 78% of the questions go to review and the total is 1,165.

lowest cost on validation: threshold 0.75, review 2.8%, 138.8 dollars per 1,000 (no review: 182.0 at 0.30)

What this tells us. Sending 2.8% of the questions to a person cuts the cost from 182 to 138.8 dollars per 1,000, by 24%. The curve is flat near its lowest point: from 0.70 to 0.80, the total stays between 138.8 and 146. Above 0.95, the cost climbs fast, because the probabilities rarely reach 0.99 even when the model is right.

What if Leila’s team can read only 2% of the questions? The cheapest threshold under that limit is 0.66: 1.8% reviewed, for 152.2 dollars per 1,000. The limit costs about 13 dollars per 1,000, which Leila can weigh against one more agent.

Now Tunde opens the test file. Every method and the threshold were chosen on validation; nothing changes after this run.

method intent_accuracy route_accuracy security_recall review_share total_per_1000
0 fine-tuned encoder, review below 0.75 NaN 0.966 0.958 0.022 92.4
1 fine-tuned encoder 0.930 0.976 0.963 0.000 141.6
2 TF-IDF + LR 0.908 0.969 0.938 0.000 200.0
3 embeddings (frozen) + LR 0.908 0.964 0.925 0.000 239.0
4 everything to a person NaN 0.000 0.000 1.000 1500.0
The 3,080 test questions, used once. Review: the fine-tuned encoder that sends questions below 0.75 to a person, the cheapest. Frozen: frozen embeddings. A person reading everything would cost 1,500.
Text description of the chart

A column chart with four columns, the cost per 1,000 test questions. Encoder with review below 0.75: 92 dollars, the lowest, in the first colour. Encoder alone: 142. TF-IDF: 200. Frozen embeddings: 239. Everything to a person, not drawn, would be 1,500.

What this tells us. The choice holds: the encoder is the best method on the test questions too (0.930 on intents, 0.976 on routes). With the review queue, it costs 92.4 dollars per 1,000 and sends 2.2% to a person. Its route accuracy reads lower (0.966) only because a reviewed question counts as “not routed”. The test costs less than validation (138.8), because each set has its own noise. Quote both.

The samples with recorded LLM answers tell the same story:

method intent_accuracy route_accuracy review_share total_per_1000
0 cascade: encoder >= 0.75, LLM >= 0.9, else a person NaN 0.984 0.005 49.4
1 fine-tuned encoder, review below 0.75 NaN 0.978 0.013 55.8
2 fine-tuned encoder 0.942 0.984 0.000 83.1
3 LLM few-shot (recorded) 0.945 0.983 0.000 88.4
4 TF-IDF + LR 0.917 0.977 0.000 135.1
5 embeddings (frozen) + LR 0.914 0.966 0.000 197.4
6 LLM zero-shot (recorded) 0.812 0.931 0.000 337.8
The 770 test questions with recorded LLM answers. Review: the encoder with review below 0.75. Few and Zero: the LLM, few-shot and zero-shot. The few-shot LLM ties with the encoder; the zero-shot LLM is the most expensive. The cascade is the cheapest, but it rests on very few questions.
Text description of the chart

A column chart with five columns, the cost per 1,000 of the 770 test sample questions. Cascade: 49 dollars. Encoder with review: 56, in the first colour. Encoder alone: 83. LLM few-shot: 88. LLM zero-shot: 338. TF-IDF, not drawn, costs 135 on this sample.

What this tells us. The few-shot LLM and the encoder tie: 0.945 against 0.942 on intents, 88.4 against 83.1 dollars. The zero-shot LLM, with the intent names only, is far behind (0.812 on intents). The examples in the prompt teach the LLM the label borders, as at Larkfield.

Would an LLM help with the unsure questions?

Section titled “Would an LLM help with the unsure questions?”

The first row is a cascade, as in Cascades. The encoder decides when its route confidence is at least 0.75. Below that, the LLM decides when its written confidence is at least 0.9. A person reads the rest. Tunde chose the 0.9 on the validation sample. On the test sample, the cascade costs 49.4 dollars against 55.8, and sends only 0.5% to a person.

But the evidence is thin. The LLM threshold was chosen on only 17 unsure questions, and the test sample has about 10. The cascade is a good next experiment: record the LLM for every unsure question for a month, and decide on hundreds of questions, not ten.

On the 3,080 test questions, the encoder makes 217 intent mistakes. 142 of them stay inside one route, so only 73 questions (2.4%) reach the wrong team:

intent predicted questions same_route
0 declined_transfer declined_card_payment 5 False
1 card_arrival card_delivery_estimate 4 True
2 top_up_reverted top_up_failed 4 True
3 balance_not_updated_after_bank_transfer transfer_timing 4 True
4 fiat_currency_support exchange_via_app 3 True
5 card_delivery_estimate card_arrival 3 True

What this tells us. The costliest border is between transfers and card payments. “My payment was declined” can be either, and only the customer knows which. By route, transfers questions sent to payments_fees are the most common route mistake (11). For the urgent team, 9 of 240 security questions went to another team: 4 to account, 3 to cards and 2 to payments_fees. Those 9 questions cost 180 of the 436 dollars of wrong routes, because each costs 20.

The last step runs the router on eight new questions that we wrote, in the style of a real inbox.

text intent route confidence decision
0 My new card still has not come. It has been 10 days. card_arrival cards 0.985 send to cards
1 There is a payment in Lisbon on my card and I have never been there. card_payment_not_recognised security 0.962 send to security
2 How long does a transfer to a euro account take? transfer_timing transfers 0.906 send to transfers
3 I paid cash in at the machine but my balance did not change. balance_not_updated_after_cheque_or_cash_deposit cash_atm 0.792 send to cash_atm
4 Why did I pay a fee when I topped up with my card? top_up_by_card_charge top_ups 0.944 send to top_ups
5 Can my 15 year old son open an account? age_limit account 0.989 send to account
6 The shop says the payment failed but the money left my account. reverted_card_payment? payments_fees 0.972 send to payments_fees
7 hello? why_verify_identity account 0.623 review by a person

What this tells us. Seven routes look right, and the Lisbon payment goes to security with 0.962. The cash deposit is the least sure (0.792), close to the threshold. “hello?” has no intent; the model still picks one, but with 0.623 it goes to a person. A question that matches none of the 77 intents is the subject of the next case study.

Tunde writes to Leila in her words:

  • Use the fine-tuned encoder, and send questions with a route confidence below 0.75 to a person. On the test questions, this cost 92.4 dollars per 1,000. Reading everything costs 1,500, and the simple TF-IDF model 200. About 2 in 100 questions still go to a person.
  • Keep the written mapping as a reviewed file. Any change to it changes the routes. Review it with the team leads, version it, and test the router again after each change.
  • Watch the security team’s misses every week. 9 of 240 urgent questions went elsewhere in the test. Ask agents to flag security questions that reach them, and add those to the training data.
  • Do not add the hosted LLM yet. It ties with the encoder, is 300 times slower, and sends customers’ questions outside the bank. Test the cascade on a month of unsure questions first.
  • If the review team can read only 2%, use 0.66. It costs about 13 dollars more per 1,000 questions than 0.75.

And what not to use it for:

  • Not to answer customers automatically. The router chooses a team; people answer.
  • Not as proof for a real bank. These questions were written for an imagined bank. Label a few thousand of your own messages and test again.
  • Real data is harder at the fine level, not at the team level. Routes: 0.969 for TF-IDF on the test, close to Larkfield’s 0.962. Intents: 0.908.
  • The order of methods changed with the data. Frozen embeddings tied with TF-IDF here, while at Larkfield they lost. Test on your own data.
  • Limits. The questions were written by crowd workers, and they are cleaner and shorter than real ones. Each has one intent, so there are no two requests. The data is English only, some labels are borderline, the cost table is an assumption, and the LLM was recorded for samples only.
  • A wrong route is a real customer who waits. For security questions, the wait can cost the customer money. Keep a way for any agent to send a question to security at once, whatever the router says.
  • Check who the router fails. Banking77 has no customer information, so we could not check groups of customers. With real messages, compare the error rate by language, by channel and, where the law allows, by customer group, as in Priority, urgency and language.
  • Customer messages are personal data. The encoder runs inside the bank. Before you send real messages to a hosted LLM, check the contract and the law, and remove names and account numbers.
  • Credit the source. CC BY 4.0 asks for credit: name PolyAI and the paper, link the licence, and say what you changed.

This case study turns a fine-grained label set into a business decision: a mapping, a method, a threshold and a cost. Make it yours with one change. For example, map to 5 teams, change the cost of security questions, or try a second encoder. Then publish it so that someone can read it in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • intent_routes.csv: your mapping, with one sentence per route about why.
  • comparison.csv: every method on validation, and the test results apart.
  • The data source and licence: the PolyAI repository, CC BY 4.0, the paper, and the SHA-256 of the files you used.

A README template:

# Routing banking questions to 7 teams, with a review queue chosen by cost
## Problem
An online bank's help desk gets questions in 77 intents. Seven teams answer them. A question in
the wrong queue costs $4, an urgent security question $20, a review by a person $1.50 (assumptions).
## Data and licence
Banking77 (Casanueva et al., PolyAI, 2020), 13,083 questions, 77 intents, CC BY 4.0,
github.com/PolyAI-LDN/task-specific-datasets. Crowd-written questions for an imagined bank.
## Approach
A written mapping from 77 intents to 7 routes. Validation 2,000 split from train; test used once.
Compared TF-IDF + logistic regression, frozen MiniLM embeddings, a fine-tuned MiniLM encoder
(CPU, 8 minutes), and a few-shot LLM (recorded, on a sample). Route confidence = sum of the
intent probabilities of the route. Review threshold chosen on validation by cost.
## Results (test, used once)
Fine-tuned encoder, review below 0.75: $92.4 per 1,000 questions, 2.2% reviewed
(everything to a person: $1,500; TF-IDF: $200). Intent accuracy 0.930, route accuracy 0.976.
## Limits
Crowd-written questions; one intent per question; English only; assumed costs.

How to describe it in an interview, in two sentences: “I routed banking questions in 77 intents to 7 teams through a written mapping, and compared TF-IDF, embeddings, a fine-tuned small encoder and an LLM on one cost measure. The fine-tuned encoder with a review threshold chosen from the cost table cut the cost per 1,000 questions from 1,500 dollars to 92, and I showed that most of the remaining intent mistakes never reach the wrong team.”

Go further:

The notebook has three tasks with checks. Predict each answer before you run it.

1. Learn the routes directly. In section 6, train the same TF-IDF pipeline on train.route and measure its route accuracy on the validation questions.

Show the answer
direct = tfidf_lr().fit(train.text, train.route)
direct_accuracy = round(float((direct.predict(validation.text) == validation.route).mean()), 3)
route accuracy, trained on the 7 routes: 0.944

Lower than 0.962, as explained in the baseline section.

2. The card pair. In section 7, count the validation questions labelled card_delivery_estimate that the baseline predicts as card_arrival.

Show the answer
crosstab = pd.crosstab(pair.intent, pd.Series(tfidf_intent_val[pair.index], index=pair.index, name="predicted"))
estimate_as_arrival = int(crosstab.loc["card_delivery_estimate", "card_arrival"])
predicted card_arrival card_delivery_estimate
intent
card_arrival 30 0
card_delivery_estimate 2 18
card_delivery_estimate questions predicted as card_arrival: 2
  1. Read the table by rows: of 20 card_delivery_estimate questions, 2 went to card_arrival. Both are in cards, so they cost nothing.

3. A smaller review team. In section 11, find the cheapest threshold when at most 2% of the questions may go to review.

Show the answer
under = curve[curve.review_share <= 0.02]
threshold_2 = float(under.loc[under.total_per_1000.idxmin(), "threshold"])
threshold with at most 2% to review: 0.66
review_share total_per_1000
0.018 152.2

0.66: 1.8% reviewed, 152.2 dollars per 1,000.

Check your understanding

The baseline learns 77 intents, although the bank needs only 7 routes. Trained on the 7 routes directly, it reaches 0.944 instead of 0.962. Why is the finer model better at the coarser task?

Show the answer

Each route mixes very different questions. “My card was stolen” and “Who set up this direct debit?” are both security, but they share few words. A model with one class per route must draw one region around all of them, which is hard. A model with one class per intent learns each kind of question on its own, and the mapping adds them up afterwards. So keep the fine labels when you have them, and map them to teams after the model, not before.

Check your understanding

For 'I lost my wallet and all my cards were in it.', the top intent has a probability of only 0.418, yet the router sends the question straight to security at a threshold of 0.75. Is that a mistake?

Show the answer

No. The model is unsure which security intent this is: lost_or_stolen_card 0.418, compromised_card 0.416, lost_or_stolen_phone 0.044. All three go to the same team. The route confidence is the sum of the probabilities of the route’s intents: 0.916 for security. The question is about the route, so the route confidence is the right number to compare with the threshold. On validation, using the sum kept 81 of 2,000 questions out of review.

Check your understanding

On the 765 validation questions of the sample, the frozen embeddings cost 193.5 dollars per 1,000 and the fine-tuned encoder 214.4. On all 2,000 validation questions, the encoder costs 182 and the embeddings 232. Which result should Tunde trust, and why?

Show the answer

The result on all 2,000 questions. The sample is smaller, and the methods are close. One urgent security question in the wrong queue changes a sample row by 20 ÷ 765 × 1,000, about 26 dollars per 1,000. That is more than the gap between the methods. A sample is useful to compare a method that has answers only for the sample, like the recorded LLM, and to find large gaps. To choose between close methods, use the largest set you have, and confirm once on the test set.

Recap

  • A written mapping turned 77 fine-grained intents into 7 team routes, with urgency deciding first. Two of every three intent mistakes stayed inside one team.
  • Learning the intents and adding them up per route beat learning the routes directly (0.962 against 0.944). The route confidence is the sum of the route’s intent probabilities.
  • On the full validation set, the fine-tuned encoder was the most accurate and the cheapest. The few-shot LLM tied with it on a sample; the zero-shot LLM was far behind.
  • The review threshold came from the cost table: 0.75 on validation, 2.8% reviewed. On the test, used once: 92.4 dollars per 1,000 questions and 2.2% reviewed, against 1,500 for reading everything.
  • Real data was harder at the intent level (0.930 for the encoder) than at the team level (0.976).

Next, the second case study: questions that match no intent at all, and how to send them to a person.

Next: Find what the bot cannot handle