Route banking questions
- Case study 1
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we routed Larkfield's synthetic messages to six teams. In this case study, we use real open data: banking questions in 77 fine-grained intents. We group the intents into team routes, compare four kinds of method on one cost measure, and choose how many questions a person reads.
Case study 1 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceRoute banking questions
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Take 13,083 banking questions with 77 fine-grained labels to a routing decision that you can defend with numbers. You will map the labels to 7 teams and compare four kinds of method on one cost measure. Then you choose how many questions a person reads, test once, and run the router on new questions.
The notebook runs every step of this page, plus three tasks with checks. Its setup cells install the packages and download the data (1 MB) and the course’s files (44 MB). The base model (about 470 MB) downloads once. It needs no account and no key: the LLM’s answers were recorded. After the install, it runs in about a minute on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole course. These lessons matter most:
- The cost table and the cost per 1,000 messages, from The cost of each mistake.
- TF-IDF with logistic regression, from Text as numbers: TF-IDF and Naive Bayes, logistic regression and linear SVM.
- Frozen embeddings and fine-tuning, from Sentence embeddings for classification and Fine-tune a small encoder on CPU.
- A few-shot LLM prompt, from Few-shot examples and label descriptions.
- Teams and topics as two levels, from Hierarchical labels.
- Sending unsure messages to a person, from Decide, review or send to a person, and cascades, from Cascades.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, sentence-transformers 6.2.0, Transformers 5.22.2 and PyTorch 2.15.0 (CPU). The setup cell makes a folder banking77. To start again, delete that folder. This case study does not change your larkroute project.
Tested on macOS (Apple silicon) with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”Imagine a small online bank. Its customers write to the help desk in the app, and each question must reach the right team. Today, a person reads every question first and moves it to a team.
Leila leads customer service. She has seven teams, from cards to transfers. One of them is urgent, security: lost cards, stolen phones and payments that the customer does not recognise. A question for that team cannot wait in the wrong queue, because someone may be using the customer’s money.
Tunde is a data scientist. Leila asks him: “Can a model send most questions straight to the right team? How many must my people still read? And what does it cost?”
We invented Leila, Tunde and the bank. The questions come from a published dataset, described below.
The case study uses the course’s cost table. These are assumptions, the same values as for Larkfield:
- A question in the wrong queue costs 4 dollars: the wrong team reads it and moves it, and the customer waits longer.
- An urgent security question in the wrong queue costs 20 dollars.
- A review by a person costs 1.50 dollars: about a minute of an agent’s time.
So “a person reads everything”, Leila’s process today, costs 1,500 dollars per 1,000 questions.
What you will build
Section titled “What you will build”You will build a router that chooses one of 77 intents (what the customer wants), and from the intent, one of 7 routes (the team that answers). When it is unsure, it sends the question to a person. You will compare four kinds of method with one measure: the cost per 1,000 questions.
The data
Section titled “The data”The data is Banking77, published by PolyAI. It has 13,083 short questions to an online bank, each labelled with one of 77 intents, such as card_arrival (“Where is my new card?”) or transfer_timing (“How long does a transfer take?”).
Crowd workers wrote the questions for an imagined bank, so no real customer is behind them. They are real in another sense: people wrote them, with their own words, typos and borderline cases. The Larkfield inbox was written by one model from 27 templates.
We compared it with four other open datasets; the dataset card says why it won.
First checks
Section titled “First checks”The notebook downloads the two files from PolyAI’s repository at a fixed version and checks their checksums. Then Tunde loads them:
raw_train = pd.read_csv("data/train.csv").rename(columns={"category": "intent"})test = pd.read_csv("data/test.csv").rename(columns={"category": "intent"})The first checks print:
train file: 10,003 questions | test file: 3,080 | intents: 77missing values: 0 | repeated questions: 0 | test questions also in train: 0words per question: median 10, 90% have at most 22, longest 79train questions per intent: from 35 (contactless_not_working) to 187 (card_payment_fee_charged) | test: 40 to 40odd intent names: ['Refund_not_showing_up', 'reverted_card_payment?']What this tells us. The files are clean, and the questions are short: half have 10 words or fewer. The test file has exactly 40 questions per intent. Two intent names break the pattern. We keep them as they are; renaming labels quietly is how two files stop agreeing.
Tunde also reads 100 random questions and the longest ones. A few of the longest rows are answers written by a bank agent, not questions. Real data has this kind of noise.
From 77 intents to 7 routes
Section titled “From 77 intents to 7 routes”Think of a post office. Letters have exact addresses, but the sorting office has only a few bags, one per van. The office writes down once which streets go into which bag, and every sorter uses that list.
Banking77’s intents are the exact addresses. Leila’s teams are the bags. Tunde and Leila write down which team answers each intent. This is the written mapping: a file, intent_routes.csv, with one row per intent. It is a business decision, read by people and by code.
Text description of the diagram
On the left, one box: the 77 intents, Banking77’s labels, one per question. Seven arrows lead to seven route boxes. security has 6 intents and is urgent, for example lost_or_stolen_card and card_payment_not_recognised. cards has 20 intents, for example card_arrival, card_delivery_estimate and change_pin. payments_fees has 14 intents, for example transaction_charged_twice, extra_charge_on_statement and exchange_rate. transfers has 11, for example pending_transfer and failed_transfer. top_ups has 10, for example pending_top_up and top_up_failed. cash_atm has 7, for example atm_support and declined_cash_withdrawal. account has 9, for example verify_my_identity and terminate_account.
They followed three rules:
- One team per kind of work. A team gets the intents that its people can answer without help. The
cardsteam handles everything about the card itself: ordering, delivery, the PIN, virtual cards. - Urgency decides first. Every intent where someone else may be using the money goes to
security, even when another team knows the topic. Socard_payment_not_recognisedgoes tosecurity, not topayments_fees. - Write the hard choices down.
card_swallowed(a cash machine kept the card) could becash_atmorcards. The customer needs a new card, so it goes tocards.supported_cards_and_currenciesmixes two topics; it goes topayments_fees, with the other currency questions.
The notebook reads the file and checks it:
77 intents mapped to 7 routes; intents without a route: 0 intents priority train_questions shareroutesecurity 6 urgent 799 0.080cards 20 normal 2169 0.217payments_fees 14 normal 2105 0.210transfers 11 normal 1581 0.158top_ups 10 normal 1232 0.123cash_atm 7 normal 1104 0.110account 9 normal 1013 0.101What this tells us. The routes are not equal: cards and payments_fees get about a fifth of the questions each, and security 8%. This is the two-level shape of Hierarchical labels.
Frame the problem
Section titled “Frame the problem”Tunde writes down his answers before he trains anything:
- Unit: one question. It gets one route, or goes to a person.
- Labels: the model learns the 77 intents; the route comes from the mapping. We test both levels.
- One measure for the choice: the cost per 1,000 questions, with the course’s cost table. Route accuracy, the intent accuracy, the recall of
securityand the share sent to review stand next to it. - The split: 2,000 of the training questions become a validation set, stratified by intent. All choices are made on it. The test file stays closed until the end, and we use it once.
- A fair comparison: the hosted LLM was recorded for a sample only: up to 10 questions per intent, 765 validation and 770 test questions. Every method is also scored on the same sample.
train 8,003 | validation 2,000 | test 3,080 (used once, at the end)The course’s helper routing.py scores every method, with the bank’s 7 routes and every security question marked urgent.
A baseline: TF-IDF and logistic regression
Section titled “A baseline: TF-IDF and logistic regression”Tunde starts with the course project’s baseline: TF-IDF of words, word pairs and letter groups, and logistic regression. It learns the 77 intents in 5 seconds. On the validation questions:
method intent_accuracy route_accuracy security_recall review_share total_per_10000 TF-IDF + LR 0.896 0.962 0.919 0.0 256.01 everything to a person NaN 0.000 0.000 1.0 1500.0What this tells us. The baseline gets about 9 in 10 intents and 96 in 100 routes right. Many intent mistakes still land in the right team. The baseline costs 256 dollars per 1,000 questions, against 1,500 when a person reads everything.
On Larkfield’s synthetic inbox, the same pipeline also reached 0.962 on 7 routes. But the 77 intents are much harder (0.896). Fine-grained labels written by many people are where real data bites.
Learn the intents, or only the routes?
Section titled “Learn the intents, or only the routes?”If the bank needs only 7 routes, why learn 77 intents? Tunde trains the same pipeline on the routes directly:
route accuracy, trained on the 7 routes: 0.944What this tells us. Learning the 7 routes directly is worse: 0.944 against 0.962. Each route mixes very different questions, so one class per route is a harder job. Keep the fine labels, and map them afterwards.
Route confidence: add the intents of each route
Section titled “Route confidence: add the intents of each route”To send unsure questions to a person, we need a confidence for the route. The model gives a probability for each of the 77 intents. Which number is the route’s confidence?
Think of a vote with 77 candidates from 7 parties. Two candidates of one party get 42% and 41%. No candidate has a majority, but the party has 83%, and the party is what you asked about.
The same holds for routes. The route confidence is the sum of the probabilities of all the intents of that route. The function route_proba() in the notebook does this sum for every question.
Text description of the diagram
The question is “I lost my wallet and all my cards were in it.” The fine-tuned encoder gives 77 intent probabilities: lost_or_stolen_card 0.418, compromised_card 0.416, lost_or_stolen_phone 0.044, and the 74 others 0.122 together. One arrow, “old way”, takes the top intent only: 0.418 is below the threshold of 0.75, so a person must read it. The other arrow, “this case study”, adds the intents of each route: security 0.916, and the other 6 routes 0.084 together. 0.916 is above 0.75, so the question goes to security.
The model is not sure which security intent this is, but all three go to the same team. On the 2,000 validation questions, at a threshold of 0.75, the top intent is below the threshold for 138 questions, and the route confidence for only 57. The sum keeps 81 questions out of the review queue, and they still go to the right team.
Where the intents get confused
Section titled “Where the intents get confused”The baseline makes 207 intent mistakes on the validation questions. Tunde counts them by pair:
intent mistakes: 207, of which inside the same route: 132 (64%); they cost nothing at the route level intent predicted questions same_route0 top_up_reverted top_up_failed 5 True1 top_up_failed pending_top_up 4 True2 wrong_exchange_rate_for_cash_withdrawal card_payment_wrong_exchange_rate 3 False3 pending_transfer transfer_not_received_by_recipient 3 True4 balance_not_updated_after_bank_transfer transfer_not_received_by_recipient 3 True5 card_payment_wrong_exchange_rate wrong_exchange_rate_for_cash_withdrawal 3 FalseWhat this tells us. Two of every three intent mistakes stay inside one team, so the mapping makes them free. The costly mistakes cross a team border: a wrong exchange rate on a cash withdrawal (cash_atm) against one on a card payment (payments_fees).
The best-known pair in Banking77 is card_arrival against card_delivery_estimate. Read two training questions:
- “I ordered my card but it still isn’t here” is labelled
card_arrival. - “I ordered my card a couple weeks ago and haven’t received it yet. When can I expect it?” is labelled
card_delivery_estimate.
A person could label either one either way. On validation, the baseline confuses this pair only twice. On the test questions, the chosen model will confuse it 7 times. Both intents go to cards, so at the route level it costs nothing. If the bank later sends a different automatic answer for each intent, this border will matter again.
Common mistake
The labels that are both right
Symptom: Tunde reads the questions that the model gets “wrong” between card_arrival and card_delivery_estimate. For most of them, he agrees with the model.
Cause: the label border is not sharp. Many crowd workers labelled similar questions in different ways. A model cannot be more consistent than its labels, as Measure agreement showed.
Fix: do not chase these points with a bigger model. Group the two intents into one team, as the mapping does. If the difference matters to the business, write a labelling guideline for the border and relabel a sample, as in Labelling guidelines.
Consequence in AI work: part of every “error rate” on a public dataset is label noise. Read the mistakes before you buy more accuracy.
Three more methods
Section titled “Three more methods”Each method learns on the same 8,003 training questions and is measured on the same 2,000 validation questions.
Frozen embeddings and logistic regression
Section titled “Frozen embeddings and logistic regression”The course’s older embedding model turns each question into 384 numbers that capture its meaning. A logistic regression learns the 77 intents on them, as in Sentence embeddings for classification. Embedding all 13,083 questions takes 13 seconds on a laptop.
The result: intent accuracy 0.900, route accuracy 0.962, 232 dollars per 1,000. On Larkfield’s inbox, frozen embeddings lost clearly to TF-IDF (0.920 against 0.962). Here they tie on routes and cost a little less. Short questions written by many people use many words for one meaning, and that is where meaning-based numbers help.
The fine-tuned encoder
Section titled “The fine-tuned encoder”Next, the same model, fine-tuned. Its 12 layers and a new classification head with 77 outputs learn from the training questions. The large word table stays frozen, as in Fine-tune a small encoder on CPU. 21.7 million of its 117.7 million parameters learn.
The first run used Larkfield’s recipe with 4 epochs. Its validation loss was still falling at the end (0.706), and it reached 0.890 on intents: it was under-trained. The second run used 6 epochs and a higher learning rate (0.0002). It trained for 484 seconds, about 8 minutes, on a laptop CPU (4 threads), and its validation loss stopped falling at about 0.37. That run is the ready checkpoint: 43.4 MB of changed weights in the labs repository. The notebook loads it, and an optional cell trains it again.
missing: ['bert.embeddings.word_embeddings.weight'] | unexpected: [] method intent_accuracy route_accuracy security_recall total_per_1000 ms_per_message0 fine-tuned encoder 0.922 0.968 0.956 182.0 4.36What this tells us. Only the frozen word table is missing, as expected. The fine-tuned encoder is the best so far on every column. It gets 0.922 on intents and 0.968 on routes, finds 95.6% of the security questions, and costs 182 dollars per 1,000. It needs about 4 milliseconds per question.
The hosted LLM, few-shot and recorded
Section titled “The hosted LLM, few-shot and recorded”A hosted LLM (Azure chat-small) chose an intent for the sample of questions. The prompt lists the 77 intents grouped by route, and adds the 8 most similar training questions with their intents, as in Few-shot examples and label descriptions. The answer is JSON: one intent from the list and a confidence. We recorded each answer once, so you replay them with no key. Here is the prompt for one test question:
Labelled examples:- "I was issued a new card a week ago but still haven't received it. How long will it be until I receive it?" -> card_arrival- "I ordered my card a couple weeks ago and haven't received it yet. When can I expect it?" -> card_delivery_estimate- "My card hasn't arrived in the mail yet. I ordered it two weeks ago. What can I do?" -> card_arrival...Question:I still have not received my new card, I ordered over a week ago.few-shot: 1535 answers, all valid: True | tokens in 1083, out 20 | 0.118 dollars per 1,000 | median 1378 msEach of the 1,535 answers costs about a tenth of a cent and takes about 1.4 seconds.
A fair comparison
Section titled “A fair comparison”The LLM has answers only for the sample, so every method is scored on the same 765 validation questions:
method intent_accuracy route_accuracy security_recall total_per_10000 embeddings (frozen) + LR 0.901 0.962 0.967 193.51 LLM few-shot (recorded) 0.898 0.963 0.950 209.32 fine-tuned encoder 0.920 0.967 0.933 214.43 TF-IDF + LR 0.885 0.954 0.900 308.5What this tells us. On this sample, the frozen embeddings are cheapest and the encoder is third. But the three are within 21 dollars, and one urgent security question in the wrong queue changes a row by 26 dollars per 1,000. On the full 2,000 questions, the encoder was clearly cheapest (182 against 232 and 256). Use a small sample to find large gaps, not small ones.
The LLM is as good as the trained models, with no training. But it is not better, it takes over a second per question, and every question leaves the bank.
Tunde chooses the fine-tuned encoder: the most accurate and cheapest on the full validation set, fast, and it runs inside the bank.
A review queue from the cost table
Section titled “A review queue from the cost table”Now the second decision: which questions go to a person? A question whose route confidence is below a threshold goes to review, as in Decide, review or send to a person. Each review costs 1.50 dollars, but it removes the risk of a 4- or 20-dollar mistake. Tunde tries every threshold from 0.30 to 0.99 on the validation questions:
def with_review(P, threshold): """The routes, with "review" where the highest route probability is below the threshold.""" return np.where(P.max(axis=1) >= threshold, P.idxmax(axis=1), "review")- Total cost per 1,000 questions
- Wrong routes
- Reviews (1.50 dollars each)
Text description of the chart
Three lines against the threshold, from 0.30 to 0.97. The cost of wrong routes falls from 182 dollars per 1,000 at 0.30 to 154 at 0.60, 110 at 0.70, 96.8 at 0.75, 56 at 0.90 and 23.2 at 0.95. The cost of reviews rises from 0 at 0.30 to 15 at 0.60, 33 at 0.70, 42 at 0.75, 91.5 at 0.90 and 141 at 0.95. The total is 182 at 0.30, 169 at 0.60, 143 at 0.70, lowest at 0.75 with 138.8, then 147.5 at 0.90, 164.2 at 0.95 and 256 at 0.97. At 0.99, not shown, 78% of the questions go to review and the total is 1,165.
lowest cost on validation: threshold 0.75, review 2.8%, 138.8 dollars per 1,000 (no review: 182.0 at 0.30)What this tells us. Sending 2.8% of the questions to a person cuts the cost from 182 to 138.8 dollars per 1,000, by 24%. The curve is flat near its lowest point: from 0.70 to 0.80, the total stays between 138.8 and 146. Above 0.95, the cost climbs fast, because the probabilities rarely reach 0.99 even when the model is right.
What if Leila’s team can read only 2% of the questions? The cheapest threshold under that limit is 0.66: 1.8% reviewed, for 152.2 dollars per 1,000. The limit costs about 13 dollars per 1,000, which Leila can weigh against one more agent.
The test, once
Section titled “The test, once”Now Tunde opens the test file. Every method and the threshold were chosen on validation; nothing changes after this run.
method intent_accuracy route_accuracy security_recall review_share total_per_10000 fine-tuned encoder, review below 0.75 NaN 0.966 0.958 0.022 92.41 fine-tuned encoder 0.930 0.976 0.963 0.000 141.62 TF-IDF + LR 0.908 0.969 0.938 0.000 200.03 embeddings (frozen) + LR 0.908 0.964 0.925 0.000 239.04 everything to a person NaN 0.000 0.000 1.000 1500.0Text description of the chart
A column chart with four columns, the cost per 1,000 test questions. Encoder with review below 0.75: 92 dollars, the lowest, in the first colour. Encoder alone: 142. TF-IDF: 200. Frozen embeddings: 239. Everything to a person, not drawn, would be 1,500.
What this tells us. The choice holds: the encoder is the best method on the test questions too (0.930 on intents, 0.976 on routes). With the review queue, it costs 92.4 dollars per 1,000 and sends 2.2% to a person. Its route accuracy reads lower (0.966) only because a reviewed question counts as “not routed”. The test costs less than validation (138.8), because each set has its own noise. Quote both.
The samples with recorded LLM answers tell the same story:
method intent_accuracy route_accuracy review_share total_per_10000 cascade: encoder >= 0.75, LLM >= 0.9, else a person NaN 0.984 0.005 49.41 fine-tuned encoder, review below 0.75 NaN 0.978 0.013 55.82 fine-tuned encoder 0.942 0.984 0.000 83.13 LLM few-shot (recorded) 0.945 0.983 0.000 88.44 TF-IDF + LR 0.917 0.977 0.000 135.15 embeddings (frozen) + LR 0.914 0.966 0.000 197.46 LLM zero-shot (recorded) 0.812 0.931 0.000 337.8Text description of the chart
A column chart with five columns, the cost per 1,000 of the 770 test sample questions. Cascade: 49 dollars. Encoder with review: 56, in the first colour. Encoder alone: 83. LLM few-shot: 88. LLM zero-shot: 338. TF-IDF, not drawn, costs 135 on this sample.
What this tells us. The few-shot LLM and the encoder tie: 0.945 against 0.942 on intents, 88.4 against 83.1 dollars. The zero-shot LLM, with the intent names only, is far behind (0.812 on intents). The examples in the prompt teach the LLM the label borders, as at Larkfield.
Would an LLM help with the unsure questions?
Section titled “Would an LLM help with the unsure questions?”The first row is a cascade, as in Cascades. The encoder decides when its route confidence is at least 0.75. Below that, the LLM decides when its written confidence is at least 0.9. A person reads the rest. Tunde chose the 0.9 on the validation sample. On the test sample, the cascade costs 49.4 dollars against 55.8, and sends only 0.5% to a person.
But the evidence is thin. The LLM threshold was chosen on only 17 unsure questions, and the test sample has about 10. The cascade is a good next experiment: record the LLM for every unsure question for a month, and decide on hundreds of questions, not ten.
Where the router still fails
Section titled “Where the router still fails”On the 3,080 test questions, the encoder makes 217 intent mistakes. 142 of them stay inside one route, so only 73 questions (2.4%) reach the wrong team:
intent predicted questions same_route0 declined_transfer declined_card_payment 5 False1 card_arrival card_delivery_estimate 4 True2 top_up_reverted top_up_failed 4 True3 balance_not_updated_after_bank_transfer transfer_timing 4 True4 fiat_currency_support exchange_via_app 3 True5 card_delivery_estimate card_arrival 3 TrueWhat this tells us. The costliest border is between transfers and card payments. “My payment was declined” can be either, and only the customer knows which. By route, transfers questions sent to payments_fees are the most common route mistake (11). For the urgent team, 9 of 240 security questions went to another team: 4 to account, 3 to cards and 2 to payments_fees. Those 9 questions cost 180 of the 436 dollars of wrong routes, because each costs 20.
New questions
Section titled “New questions”The last step runs the router on eight new questions that we wrote, in the style of a real inbox.
text intent route confidence decision0 My new card still has not come. It has been 10 days. card_arrival cards 0.985 send to cards1 There is a payment in Lisbon on my card and I have never been there. card_payment_not_recognised security 0.962 send to security2 How long does a transfer to a euro account take? transfer_timing transfers 0.906 send to transfers3 I paid cash in at the machine but my balance did not change. balance_not_updated_after_cheque_or_cash_deposit cash_atm 0.792 send to cash_atm4 Why did I pay a fee when I topped up with my card? top_up_by_card_charge top_ups 0.944 send to top_ups5 Can my 15 year old son open an account? age_limit account 0.989 send to account6 The shop says the payment failed but the money left my account. reverted_card_payment? payments_fees 0.972 send to payments_fees7 hello? why_verify_identity account 0.623 review by a personWhat this tells us. Seven routes look right, and the Lisbon payment goes to security with 0.962. The cash deposit is the least sure (0.792), close to the threshold. “hello?” has no intent; the model still picks one, but with 0.623 it goes to a person. A question that matches none of the 77 intents is the subject of the next case study.
Recommendation
Section titled “Recommendation”Tunde writes to Leila in her words:
- Use the fine-tuned encoder, and send questions with a route confidence below 0.75 to a person. On the test questions, this cost 92.4 dollars per 1,000. Reading everything costs 1,500, and the simple TF-IDF model 200. About 2 in 100 questions still go to a person.
- Keep the written mapping as a reviewed file. Any change to it changes the routes. Review it with the team leads, version it, and test the router again after each change.
- Watch the security team’s misses every week. 9 of 240 urgent questions went elsewhere in the test. Ask agents to flag security questions that reach them, and add those to the training data.
- Do not add the hosted LLM yet. It ties with the encoder, is 300 times slower, and sends customers’ questions outside the bank. Test the cascade on a month of unsure questions first.
- If the review team can read only 2%, use 0.66. It costs about 13 dollars more per 1,000 questions than 0.75.
And what not to use it for:
- Not to answer customers automatically. The router chooses a team; people answer.
- Not as proof for a real bank. These questions were written for an imagined bank. Label a few thousand of your own messages and test again.
Insights and limits
Section titled “Insights and limits”- Real data is harder at the fine level, not at the team level. Routes: 0.969 for TF-IDF on the test, close to Larkfield’s 0.962. Intents: 0.908.
- The order of methods changed with the data. Frozen embeddings tied with TF-IDF here, while at Larkfield they lost. Test on your own data.
- Limits. The questions were written by crowd workers, and they are cleaner and shorter than real ones. Each has one intent, so there are no two requests. The data is English only, some labels are borderline, the cost table is an assumption, and the LLM was recorded for samples only.
Responsible use
Section titled “Responsible use”- A wrong route is a real customer who waits. For security questions, the wait can cost the customer money. Keep a way for any agent to send a question to
securityat once, whatever the router says. - Check who the router fails. Banking77 has no customer information, so we could not check groups of customers. With real messages, compare the error rate by language, by channel and, where the law allows, by customer group, as in Priority, urgency and language.
- Customer messages are personal data. The encoder runs inside the bank. Before you send real messages to a hosted LLM, check the contract and the law, and remove names and account numbers.
- Credit the source. CC BY 4.0 asks for credit: name PolyAI and the paper, link the licence, and say what you changed.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study turns a fine-grained label set into a business decision: a mapping, a method, a threshold and a cost. Make it yours with one change. For example, map to 5 teams, change the cost of security questions, or try a second encoder. Then publish it so that someone can read it in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
intent_routes.csv: your mapping, with one sentence per route about why.comparison.csv: every method on validation, and the test results apart.- The data source and licence: the PolyAI repository, CC BY 4.0, the paper, and the SHA-256 of the files you used.
A README template:
# Routing banking questions to 7 teams, with a review queue chosen by cost
## ProblemAn online bank's help desk gets questions in 77 intents. Seven teams answer them. A question inthe wrong queue costs $4, an urgent security question $20, a review by a person $1.50 (assumptions).
## Data and licenceBanking77 (Casanueva et al., PolyAI, 2020), 13,083 questions, 77 intents, CC BY 4.0,github.com/PolyAI-LDN/task-specific-datasets. Crowd-written questions for an imagined bank.
## ApproachA written mapping from 77 intents to 7 routes. Validation 2,000 split from train; test used once.Compared TF-IDF + logistic regression, frozen MiniLM embeddings, a fine-tuned MiniLM encoder(CPU, 8 minutes), and a few-shot LLM (recorded, on a sample). Route confidence = sum of theintent probabilities of the route. Review threshold chosen on validation by cost.
## Results (test, used once)Fine-tuned encoder, review below 0.75: $92.4 per 1,000 questions, 2.2% reviewed(everything to a person: $1,500; TF-IDF: $200). Intent accuracy 0.930, route accuracy 0.976.
## LimitsCrowd-written questions; one intent per question; English only; assumed costs.How to describe it in an interview, in two sentences: “I routed banking questions in 77 intents to 7 teams through a written mapping, and compared TF-IDF, embeddings, a fine-tuned small encoder and an LLM on one cost measure. The fine-tuned encoder with a review threshold chosen from the cost table cut the cost per 1,000 questions from 1,500 dollars to 92, and I showed that most of the remaining intent mistakes never reach the wrong team.”
Go further:
- Change the mapping: merge
top_upsandcash_atm, or splitpayments_fees. Does the cost change? - Give security questions their own, lower threshold, as the takeover flag in Imbalance and rare classes.
- Calibrate the route confidence on validation, and compare the reliability before and after, as in Is the confidence true?.
- Compare with the other case studies: Find what the bot cannot handle and Sort requests with few labels.
Your turn
Section titled “Your turn”The notebook has three tasks with checks. Predict each answer before you run it.
1. Learn the routes directly. In section 6, train the same TF-IDF pipeline on train.route and measure its route accuracy on the validation questions.
Show the answer
direct = tfidf_lr().fit(train.text, train.route)direct_accuracy = round(float((direct.predict(validation.text) == validation.route).mean()), 3)route accuracy, trained on the 7 routes: 0.944Lower than 0.962, as explained in the baseline section.
2. The card pair. In section 7, count the validation questions labelled card_delivery_estimate that the baseline predicts as card_arrival.
Show the answer
crosstab = pd.crosstab(pair.intent, pd.Series(tfidf_intent_val[pair.index], index=pair.index, name="predicted"))estimate_as_arrival = int(crosstab.loc["card_delivery_estimate", "card_arrival"])predicted card_arrival card_delivery_estimateintentcard_arrival 30 0card_delivery_estimate 2 18card_delivery_estimate questions predicted as card_arrival: 2- Read the table by rows: of 20
card_delivery_estimatequestions, 2 went tocard_arrival. Both are incards, so they cost nothing.
3. A smaller review team. In section 11, find the cheapest threshold when at most 2% of the questions may go to review.
Show the answer
under = curve[curve.review_share <= 0.02]threshold_2 = float(under.loc[under.total_per_1000.idxmin(), "threshold"])threshold with at most 2% to review: 0.66 review_share total_per_1000 0.018 152.20.66: 1.8% reviewed, 152.2 dollars per 1,000.
Knowledge check
Section titled “Knowledge check”Check your understanding
The baseline learns 77 intents, although the bank needs only 7 routes. Trained on the 7 routes directly, it reaches 0.944 instead of 0.962. Why is the finer model better at the coarser task?
Show the answer
Each route mixes very different questions. “My card was stolen” and “Who set up this direct debit?” are both security, but they share few words. A model with one class per route must draw one region around all of them, which is hard. A model with one class per intent learns each kind of question on its own, and the mapping adds them up afterwards. So keep the fine labels when you have them, and map them to teams after the model, not before.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
For 'I lost my wallet and all my cards were in it.', the top intent has a probability of only 0.418, yet the router sends the question straight to security at a threshold of 0.75. Is that a mistake?
Show the answer
No. The model is unsure which security intent this is: lost_or_stolen_card 0.418, compromised_card 0.416, lost_or_stolen_phone 0.044. All three go to the same team. The route confidence is the sum of the probabilities of the route’s intents: 0.916 for security. The question is about the route, so the route confidence is the right number to compare with the threshold. On validation, using the sum kept 81 of 2,000 questions out of review.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the 765 validation questions of the sample, the frozen embeddings cost 193.5 dollars per 1,000 and the fine-tuned encoder 214.4. On all 2,000 validation questions, the encoder costs 182 and the embeddings 232. Which result should Tunde trust, and why?
Show the answer
The result on all 2,000 questions. The sample is smaller, and the methods are close. One urgent security question in the wrong queue changes a sample row by 20 ÷ 765 × 1,000, about 26 dollars per 1,000. That is more than the gap between the methods. A sample is useful to compare a method that has answers only for the sample, like the recorded LLM, and to find large gaps. To choose between close methods, use the largest set you have, and confirm once on the test set.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A written mapping turned 77 fine-grained intents into 7 team routes, with urgency deciding first. Two of every three intent mistakes stayed inside one team.
- Learning the intents and adding them up per route beat learning the routes directly (0.962 against 0.944). The route confidence is the sum of the route’s intent probabilities.
- On the full validation set, the fine-tuned encoder was the most accurate and the cheapest. The few-shot LLM tied with it on a sample; the zero-shot LLM was far behind.
- The review threshold came from the cost table: 0.75 on validation, 2.8% reviewed. On the test, used once: 92.4 dollars per 1,000 questions and 2.2% reviewed, against 1,500 for reading everything.
- Real data was harder at the intent level (0.930 for the encoder) than at the team level (0.976).
Next, the second case study: questions that match no intent at all, and how to send them to a person.