An IVR for spoken intents
- Case study 4
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we built and tested the Larkfield IVR. In this case study, we build a menu-and-speech IVR for a bank's common requests, test it with real spoken-intent recordings, and compare three ways to find the intent.
Case study 4 of 4 · Speech AI: Transcribe, Understand and SpeakAn IVR for spoken intents
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Choose the intent step for a phone line that understands spoken requests. You will test keyword rules, a classifier and an LLM on real people’s voices in four language varieties. You will turn accuracy into calls routed without a person and into money. Then you will choose a design (menu, speech or both) and name the callers it serves worst.
The notebook runs the steps of this page with the same code. It downloads a few megabytes: the text and labels of every request, the recorded transcripts and LLM answers, and 12 short recordings. At the end, it runs Whisper base live on those 12 recordings. It needs no account and no key.
Before you startWhat to know and to have ready
What you need to know
This case study uses the understanding and IVR parts of the course. These lessons matter most:
- Keyword rules, the TF-IDF classifier and an LLM that answers in JSON, from Why did they call?.
- The IVR’s intent step and its keyword rules, from Let callers say what they need.
- Containment and results by slice, from Test the IVR like a product, and costs per call, from Menu, agent or both?.
- The word error rate and the normalisation guide, from Word error rate, step by step.
What you need on your computer (only for the local route; the notebook needs none of it)
- A virtual environment with scikit-learn and faster-whisper, as in the project. The
larkspeechproject works; this case study does not change its files. - The notebook, downloaded from the link above. Its setup cell makes a new folder
minds-ivr. To start again, delete that folder.
Tested on macOS with Python 3.12. Windows, Linux, Colab and Kaggle were not tested.
The situation
Section titled “The situation”Larkfield’s IVR understands four intents: order status, opening hours, a return, and “a person”. Omar, the product manager, wants more. Callers ask about invoices, payment cards, account changes and much more. “Can the IVR route twenty requests by speech? Or do we need a long keypad menu? Or both?”
Priya sees two problems. Every Larkfield call is synthetic, and real people speak in their own words. And Larkfield has no labelled recordings of real requests yet. So she tests on public data first: real voices, many intents, phone audio. The closest open set is about banking, not garden products, but the question is the same: which intent step, with which safety net, for which callers?
Two mistakes matter, and they cost different amounts:
- A wrong route. The caller lands in the wrong queue, waits, explains again and is transferred. The course’s assumption is US$15 for a wrong action.
- A hand-over that was not needed. A person answers a call that the IVR could have routed. The course’s assumption is US$4.00 a call (6 minutes at US$40 an hour).
What you will build
Section titled “What you will build”You will build the intent step of an IVR and its safety net, then compare three methods on the same requests with the same measure.
Text description of the diagram
A caller says a request. Whisper writes the words: base, the recogniser of the Larkfield IVR, or small. The intent step chooses one of 14 intents. If it is sure enough, the IVR routes the call. If it is not sure, there are two designs. Speech only gives the call to a person. Both (speech and a menu) first plays a short keypad menu with the three most likely intents: “Press 1 for a lost or blocked card, 2 for …, 3 for …, or 0 for a person.” If the caller finds their intent there, the call is routed. If not, a person takes it.
The data
Section titled “The data”The data is MInDS-14 (Multilingual Intent Detection from Speech), published by PolyAI. Crowd workers recorded spoken requests to a bank’s phone line, with 14 intents, in 14 language varieties. Each worker got an intent, a short description and three examples, then said a new request in their own words. Only native speakers of each variety took part. Some Australian requests went through a real phone voice system; the others were recorded online. All are 8 kHz phone audio, like Larkfield’s phone calls.
We use four varieties, 2,348 requests and 5.47 hours in all:
| Variety | Requests | Hours | Median length |
|---|---|---|---|
| US English (en-US) | 563 | 1.34 | 6.4 s |
| British English (en-GB) | 592 | 1.35 | 6.3 s |
| Australian English (en-AU) | 654 | 1.53 | 6.6 s |
| French (fr-FR) | 539 | 1.25 | 6.6 s |
The 14 intents are: abroad (the card in another country), address, app error, ATM limit, balance, business loan, card issues, cash deposit, direct debit, freeze, high-value payment, joint account, latest transactions and pay a bill. Each variety has about 30 to 50 requests per intent.
Listen first
Section titled “Listen first”Four requests, one per variety. Each is short. Under each player are two transcripts, because no person wrote down these words. The dataset’s transcript was made by Google’s recogniser (the paper says so). The second is Whisper base’s, recorded for this case study.
The two recognisers agree on the first request. On the British one, Whisper base turns “balance” into a name. In the French one, it loses the key word “solde” (balance).
Checks before we trust the data
Section titled “Checks before we trust the data”The transcripts are machine-made. So we cannot measure a true word error rate. We can measure how often two recognisers disagree: the WER calculation, with the dataset’s text in the place of the reference, after the course’s normalisation guide. A disagreement means that at least one of the two is wrong.
| Variety | Whisper base disagrees on | Whisper small disagrees on |
|---|---|---|
| US English | 35.6% of the words | 35.0% |
| British English | 24.4% of the words | 22.6% |
| Australian English | 19.5% of the words | 18.0% |
| French | 45.7% of the words | 34.9% |
French has the most disagreement. On Larkfield’s synthetic calls, Whisper base had a WER of 5.2% in English and 23.8% in French. Real voices are harder, and French is still the hardest.
The intent is the truth. The labels come from the task each worker was given. Some labels overlap. “Freeze my card” is labelled “freeze”, and “my card is blocked” can be “card issues”. We keep the labels as they are.
There are no speaker IDs. One worker may have recorded several requests. The only group we can slice by is the variety. There is no gender or age.
A fair test
Section titled “A fair test”New input, kept apart
Section titled “New input, kept apart”First, we keep 20% of each variety apart, the same share of each intent: 471 requests. They are the new input for the end-to-end run, and we never use them to choose anything. The comparison uses the other 1,877.
A new tool: cross-validation
Section titled “A new tool: cross-validation”Think of a teacher with only one set of exam questions. If the class practises on all of them, the exam measures memory, not skill. A fair teacher hides a few questions each time.
We have only about 450 labelled requests per variety, so one test split would be small and noisy. Cross-validation uses every request for testing, but never with a model that saw it. We cut the requests into 5 parts. A model learns from 4 parts and scores the fifth. We repeat this 5 times, so each part is scored once.
Text description of the diagram
Five rounds, one per row. Each row has five parts. In round 1, the model learns from parts 2 to 5 and scores part 1. In round 2, it learns from parts 1, 3, 4 and 5 and scores part 2. After five rounds, every request has one score from a model that never saw it.
We keep each part’s mix of varieties and intents the same as the whole (“stratified”). The machine learning course explains it in depth in Train, validation and test sets.
The baseline
Section titled “The baseline”The simplest intent step always answers the most common intent. It is right for 7.6% of the requests, about 1 in 14. Today’s Larkfield-style baseline is also simple: every caller talks to a person. That routes nothing wrong and costs US$4,000 per 1,000 calls. Every design below must beat both.
Three methods for the intent step
Section titled “Three methods for the intent step”All three read the Whisper base transcript, as the Larkfield IVR does.
Method 1: keyword rules. As in the Larkfield IVR, each intent has a list of words in English and French, and the last intent the caller names wins. We wrote the lists once, from the intent names, before we saw any result. For example, “freeze” has freeze, frozen, suspend, block my account, geler, bloquer.
Method 2: a classifier. The TF-IDF and logistic regression classifier of Module 6, now with a probability for each intent. One model learns from the three English varieties together, another from French, with 5-fold cross-validation.
Method 3: an LLM. Azure chat-small reads the transcript (text only, never the audio). It returns JSON: one intent from the list, or “unclear”, and up to two alternatives. It has no examples, only the list with one line about each intent. Its answers were recorded once, so the notebook needs no key. We used two versions of the list (see below); the charts use the second.
Text description of the diagram
Four groups of three bars, one group per variety. Keyword rules: US English 73%, British English 74%, Australian English 71%, French 49%. Classifier: US English 98%, British English 97%, Australian English 96%, French 88%. LLM: US English 96%, British English 96%, Australian English 94%, French 90%. The baseline, the most common intent, is at 8%.
What each method gets wrong
Section titled “What each method gets wrong”Keyword rules (67.1%) fail in two ways. Many requests use none of the listed words: US English 15%, British English 17%, Australian English 17%, French 39%. Callers say “my card keeps getting declined”, “can I pay in some cash” or, in French, “montre-moi mes dernières opérations” with a recogniser error in the middle. And some requests name two intents, so “the last one wins” chooses the wrong one (see the failure case below).
The classifier (95.0%) learned the words that callers really use, including the recogniser’s usual mistakes. Its most common mistake is “card issues” for an “app error” request (7 times). Many app problems are about the card. The right intent is in its top three for 99.1% of the requests. That makes a short menu a strong safety net.
The LLM needs no labelled data, which is a real advantage on day one. But it knows only what the list tells it. With our first list, it got 85.4%. Its biggest mistake: 117 “freeze” requests went to “card issues”. Our list described “freeze” as the account and “card issues” as a card that is “lost, stolen, blocked”. The annotators’ “freeze” is mostly “freeze my card”. The model followed our words, not the labels.
So we rewrote five lines of the list to say what the labels mean, for example “freeze: freezing, blocking or stopping a card or the account”. We read only the mistakes on the comparison set. With the second list it got 94.2%, and in French 90.5%: better than the classifier there (88.2%). It still said “unclear” for 3.7% of the requests, mostly business loans and cash deposits. It costs about US$0.040 per 1,000 requests at the deployment’s price, and a median 1.09 s per answer: a wait that the caller hears.
Change one thing: a bigger recogniser. With Whisper small instead of base, the classifier gets 97.0% instead of 95.0%. The gain is mostly French: 93.0% against 88.2%. Whisper small disagrees with the dataset on 34.9% of the French words, against 45.7% for base. The price is time: on the laptop CPU, the median request took 0.93 s to transcribe with small and 0.3 s with base. In Let callers say what they need, tiny and base did equally well on short menu answers. Here the answers are open sentences, and the bigger model earns its place. Measure the size on the task.
A fair comparison under one measure
Section titled “A fair comparison under one measure”Accuracy does not say what happens to the caller. The measure we compare is the cost per 1,000 calls:
- a call that a person takes: US$4.00 (assumption);
- a call routed wrong: US$15 (assumption);
- compute: Whisper on a CPU server at US$0.17 an hour and the IVR’s own work at US$0.002 a call (assumptions); the LLM’s recorded price where it is used.
We also report containment: the share of calls routed right with no person. This is a proxy. It measures only the first question of the call, not the whole conversation of Test the IVR like a product.
For the menu design we need one more assumption, labelled: when the right intent is on the short menu, 9 callers in 10 press its key. The others press 0 or give up, and a person takes the call.
Choose the threshold by cost
Section titled “Choose the threshold by cost”The classifier gives a probability. Below a threshold, the IVR does not route. Which threshold is best? We tried every value from 0.10 to 0.90 and computed the cost.
Text description of the diagram
Without a menu, the cost is lowest at a threshold of 0.30: US$552 per 1,000 calls. A higher threshold sends too many right answers to a person, and a lower one routes too many calls wrong. With a short menu, the cost falls to US$156 at a threshold of 0.60, and stays below US$220 from 0.40 to 0.80. Everything to a person costs US$4,000.
With a safety net, the IVR can afford to be less bold. Without one, the best threshold is low, because a person is the only alternative to a guess.
The designs side by side
Section titled “The designs side by side”Text description of the diagram
Everything to a person: routed right 0%, to a person 100%, routed wrong 0%, US$4,000 per 1,000 calls. Keyword rules: routed right 67%, to a person 21%, routed wrong 11%, US$2,579 per 1,000 calls. Classifier, route every call: routed right 95%, to a person 0%, routed wrong 5%, US$745 per 1,000 calls. Classifier, sure, else a person: routed right 91%, to a person 7%, routed wrong 2%, US$552 per 1,000 calls. Classifier, sure, else a short menu: routed right 97%, to a person 3%, routed wrong 0%, US$156 per 1,000 calls. The same on Whisper small: routed right 97%, to a person 2%, routed wrong 0%, US$118 per 1,000 calls. LLM, route every answer: routed right 94%, to a person 4%, routed wrong 2%, US$469 per 1,000 calls. LLM, "unclear": a menu of its alternatives: routed right 95%, to a person 2%, routed wrong 2%, US$421 per 1,000 calls.| Design (Whisper base unless said) | Routed right | To a person | Routed wrong | Hear the menu | US$ per 1,000 calls |
|---|---|---|---|---|---|
| Everything to a person | 0% | 100% | 0% | 0% | US$4,000 |
| Keyword rules (no match: a person) | 67% | 21% | 11.5% | 0% | US$2,579 |
| LLM, first list, route every answer (“unclear”: a person) | 85% | 4% | 10.4% | 0% | US$1,737 |
| LLM, second list, route every answer (“unclear”: a person) | 94% | 4% | 2.1% | 0% | US$469 |
| LLM, second list, “unclear”: a menu of its alternatives | 95% | 2% | 2.1% | 2% | US$421 |
| Classifier, route every call | 95% | 0% | 5.0% | 0% | US$745 |
| Classifier, sure (≥ 0.30), else a person | 91% | 7% | 1.7% | 0% | US$552 |
| Classifier, sure (≥ 0.60), else a short menu | 97% | 3% | 0.2% | 22% | US$156 |
| The same on Whisper small (≥ 0.60) | 97% | 2% | 0.1% | 18% | US$118 |
What this tells us. Wrong routes drive the cost: one costs as much as almost four hand-overs. Keyword rules route about one call in nine to the wrong place. The classifier with a short menu almost never routes wrong, because it routes only when it is sure, and its top three almost always hold the right intent. The price is a longer call for 22% of the callers, who hear one extra question.
The LLM with the second list is almost as accurate as the classifier, but it costs more here. It says “unclear” only for a few requests and is sure of every other answer. So all its wrong routes (2.1%) reach callers. Our design has no probability to set a threshold on. A short list of words in a prompt also changed its accuracy by 9 points: a lesson about how fragile that setting is.
What about a menu alone? Fourteen intents do not fit in one keypad menu. A caller can hold only a few options in mind, so a keypad-only IVR needs two levels: “card and payments, press 1…”, then a second menu. Every caller hears two menus, and the menu’s words are not the caller’s words. We cannot measure that on this data (the recordings are speech, not key presses). On Larkfield’s synthetic calls, the keypad-only setting contained only 10% of calls, against 35% with speech (Let callers say what they need).
Each group, not only the average
Section titled “Each group, not only the average”| Variety | Rules, right | Classifier, right | LLM, right | Short menu, Whisper small: routed right | To a person | Hear the menu |
|---|---|---|---|---|---|---|
| US English | 73% | 98% | 96% | 99% | 1.4% | 10% |
| British English | 74% | 97% | 96% | 98% | 1.8% | 12% |
| Australian English | 71% | 96% | 94% | 98% | 2.1% | 13% |
| French | 49% | 88% | 90% | 95% | 4.9% | 40% |
The first three columns use Whisper base. French callers are served worst by every method. With the best design, 40% of them hear the extra menu (against 10% of US English callers), and 4.9% end with a person. With Whisper base it was worse: 49% and 7.0%.
Change one thing: is it the language, or the amount of data? The English model learned from three varieties, about three times as many requests as the French model. So we gave each variety its own model, with about the same number of requests:
| Variety, its own model | Requests to learn from (per round) | Accuracy |
|---|---|---|
| US English | about 360 | 96.7% |
| British English | about 378 | 96.0% |
| Australian English | about 418 | 95.4% |
| French | about 345 | 88.2% |
Each English variety alone still reaches about 95% or more; French stays near 88%. So the gap is not only data. The French transcripts are worse: Whisper base disagrees with the dataset on 45.7% of the French words.
Does a model travel between accents? Trained on two English varieties and tested on the third, it scored 97.1% (US English), 96.2% (British English), 96.4% (Australian English): nearly as well as with all three.
The recommendation
Section titled “The recommendation”For a bank’s common requests (or Larkfield’s next twenty), Priya recommends both: speech first, with a short keypad menu as the safety net.
- The welcome asks an open question: “Tell me in a few words what you need.” Keep the keypad: “0 for a person” works at every step.
- Whisper small writes the words of the open answer. It costs about 0.93 s of compute per request on a laptop CPU, against 0.3 s for base, and it helps French most. Keep Whisper base for the short menu answers, as the Larkfield IVR does.
- A TF-IDF classifier per language chooses the intent, trained on labelled requests from your own line.
- Route when its top probability is at least 0.60. Below that, play a short menu with its top three intents and 0 for a person.
- Keep the LLM out of the routing step for now. It was better in French, so test it there as a second opinion when the classifier is unsure. Use it also to suggest labels for new requests, which a person then checks.
What a person still checks:
- Every week, a sample of routed calls, by language: the wrong routes are rare but cost the most.
- The French line first. 40% of the French callers hear the extra menu. Listen to those calls, and add labelled French requests.
- New intents. When callers ask for something new, the classifier forces it into an old intent. Read the calls that go to a person.
Cost per 1,000 calls: US$118 on the comparison set, against US$4,000 when a person answers every call (the course’s assumptions). Compute is a small part: Whisper small and the IVR cost about US$2.05 per 1,000 calls. Labelling is the hidden cost: about 450 labelled requests per language before you start.
End to end on new input
Section titled “End to end on new input”The 471 requests kept apart were never used to choose anything. We trained the classifiers on all comparison requests, then ran the recommended design once.
Text description of the diagram
US English, 113 requests: routed right 97.1%, to a person 2.0%, routed wrong 0.9%. British English, 119 requests: routed right 97.4%, to a person 2.6%, routed wrong 0.0%. Australian English, 131 requests: routed right 99.0%, to a person 1.0%, routed wrong 0.0%. French, 108 requests: routed right 93.6%, to a person 5.5%, routed wrong 0.9%.All together: routed right 96.9%, routed wrong 0.4%, a person 2.7%, US$173 per 1,000 calls. That is close to the comparison (US$118), so the choice holds. French again has the most hand-overs and hears the menu most. With about 110 requests per variety, one wrong route moves a variety’s cost by about US$140 per 1,000 calls, so small differences between the English varieties are noise.
The notebook then runs the whole chain live on 12 of these recordings: the audio, Whisper small on the laptop, the classifier, then “route” or “menu”. Five lines of its output (real output; the intent in brackets is the label):
en-GB-316 [balance] 'What is my current account, Holland?' -> balance (0.92)en-GB-259 [joint_account] 'I would like to start a joint succumb.' -> menu: joint_account, pay_bill, direct_debit (0.44)en-AU-553 [direct_debit] 'Hello, I want to return a direct visit.' -> direct_debit (0.79)fr-FR-256 [balance] 'Montre-moi le sol de mon compte.' -> balance (0.99)fr-FR-219 [freeze] 'je souhaite bloquer ma carte' -> freeze (0.95)Whisper small still turns “balance” into a name (“Holland”) and “account” into “succumb”. The classifier routes the first anyway: “current account” is enough. It is not sure about the second (0.44), so the caller hears a short menu, and the right intent is first on it. The French request that Whisper base lost is now right (“le sol” for “le solde”). A model that learned from the recogniser’s real output forgives its usual mistakes.
Insights and limits
Section titled “Insights and limits”- Real voices are harder than Larkfield’s synthetic calls. There, the call-reason test was too easy (rules 90%, classifier 96.7%, LLM 100%). Here the rules fall to 67%.
- A safety net changes the best threshold, and wrong routes, not hand-overs, drive the cost.
- French is the worst group for every method, and not only because of less data.
Limits. Each person knew the intent and saw three examples, so the requests are tidier than real callers’ requests. There is no follow-up turn. The transcripts are machine-made, so we report disagreement, not WER. Without speaker IDs, one person’s requests can be on both sides of a split, so the classifier’s numbers may be a little high. The costs are assumptions, and “9 in 10 callers press the right key” is ours. We did not measure a keypad-only menu or real callers’ patience.
Common mistake
The last word wins, and loses
Symptom: the keyword rules sent 45 “freeze” requests to “latest transactions”. For example: “Could you please freeze my card and make sure no transactions happen from now on?”
Cause: the Larkfield rule keeps the last intent the caller names, because Larkfield callers correct themselves at the end (“return… no, track my order”). These callers do the opposite. They name the intent first and then explain, and the explanation has another intent’s word.
Fix: do not move a rule from one line to another without testing it on that line’s callers. A classifier learns which words matter from labelled requests, and handled most of these (96.9% of “freeze” right).
Consequence in AI work: a rule is a guess about how people talk. Each group of callers needs its own test.
Responsible use
Section titled “Responsible use”- Real people’s voices. We play only short clips, each with its transcripts, and use the dataset’s IDs. We never try to find out who spoke. No audio went to a hosted model: the LLM read Whisper’s text.
- Tell callers they talk to a machine, and keep “0 for a person” at every step, as in Voice cloning, speaker checks and misuse.
- Do not punish an accent. If one group hears the extra menu much more often, that group waits longer for the same service. Measure it by language and by line, and fix the worst group first.
- Banking requests are personal data. In a real bank, the transcripts of “balance” and “freeze” calls need the same rules as the audio: retention, deletion and access.
- Labels from your own calls need consent. Training on real requests means recording them, with a notice and a reason.
Make it a portfolio piece
Section titled “Make it a portfolio piece”Put the case study in a public repository:
- README: the problem, the data with its licence and attribution, the three methods, the cost table with the baseline, the recommendation, and the worst group.
- The notebook, run from the top, with the package versions.
- A short design note: the IVR’s welcome, the menu wording, the threshold, and what a person checks each week.
In an interview, say it in two sentences: “I compared keyword rules, a TF-IDF classifier and an LLM as an IVR’s intent step on MInDS-14, real spoken banking requests in four varieties. A classifier that routes only when it is sure and otherwise offers its top three intents cut the assumed cost from US$4,000 to US$118 per 1,000 calls. French callers were served worst by every method.”
Go further: add more varieties of MInDS-14, try a sentence-embedding model as the paper’s authors did, or write a two-level keypad menu and test it with five friends.
Knowledge check
Section titled “Knowledge check”Check your understanding
The classifier chooses the right intent for 95% of the calls. Why not route every call?
Show the answer
Because the 5% of wrong routes cost the most. At the course’s assumptions, a wrong route costs US$15 and a hand-over US$4.00. Routing every call costs US$745 per 1,000 calls; routing only when sure, with a short menu for the rest, costs US$156. The right intent is in the top three for 99.1% of the requests, so the menu catches most of the unsure calls.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
French callers get the extra menu far more often. Tomás says: 'Just add more French data.' Is that enough?
Show the answer
Probably not. We gave each variety its own model with about the same amount of data. Every English variety still scored about 95% or more, and French about 88%. The French transcripts are worse. More labelled French requests help, but so may a better French recogniser. Measure both changes on the French line.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why can this case study not report a word error rate for Whisper?
Show the answer
Because no person wrote down what the callers said. The dataset’s transcripts come from Google’s recogniser. Comparing Whisper with them gives a disagreement rate: at least one of the two is wrong, but we cannot say which. The intent labels, which come from the task each person was given, are the truth we measure against.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On real voices, the intent step is much harder than on Larkfield’s synthetic calls: keyword rules fell to 67%.
- Cross-validation scores every request with a model that never saw it: the right tool for small labelled sets.
- Compare designs by cost, not accuracy: a wrong route costs more than a hand-over.
- Speech first, with a short keypad menu as the safety net, gave the lowest cost: US$118 per 1,000 calls with Whisper small, against US$4,000.
- French callers were served worst by every method; check every group, and fix the worst first.
Next, the final assignment: Larkfield’s new French-language line, with everything you built in this course.
Next: Final assignment