Skip to content
Français

An IVR for spoken intents

  • Case study 4
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we built and tested the Larkfield IVR. In this case study, we build a menu-and-speech IVR for a bank's common requests, test it with real spoken-intent recordings, and compare three ways to find the intent.

Case study 4 of 4 · Speech AI: Transcribe, Understand and SpeakAn IVR for spoken intents

What you will be able to do

Choose the intent step for a phone line that understands spoken requests. You will test keyword rules, a classifier and an LLM on real people’s voices in four language varieties. You will turn accuracy into calls routed without a person and into money. Then you will choose a design (menu, speech or both) and name the callers it serves worst.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Keyword rules
  • TF-IDF classifier with cross-validation
  • Recorded LLM intents
  • Containment and cost per 1,000 calls
  • Thresholds and a fallback menu
  • Results by speaker group
  • End-to-end run on new input

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs the steps of this page with the same code. It downloads a few megabytes: the text and labels of every request, the recorded transcripts and LLM answers, and 12 short recordings. At the end, it runs Whisper base live on those 12 recordings. It needs no account and no key.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the understanding and IVR parts of the course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • A virtual environment with scikit-learn and faster-whisper, as in the project. The larkspeech project works; this case study does not change its files.
  • The notebook, downloaded from the link above. Its setup cell makes a new folder minds-ivr. To start again, delete that folder.

Tested on macOS with Python 3.12. Windows, Linux, Colab and Kaggle were not tested.

Larkfield’s IVR understands four intents: order status, opening hours, a return, and “a person”. Omar, the product manager, wants more. Callers ask about invoices, payment cards, account changes and much more. “Can the IVR route twenty requests by speech? Or do we need a long keypad menu? Or both?”

Priya sees two problems. Every Larkfield call is synthetic, and real people speak in their own words. And Larkfield has no labelled recordings of real requests yet. So she tests on public data first: real voices, many intents, phone audio. The closest open set is about banking, not garden products, but the question is the same: which intent step, with which safety net, for which callers?

Two mistakes matter, and they cost different amounts:

  • A wrong route. The caller lands in the wrong queue, waits, explains again and is transferred. The course’s assumption is US$15 for a wrong action.
  • A hand-over that was not needed. A person answers a call that the IVR could have routed. The course’s assumption is US$4.00 a call (6 minutes at US$40 an hour).

You will build the intent step of an IVR and its safety net, then compare three methods on the same requests with the same measure.

A flow from left to right. The caller says 'I'd like to freeze my card'. Whisper writes the words. The intent step (keyword rules, a classifier or an LLM) gives an intent. If it is sure enough, the call is routed. If not, a short keypad menu offers three intents and 0 for a person. If the right intent is on the menu, the call is routed; if not, a person takes it.
The intent step and its safety net. 'Speech only' sends every unsure call to a person; 'both' first offers a short keypad menu.
Text description of the diagram

A caller says a request. Whisper writes the words: base, the recogniser of the Larkfield IVR, or small. The intent step chooses one of 14 intents. If it is sure enough, the IVR routes the call. If it is not sure, there are two designs. Speech only gives the call to a person. Both (speech and a menu) first plays a short keypad menu with the three most likely intents: “Press 1 for a lost or blocked card, 2 for …, 3 for …, or 0 for a person.” If the caller finds their intent there, the call is routed. If not, a person takes it.

The data is MInDS-14 (Multilingual Intent Detection from Speech), published by PolyAI. Crowd workers recorded spoken requests to a bank’s phone line, with 14 intents, in 14 language varieties. Each worker got an intent, a short description and three examples, then said a new request in their own words. Only native speakers of each variety took part. Some Australian requests went through a real phone voice system; the others were recorded online. All are 8 kHz phone audio, like Larkfield’s phone calls.

We use four varieties, 2,348 requests and 5.47 hours in all:

Variety Requests Hours Median length
US English (en-US) 563 1.34 6.4 s
British English (en-GB) 592 1.35 6.3 s
Australian English (en-AU) 654 1.53 6.6 s
French (fr-FR) 539 1.25 6.6 s

The 14 intents are: abroad (the card in another country), address, app error, ATM limit, balance, business loan, card issues, cash deposit, direct debit, freeze, high-value payment, joint account, latest transactions and pay a bill. Each variety has about 30 to 50 requests per intent.

Four requests, one per variety. Each is short. Under each player are two transcripts, because no person wrote down these words. The dataset’s transcript was made by Google’s recogniser (the paper says so). The second is Whisper base’s, recorded for this case study.

en-US-151, intent “freeze”, 3.9 s. A real person, from MInDS-14 (CC BY 4.0). Dataset transcript: “I’d like to freeze my card”. Whisper base: “I’d like to freeze my card.”
en-GB-316, intent “balance”, 4.3 s. A real person, from MInDS-14 (CC BY 4.0). Dataset transcript: “what is my current account balance”. Whisper base: “What is my current account, Helen?”
en-AU-553, intent “direct debit”, 3.5 s. A real person, from MInDS-14 (CC BY 4.0). Dataset transcript: “hello I want to return a direct debit”. Whisper base: “Hello, I want to return a direct delete.”
fr-FR-256, intent “balance”, 4.2 s. A real person, from MInDS-14 (CC BY 4.0). Dataset transcript: “montre-moi le solde de mon compte” (“show me my account balance”). Whisper base: “Montre-en-warre est sur le démoncompte.”

The two recognisers agree on the first request. On the British one, Whisper base turns “balance” into a name. In the French one, it loses the key word “solde” (balance).

The transcripts are machine-made. So we cannot measure a true word error rate. We can measure how often two recognisers disagree: the WER calculation, with the dataset’s text in the place of the reference, after the course’s normalisation guide. A disagreement means that at least one of the two is wrong.

Variety Whisper base disagrees on Whisper small disagrees on
US English 35.6% of the words 35.0%
British English 24.4% of the words 22.6%
Australian English 19.5% of the words 18.0%
French 45.7% of the words 34.9%

French has the most disagreement. On Larkfield’s synthetic calls, Whisper base had a WER of 5.2% in English and 23.8% in French. Real voices are harder, and French is still the hardest.

The intent is the truth. The labels come from the task each worker was given. Some labels overlap. “Freeze my card” is labelled “freeze”, and “my card is blocked” can be “card issues”. We keep the labels as they are.

There are no speaker IDs. One worker may have recorded several requests. The only group we can slice by is the variety. There is no gender or age.

First, we keep 20% of each variety apart, the same share of each intent: 471 requests. They are the new input for the end-to-end run, and we never use them to choose anything. The comparison uses the other 1,877.

Think of a teacher with only one set of exam questions. If the class practises on all of them, the exam measures memory, not skill. A fair teacher hides a few questions each time.

We have only about 450 labelled requests per variety, so one test split would be small and noisy. Cross-validation uses every request for testing, but never with a model that saw it. We cut the requests into 5 parts. A model learns from 4 parts and scores the fifth. We repeat this 5 times, so each part is scored once.

Five rows of five boxes. In row 1 the first box is orange (score) and the others are light (learn). In row 2 the second box is orange, and so on to row 5.
5-fold cross-validation: five rounds, each part scored once by a model that learned from the other four.
Text description of the diagram

Five rounds, one per row. Each row has five parts. In round 1, the model learns from parts 2 to 5 and scores part 1. In round 2, it learns from parts 1, 3, 4 and 5 and scores part 2. After five rounds, every request has one score from a model that never saw it.

We keep each part’s mix of varieties and intents the same as the whole (“stratified”). The machine learning course explains it in depth in Train, validation and test sets.

The simplest intent step always answers the most common intent. It is right for 7.6% of the requests, about 1 in 14. Today’s Larkfield-style baseline is also simple: every caller talks to a person. That routes nothing wrong and costs US$4,000 per 1,000 calls. Every design below must beat both.

All three read the Whisper base transcript, as the Larkfield IVR does.

Method 1: keyword rules. As in the Larkfield IVR, each intent has a list of words in English and French, and the last intent the caller names wins. We wrote the lists once, from the intent names, before we saw any result. For example, “freeze” has freeze, frozen, suspend, block my account, geler, bloquer.

Method 2: a classifier. The TF-IDF and logistic regression classifier of Module 6, now with a probability for each intent. One model learns from the three English varieties together, another from French, with 5-fold cross-validation.

Method 3: an LLM. Azure chat-small reads the transcript (text only, never the audio). It returns JSON: one intent from the list, or “unclear”, and up to two alternatives. It has no examples, only the list with one line about each intent. Its answers were recorded once, so the notebook needs no key. We used two versions of the list (see below); the charts use the second.

Grouped horizontal bars of accuracy for each variety and method. The classifier and the LLM are close, above 90% in English. Keyword rules are much lower, and lowest in French. In French the LLM is a little above the classifier. A dashed line near 8% marks the most common intent.
Intent right, by variety and method (Whisper base transcripts; the classifier with 5-fold cross-validation; the LLM with the second list). French is the worst variety for every method.
Text description of the diagram

Four groups of three bars, one group per variety. Keyword rules: US English 73%, British English 74%, Australian English 71%, French 49%. Classifier: US English 98%, British English 97%, Australian English 96%, French 88%. LLM: US English 96%, British English 96%, Australian English 94%, French 90%. The baseline, the most common intent, is at 8%.

Keyword rules (67.1%) fail in two ways. Many requests use none of the listed words: US English 15%, British English 17%, Australian English 17%, French 39%. Callers say “my card keeps getting declined”, “can I pay in some cash” or, in French, “montre-moi mes dernières opérations” with a recogniser error in the middle. And some requests name two intents, so “the last one wins” chooses the wrong one (see the failure case below).

The classifier (95.0%) learned the words that callers really use, including the recogniser’s usual mistakes. Its most common mistake is “card issues” for an “app error” request (7 times). Many app problems are about the card. The right intent is in its top three for 99.1% of the requests. That makes a short menu a strong safety net.

The LLM needs no labelled data, which is a real advantage on day one. But it knows only what the list tells it. With our first list, it got 85.4%. Its biggest mistake: 117 “freeze” requests went to “card issues”. Our list described “freeze” as the account and “card issues” as a card that is “lost, stolen, blocked”. The annotators’ “freeze” is mostly “freeze my card”. The model followed our words, not the labels.

So we rewrote five lines of the list to say what the labels mean, for example “freeze: freezing, blocking or stopping a card or the account”. We read only the mistakes on the comparison set. With the second list it got 94.2%, and in French 90.5%: better than the classifier there (88.2%). It still said “unclear” for 3.7% of the requests, mostly business loans and cash deposits. It costs about US$0.040 per 1,000 requests at the deployment’s price, and a median 1.09 s per answer: a wait that the caller hears.

Change one thing: a bigger recogniser. With Whisper small instead of base, the classifier gets 97.0% instead of 95.0%. The gain is mostly French: 93.0% against 88.2%. Whisper small disagrees with the dataset on 34.9% of the French words, against 45.7% for base. The price is time: on the laptop CPU, the median request took 0.93 s to transcribe with small and 0.3 s with base. In Let callers say what they need, tiny and base did equally well on short menu answers. Here the answers are open sentences, and the bigger model earns its place. Measure the size on the task.

Accuracy does not say what happens to the caller. The measure we compare is the cost per 1,000 calls:

  • a call that a person takes: US$4.00 (assumption);
  • a call routed wrong: US$15 (assumption);
  • compute: Whisper on a CPU server at US$0.17 an hour and the IVR’s own work at US$0.002 a call (assumptions); the LLM’s recorded price where it is used.

We also report containment: the share of calls routed right with no person. This is a proxy. It measures only the first question of the call, not the whole conversation of Test the IVR like a product.

For the menu design we need one more assumption, labelled: when the right intent is on the short menu, 9 callers in 10 press its key. The others press 0 or give up, and a person takes the call.

The classifier gives a probability. Below a threshold, the IVR does not route. Which threshold is best? We tried every value from 0.10 to 0.90 and computed the cost.

Two lines of US$ per 1,000 calls against the threshold. 'Else a person' has a U shape with its lowest point at a low threshold. 'Else a 3-item menu' falls fast and stays low over a wide range. A dashed line at US$4,000 marks everything to a person.
Cost per 1,000 calls for each threshold (classifier on Whisper base; the course's cost assumptions). With a short menu as the safety net, the cost is low over a wide range.
Text description of the diagram

Without a menu, the cost is lowest at a threshold of 0.30: US$552 per 1,000 calls. A higher threshold sends too many right answers to a person, and a lower one routes too many calls wrong. With a short menu, the cost falls to US$156 at a threshold of 0.60, and stays below US$220 from 0.40 to 0.80. Everything to a person costs US$4,000.

With a safety net, the IVR can afford to be less bold. Without one, the best threshold is low, because a person is the only alternative to a guess.

Eight stacked bars of routed right, to a person and routed wrong, with the cost per 1,000 calls at the right. Everything to a person: all grey, US$4,000. Keyword rules: about two thirds routed right and a red part. The classifier with a short menu has the longest green part and the lowest cost, lowest of all on Whisper small.
What happens to the calls under each design (comparison set, Whisper base unless said, cross-validation). At the right: US$ per 1,000 calls, with the course's cost assumptions.
Text description of the diagramEverything to a person: routed right 0%, to a person 100%, routed wrong 0%, US$4,000 per 1,000 calls. Keyword rules: routed right 67%, to a person 21%, routed wrong 11%, US$2,579 per 1,000 calls. Classifier, route every call: routed right 95%, to a person 0%, routed wrong 5%, US$745 per 1,000 calls. Classifier, sure, else a person: routed right 91%, to a person 7%, routed wrong 2%, US$552 per 1,000 calls. Classifier, sure, else a short menu: routed right 97%, to a person 3%, routed wrong 0%, US$156 per 1,000 calls. The same on Whisper small: routed right 97%, to a person 2%, routed wrong 0%, US$118 per 1,000 calls. LLM, route every answer: routed right 94%, to a person 4%, routed wrong 2%, US$469 per 1,000 calls. LLM, "unclear": a menu of its alternatives: routed right 95%, to a person 2%, routed wrong 2%, US$421 per 1,000 calls.
Design (Whisper base unless said) Routed right To a person Routed wrong Hear the menu US$ per 1,000 calls
Everything to a person 0% 100% 0% 0% US$4,000
Keyword rules (no match: a person) 67% 21% 11.5% 0% US$2,579
LLM, first list, route every answer (“unclear”: a person) 85% 4% 10.4% 0% US$1,737
LLM, second list, route every answer (“unclear”: a person) 94% 4% 2.1% 0% US$469
LLM, second list, “unclear”: a menu of its alternatives 95% 2% 2.1% 2% US$421
Classifier, route every call 95% 0% 5.0% 0% US$745
Classifier, sure (≥ 0.30), else a person 91% 7% 1.7% 0% US$552
Classifier, sure (≥ 0.60), else a short menu 97% 3% 0.2% 22% US$156
The same on Whisper small (≥ 0.60) 97% 2% 0.1% 18% US$118

What this tells us. Wrong routes drive the cost: one costs as much as almost four hand-overs. Keyword rules route about one call in nine to the wrong place. The classifier with a short menu almost never routes wrong, because it routes only when it is sure, and its top three almost always hold the right intent. The price is a longer call for 22% of the callers, who hear one extra question.

The LLM with the second list is almost as accurate as the classifier, but it costs more here. It says “unclear” only for a few requests and is sure of every other answer. So all its wrong routes (2.1%) reach callers. Our design has no probability to set a threshold on. A short list of words in a prompt also changed its accuracy by 9 points: a lesson about how fragile that setting is.

What about a menu alone? Fourteen intents do not fit in one keypad menu. A caller can hold only a few options in mind, so a keypad-only IVR needs two levels: “card and payments, press 1…”, then a second menu. Every caller hears two menus, and the menu’s words are not the caller’s words. We cannot measure that on this data (the recordings are speech, not key presses). On Larkfield’s synthetic calls, the keypad-only setting contained only 10% of calls, against 35% with speech (Let callers say what they need).

Variety Rules, right Classifier, right LLM, right Short menu, Whisper small: routed right To a person Hear the menu
US English 73% 98% 96% 99% 1.4% 10%
British English 74% 97% 96% 98% 1.8% 12%
Australian English 71% 96% 94% 98% 2.1% 13%
French 49% 88% 90% 95% 4.9% 40%

The first three columns use Whisper base. French callers are served worst by every method. With the best design, 40% of them hear the extra menu (against 10% of US English callers), and 4.9% end with a person. With Whisper base it was worse: 49% and 7.0%.

Change one thing: is it the language, or the amount of data? The English model learned from three varieties, about three times as many requests as the French model. So we gave each variety its own model, with about the same number of requests:

Variety, its own model Requests to learn from (per round) Accuracy
US English about 360 96.7%
British English about 378 96.0%
Australian English about 418 95.4%
French about 345 88.2%

Each English variety alone still reaches about 95% or more; French stays near 88%. So the gap is not only data. The French transcripts are worse: Whisper base disagrees with the dataset on 45.7% of the French words.

Does a model travel between accents? Trained on two English varieties and tested on the third, it scored 97.1% (US English), 96.2% (British English), 96.4% (Australian English): nearly as well as with all three.

For a bank’s common requests (or Larkfield’s next twenty), Priya recommends both: speech first, with a short keypad menu as the safety net.

  1. The welcome asks an open question: “Tell me in a few words what you need.” Keep the keypad: “0 for a person” works at every step.
  2. Whisper small writes the words of the open answer. It costs about 0.93 s of compute per request on a laptop CPU, against 0.3 s for base, and it helps French most. Keep Whisper base for the short menu answers, as the Larkfield IVR does.
  3. A TF-IDF classifier per language chooses the intent, trained on labelled requests from your own line.
  4. Route when its top probability is at least 0.60. Below that, play a short menu with its top three intents and 0 for a person.
  5. Keep the LLM out of the routing step for now. It was better in French, so test it there as a second opinion when the classifier is unsure. Use it also to suggest labels for new requests, which a person then checks.

What a person still checks:

  • Every week, a sample of routed calls, by language: the wrong routes are rare but cost the most.
  • The French line first. 40% of the French callers hear the extra menu. Listen to those calls, and add labelled French requests.
  • New intents. When callers ask for something new, the classifier forces it into an old intent. Read the calls that go to a person.

Cost per 1,000 calls: US$118 on the comparison set, against US$4,000 when a person answers every call (the course’s assumptions). Compute is a small part: Whisper small and the IVR cost about US$2.05 per 1,000 calls. Labelling is the hidden cost: about 450 labelled requests per language before you start.

The 471 requests kept apart were never used to choose anything. We trained the classifiers on all comparison requests, then ran the recommended design once.

Four stacked bars, one per variety, of routed right, to a person and routed wrong. All four are almost entirely green. French has the largest grey part.
The recommended design on the new input (Whisper small; route at the chosen threshold, else a short menu).
Text description of the diagramUS English, 113 requests: routed right 97.1%, to a person 2.0%, routed wrong 0.9%. British English, 119 requests: routed right 97.4%, to a person 2.6%, routed wrong 0.0%. Australian English, 131 requests: routed right 99.0%, to a person 1.0%, routed wrong 0.0%. French, 108 requests: routed right 93.6%, to a person 5.5%, routed wrong 0.9%.

All together: routed right 96.9%, routed wrong 0.4%, a person 2.7%, US$173 per 1,000 calls. That is close to the comparison (US$118), so the choice holds. French again has the most hand-overs and hears the menu most. With about 110 requests per variety, one wrong route moves a variety’s cost by about US$140 per 1,000 calls, so small differences between the English varieties are noise.

The notebook then runs the whole chain live on 12 of these recordings: the audio, Whisper small on the laptop, the classifier, then “route” or “menu”. Five lines of its output (real output; the intent in brackets is the label):

en-GB-316 [balance] 'What is my current account, Holland?' -> balance (0.92)
en-GB-259 [joint_account] 'I would like to start a joint succumb.' -> menu: joint_account, pay_bill, direct_debit (0.44)
en-AU-553 [direct_debit] 'Hello, I want to return a direct visit.' -> direct_debit (0.79)
fr-FR-256 [balance] 'Montre-moi le sol de mon compte.' -> balance (0.99)
fr-FR-219 [freeze] 'je souhaite bloquer ma carte' -> freeze (0.95)

Whisper small still turns “balance” into a name (“Holland”) and “account” into “succumb”. The classifier routes the first anyway: “current account” is enough. It is not sure about the second (0.44), so the caller hears a short menu, and the right intent is first on it. The French request that Whisper base lost is now right (“le sol” for “le solde”). A model that learned from the recogniser’s real output forgives its usual mistakes.

  • Real voices are harder than Larkfield’s synthetic calls. There, the call-reason test was too easy (rules 90%, classifier 96.7%, LLM 100%). Here the rules fall to 67%.
  • A safety net changes the best threshold, and wrong routes, not hand-overs, drive the cost.
  • French is the worst group for every method, and not only because of less data.

Limits. Each person knew the intent and saw three examples, so the requests are tidier than real callers’ requests. There is no follow-up turn. The transcripts are machine-made, so we report disagreement, not WER. Without speaker IDs, one person’s requests can be on both sides of a split, so the classifier’s numbers may be a little high. The costs are assumptions, and “9 in 10 callers press the right key” is ours. We did not measure a keypad-only menu or real callers’ patience.

Common mistake

The last word wins, and loses

Symptom: the keyword rules sent 45 “freeze” requests to “latest transactions”. For example: “Could you please freeze my card and make sure no transactions happen from now on?”

Cause: the Larkfield rule keeps the last intent the caller names, because Larkfield callers correct themselves at the end (“return… no, track my order”). These callers do the opposite. They name the intent first and then explain, and the explanation has another intent’s word.

Fix: do not move a rule from one line to another without testing it on that line’s callers. A classifier learns which words matter from labelled requests, and handled most of these (96.9% of “freeze” right).

Consequence in AI work: a rule is a guess about how people talk. Each group of callers needs its own test.

  • Real people’s voices. We play only short clips, each with its transcripts, and use the dataset’s IDs. We never try to find out who spoke. No audio went to a hosted model: the LLM read Whisper’s text.
  • Tell callers they talk to a machine, and keep “0 for a person” at every step, as in Voice cloning, speaker checks and misuse.
  • Do not punish an accent. If one group hears the extra menu much more often, that group waits longer for the same service. Measure it by language and by line, and fix the worst group first.
  • Banking requests are personal data. In a real bank, the transcripts of “balance” and “freeze” calls need the same rules as the audio: retention, deletion and access.
  • Labels from your own calls need consent. Training on real requests means recording them, with a notice and a reason.

Put the case study in a public repository:

  • README: the problem, the data with its licence and attribution, the three methods, the cost table with the baseline, the recommendation, and the worst group.
  • The notebook, run from the top, with the package versions.
  • A short design note: the IVR’s welcome, the menu wording, the threshold, and what a person checks each week.

In an interview, say it in two sentences: “I compared keyword rules, a TF-IDF classifier and an LLM as an IVR’s intent step on MInDS-14, real spoken banking requests in four varieties. A classifier that routes only when it is sure and otherwise offers its top three intents cut the assumed cost from US$4,000 to US$118 per 1,000 calls. French callers were served worst by every method.”

Go further: add more varieties of MInDS-14, try a sentence-embedding model as the paper’s authors did, or write a two-level keypad menu and test it with five friends.

Check your understanding

The classifier chooses the right intent for 95% of the calls. Why not route every call?

Show the answer

Because the 5% of wrong routes cost the most. At the course’s assumptions, a wrong route costs US$15 and a hand-over US$4.00. Routing every call costs US$745 per 1,000 calls; routing only when sure, with a short menu for the rest, costs US$156. The right intent is in the top three for 99.1% of the requests, so the menu catches most of the unsure calls.

Check your understanding

French callers get the extra menu far more often. Tomás says: 'Just add more French data.' Is that enough?

Show the answer

Probably not. We gave each variety its own model with about the same amount of data. Every English variety still scored about 95% or more, and French about 88%. The French transcripts are worse. More labelled French requests help, but so may a better French recogniser. Measure both changes on the French line.

Check your understanding

Why can this case study not report a word error rate for Whisper?

Show the answer

Because no person wrote down what the callers said. The dataset’s transcripts come from Google’s recogniser. Comparing Whisper with them gives a disagreement rate: at least one of the two is wrong, but we cannot say which. The intent labels, which come from the task each person was given, are the truth we measure against.

Recap

  • On real voices, the intent step is much harder than on Larkfield’s synthetic calls: keyword rules fell to 67%.
  • Cross-validation scores every request with a model that never saw it: the right tool for small labelled sets.
  • Compare designs by cost, not accuracy: a wrong route costs more than a hand-over.
  • Speech first, with a short keypad menu as the safety net, gave the lowest cost: US$118 per 1,000 calls with Whisper small, against US$4,000.
  • French callers were served worst by every method; check every group, and fix the worst first.

Next, the final assignment: Larkfield’s new French-language line, with everything you built in this course.

Next: Final assignment