Let callers say what they need
- Lesson 25
- 40 min
- Module 7: Build a small IVR
Where we are
Our keypad menu works, but callers must listen to every option. In this lesson, we let callers say what they need: VAD finds the end of their turn, the recogniser writes their words, and rules or a classifier find the intent.
Lesson 25 of 36 · Speech AI: Transcribe, Understand and SpeakLet callers say what they need
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Let callers of the Larkfield IVR say what they need in their own words, and keep the keypad as the way out when speech fails.
NotebookDo this lesson in a notebook
The notebook runs the IVR’s keyword rules on the recorded menu answers of the test calls, and finds the answers they miss. Then you make the rules better without breaking the answers that worked. It downloads only small text files and runs no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Voice activity detection (VAD) with Silero, from WebRTC VAD and Silero VAD.
- How an initial prompt changes what Whisper writes, from Help the model with your words.
- Keyword rules and a classifier for the call reason, from Why did they call?.
- The keypad menu and its misses, from A keypad menu first.
What you need
- Your
larkspeechfolder at the end-of-Module-7 snapshot, from the last lesson. - Whisper base and tiny (Module 4 downloaded them).
- Tested on macOS (Apple M4) with Python 3.12. The Windows commands are the same in PowerShell, but were not run.
A receptionist, not a form
Section titled “A receptionist, not a form”Call a small hotel. The receptionist says: “Good evening, how can I help?” You say: “I’d like to know if you have a room on Friday.” She does not read you a list of eight options. She listens until you stop, understands what you want, and answers.
To do that, she does three things without thinking. She waits until you have finished your sentence. She hears your words. And she understands what you want: a booking, not the restaurant. The IVR must do the same three things, one after the other.
The problem: a keypad menu loses most callers
Section titled “The problem: a keypad menu loses most callers”In the last lesson, the IVR ignored every spoken word and used only the keypad. On the 60 recorded test calls, it finished 10% of the calls without a person. Most test callers answered “What do you need?” with words, as many people do. A keypad-only menu passed them all to a person.
Speech is harder than keys. A key is never misheard. Words are misheard, cut off, or said in another language. So the IVR adds speech in three steps, and it keeps the keypad for when speech fails.
Text description of the diagram
Each 20 ms frame of caller audio is first checked for a keypad tone. Keys go straight to the dialogue. Otherwise, the IVR keeps the audio, and every 200 ms Silero VAD looks for the speech in it. When 0.8 seconds of silence follow the speech, the caller’s turn has ended. Whisper base writes the words, with a short prompt for the current state. Then the intent rules turn the words into an intent, and the dialogue chooses the next prompt.
Step 1: when has the caller finished?
Section titled “Step 1: when has the caller finished?”The IVR cannot wait for the caller to hang up. It must decide when the caller’s turn has ended, then answer. This is called end-of-turn detection, or endpointing.
The simplest rule works well for short answers: when the caller has spoken, then been silent for a fixed time, the turn has ended. The Larkfield IVR waits for 0.8 seconds of silence. To find speech and silence, it uses Silero VAD, the voice activity detector from Module 3. Every 200 ms, the engine runs it on all the audio of the turn so far, and looks at where the last speech ended.
A tiny worked example: “Where is my order?”
Section titled “A tiny worked example: “Where is my order?””Here is the caller’s answer to the menu in call I-009:
Transcript: “Where is my order?”
The engine checks every 200 ms. These are its last checks on this turn, in seconds after the menu prompt ended:
| Check at | Speech found from | to | Silence since the speech ended | Turn over? |
|---|---|---|---|---|
| 2.4 s | 0.86 s | 1.82 s | 0.58 s | no |
| 2.6 s | 0.86 s | 1.82 s | 0.78 s | no (under 0.8 s) |
| 2.8 s | 0.86 s | 1.82 s | 0.98 s | yes |
Text description of the diagram
The caller’s voice starts at about 0.7 seconds. Silero VAD finds speech from 0.86 to 1.82 seconds. At 2.6 seconds, the silence after the speech is 0.78 seconds: not enough. At the next check, 2.8 seconds, it is 0.98 seconds, so the turn ends and the audio goes to Whisper.
Two details matter here. First, the turn ends at the first check after 0.8 seconds of silence, so the caller waits 0.8 to 1.0 seconds. Second, Silero found the speech only from 0.86 seconds, although the voice starts a little earlier. VAD often misses the quiet start of a word. So the engine sends Whisper the audio from 0.2 seconds before the speech to 0.25 seconds after it.
How long to wait
Section titled “How long to wait”A shorter wait makes the IVR feel faster. A longer wait lets callers pause in the middle of a sentence, or between the groups of an order number. The course measured three values on the 60 test calls:
Text description of the diagram
With speech, the IVR contains 33% to 35% of the calls with every setting. The settings are Whisper base or tiny, and an end of turn of 0.5, 0.8 or 1.5 seconds. With the keypad only, it contains 10%. The end of turn of 0.5 seconds passed one test fewer (38 of 60) than 0.8 and 1.5 seconds (39 of 60).
On these calls, the wait changes little. Speech itself makes the big difference: from 10% to 35%. Whisper tiny does as well as base here, because the answers are short and each state expects only a few words. The course keeps base and 0.8 seconds. Module 8 comes back to the wait, because there every tenth of a second counts.
Step 2: hear the words, with a small grammar per state
Section titled “Step 2: hear the words, with a small grammar per state”Whisper writes the caller’s words, as in Module 4. In Module 4, a prompt with Larkfield’s vocabulary helped Whisper small with product names. The first version of the IVR used that same vocabulary prompt in every state, and it failed at the very first question. Callers said “Français”, and Whisper base wrote “Forfeit” and “Enbush”.
The fix was to give Whisper a short prompt for each state: only the words the IVR expects there. Phone systems call such a list of expected answers a grammar. The IVR’s grammars are in engine.py:
STATE_PROMPTS = { "language": {None: "English. Français."}, "menu": {"en": "Order status. Opening hours. Start a return. Talk to a person.", "fr": "Suivi de commande. Heures d'ouverture. Faire un retour. Parler à quelqu'un."}, "number": {"en": "My order number is 742155. Seven four two, one five five.", "fr": "Mon numéro de commande est le 742155. Sept quatre deux, un cinq cinq."}, "confirm": {"en": "Yes. No.", "fr": "Oui. Non."},}In the language state, the IVR does not know the language yet. So it does not tell Whisper the language, and the prompt has a word in each one. In the other states, it tells Whisper the language that the caller chose.
The team also added two French words to the menu rules. With both changes, the share of test calls that the IVR finished without a person rose from 28% to 35%. A prompt is a setting like any other: measure it in each state where you use it. The next lesson finds a state where this prompt does not help.
Step 3: from words to an intent
Section titled “Step 3: from words to an intent”The words “Where’s my order?” are not yet a decision. The IVR needs the intent: one of order_status, opening_hours, start_return or agent.
Keyword rules, and the last one wins
Section titled “Keyword rules, and the last one wins”The IVR uses keyword rules, as in Module 6: a list of words and phrases for each intent.
INTENT_WORDS = { "order_status": ["order status", "where is my order", "track", "tracking", "delivery", "hasn't arrived", "has not arrived", "suivi", "suivre", "où est ma commande", "où est la commande", "livraison"], "opening_hours": ["opening hours", "hours", "when are you open", "open", "heures", "ouverture", "ouverts", "ouvert"], "start_return": ["return", "send back", "send it back", "retour", "retourner", "renvoyer"], "agent": ["person", "agent", "someone", "human", "representative", "quelqu'un", "quelqu un", "personne", "agent", "conseiller"],}Callers sometimes change their mind in one breath: “I want to return something… no, track my order.” Two intents are in that sentence. The rule find_intent keeps the last one the caller names, because people correct themselves at the end:
| Caller’s words | Intents found, in order | Intent |
|---|---|---|
| “I want to return something... no, track my order.” | return, then track | order_status |
| “Where is my order?” | where is my order | order_status |
| “What are your opening hours?” | opening hours | opening_hours |
| “Can I speak to someone?” | someone | agent |
| “My neighbour said I should call you about the garden thing.” | none | no match |
When no rule matches, the menu state records a no-match and plays its retry: “Sorry, I did not get that. Please use your keypad: press 1 for orders …”. So after one failed spoken answer, the IVR offers the keypad. After a second miss, a person takes over.
And a classifier?
Section titled “And a classifier?”In Module 6, a TF-IDF classifier found the call reason better than keyword rules (96.7% against 90% on the test calls). Why not use it here?
That classifier learned from whole conversations, labelled with seven call reasons. “Opening hours” is not one of them, and a menu answer has two to six words. A classifier for the menu would need its own training data: many real menu answers, each labelled with its intent. Larkfield has none yet. The test calls are for testing, not for training.
So the IVR starts with rules, which need no data and are easy to read. A classifier becomes worth it when the menu has many intents. It also needs labelled answers to learn from, for example from the call records of a few months. Then measure it the same way: on recorded test calls, per language.
Larkfield: what speech adds
Section titled “Larkfield: what speech adds”With speech, the IVR finished 35% of the 60 test calls without a person, against 10% with the keypad only. By language, it contained 41% of the English calls and 30% of the French calls, on Larkfield’s synthetic calls.
The French gap starts at the first question. In six French test calls, the IVR did not get past “Français”. Whisper wrote words like “We are so enough.” or “Perfect.” instead. The IVR then asked “Sorry. Press 1 for English. Appuyez sur 2 pour le français.” A real caller would press 2. The recorded caller cannot: they say their next recorded line. So the second miss sends them to a person.
Guided practice: speech through the IVR
Section titled “Guided practice: speech through the IVR”-
In
larkspeech, with the environment active, replay five spoken test calls: two that should work (I-009, I-029), one that asks for a person (I-073), and two French calls (I-008, I-011).Terminal window speech ivr replay --calls I-009,I-029,I-073,I-008,I-011IVR replay: 5 recorded test calls, Whisper base, end of turn 0.8 scalls passed contained wrong intent to a person s to answerall 5 3 40% 0% 60% 35.2Failed: I-008 (status_speech: handover, expected status_given), I-011 (status_speech: handover, expected status_given)Wrote outputs/ivr-replay.jsonl (one call record per line) -
Print what the IVR heard in each call:
Terminal window python -c "import json; [print(r['call_id'], r['outcome'], r['handoff_reason'], r['heard']) for r in map(json.loads, open('outputs/ivr-replay.jsonl'))]"I-008 handover no_match ['We are so enough.']I-009 status_given None ['English.', 'Where is my order?', '479907.', 'Yes. Yes.']I-011 handover no_input ['Prowa C.']I-029 hours_given None ['English.', 'What are your opening hours?']I-073 handover no_input ['English.', 'I want to talk to her.']I-008 and I-011 never got past “Français”. Look at I-073: the caller said “I want to talk to a person”, and Whisper wrote “her”. No rule matched, the IVR offered the keypad, and the recorded caller said nothing more. The call passed, because a person took over. But it passed for the wrong reason:
no_input, notcaller_asked. -
Try the intent rules alone, with no audio:
Terminal window python -c "from larkspeech.ivr.dialogue import find_intent; print(find_intent('I want to return something... no, track my order.', 'en')); print(find_intent('Je veux faire un retour... non, suivre ma commande.', 'fr')); print(find_intent('I want to talk to her.', 'en'))"order_statusorder_statusNone
Your turn: Whisper tiny
Section titled “Your turn: Whisper tiny”Predict: Whisper tiny is faster than base, and in Module 4 it made many more errors on whole calls. On the same five calls, does it pass fewer? Run the replay with --model tiny.
Check your answer
speech ivr replay --calls I-009,I-029,I-073,I-008,I-011 --model tinyIVR replay: 5 recorded test calls, Whisper tiny, end of turn 0.8 s calls passed contained wrong intent to a person s to answerall 5 3 40% 0% 60% 34.5Failed: I-008 (status_speech: handover, expected status_given), I-011 (status_speech: handover, expected status_given)Wrote outputs/ivr-replay.jsonl (one call record per line)The same three pass. On all 60 test calls, tiny and base also passed the same number (39), with the same containment (35%). Each state’s grammar keeps the expected answers short and few, so the smaller model is enough here. On whole conversations, in Module 4, the difference was large. Measure the size on the task, not in general.
Common mistake
The caller is cut off in the middle of a number
Symptom: in test call I-049, the caller says the order number in two groups: “seven nine eight … zero nine zero”. The IVR heard only “Seven nine eight.” and replied “I heard 7 9 8. Please say the rest of the number.”
Cause: the pause between the two groups is longer than the 0.8 seconds of silence that end a turn. The IVR decided, in the middle of the number, that the caller had finished.
Fix: callers pause by nature in some states (the order number), and give one word in others (yes or no). Use a longer wait where people pause, or ask for the rest, as the Larkfield IVR does. In I-049, the caller then said “zero nine zero”, and the IVR joined the two parts. Measure every change on the recorded calls with pauses.
Consequence in AI work: a single setting for every situation is rarely best. Tune by state or slice, and measure each one.
Check your understanding
Silero VAD finds speech in a caller's turn from 0.5 s to 1.6 s. The engine checks every 200 ms and waits for 0.8 s of silence. At which check does the turn end?
Show the answer
At 2.4 s: at 2.2 s the silence is 0.6 s (not enough); at 2.4 s it is 0.8 s, which meets the rule. The caller waits 0.8 s of silence, plus the recognition time, before the IVR answers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the language state give Whisper the prompt "English. Français." and no language, while the menu state gives it the chosen language and the menu phrases?
Show the answer
In the language state, the IVR does not know the language yet. So Whisper must find it, and the prompt has one expected word in each language. After the choice, Whisper gets the language and the few phrases expected in that state, which makes the short answers easier to recognise. With one big vocabulary prompt everywhere, Whisper base wrote “Forfeit” for “Français”, and containment was 28% instead of 35% (with two French menu words added too).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A caller says "Can I check my delivery, or no, actually, what are your hours?" Which intent do the rules return, and why?
Show the answer
opening_hours. The rules find “delivery” (order status) and “hours” (opening hours). They keep the last intent the caller names, because people correct themselves at the end of a sentence.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Speech takes three steps: find the end of the turn (Silero VAD every 200 ms, 0.8 s of silence), recognise the words (Whisper base), and find the intent (keyword rules, the last intent wins).
- A short prompt per state (a grammar) works better than one big vocabulary: with two French menu words, containment rose from 28% to 35%.
- With speech, the IVR contained 35% of the test calls, against 10% with the keypad only. English 41%, French 30%.
- Whisper tiny did as well as base, and the end of turn changed little: on short answers with a grammar, a small model is enough.
- After one failed spoken answer, the IVR offers the keypad. After two misses, a person takes over.
You can now let callers say what they need. But the hardest answer is still ahead: the order number, six digits said in groups, with pauses, “oh” for zero, on a noisy line. Next, you get the number right, read it back, and recover when it is wrong.