Where a menu stops working
- Lesson 28
- 30 min
- Module 8: From IVR to voice agent
Where we are
Our tests show which calls the IVR handles and which it hands over. Some callers ask two things or change their mind. In this lesson, we look at those calls, and at what an LLM could add and what it would cost.
Lesson 28 of 36 · Speech AI: Transcribe, Understand and SpeakWhere a menu stops working
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Find the calls that a phone menu cannot handle, and decide what a large language model (LLM) could add there. Then count what it would cost in waiting time, money and mistakes that callers hear.
Before you startWhat to know and to have ready
What you need to know
- What Larkfield’s IVR does: it greets the caller, understands speech and keypad tones, and reads the order number back. It hands over to a person after two misses. See What an IVR is, and why companies still use one and Test the IVR like a product.
- What an LLM is: a model that reads text and writes text. The LLM application course teaches how to call one from code. You do not need it for this lesson.
What you need on your computer
Nothing. This lesson reads recorded calls and recorded answers. You run code in the next lesson.
A form and a person
Section titled “A form and a person”Think of a printed form at a post office. It has boxes: name, address, the parcel’s weight. It works well when your question fits the boxes. Then a customer writes “I sent a parcel last week, no, wait, I want to send a new one”. The form has no box for “no, wait”. A clerk behind the desk reads the whole sentence and understands it.
A phone menu is the form. Its prompts are the boxes: “say order status, opening hours or start a return”. The clerk is what Tomás, the AI engineer, wants: a voice agent. It is a phone line where an LLM listens to the caller’s words and decides what to say and do.
Tomás says that an LLM agent always beats a menu. Omar, the product manager, owns the decision for the out-of-hours line. He asks a better question: on which calls does the menu fail, and would an agent do better on those calls?
The problem: where the IVR fails
Section titled “The problem: where the IVR fails”In Module 7, Amira tested the IVR with 60 recorded test calls. A test call passes when the IVR ends it the way it should. For some calls, the right ending is the answer, such as the order status. For other calls, the right ending is a person, for example when the caller asks for one.
The IVR passed 39 of the 60 calls. (A stricter test, which also checks the reason for each hand-over, passes 33. This module uses the first rule for both designs.) Of all 60 calls, it contained 35%: it gave the answer with no person. It handed 63% of the calls to a person. The chart shows the passes for each kind of test call. These numbers are on Larkfield’s synthetic calls, as all the numbers in this lesson.
Text description of the diagram
Each bar is one kind of test call. The blue part is the share that the IVR passed, and the text gives the count. Seven kinds pass every call: a return by speech, opening hours by speech or keypad, order status by keypad, an unclear request, a caller who asks for a person, and a caller who presses 0 again. The kinds at the bottom fail most. A caller who changes their mind passes 0 of 3. An order status said aloud passes 2 of 12. A number said with a pause passes 1 of 4, and a wrong number that the caller then fixes passes 1 of 3.
Where the failed calls stop
Section titled “Where the failed calls stop”The IVR failed 21 of the 60 calls, and 15 of the 21 are French. Each failed call stops at one step of the menu. It stops at the language question in 5 calls, at the menu in 9 calls, and at the order number in 6 calls. In one call, the caller says nothing and the IVR hangs up.
Look at what Whisper base wrote in some of these calls:
| Call | The caller said | Whisper base wrote | Where the call stopped |
|---|---|---|---|
| I-056 | “Où est ma commande?” | “Le matemain.” | the menu |
| I-011 | “Français.” | “Prowa C.” | the language question |
| I-003 | “522912” | “Two two nine one two.” | the order number |
| I-006 | “Where is my order?” | “Where’s my order?” | the menu |
The first three calls fail on recognition: the recogniser heard the wrong words. The last call is different. Whisper base heard the caller well, but “Where’s my order?” is not one of the menu’s phrases. That is a failure of understanding: the words are right, but the menu cannot use them.
In our test calls, most failures are recognition, and most of them are French. Accents, noise and two languages measured it. Whisper base has a WER of 23.8% on the French test calls and 5.2% on the English ones. Failures of understanding are fewer, but a menu cannot fix them at all. They happen when a caller says the request in other words, changes their mind, or does not know what to ask.
Listen to two callers
Section titled “Listen to two callers”Here is the caller of test call I-076. The IVR asked “How can I help?”.
And here is a French caller, test call I-045, who changes their mind in the middle of the sentence.
A person at Larkfield’s help desk handles both callers in a few seconds. The first caller has a question about a product, and the agent asks which one. The second caller wants the order status, and the agent ignores the word “retour”.
The idea: let a model choose the next step
Section titled “The idea: let a model choose the next step”A menu has a fixed list of steps. Amira wrote each prompt and each next step. A voice agent works in another way:
- It asks an open question: “How can I help?”
- An LLM reads the caller’s words and everything said so far in the call.
- The LLM chooses the next step. It can answer, ask a question, or call a tool.
A tool is a function that the program lets the LLM call. Larkfield’s agent has four tools: look up an order, start a return, transfer the caller to a person, and end the call. The LLM does not change the order system itself. It asks for a tool call, and the program runs it. The agents course teaches tools in depth: Tool contracts and Approval boundaries.
Text description of the diagram
On the left is the IVR. It says a fixed prompt, then checks whether the caller’s words match a menu item. If they match, it runs the fixed next step that Amira wrote. If the words do not match twice, it hands the call to a person. On the right is the voice agent. It asks an open question. The LLM reads the caller’s words and the call so far, then chooses one of three things. It can answer or ask in its own words. It can call the order system as a tool. Or it can call the tool that hands the call to a person.
So the agent adds two things. It understands sentences that are not on the menu. And it can mix steps freely: confirm a number, look it up, and answer a second question in the same call.
A worked example: the same caller, two designs
Section titled “A worked example: the same caller, two designs”The course team ran a voice agent on the same 60 recorded test calls. It used the hosted model chat-small on Azure, with the four tools. Every answer was recorded once. You replay the recordings, so you need no key. Here is test call I-045 again, turn by turn. The left column is what Whisper base wrote from the caller’s audio.
| Caller (as recognised) | The agent’s answer (recorded) |
|---|---|
| [the caller pressed 2 on the keypad] | “Sorry, I didn’t understand. How can I help? Vous pouvez aussi parler en français.” |
| “Je veux faire un retour non suivre ma commande.” | “Bien sûr. Pour commencer un retour, dites-moi votre numéro de commande.” |
| “189 710.” | “Votre numéro de commande est le 1 8 9, 7 1 0. Est-ce bien cela ?” |
| “Oui c’est ça !” | (calls the tool look up order 189710: no such order) |
| “Je ne trouve pas de commande sous ce numéro. Voulez-vous vérifier le numéro ou parler à une personne ?” |
Read it slowly. Three things went wrong, and each one teaches something:
- The keypad press. The recorded caller pressed 2 to choose French, because the IVR’s greeting asks for it. The agent never offered a keypad menu. So the key meant nothing to it. The replay is unfair to the agent here.
- The change of mind. The caller said “no, track my order”. The agent answered “to start a return, tell me your order number”. The menu did not understand this sentence either, but it only asked again. The agent chose the wrong task and said it aloud.
- The order number. The caller said 187971. Whisper base wrote “189 710”. The agent read the wrong number back, and the recorded caller said “yes”. The order system found no such order, so nothing bad happened.
The replay has one more limit. The recorded caller cannot react. In a real call, the caller would say “no, I said track my order”. Here, the next recorded line plays, whatever the agent said. So the replay is unfair to both designs: to the IVR in Module 7, and to the agent here.
What the agent costs
Section titled “What the agent costs”An agent is not free. Omar writes down three costs before anyone builds it.
Waiting time
Section titled “Waiting time”The agent adds a step that the menu does not have: the LLM must write each answer. The menu’s prompts are made in advance. On the recorded calls, the median wait from “the caller stops talking” to “the first sound back” was 2.10 s. No caller turn was under one second. The next lesson takes this wait apart.
The LLM charges for each word it reads and writes. The 60 recorded test calls cost US$0.011 in all, about two hundredths of a cent a call. That is tiny next to US$4.00 for a person. The money that matters is the people. Every call that the agent hands over still costs US$4.00. Prices change: check the current page of the model you use.
Mistakes said aloud
Section titled “Mistakes said aloud”A menu can only say what Amira wrote. An agent writes new sentences. The system prompt asked it to read order numbers digit by digit. On test call I-054, it said “six hundred three, one hundred thirty-seven” instead. On call I-045, it said “pour commencer un retour” to a caller who wanted the order status. A wrong sentence in text can be fixed before anyone reads it. A wrong sentence in a phone call is heard at once.
| IVR (menu) | Voice agent | |
|---|---|---|
| Understands | the menu’s words and the keypad | free sentences |
| Says | only prompts that Amira wrote | new sentences each time |
| Waits after the caller (median) | no LLM step | 2.10 s on the recorded calls |
| LLM cost per call | none | about US$0.0002 (60 calls: US$0.011) |
| Can act wrongly | yes: 2 wrong actions in 60 calls (in the recorded replay of the first dialogue.py; none with the current IVR) |
yes, through its tools |
Costs of people and mistakes are assumptions (Grace’s cost table). LLM times and words are recorded answers from chat-small.
When an agent is worth trying
Section titled “When an agent is worth trying”Try an agent where the menu fails because callers say more than one menu item: a change of mind, two questions, an unclear request. Our 60 test calls have no caller with two questions. Add such calls to the test set before you judge an agent on them.
Do not expect an agent to fix the recognition errors. On “order status, speech”, the agent passed 2 of 12 calls, the same as the IVR. Both used Whisper base, and both heard the wrong digits. A better recogniser, or the keypad for the number, fixes that. An LLM does not.
And keep the menu where it works. Keypad calls, opening hours and simple returns pass every time with the IVR. An agent there adds waiting time and new chances for mistakes, and gains nothing.
Common mistake
Judging the agent on one good call
Symptom: Tomás plays one recorded agent call in a meeting. The caller says something vague, and the agent politely hands the call to a person. Everybody agrees that the agent “understands people”. Nobody asks how the agent does on the other 59 calls.
Cause: one call is an anecdote. The agent’s answers change with each wording. Some kinds of call go well (an unclear request: 4 of 4 handed over correctly). Others go badly (an order number by keypad: 0 of 6, because the agent never offered the keypad).
Fix: judge the agent on the same recorded test calls as the IVR, with the same pass rules. Report the result for each kind of call. The chart above does it for the IVR, and Speech-to-speech models and evaluation does it for both.
Consequence in AI work: a demo chooses its best example. A test set shows the average and the worst cases. A decision made on a demo goes live with failures that nobody measured.
Guided practice: sort the failures
Section titled “Guided practice: sort the failures”Use the table below: more of the IVR’s failed test calls, with the recorded transcripts of Whisper base. No code is needed.
| Call | Kind | The caller said | Whisper base wrote |
|---|---|---|---|
| I-008 | order status, speech | “Français.” | “We are so enough.” |
| I-010 | order status, speech | “303362” | “Three, both three, three.” |
| I-046 | change of mind | “Je veux faire un retour… non, suivre ma commande.” | “Un retour à quelqu’un.” |
| I-047 | change of mind | “Je veux faire un retour… non, suivre ma commande.” | “Je veux faire un retournant. Five-rena commande.” |
| I-055 | wrong number, then fixed | “Je veux suivre ma commande.” | “Je suis vraiment commande.” |
- For each call, compare the two texts.
- Mark the call recognition if the words were heard wrong. Mark it understanding if the words were right but the menu could not use them.
- For each call, write what would help most: a better recogniser, the keypad, or an LLM.
- Compare your answers with the answer below.
Show the answer
All five are recognition failures. In each one, the text that Whisper base wrote is not what the caller said.
- I-008 and I-055 are French requests that came out as other words. A better recogniser helps, or the keypad: the IVR offers “press 2” for French and keys for the menu.
- I-010 is a spoken order number. The keypad helps most.
- I-046 and I-047 are the change of mind. An LLM could understand “un retour… non, suivre ma commande”. But here it never gets those words. I-046 came out as “a return to someone”, so the IVR heard “someone” and put the caller through to a person. An LLM that reads a wrong text makes a wrong choice too.
So an LLM agent helps with failures of understanding, such as I-006 above. It does not help with the recognition failures, which are most of the failures in our test calls.
Your turn: test call I-076
Section titled “Your turn: test call I-076”Read the recorded agent answers for test call I-076, the caller of the first clip:
| Caller (as recognised) | The agent’s answer (recorded) |
|---|---|
| “End of.” | “I’m sorry, I didn’t understand. Could you tell me what you need help with?” |
| “My neighbor said I should call you about the garden thing.” | “Sorry, I’m not sure what you mean by ‘the garden thing.’ Are you calling about an order, a return, our opening hours, or would you like to speak to someone?” |
| “My neighbor said I should call you about the garden thing.” | (calls the tool transfer to a person: “Caller’s request is unclear after two attempts to clarify.”) |
Before you read the answer, decide: did the agent do better than the IVR on this call? What did each design cost?
Show the answer
Both designs passed: the right ending for an unclear request is a person. Both handed over after two misses. The IVR’s rule is in its code. The agent followed the same rule because its system prompt asked for it. So on this call, the agent added waiting time and an LLM cost, and gave the same result. Its second question was better: it named the menu items in a friendly way. The first line, “End of.”, is Whisper base’s text for the caller saying “English”: a recognition error again.
Knowledge check
Section titled “Knowledge check”Check your understanding
The IVR passed 39 of 60 test calls but contained only 35% of them. How can both be true?
Show the answer
A test call passes when it ends the way it should. For some calls, the right ending is a person: a caller who asks for one, presses 0, or makes an unclear request. Those calls pass with a hand-over, but they are not contained. Containment counts only the calls that the IVR finished itself, with the answer and no person: 21 of 60, which is 35%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: "An LLM agent will fix our order-status calls by speech." What do the recorded calls say?
Show the answer
They say no. On “order status, speech”, the agent passed 2 of 12 calls, the same as the IVR. Both used Whisper base, and both heard the wrong digits. The agent even read a wrong number back, and the recorded caller said yes. The fix for this kind is in recognition: a better recogniser, or the keypad for the number. An LLM helps where the words are heard right but do not fit the menu, such as a change of mind.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Name three costs of a voice agent that a menu does not have, with a number for each from the recorded calls.
Show the answer
Waiting time: a median of 2.10 s from “the caller stops talking” to “the first sound back”. No turn was under 1 s. Money: an LLM cost of US$0.011 for the 60 calls, small next to US$4.00 for each call that a person handles (an assumption). Mistakes said aloud: the agent read an order number as “six hundred three, one hundred thirty-seven”, although its prompt asked for digits. It also offered a return to a caller who said “no, track my order”.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- The IVR passed 39 of 60 recorded test calls and contained 35%. Of its 21 failures, 15 are French calls, and most start with a word that Whisper base heard wrong.
- Some failures are recognition (the words were heard wrong). Others are understanding (the words do not fit the menu). An LLM helps only with the second kind.
- A voice agent asks an open question and lets an LLM choose the next step. The LLM can answer, ask, or call a tool, such as the order system or a hand-over.
- On the same calls, the recorded agent waited a median of 2.10 s before its first sound, and it said some mistakes aloud.
- The replay’s callers cannot react, so the replay is unfair to both designs. Judge both on the same calls, kind by kind.
An agent can understand the calls that the menu cannot, but every answer comes 2.10 s after the caller stops talking. Next, you take that wait apart, step by step, and find where the time goes.
Next: The voice-agent pipeline