Skip to content
Français

Where a menu stops working

  • Lesson 28
  • 30 min
  • Module 8: From IVR to voice agent

Where we are

Our tests show which calls the IVR handles and which it hands over. Some callers ask two things or change their mind. In this lesson, we look at those calls, and at what an LLM could add and what it would cost.

Lesson 28 of 36 · Speech AI: Transcribe, Understand and SpeakWhere a menu stops working

What you will be able to do

Find the calls that a phone menu cannot handle, and decide what a large language model (LLM) could add there. Then count what it would cost in waiting time, money and mistakes that callers hear.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson reads recorded calls and recorded answers. You run code in the next lesson.

Think of a printed form at a post office. It has boxes: name, address, the parcel’s weight. It works well when your question fits the boxes. Then a customer writes “I sent a parcel last week, no, wait, I want to send a new one”. The form has no box for “no, wait”. A clerk behind the desk reads the whole sentence and understands it.

A phone menu is the form. Its prompts are the boxes: “say order status, opening hours or start a return”. The clerk is what Tomás, the AI engineer, wants: a voice agent. It is a phone line where an LLM listens to the caller’s words and decides what to say and do.

Tomás says that an LLM agent always beats a menu. Omar, the product manager, owns the decision for the out-of-hours line. He asks a better question: on which calls does the menu fail, and would an agent do better on those calls?

In Module 7, Amira tested the IVR with 60 recorded test calls. A test call passes when the IVR ends it the way it should. For some calls, the right ending is the answer, such as the order status. For other calls, the right ending is a person, for example when the caller asks for one.

The IVR passed 39 of the 60 calls. (A stricter test, which also checks the reason for each hand-over, passes 33. This module uses the first rule for both designs.) Of all 60 calls, it contained 35%: it gave the answer with no person. It handed 63% of the calls to a person. The chart shows the passes for each kind of test call. These numbers are on Larkfield’s synthetic calls, as all the numbers in this lesson.

Horizontal bars, one for each of 13 kinds of test call, sorted by the share passed. Passed all: return by speech 6 of 6, hours by speech 6 of 6, order status by keypad 6 of 6, unclear request 4 of 4, asks for a person 3 of 3, presses 0 again 3 of 3, hours by keypad 3 of 3. Passed some: caller says nothing 2 of 3, noisy line then keypad 2 of 4, wrong number then fixed 1 of 3, number with a pause 1 of 4, order status by speech 2 of 12. Passed none: change of mind 0 of 3.
The IVR on its 60 recorded test calls (Whisper base, end-of-turn wait 0.8 s), passed by kind. It passes every keypad call and every simple spoken request. It fails where the caller says more than a menu item.
Text description of the diagram

Each bar is one kind of test call. The blue part is the share that the IVR passed, and the text gives the count. Seven kinds pass every call: a return by speech, opening hours by speech or keypad, order status by keypad, an unclear request, a caller who asks for a person, and a caller who presses 0 again. The kinds at the bottom fail most. A caller who changes their mind passes 0 of 3. An order status said aloud passes 2 of 12. A number said with a pause passes 1 of 4, and a wrong number that the caller then fixes passes 1 of 3.

The IVR failed 21 of the 60 calls, and 15 of the 21 are French. Each failed call stops at one step of the menu. It stops at the language question in 5 calls, at the menu in 9 calls, and at the order number in 6 calls. In one call, the caller says nothing and the IVR hangs up.

Look at what Whisper base wrote in some of these calls:

Call The caller said Whisper base wrote Where the call stopped
I-056 “Où est ma commande?” “Le matemain.” the menu
I-011 “Français.” “Prowa C.” the language question
I-003 “522912” “Two two nine one two.” the order number
I-006 “Where is my order?” “Where’s my order?” the menu

The first three calls fail on recognition: the recogniser heard the wrong words. The last call is different. Whisper base heard the caller well, but “Where’s my order?” is not one of the menu’s phrases. That is a failure of understanding: the words are right, but the menu cannot use them.

In our test calls, most failures are recognition, and most of them are French. Accents, noise and two languages measured it. Whisper base has a WER of 23.8% on the French test calls and 5.2% on the English ones. Failures of understanding are fewer, but a menu cannot fix them at all. They happen when a caller says the request in other words, changes their mind, or does not know what to ask.

Here is the caller of test call I-076. The IVR asked “How can I help?”.

Test call I-076, the caller’s second turn, phone quality (8 kHz, landline). A synthetic voice: Piper, trained on the VCTK corpus. Transcript: “My neighbour said I should call you about the garden thing.”

And here is a French caller, test call I-045, who changes their mind in the middle of the sentence.

Test call I-045, the caller’s second turn, phone quality (8 kHz, mobile line). A synthetic voice: Kokoro (voice ff_siwis), learned from the SIWIS French corpus. Transcript: “Je veux faire un retour… non, suivre ma commande.” In English: “I want to make a return… no, track my order.”

A person at Larkfield’s help desk handles both callers in a few seconds. The first caller has a question about a product, and the agent asks which one. The second caller wants the order status, and the agent ignores the word “retour”.

The idea: let a model choose the next step

Section titled “The idea: let a model choose the next step”

A menu has a fixed list of steps. Amira wrote each prompt and each next step. A voice agent works in another way:

  • It asks an open question: “How can I help?”
  • An LLM reads the caller’s words and everything said so far in the call.
  • The LLM chooses the next step. It can answer, ask a question, or call a tool.

A tool is a function that the program lets the LLM call. Larkfield’s agent has four tools: look up an order, start a return, transfer the caller to a person, and end the call. The LLM does not change the order system itself. It asks for a tool call, and the program runs it. The agents course teaches tools in depth: Tool contracts and Approval boundaries.

Two flows side by side. IVR (menu): a fixed prompt goes to a question, do the words match a menu item? Yes leads to a fixed next step written by Amira. No, twice, leads to a person. Voice agent: an open question goes to the LLM, which reads the words and the call so far. The LLM chooses the next step: answer or ask in its own words, call the order system as a tool, or call the tool that hands over to a person.
The menu decides with fixed rules that a person wrote. The agent lets the LLM choose the next step, and the program runs the tools that it asks for.
Text description of the diagram

On the left is the IVR. It says a fixed prompt, then checks whether the caller’s words match a menu item. If they match, it runs the fixed next step that Amira wrote. If the words do not match twice, it hands the call to a person. On the right is the voice agent. It asks an open question. The LLM reads the caller’s words and the call so far, then chooses one of three things. It can answer or ask in its own words. It can call the order system as a tool. Or it can call the tool that hands the call to a person.

So the agent adds two things. It understands sentences that are not on the menu. And it can mix steps freely: confirm a number, look it up, and answer a second question in the same call.

A worked example: the same caller, two designs

Section titled “A worked example: the same caller, two designs”

The course team ran a voice agent on the same 60 recorded test calls. It used the hosted model chat-small on Azure, with the four tools. Every answer was recorded once. You replay the recordings, so you need no key. Here is test call I-045 again, turn by turn. The left column is what Whisper base wrote from the caller’s audio.

Caller (as recognised) The agent’s answer (recorded)
[the caller pressed 2 on the keypad] “Sorry, I didn’t understand. How can I help? Vous pouvez aussi parler en français.”
“Je veux faire un retour non suivre ma commande.” “Bien sûr. Pour commencer un retour, dites-moi votre numéro de commande.”
“189 710.” “Votre numéro de commande est le 1 8 9, 7 1 0. Est-ce bien cela ?”
“Oui c’est ça !” (calls the tool look up order 189710: no such order)
“Je ne trouve pas de commande sous ce numéro. Voulez-vous vérifier le numéro ou parler à une personne ?”

Read it slowly. Three things went wrong, and each one teaches something:

  1. The keypad press. The recorded caller pressed 2 to choose French, because the IVR’s greeting asks for it. The agent never offered a keypad menu. So the key meant nothing to it. The replay is unfair to the agent here.
  2. The change of mind. The caller said “no, track my order”. The agent answered “to start a return, tell me your order number”. The menu did not understand this sentence either, but it only asked again. The agent chose the wrong task and said it aloud.
  3. The order number. The caller said 187971. Whisper base wrote “189 710”. The agent read the wrong number back, and the recorded caller said “yes”. The order system found no such order, so nothing bad happened.

The replay has one more limit. The recorded caller cannot react. In a real call, the caller would say “no, I said track my order”. Here, the next recorded line plays, whatever the agent said. So the replay is unfair to both designs: to the IVR in Module 7, and to the agent here.

An agent is not free. Omar writes down three costs before anyone builds it.

The agent adds a step that the menu does not have: the LLM must write each answer. The menu’s prompts are made in advance. On the recorded calls, the median wait from “the caller stops talking” to “the first sound back” was 2.10 s. No caller turn was under one second. The next lesson takes this wait apart.

The LLM charges for each word it reads and writes. The 60 recorded test calls cost US$0.011 in all, about two hundredths of a cent a call. That is tiny next to US$4.00 for a person. The money that matters is the people. Every call that the agent hands over still costs US$4.00. Prices change: check the current page of the model you use.

A menu can only say what Amira wrote. An agent writes new sentences. The system prompt asked it to read order numbers digit by digit. On test call I-054, it said “six hundred three, one hundred thirty-seven” instead. On call I-045, it said “pour commencer un retour” to a caller who wanted the order status. A wrong sentence in text can be fixed before anyone reads it. A wrong sentence in a phone call is heard at once.

IVR (menu) Voice agent
Understands the menu’s words and the keypad free sentences
Says only prompts that Amira wrote new sentences each time
Waits after the caller (median) no LLM step 2.10 s on the recorded calls
LLM cost per call none about US$0.0002 (60 calls: US$0.011)
Can act wrongly yes: 2 wrong actions in 60 calls (in the recorded replay of the first dialogue.py; none with the current IVR) yes, through its tools

Costs of people and mistakes are assumptions (Grace’s cost table). LLM times and words are recorded answers from chat-small.

Try an agent where the menu fails because callers say more than one menu item: a change of mind, two questions, an unclear request. Our 60 test calls have no caller with two questions. Add such calls to the test set before you judge an agent on them.

Do not expect an agent to fix the recognition errors. On “order status, speech”, the agent passed 2 of 12 calls, the same as the IVR. Both used Whisper base, and both heard the wrong digits. A better recogniser, or the keypad for the number, fixes that. An LLM does not.

And keep the menu where it works. Keypad calls, opening hours and simple returns pass every time with the IVR. An agent there adds waiting time and new chances for mistakes, and gains nothing.

Common mistake

Judging the agent on one good call

Symptom: Tomás plays one recorded agent call in a meeting. The caller says something vague, and the agent politely hands the call to a person. Everybody agrees that the agent “understands people”. Nobody asks how the agent does on the other 59 calls.

Cause: one call is an anecdote. The agent’s answers change with each wording. Some kinds of call go well (an unclear request: 4 of 4 handed over correctly). Others go badly (an order number by keypad: 0 of 6, because the agent never offered the keypad).

Fix: judge the agent on the same recorded test calls as the IVR, with the same pass rules. Report the result for each kind of call. The chart above does it for the IVR, and Speech-to-speech models and evaluation does it for both.

Consequence in AI work: a demo chooses its best example. A test set shows the average and the worst cases. A decision made on a demo goes live with failures that nobody measured.

Use the table below: more of the IVR’s failed test calls, with the recorded transcripts of Whisper base. No code is needed.

Call Kind The caller said Whisper base wrote
I-008 order status, speech “Français.” “We are so enough.”
I-010 order status, speech “303362” “Three, both three, three.”
I-046 change of mind “Je veux faire un retour… non, suivre ma commande.” “Un retour à quelqu’un.”
I-047 change of mind “Je veux faire un retour… non, suivre ma commande.” “Je veux faire un retournant. Five-rena commande.”
I-055 wrong number, then fixed “Je veux suivre ma commande.” “Je suis vraiment commande.”
  1. For each call, compare the two texts.
  2. Mark the call recognition if the words were heard wrong. Mark it understanding if the words were right but the menu could not use them.
  3. For each call, write what would help most: a better recogniser, the keypad, or an LLM.
  4. Compare your answers with the answer below.
Show the answer

All five are recognition failures. In each one, the text that Whisper base wrote is not what the caller said.

  • I-008 and I-055 are French requests that came out as other words. A better recogniser helps, or the keypad: the IVR offers “press 2” for French and keys for the menu.
  • I-010 is a spoken order number. The keypad helps most.
  • I-046 and I-047 are the change of mind. An LLM could understand “un retour… non, suivre ma commande”. But here it never gets those words. I-046 came out as “a return to someone”, so the IVR heard “someone” and put the caller through to a person. An LLM that reads a wrong text makes a wrong choice too.

So an LLM agent helps with failures of understanding, such as I-006 above. It does not help with the recognition failures, which are most of the failures in our test calls.

Read the recorded agent answers for test call I-076, the caller of the first clip:

Caller (as recognised) The agent’s answer (recorded)
“End of.” “I’m sorry, I didn’t understand. Could you tell me what you need help with?”
“My neighbor said I should call you about the garden thing.” “Sorry, I’m not sure what you mean by ‘the garden thing.’ Are you calling about an order, a return, our opening hours, or would you like to speak to someone?”
“My neighbor said I should call you about the garden thing.” (calls the tool transfer to a person: “Caller’s request is unclear after two attempts to clarify.”)

Before you read the answer, decide: did the agent do better than the IVR on this call? What did each design cost?

Show the answer

Both designs passed: the right ending for an unclear request is a person. Both handed over after two misses. The IVR’s rule is in its code. The agent followed the same rule because its system prompt asked for it. So on this call, the agent added waiting time and an LLM cost, and gave the same result. Its second question was better: it named the menu items in a friendly way. The first line, “End of.”, is Whisper base’s text for the caller saying “English”: a recognition error again.

Check your understanding

The IVR passed 39 of 60 test calls but contained only 35% of them. How can both be true?

Show the answer

A test call passes when it ends the way it should. For some calls, the right ending is a person: a caller who asks for one, presses 0, or makes an unclear request. Those calls pass with a hand-over, but they are not contained. Containment counts only the calls that the IVR finished itself, with the answer and no person: 21 of 60, which is 35%.

Check your understanding

Tomás says: "An LLM agent will fix our order-status calls by speech." What do the recorded calls say?

Show the answer

They say no. On “order status, speech”, the agent passed 2 of 12 calls, the same as the IVR. Both used Whisper base, and both heard the wrong digits. The agent even read a wrong number back, and the recorded caller said yes. The fix for this kind is in recognition: a better recogniser, or the keypad for the number. An LLM helps where the words are heard right but do not fit the menu, such as a change of mind.

Check your understanding

Name three costs of a voice agent that a menu does not have, with a number for each from the recorded calls.

Show the answer

Waiting time: a median of 2.10 s from “the caller stops talking” to “the first sound back”. No turn was under 1 s. Money: an LLM cost of US$0.011 for the 60 calls, small next to US$4.00 for each call that a person handles (an assumption). Mistakes said aloud: the agent read an order number as “six hundred three, one hundred thirty-seven”, although its prompt asked for digits. It also offered a return to a caller who said “no, track my order”.

Recap

  • The IVR passed 39 of 60 recorded test calls and contained 35%. Of its 21 failures, 15 are French calls, and most start with a word that Whisper base heard wrong.
  • Some failures are recognition (the words were heard wrong). Others are understanding (the words do not fit the menu). An LLM helps only with the second kind.
  • A voice agent asks an open question and lets an LLM choose the next step. The LLM can answer, ask, or call a tool, such as the order system or a hand-over.
  • On the same calls, the recorded agent waited a median of 2.10 s before its first sound, and it said some mistakes aloud.
  • The replay’s callers cannot react, so the replay is unfair to both designs. Judge both on the same calls, kind by kind.

An agent can understand the calls that the menu cannot, but every answer comes 2.10 s after the caller stops talking. Next, you take that wait apart, step by step, and find where the time goes.

Next: The voice-agent pipeline