Menu, agent or both?
- Lesson 32
- 40 min
- Module 8: From IVR to voice agent
Where we are
We have an IVR and a voice agent, each tested on its own, and a fair way to judge them. Omar must choose one for the out-of-hours line. In this lesson, we run the IVR, the agent and a hybrid on the same recorded calls and choose with the numbers.
Lesson 32 of 36 · Speech AI: Transcribe, Understand and SpeakMenu, agent or both?
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Choose a design for Larkfield’s out-of-hours phone line: the menu, the voice agent, or both together. Compare them on the same recorded calls, count every cost, including the ones a table leaves out, and write the recommendation with its risks.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It builds the three designs from the two recordings and splits the cost per call into its parts. It counts the calls that end with no answer, and changes the cost assumptions. It needs no model download and no account. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The five measures of a voice agent, and the calls that must go to a person: Speech-to-speech models and evaluation.
- What containment is: the share of calls that the system finishes itself, with no person.
What you need on your computer (only for the local route; the notebook needs none of it)
- Your project folder
larkspeechwith the files of The voice-agent pipeline:speech agent --helpworks in it.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.
A self-checkout with a person nearby
Section titled “A self-checkout with a person nearby”Think of a supermarket. A self-checkout is fast for a basket of simple things. It fails on a bag of loose apples or a bottle of wine, and then it calls a member of staff. A staffed till handles everything, but it costs a person for every customer. Most shops do both: the machine first, a person when the machine is stuck.
Larkfield has the same three choices for its phone line out of hours. It can use the menu, the voice agent, or the menu first and the agent when the menu is stuck. About 900 calls a month come out of hours.
The problem: two designs, two separate tests
Section titled “The problem: two designs, two separate tests”Amira tested the IVR in Module 7. The course team then ran the voice agent on the same calls. Tomás says that the agent is newer, so it must be better. Amira says that the menu passed more test calls. Both look at their own numbers.
Omar needs one table: the same calls, the same rules, the same costs. And he needs the third design that nobody has tested yet.
The idea: a hybrid, and one comparison
Section titled “The idea: a hybrid, and one comparison”A hybrid design puts the menu first. The menu handles what it can. When it fails twice to match the caller’s words, the call goes to the voice agent instead of a person. A caller who asks for a person, or presses 0, still gets a person at once.
Text description of the diagram
A call starts at the IVR menu, which understands speech and the keypad. If the caller chooses a menu item, the IVR does the task: order status, opening hours or a return, and the call ends. If the caller presses 0 or asks for a person, the call goes to a person. If the menu does not match the caller’s words twice, the call goes to the voice agent, an LLM with tools and the same guardrails. The agent ends the call when the task is done. When it cannot help, it puts the caller through to a person.
To compare the three, use the same recorded test calls and one set of rules:
- Contained: the call ended with the answer (status, hours or a return started), with no wrong action and no person.
- To a person: the call was handed over.
- Wrong actions: the design acted on a wrong intent or a wrong order.
- Cost per call: people, wrong actions and compute, from Grace’s cost table. These are assumptions: a call handled by a person US$4.00, a wrong action US$15, IVR compute US$0.002 a call. The LLM cost is the recorded cost of each agent call.
Larkfield: compare the three designs
Section titled “Larkfield: compare the three designs”You need no new file: speech agent compare came with the snapshot end-of-m08 in The voice-agent pipeline.
-
In your folder
larkspeech, with the virtual environment active, run the comparison:Terminal window speech agent comparedesign calls contained to a person wrong actions mean s US$ a callIVR (menu) 60 35% 63% 2 45.1 3.035voice agent 60 17% 70% 0 24.3 2.800hybrid 60 37% 55% 2 51.9 2.702Costs are assumptions (data/costs.json); agent answers and times are recorded (chat-small).It uses the IVR’s recorded replay and the recorded agent calls, so it takes about a second.
The IVR’s recorded replay was made once, with the first version of
dialogue.py, before Module 7’s last changes. In it, two French calls (I-046 and I-051) went to a person for a wrong reason. The IVR heard a request for a person, but the caller wanted an order status. Those are its 2 wrong actions. When we replayed the calls again with the current project, Whisper heard those turns differently, and no call had a wrong intent. To compare your own IVR, runspeech ivr replayfirst, thenspeech agent compare --ivr-records outputs/ivr-replay.jsonl. On our run, the IVR then cost US$2.535 a call and the hybrid US$2.202, with the same containment. The ranking with hang-ups, later in this lesson, does not change. -
The command also writes the table to
outputs/designs.jsonl, one design a line. Look at the first line:Terminal window head -1 outputs/designs.jsonlTerminal window Get-Content outputs\designs.jsonl -TotalCount 1{"design": "IVR (menu)", "calls": 60, "containment": 0.35, "handover": 0.6333333333333333, "wrong_actions": 2, "mean_seconds": 45.07233333333333, "usd_per_call": 3.0353333333333334}The file keeps every digit. The table rounds them.
On this table, the hybrid wins: the most calls contained, the fewest to a person, and the lowest cost. Tomás’s agent alone is the worst at containment. Before Omar decides, Priya checks where each number comes from.
A worked example: the IVR’s cost per call
Section titled “A worked example: the IVR’s cost per call”Every cost per call is the sum of three parts, divided by the 60 calls:
| Part | Count | Cost each | Total |
|---|---|---|---|
| Calls to a person | 38 | US$4.00 | US$152.00 |
| Wrong actions | 2 | US$15.00 | US$30.00 |
| IVR compute | 60 | US$0.002 | US$0.12 |
| All 60 calls | US$182.12 |
US$182.12 / 60 = US$3.035 a call. The 38 calls to a person include the 2 wrong actions, because both wrong actions were hand-overs (more on them below). So those 2 calls pay for a person and for a wrong action. People are 83% of the cost. The LLM costs of the agent are even smaller than the IVR’s compute: US$0.011 for all 60 calls. So the design that wins on cost is the one that sends the fewest calls to a person.
Where each call ended
Section titled “Where each call ended”Text description of the diagram
Each bar is one design, split into the 60 test calls. The IVR contained 21 calls, handed 36 to a person, made 2 wrong actions and left 1 call with no end. The voice agent contained 10, handed 42 to a person and left 8 with no end. The hybrid contained 22, handed 31 to a person, made 2 wrong actions and left 5 with no end.
Look at the orange part: no end. These calls ended with no answer and no person. In the agent’s replay, the recorded caller ran out of lines while the agent was still asking a question. A real caller would hang up, or call again. The IVR has 1 such call: the silent caller of the previous lesson. The agent has 8 (7 with no end and 1 that it ended with no answer), and the hybrid 5.
| Calls with no end | IVR | Voice agent | Hybrid |
|---|---|---|---|
| The caller hung up or ran out of lines (counted as a hang-up below) | 1 | 7 | 4 |
| The agent ended the call with no answer | 0 | 1 | 1 |
| All “no end” calls in the chart | 1 | 8 | 5 |
Now look at the hybrid. Of the 16 calls that the IVR handed over after two misses, the agent contained only 1. It handed 11 to a person, and left 4 with no end. So the hybrid saves 5 hand-overs, but 4 of those 5 callers got no answer at all.
What the table leaves out
Section titled “What the table leaves out”The command’s cost counts people, wrong actions and compute. It does not count a caller who hangs up. Grace’s cost table has a number for that: US$8 for a caller who hangs up. It is an assumption: a second call, and an unhappy customer. Count every call that ended with a hang-up at US$8:
Text description of the diagram
Each bar is the cost per call of one design. The solid parts are what the command counts: people, wrong actions, and compute with the LLM. The hatched part is the cost of hang-ups, which the command leaves out. IVR: people US$2.533, wrong actions US$0.500, compute US$0.002, total US$3.035; hang-ups add US$0.133 (1 call). Voice agent: people US$2.800, total US$2.800; hang-ups add US$0.933 (7 calls). Hybrid: people US$2.200, wrong actions US$0.500, compute US$0.002, total US$2.702; hang-ups add US$0.533 (4 calls).
| Design | US$ a call (the command) | Hang-ups | US$ a call with hang-ups |
|---|---|---|---|
| IVR (menu) | 3.035 | 1 | 3.169 |
| Hybrid | 2.702 | 4 | 3.235 |
| Voice agent | 2.800 | 7 | 3.734 |
The ranking changes. With hang-ups counted, the IVR is the cheapest, the hybrid is close, and the agent is the most expensive. Nothing in the data changed: one cost that the table did not show changed the answer.
Two more things to check
Section titled “Two more things to check”The wrong actions. The IVR’s 2 wrong actions are the two French calls where Whisper base heard “quelqu’un” (someone). The IVR put those callers through to a person. The command charges US$15 each, as for a return started by mistake. A wrong hand-over costs less than that in real life. So the IVR’s cost is, if anything, a little too high.
The time. The agent’s mean call is the shortest: 24.3 s. But many of its calls are short because it handed over early. The hybrid’s calls are the longest, 51.9 s: a caller who reaches the agent has already spent the menu’s time. The two clocks are also made in different ways. The IVR’s time comes from its replay, with the real length of each Piper prompt. The agent’s time is a sum: the caller’s audio, each wait, each spoken answer, and a fixed 4 s for its greeting. Compare the times only roughly. And the IVR’s 45.1 s includes extra retries from a known bug in the replay: after a key press, it treats the rest of the recording as a new, silent turn. Test the IVR like a product explains it. With the fix, the IVR’s mean is about 35.8 s, and its passes and containment do not change.
How sure is the choice?
Section titled “How sure is the choice?”The costs are assumptions, so test them. Change the cost of a person, and keep the hang-ups at US$8:
| Cost of a person | IVR | Hybrid | Agent |
|---|---|---|---|
| US$3.00 | 2.54 | 2.69 | 3.03 |
| US$4.00 | 3.17 | 3.24 | 3.73 |
| US$6.00 | 4.44 | 4.34 | 5.13 |
The hybrid sends 5 fewer calls to a person, but leaves 3 more callers with no answer. The two cost the same when a person costs about US$4.80 a call. Above that, the hybrid is cheaper. And the difference in containment is one call in 60: within noise. On 900 calls a month, the IVR and the hybrid differ by about US$60 a month with hang-ups counted. The recorded calls cannot separate these two designs. Only callers who react can.
The agent’s wait in the hybrid
Section titled “The agent’s wait in the hybrid”In the hybrid, a caller reaches the agent only after the menu failed twice. That caller has already waited, and may already be unhappy. Each of the agent’s answers then comes after a silence.
Explore
The agent's wait, for a caller who reaches it from the menu
Predict first: A caller reaches the agent after two menu misses. With the fastest settings you can choose, how long is the silence before each answer: under 1 s, about 1.4 s, or about 2 s?
- End-of-turn wait: 0.80 s
- Recognition (Whisper base): 0.24 s
- LLM, first words: 1.02 s
- Voice, first sound: 0.05 s
- Total wait
- 2.11 s
- Against the 1 s target
- 1.11 s over
End-of-turn wait 0.8 s, Whisper base, streaming: the parts add up to 2.11 s, 1.11 s over the 1 s target. The largest part is "LLM, first words" (1.02 s). Even with no wait and no recognition at all, the LLM and the voice alone take 1.06 s: more than the target.
| Part | Median | 90th percentile | Range | Source |
|---|---|---|---|---|
| End-of-turn wait | 0.80 s | 0.80 s | fixed | a setting |
| Recognition (Whisper base) | 0.24 s | 0.30 s | 0.23 to 1.01 s | measured on the course laptop |
| LLM, first words | 1.02 s | 1.49 s | 0.69 to 2.45 s | recorded (chat-small) |
| Voice, first sound | 0.05 s | 0.08 s | 0.01 to 0.76 s | measured on the course laptop |
Text description of the explorer
A stacked bar from 0 to 3 seconds, with a dashed line at the 1-second target. With the starting settings (end-of-turn wait 0.8 s, Whisper base, streaming on), the parts add up to 2.11 s: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first words 1.02 s, voice first sound 0.05 s. The fastest settings in the explorer (a wait of 0.2 s, Whisper tiny, streaming) give 1.38 s. The previous lesson showed that a wait under 0.7 s cuts more callers off. With a wait of 0.7 s and Whisper tiny, the sum is 1.88 s. The LLM’s first words, 1.02 s, are over the target on their own.
Even with the fastest settings, the caller waits about 1.4 s before each answer. With the wait that does not cut callers off, 0.7 s, it is about 1.9 s. The IVR answers after about 1.05 s, because its prompts are made in advance. So the hybrid gives the slowest answers to the callers who already had trouble. That is a risk to test with real callers.
Larkfield’s choice
Section titled “Larkfield’s choice”Omar asks for one recommendation for the out-of-hours line. Priya writes it with the numbers, all on Larkfield’s synthetic calls:
| Design | For | Against | Main risk |
|---|---|---|---|
| IVR (menu) | 39 of 60 passed; cheapest with hang-ups (US$3.169); answers in about 1.05 s | contains only 35%; fails 15 French calls; hung up on one silent caller | French callers sent to a person, or worse, to silence |
| Voice agent | 0 wrong actions; every must-go-to-a-person call passed | 25 of 60 passed; 8 calls with no end; 2.10 s per answer | slow answers, and mistakes said aloud |
| Hybrid | most contained (37%) and fewest to a person (55%) | the agent contains 1 of the 16 calls it gets; 4 callers left with no answer | the slowest answers for callers who already had trouble |
Recommendation: keep the IVR for the out-of-hours line now, and run the hybrid as a measured pilot.
- Fix the IVR first. Hand over a silent caller instead of hanging up. Work on French recognition, which causes most failures. Accents, noise and two languages measured a French WER of 15.1% for Whisper small against 23.8% for base. Each fix helps every design, because the hybrid starts with the menu.
- Pilot the hybrid on a share of real out-of-hours calls. Keep the guardrails in code, and hand over when the agent has no answer. Measure the five measures of the previous lesson, with hang-ups counted.
- Decide again with the pilot. The recorded calls cannot separate the IVR from the hybrid. The hybrid may win if a person costs more than about US$4.80 a call. It may also win if real callers fail the menu more often than the recorded ones.
Tomás is not happy, but he agrees with the method. The agent did well where it mattered most: no wrong action, and every caller who needed a person got one. It needs callers who can answer its questions.
Common mistake
The cheapest line in the table
Symptom: Tomás reads the command’s table. He writes “the hybrid saves US$0.33 a call against the IVR: US$300 a month on 900 calls”. Omar almost signs.
IVR (menu) 60 35% 63% 2 45.1 3.035hybrid 60 37% 55% 2 51.9 2.702Cause: the cost counts a call handed to a person, but not a call that ended with no answer. In the hybrid, the agent left 4 callers with no answer in calls where the IVR had handed them to a person. Those calls became “free”.
Fix: list how every call ended, not only the contained ones and the hand-overs. Give every ending a cost, including hang-ups. With hang-ups at US$8, the hybrid costs US$3.235 a call and the IVR US$3.169.
Consequence in AI work: a system that fails in a new way can look cheaper. The old cost table has no line for the new failure. When you add a design, check that every way it can end has a cost.
Guided practice: the designs in the notebook
Section titled “Guided practice: the designs in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The setup cell ends with
Ready: 4 files in larkfield-calls. - In section 2, run the three designs. The table matches the command, with one more column:
no end, which is 1, 7 and 4. - In section 3, run the cell for the hybrid. 16 calls go to the agent: 11 to a person, 3 with no end, 1 that the agent ended with no answer, and 1 with the order status.
- In section 4, read the parts of the cost. People are almost all of it.
- In section 6, change the cost of a person. At US$6, the hybrid becomes the cheapest.
Your turn: the recommendation
Section titled “Your turn: the recommendation”In section 5 of the notebook, count each call with no end at the cost of a hang-up, and find the cheapest design. Then write Omar’s recommendation in five lines. Give the design, two numbers for it, one number against it, its main risk, and what would change your mind.
Show the answer
with_hangups = {name: round(table(rows, costs, hangup_cost=costs["caller_hangs_up"])["usd"], 3) for name, rows in D.items()}cheapest = min(with_hangups, key=with_hangups.get)print(with_hangups, "->", cheapest){'IVR (menu)': 3.169, 'voice agent': 3.734, 'hybrid': 3.235} -> IVR (menu)A good recommendation, for example:
- Design: the IVR now, with the hybrid as a measured pilot.
- For: it passed 39 of 60 test calls, and it is the cheapest with hang-ups counted (US$3.169 a call against US$3.235 for the hybrid).
- Against: it contains only 35% of calls, and 15 of its 21 failures are French calls.
- Main risk: French callers sent to a person, or to silence: fix the silent-caller hang-up first.
- What would change my mind: a cost of a person above about US$4.80. Or a pilot with real callers where the hybrid contains clearly more calls, with no more hang-ups.
Knowledge check
Section titled “Knowledge check”Check your understanding
Compute the hybrid's cost per call from its parts: 33 calls to a person at US$4, 2 wrong actions at US$15, and US$0.123 of compute and LLM for all 60 calls. Then add 4 hang-ups at US$8.
Show the answer
People: 33 × 4 = US$132. Wrong actions: 2 × 15 = US$30. Compute and LLM: US$0.123. The sum is US$162.123, so US$162.123 / 60 = about US$2.702 a call, as the command prints. Hang-ups: 4 × 8 = US$32, or US$0.533 a call. With them, the hybrid costs about US$3.235 a call, more than the IVR’s US$3.169.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The hybrid contains 37% of the test calls and the IVR 35%. Is the hybrid better at containment?
Show the answer
Not clearly. The difference is 22 calls against 21: one call in 60. One call can change with the next recording, or with one recognition error. And the hybrid gets its one extra call with 4 callers left with no answer. To separate the two designs, you need more calls, or a pilot with real callers who can answer the agent’s questions.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why are the agent's calls the shortest on average (24.3 s), and why is that not a reason to choose it?
Show the answer
Many of the agent’s calls end early: it handed 42 of 60 calls to a person, often after a few turns. A short call that ends with a hand-over is not a good call for the caller. The two clocks are also made differently. The IVR’s time comes from its replay. The agent’s time is a sum of the caller’s audio, the waits, the spoken answers and a fixed greeting. Compare outcomes and costs first, and the times only roughly.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A hybrid puts the menu first and sends a call to the voice agent only after two misses.
- On the 60 recorded test calls: IVR 35% contained, US$3.035 a call; agent 17%, US$2.800; hybrid 37%, US$2.702. Costs are assumptions; agent answers are recorded.
- The command’s cost leaves out the calls that ended with no answer. Counted at US$8 a hang-up: IVR US$3.169, hybrid US$3.235, agent US$3.734. The ranking changes.
- These calls cannot separate the IVR from the hybrid. They differ by one contained call, and the cost order flips at about US$4.80 for a person.
- Larkfield keeps the IVR, fixes its French recognition and its silent-caller hang-up, and pilots the hybrid with real callers.
You can now build a phone line, a voice agent and a hybrid, and choose between them with evidence. But everything so far ran on a laptop, one call at a time. Next, you run speech as a service. You use a queue for recorded calls and a streaming path for live ones, and you find the cost of each per audio hour.
Next: Batch and streaming