Skip to content
Français

Menu, agent or both?

  • Lesson 32
  • 40 min
  • Module 8: From IVR to voice agent

Where we are

We have an IVR and a voice agent, each tested on its own, and a fair way to judge them. Omar must choose one for the out-of-hours line. In this lesson, we run the IVR, the agent and a hybrid on the same recorded calls and choose with the numbers.

What you will be able to do

Choose a design for Larkfield’s out-of-hours phone line: the menu, the voice agent, or both together. Compare them on the same recorded calls, count every cost, including the ones a table leaves out, and write the recommendation with its risks.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It builds the three designs from the two recordings and splits the cost per call into its parts. It counts the calls that end with no answer, and changes the cost assumptions. It needs no model download and no account. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • The five measures of a voice agent, and the calls that must go to a person: Speech-to-speech models and evaluation.
  • What containment is: the share of calls that the system finishes itself, with no person.

What you need on your computer (only for the local route; the notebook needs none of it)

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.

Think of a supermarket. A self-checkout is fast for a basket of simple things. It fails on a bag of loose apples or a bottle of wine, and then it calls a member of staff. A staffed till handles everything, but it costs a person for every customer. Most shops do both: the machine first, a person when the machine is stuck.

Larkfield has the same three choices for its phone line out of hours. It can use the menu, the voice agent, or the menu first and the agent when the menu is stuck. About 900 calls a month come out of hours.

The problem: two designs, two separate tests

Section titled “The problem: two designs, two separate tests”

Amira tested the IVR in Module 7. The course team then ran the voice agent on the same calls. Tomás says that the agent is newer, so it must be better. Amira says that the menu passed more test calls. Both look at their own numbers.

Omar needs one table: the same calls, the same rules, the same costs. And he needs the third design that nobody has tested yet.

A hybrid design puts the menu first. The menu handles what it can. When it fails twice to match the caller’s words, the call goes to the voice agent instead of a person. A caller who asks for a person, or presses 0, still gets a person at once.

A flow. A call goes to the IVR menu, with speech and keypad. With a menu item, the IVR does the task: status, hours or a return, and the call ends. With 0, or when the caller asks for a person, the call goes to a person. With no match twice, the call goes to the voice agent, an LLM with tools and the same guardrails. If the task is done, the call ends. If the agent cannot help, the call goes to a person.
The hybrid: the menu first; the voice agent only after the menu fails twice to match; a person whenever the caller asks.
Text description of the diagram

A call starts at the IVR menu, which understands speech and the keypad. If the caller chooses a menu item, the IVR does the task: order status, opening hours or a return, and the call ends. If the caller presses 0 or asks for a person, the call goes to a person. If the menu does not match the caller’s words twice, the call goes to the voice agent, an LLM with tools and the same guardrails. The agent ends the call when the task is done. When it cannot help, it puts the caller through to a person.

To compare the three, use the same recorded test calls and one set of rules:

  • Contained: the call ended with the answer (status, hours or a return started), with no wrong action and no person.
  • To a person: the call was handed over.
  • Wrong actions: the design acted on a wrong intent or a wrong order.
  • Cost per call: people, wrong actions and compute, from Grace’s cost table. These are assumptions: a call handled by a person US$4.00, a wrong action US$15, IVR compute US$0.002 a call. The LLM cost is the recorded cost of each agent call.

You need no new file: speech agent compare came with the snapshot end-of-m08 in The voice-agent pipeline.

  1. In your folder larkspeech, with the virtual environment active, run the comparison:

    Terminal window
    speech agent compare
    design calls contained to a person wrong actions mean s US$ a call
    IVR (menu) 60 35% 63% 2 45.1 3.035
    voice agent 60 17% 70% 0 24.3 2.800
    hybrid 60 37% 55% 2 51.9 2.702
    Costs are assumptions (data/costs.json); agent answers and times are recorded (chat-small).

    It uses the IVR’s recorded replay and the recorded agent calls, so it takes about a second.

    The IVR’s recorded replay was made once, with the first version of dialogue.py, before Module 7’s last changes. In it, two French calls (I-046 and I-051) went to a person for a wrong reason. The IVR heard a request for a person, but the caller wanted an order status. Those are its 2 wrong actions. When we replayed the calls again with the current project, Whisper heard those turns differently, and no call had a wrong intent. To compare your own IVR, run speech ivr replay first, then speech agent compare --ivr-records outputs/ivr-replay.jsonl. On our run, the IVR then cost US$2.535 a call and the hybrid US$2.202, with the same containment. The ranking with hang-ups, later in this lesson, does not change.

  2. The command also writes the table to outputs/designs.jsonl, one design a line. Look at the first line:

    Terminal window
    head -1 outputs/designs.jsonl
    {"design": "IVR (menu)", "calls": 60, "containment": 0.35, "handover": 0.6333333333333333, "wrong_actions": 2, "mean_seconds": 45.07233333333333, "usd_per_call": 3.0353333333333334}

    The file keeps every digit. The table rounds them.

On this table, the hybrid wins: the most calls contained, the fewest to a person, and the lowest cost. Tomás’s agent alone is the worst at containment. Before Omar decides, Priya checks where each number comes from.

A worked example: the IVR’s cost per call

Section titled “A worked example: the IVR’s cost per call”

Every cost per call is the sum of three parts, divided by the 60 calls:

Part Count Cost each Total
Calls to a person 38 US$4.00 US$152.00
Wrong actions 2 US$15.00 US$30.00
IVR compute 60 US$0.002 US$0.12
All 60 calls US$182.12

US$182.12 / 60 = US$3.035 a call. The 38 calls to a person include the 2 wrong actions, because both wrong actions were hand-overs (more on them below). So those 2 calls pay for a person and for a wrong action. People are 83% of the cost. The LLM costs of the agent are even smaller than the IVR’s compute: US$0.011 for all 60 calls. So the design that wins on cost is the one that sends the fewest calls to a person.

Three stacked horizontal bars of 60 calls each. IVR (menu): contained 21, to a person 36, no end 1, wrong action 2. Voice agent: contained 10, to a person 42, no end 8. Hybrid: contained 22, to a person 31, no end 5, wrong action 2.
Where each of the 60 recorded test calls ended, for each design (Larkfield's synthetic calls). "No end" is a call that ended with no answer and no person.
Text description of the diagram

Each bar is one design, split into the 60 test calls. The IVR contained 21 calls, handed 36 to a person, made 2 wrong actions and left 1 call with no end. The voice agent contained 10, handed 42 to a person and left 8 with no end. The hybrid contained 22, handed 31 to a person, made 2 wrong actions and left 5 with no end.

Look at the orange part: no end. These calls ended with no answer and no person. In the agent’s replay, the recorded caller ran out of lines while the agent was still asking a question. A real caller would hang up, or call again. The IVR has 1 such call: the silent caller of the previous lesson. The agent has 8 (7 with no end and 1 that it ended with no answer), and the hybrid 5.

Calls with no end IVR Voice agent Hybrid
The caller hung up or ran out of lines (counted as a hang-up below) 1 7 4
The agent ended the call with no answer 0 1 1
All “no end” calls in the chart 1 8 5

Now look at the hybrid. Of the 16 calls that the IVR handed over after two misses, the agent contained only 1. It handed 11 to a person, and left 4 with no end. So the hybrid saves 5 hand-overs, but 4 of those 5 callers got no answer at all.

The command’s cost counts people, wrong actions and compute. It does not count a caller who hangs up. Grace’s cost table has a number for that: US$8 for a caller who hangs up. It is an assumption: a second call, and an unhappy customer. Count every call that ended with a hang-up at US$8:

Three stacked horizontal bars of cost per call. IVR: people 2.53, wrong actions 0.50, compute 0.002, total 3.035, plus hang-ups 0.133. Voice agent: people 2.80, total 2.800, plus hang-ups 0.933. Hybrid: people 2.20, wrong actions 0.50, compute 0.002, total 2.702, plus hang-ups 0.533. The hang-up parts are hatched and labelled not in the command.
The cost per call of each design in parts, with the hang-ups that the command does not count (US$8 each, an assumption). With hang-ups: IVR US$3.169, hybrid US$3.235, agent US$3.734.
Text description of the diagram

Each bar is the cost per call of one design. The solid parts are what the command counts: people, wrong actions, and compute with the LLM. The hatched part is the cost of hang-ups, which the command leaves out. IVR: people US$2.533, wrong actions US$0.500, compute US$0.002, total US$3.035; hang-ups add US$0.133 (1 call). Voice agent: people US$2.800, total US$2.800; hang-ups add US$0.933 (7 calls). Hybrid: people US$2.200, wrong actions US$0.500, compute US$0.002, total US$2.702; hang-ups add US$0.533 (4 calls).

Design US$ a call (the command) Hang-ups US$ a call with hang-ups
IVR (menu) 3.035 1 3.169
Hybrid 2.702 4 3.235
Voice agent 2.800 7 3.734

The ranking changes. With hang-ups counted, the IVR is the cheapest, the hybrid is close, and the agent is the most expensive. Nothing in the data changed: one cost that the table did not show changed the answer.

The wrong actions. The IVR’s 2 wrong actions are the two French calls where Whisper base heard “quelqu’un” (someone). The IVR put those callers through to a person. The command charges US$15 each, as for a return started by mistake. A wrong hand-over costs less than that in real life. So the IVR’s cost is, if anything, a little too high.

The time. The agent’s mean call is the shortest: 24.3 s. But many of its calls are short because it handed over early. The hybrid’s calls are the longest, 51.9 s: a caller who reaches the agent has already spent the menu’s time. The two clocks are also made in different ways. The IVR’s time comes from its replay, with the real length of each Piper prompt. The agent’s time is a sum: the caller’s audio, each wait, each spoken answer, and a fixed 4 s for its greeting. Compare the times only roughly. And the IVR’s 45.1 s includes extra retries from a known bug in the replay: after a key press, it treats the rest of the recording as a new, silent turn. Test the IVR like a product explains it. With the fix, the IVR’s mean is about 35.8 s, and its passes and containment do not change.

The costs are assumptions, so test them. Change the cost of a person, and keep the hang-ups at US$8:

Cost of a person IVR Hybrid Agent
US$3.00 2.54 2.69 3.03
US$4.00 3.17 3.24 3.73
US$6.00 4.44 4.34 5.13

The hybrid sends 5 fewer calls to a person, but leaves 3 more callers with no answer. The two cost the same when a person costs about US$4.80 a call. Above that, the hybrid is cheaper. And the difference in containment is one call in 60: within noise. On 900 calls a month, the IVR and the hybrid differ by about US$60 a month with hang-ups counted. The recorded calls cannot separate these two designs. Only callers who react can.

In the hybrid, a caller reaches the agent only after the menu failed twice. That caller has already waited, and may already be unhappy. Each of the agent’s answers then comes after a silence.

Explore

The agent's wait, for a caller who reaches it from the menu

  • End-of-turn wait: 0.80 s
  • Recognition (Whisper base): 0.24 s
  • LLM, first words: 1.02 s
  • Voice, first sound: 0.05 s
Total wait
2.11 s
Against the 1 s target
1.11 s over

End-of-turn wait 0.8 s, Whisper base, streaming: the parts add up to 2.11 s, 1.11 s over the 1 s target. The largest part is "LLM, first words" (1.02 s). Even with no wait and no recognition at all, the LLM and the voice alone take 1.06 s: more than the target.

Each part over the 127 recorded agent turns
PartMedian90th percentileRangeSource
End-of-turn wait0.80 s0.80 sfixeda setting
Recognition (Whisper base)0.24 s0.30 s0.23 to 1.01 smeasured on the course laptop
LLM, first words1.02 s1.49 s0.69 to 2.45 srecorded (chat-small)
Voice, first sound0.05 s0.08 s0.01 to 0.76 smeasured on the course laptop
The same latency budget as in The voice-agent pipeline: the median of each part over the 127 recorded agent answers (Larkfield's synthetic calls).
Text description of the explorer

A stacked bar from 0 to 3 seconds, with a dashed line at the 1-second target. With the starting settings (end-of-turn wait 0.8 s, Whisper base, streaming on), the parts add up to 2.11 s: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first words 1.02 s, voice first sound 0.05 s. The fastest settings in the explorer (a wait of 0.2 s, Whisper tiny, streaming) give 1.38 s. The previous lesson showed that a wait under 0.7 s cuts more callers off. With a wait of 0.7 s and Whisper tiny, the sum is 1.88 s. The LLM’s first words, 1.02 s, are over the target on their own.

Even with the fastest settings, the caller waits about 1.4 s before each answer. With the wait that does not cut callers off, 0.7 s, it is about 1.9 s. The IVR answers after about 1.05 s, because its prompts are made in advance. So the hybrid gives the slowest answers to the callers who already had trouble. That is a risk to test with real callers.

Omar asks for one recommendation for the out-of-hours line. Priya writes it with the numbers, all on Larkfield’s synthetic calls:

Design For Against Main risk
IVR (menu) 39 of 60 passed; cheapest with hang-ups (US$3.169); answers in about 1.05 s contains only 35%; fails 15 French calls; hung up on one silent caller French callers sent to a person, or worse, to silence
Voice agent 0 wrong actions; every must-go-to-a-person call passed 25 of 60 passed; 8 calls with no end; 2.10 s per answer slow answers, and mistakes said aloud
Hybrid most contained (37%) and fewest to a person (55%) the agent contains 1 of the 16 calls it gets; 4 callers left with no answer the slowest answers for callers who already had trouble

Recommendation: keep the IVR for the out-of-hours line now, and run the hybrid as a measured pilot.

  1. Fix the IVR first. Hand over a silent caller instead of hanging up. Work on French recognition, which causes most failures. Accents, noise and two languages measured a French WER of 15.1% for Whisper small against 23.8% for base. Each fix helps every design, because the hybrid starts with the menu.
  2. Pilot the hybrid on a share of real out-of-hours calls. Keep the guardrails in code, and hand over when the agent has no answer. Measure the five measures of the previous lesson, with hang-ups counted.
  3. Decide again with the pilot. The recorded calls cannot separate the IVR from the hybrid. The hybrid may win if a person costs more than about US$4.80 a call. It may also win if real callers fail the menu more often than the recorded ones.

Tomás is not happy, but he agrees with the method. The agent did well where it mattered most: no wrong action, and every caller who needed a person got one. It needs callers who can answer its questions.

Common mistake

The cheapest line in the table

Symptom: Tomás reads the command’s table. He writes “the hybrid saves US$0.33 a call against the IVR: US$300 a month on 900 calls”. Omar almost signs.

IVR (menu) 60 35% 63% 2 45.1 3.035
hybrid 60 37% 55% 2 51.9 2.702

Cause: the cost counts a call handed to a person, but not a call that ended with no answer. In the hybrid, the agent left 4 callers with no answer in calls where the IVR had handed them to a person. Those calls became “free”.

Fix: list how every call ended, not only the contained ones and the hand-overs. Give every ending a cost, including hang-ups. With hang-ups at US$8, the hybrid costs US$3.235 a call and the IVR US$3.169.

Consequence in AI work: a system that fails in a new way can look cheaper. The old cost table has no line for the new failure. When you add a design, check that every way it can end has a cost.

Guided practice: the designs in the notebook

Section titled “Guided practice: the designs in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The setup cell ends with Ready: 4 files in larkfield-calls.
  3. In section 2, run the three designs. The table matches the command, with one more column: no end, which is 1, 7 and 4.
  4. In section 3, run the cell for the hybrid. 16 calls go to the agent: 11 to a person, 3 with no end, 1 that the agent ended with no answer, and 1 with the order status.
  5. In section 4, read the parts of the cost. People are almost all of it.
  6. In section 6, change the cost of a person. At US$6, the hybrid becomes the cheapest.

In section 5 of the notebook, count each call with no end at the cost of a hang-up, and find the cheapest design. Then write Omar’s recommendation in five lines. Give the design, two numbers for it, one number against it, its main risk, and what would change your mind.

Show the answer
with_hangups = {name: round(table(rows, costs, hangup_cost=costs["caller_hangs_up"])["usd"], 3)
for name, rows in D.items()}
cheapest = min(with_hangups, key=with_hangups.get)
print(with_hangups, "->", cheapest)
{'IVR (menu)': 3.169, 'voice agent': 3.734, 'hybrid': 3.235} -> IVR (menu)

A good recommendation, for example:

  • Design: the IVR now, with the hybrid as a measured pilot.
  • For: it passed 39 of 60 test calls, and it is the cheapest with hang-ups counted (US$3.169 a call against US$3.235 for the hybrid).
  • Against: it contains only 35% of calls, and 15 of its 21 failures are French calls.
  • Main risk: French callers sent to a person, or to silence: fix the silent-caller hang-up first.
  • What would change my mind: a cost of a person above about US$4.80. Or a pilot with real callers where the hybrid contains clearly more calls, with no more hang-ups.

Check your understanding

Compute the hybrid's cost per call from its parts: 33 calls to a person at US$4, 2 wrong actions at US$15, and US$0.123 of compute and LLM for all 60 calls. Then add 4 hang-ups at US$8.

Show the answer

People: 33 × 4 = US$132. Wrong actions: 2 × 15 = US$30. Compute and LLM: US$0.123. The sum is US$162.123, so US$162.123 / 60 = about US$2.702 a call, as the command prints. Hang-ups: 4 × 8 = US$32, or US$0.533 a call. With them, the hybrid costs about US$3.235 a call, more than the IVR’s US$3.169.

Check your understanding

The hybrid contains 37% of the test calls and the IVR 35%. Is the hybrid better at containment?

Show the answer

Not clearly. The difference is 22 calls against 21: one call in 60. One call can change with the next recording, or with one recognition error. And the hybrid gets its one extra call with 4 callers left with no answer. To separate the two designs, you need more calls, or a pilot with real callers who can answer the agent’s questions.

Check your understanding

Why are the agent's calls the shortest on average (24.3 s), and why is that not a reason to choose it?

Show the answer

Many of the agent’s calls end early: it handed 42 of 60 calls to a person, often after a few turns. A short call that ends with a hand-over is not a good call for the caller. The two clocks are also made differently. The IVR’s time comes from its replay. The agent’s time is a sum of the caller’s audio, the waits, the spoken answers and a fixed greeting. Compare outcomes and costs first, and the times only roughly.

Recap

  • A hybrid puts the menu first and sends a call to the voice agent only after two misses.
  • On the 60 recorded test calls: IVR 35% contained, US$3.035 a call; agent 17%, US$2.800; hybrid 37%, US$2.702. Costs are assumptions; agent answers are recorded.
  • The command’s cost leaves out the calls that ended with no answer. Counted at US$8 a hang-up: IVR US$3.169, hybrid US$3.235, agent US$3.734. The ranking changes.
  • These calls cannot separate the IVR from the hybrid. They differ by one contained call, and the cost order flips at about US$4.80 for a person.
  • Larkfield keeps the IVR, fixes its French recognition and its silent-caller hang-up, and pilots the hybrid with real callers.

You can now build a phone line, a voice agent and a hybrid, and choose between them with evidence. But everything so far ran on a laptop, one call at a time. Next, you run speech as a service. You use a queue for recorded calls and a streaming path for live ones, and you find the cost of each per audio hour.

Next: Batch and streaming