Skip to content
Français

Speech-to-speech models and evaluation

  • Lesson 31
  • 30 min
  • Module 8: From IVR to voice agent

Where we are

We built a voice agent from separate parts and learned to set its waits. Newer models take speech in and give speech out directly. In this lesson, we see what they change, and we learn to evaluate a voice agent as a whole, not one part at a time.

Lesson 31 of 36 · Speech AI: Transcribe, Understand and SpeakSpeech-to-speech models and evaluation

What you will be able to do

Know what a speech-to-speech model changes in a voice agent, and what it makes harder. Then judge any voice agent as a whole, with a short list of measures that Larkfield can compare from one design to the next.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer

Nothing. This lesson reads the recorded results of the IVR and of the agent.

Think of a meeting between two people who speak different languages. You can help them in two ways.

The first way is a chain. A note-taker writes down what the first person says. A translator translates the notes. A reader reads the translation aloud. It is slow, and the reader’s voice does not carry the first person’s tone. But you have the notes and the translation on paper. You can check them afterwards.

The second way is an interpreter who listens and speaks at once. It is fast, and a good interpreter keeps the tone: a joke still sounds like a joke. But nothing is on paper. If something goes wrong, you cannot check what was heard and what was said.

The problem: the chain is slow and loses the tone

Section titled “The problem: the chain is slow and loses the tone”

The voice agent of this module is the chain. In speech work, it is called a cascade: recognition, then an LLM, then a voice, one after the other. The previous two lessons measured its limits on Larkfield’s recorded calls:

  • Time. Each step waits for the one before. The median wait was 2.10 s, and the LLM’s first token alone took about 1 s.
  • Errors carry on. When Whisper base heard “189 710” for 187971, the LLM read the wrong number back, politely and confidently.
  • The tone is lost. The LLM reads only text. It cannot hear that a caller sounds angry, unsure or in a hurry.

A speech-to-speech model takes audio in and gives audio out, with one model in the middle. It is like the interpreter. Some vendors call these models “realtime” or “native audio” models. Some can also call tools, as the LLM in our cascade does.

Two flows. Cascade, the pipeline of this module: speech in, recognition, text that you can log, check and redact, LLM with tools, text, voice, speech out. Speech-to-speech model: speech in, one model that hears the words and the tone, calls tools and speaks, speech out. A dashed line from the one model goes to text, only if the model also writes it.
A cascade keeps text between its steps. A speech-to-speech model goes from audio to audio, and gives text only if it also writes a transcript.
Text description of the diagram

One flow is the cascade of this module. Speech goes into recognition, which writes text. You can log, check and redact this text. The LLM with tools reads it and writes an answer as text. The voice turns that text into speech. The other flow is a speech-to-speech model. Speech goes into one model. That model hears the words and the tone, can call tools, and speaks the answer. A dashed line shows that text comes out only if the model also writes a transcript.

Vendors make three claims for these models. They are vendor claims: the course did not run or measure any speech-to-speech model.

  • Less waiting. One model replaces three handoffs, and it can start to answer while the caller is still finishing.
  • The tone is kept. The model hears hesitation and anger, and its voice can sound calm or warm.
  • Turn-taking in the model. Some models decide the end of the turn and handle a barge-in themselves.

Each of these is a real cost for Larkfield, and each comes from the missing text in the middle:

  • Checks and logs. Grace’s team reads call records. Module 9 removes card numbers from transcripts. With no text in the middle, you need the model to write a transcript too, and you must trust that it matches the audio.
  • Guardrails. The read-back rule needs the order number as text, so that code can check it. Make the model call a tool with the number. Then the guardrail in code still works.
  • Measurement. You cannot score the recognition step alone, because there is none. You can only score the whole call.
  • Privacy and price. The caller’s audio goes to the vendor, not only the text. Hosted audio models are priced by the amount of audio in and out. Prices change: check the current page.

So a speech-to-speech model is a different trade, not a free upgrade. To choose, you need one way to judge any voice agent, whatever is inside it.

A part can be good while the whole is bad. Whisper base has a WER of 5.2% on the English test calls. But the agent still passed only 2 of 12 “order status, speech” calls. So judge the whole call, from the caller’s side. Five measures cover it:

Measure The question How to measure it on recorded calls
Task success Did the call end the way it should? the expected ending of each test call: pass or fail, by kind
Latency How long is the silence after the caller stops? the median and the 90th percentile of the wait per turn
Interruptions handled When the caller talks over the agent, does it stop and listen? needs a caller who interrupts: not possible with a fixed replay
Wrong actions Did it do something that nobody asked for? tool calls against the expected order and intent
Must go to a person Does every caller who needs a person get one? the calls whose right ending is a hand-over: all of them must pass

The last measure is not an average. It is a gate: a design that fails one of these calls is not ready, whatever its other scores. In Larkfield’s test set, the gate has 13 calls of four kinds: asks for a person, presses 0 again, unclear request, and a caller who says nothing. The six return calls also end with a person, because their orders cannot be returned. They are not in the gate: those callers asked for a return, not for a person. The agents course uses the same idea for actions that need a person’s approval: Approval boundaries and Task success measures.

A worked example: the IVR and the agent, scored

Section titled “A worked example: the IVR and the agent, scored”

Here are the two designs of this module on the same 60 recorded test calls. All numbers are on Larkfield’s synthetic calls.

Measure IVR (menu) Voice agent (recorded, chat-small)
Task success 39 of 60 passed 25 of 60 passed
Latency after a spoken turn (median) about 1.05 s: the 0.8 s wait and recognition; its prompts are made in advance 2.10 s (90th percentile 2.74 s)
Interruptions handled does not listen while it speaks not tested: the replay’s callers never interrupt
Wrong actions 2: twice it heard “quelqu’un” (someone) and put a caller through to a person (in the recorded replay of the first dialogue.py; the current IVR made none) 0
Must go to a person (13 calls) 12 of 13: it hung up on one silent caller 13 of 13

Read the table row by row. The IVR wins on task success and on latency. The agent wins on wrong actions and on the gate: every caller who needed a person got one. Neither design was tested on interruptions. A score that adds these rows into one number would hide all of this.

The total hides where each design wins. The chart splits the passes by kind of test call.

Pairs of horizontal bars, one pair for each of 13 kinds of test call: the IVR in blue, the voice agent in orange. The agent does worse on order status by keypad (IVR 6 of 6, agent 0 of 6), hours by keypad (3 of 3 against 0 of 3), return by speech (6 of 6 against 4 of 6), wrong number then fixed (1 of 3 against 0 of 3), number with a pause (1 of 4 against 0 of 4), noisy line then keypad (2 of 4 against 1 of 4) and hours by speech (6 of 6 against 5 of 6). Both pass 2 of 12 order-status calls by speech and 0 of 3 change-of-mind calls. Both pass all calls that ask for a person, press 0 again or make an unclear request. On a silent caller, the IVR passes 2 of 3 and the agent 3 of 3.
Passes by kind of test call, the IVR (39 of 60) and the recorded voice agent (25 of 60), on the same 60 calls. Sorted from the agent's biggest loss to its only gain.
Text description of the diagram

Each kind of test call has two bars: the IVR on top in blue, the agent below in orange. At the top are the kinds where the agent does worse. With the keypad, it passes 0 of 6 order-status calls and 0 of 3 opening-hours calls, where the IVR passes all of them. It also does worse on returns by speech, a wrong number that the caller fixes, a number with a pause, a noisy line and hours by speech. In the middle, both designs score the same: 2 of 12 order-status calls by speech and 0 of 3 changes of mind. Both pass every caller who asks for a person, presses 0 again or makes an unclear request. At the bottom, the agent does better on one kind: a silent caller, 3 of 3 against 2 of 3.

Look at the keypad calls again: the agent passed 0 of 9. The reason is not the LLM. The recorded caller presses 1 or 2 at the start, because the IVR’s greeting asks for it. The agent never offered a keypad, so “the caller pressed 1” meant nothing to it. The replay is unfair to the agent here, as it is unfair to the IVR when a recorded caller cannot answer a new question.

A fairer test needs a caller who reacts. Teams use two ways:

  • People call the agent with a list of tasks: “find the status of order 742155”. It is slow and costs people’s time, but it is real.
  • A simulated caller. A second LLM plays the caller. It gets a goal and a script, and it answers the agent’s real questions. It is cheap and repeatable, but it is another model to test. A caller that is too patient or too clever gives results that are too good.

Use the recorded replay for what it does well: the same calls, every time, for every change. Then check the decision with reacting callers before the agent goes live.

Try one when the latency budget cannot be met with the cascade, and when the tone of voice matters: for example, callers who are upset. Measure it with the five measures above, on the same test calls.

Keep the cascade when you need the text: for call records, redaction of card numbers, checks in code, and tests of each step. Larkfield needs all four, so it keeps the cascade for now. If it tries a speech-to-speech model later, it makes the model call tools for every action and write a transcript.

Common mistake

Containment as the only score

Symptom: Tomás reports one number for each design: containment. The IVR contains 35%, the hybrid of the next lesson 37%. Nobody notices that the IVR hung up on a silent caller.

I-060 caller says nothing expected: hand over (no input) IVR: hung up

Cause: containment counts the calls that the system finished itself. A hang-up is not contained, so it lowers containment a little, but it looks the same as any other call that is not contained. A caller who needed a person and got silence is a much worse result than a hand-over.

Fix: keep the must-go-to-a-person calls as a gate, and report them apart: “12 of 13, failed: I-060”. Then report task success by kind, latency and wrong actions next to containment.

Consequence in AI work: one number makes a decision easy and wrong. A score of averages lets a system pass while it fails the callers who need help most.

Use the chart and the tables above. No code is needed.

  1. Copy the five measures into a table with three columns: the measure, the IVR, the agent.
  2. Fill in task success and latency from the worked example.
  3. From the chart, add the passes of the four kinds that must go to a person. They are: asks for a person, presses 0 again, unclear request, caller says nothing.
  4. Mark the gate: does each design pass all 13 calls?
  5. Write one sentence for each design: where it wins, and where it fails.
Show the answer
Measure IVR Agent
Task success 39 of 60 25 of 60
Latency (median) about 1.05 s 2.10 s
Interruptions handled does not listen while it speaks not tested
Wrong actions 2 0
Must go to a person 3 + 3 + 4 + 2 = 12 of 13: fails the gate 3 + 3 + 4 + 3 = 13 of 13

The IVR wins on task success and latency, and fails the gate on one silent caller. Fix that first: hand over after silence, never hang up. The agent passes the gate and makes no wrong action, but it is slow, and it loses most keypad and number calls in this replay.

Your turn: plan a test of a speech-to-speech model

Section titled “Your turn: plan a test of a speech-to-speech model”

Omar asks you to test a hosted speech-to-speech model for the out-of-hours line. Write a plan of five or six lines: which calls, which measures, and one check that the replay cannot do.

Show the answer

A good plan has these parts:

  • The same 60 test calls, played as audio (the model has no text input step), with the same expected endings.
  • The five measures, with the 13 must-go-to-a-person calls as a gate.
  • Latency measured from the end of the caller’s audio to the first sound back, on the same machine and network, median and 90th percentile.
  • Tool calls for every action, so that the read-back and the order checks stay in code; count wrong actions from the tool calls.
  • A transcript from the model, compared with the reference, so that Grace’s call records and Module 9’s redaction still work.
  • A check with reacting callers, people or a simulated caller, for interruptions and for the questions that the recorded callers cannot answer.

And one line on cost: the vendor’s price for audio in and out, with the date, and where the audio goes.

Check your understanding

Name two things that a speech-to-speech model can do better than a cascade, and two things that it makes harder for Larkfield.

Show the answer

Better (vendor claims, not measured here): less waiting, because one model replaces three handoffs. It also keeps the tone of voice, in what it hears and in how it speaks. Harder: there is no text in the middle. So call records, redaction of card numbers and checks in code need a transcript that you must trust. And you cannot measure each step, only the whole call. The caller’s audio also goes to the vendor, not only the text.

Check your understanding

The agent passed 0 of 9 keypad test calls in the replay. Does this show that LLM agents cannot handle keypad tones?

Show the answer

No. The recorded callers press 1 or 2 at the start because the IVR’s greeting asks for it. The agent never offered a keypad, so the key meant nothing to it. The replay is unfair here. A fair test of keypad use would give the agent the same greeting, or use a caller who reacts to the agent’s own questions.

Check your understanding

Why is "must go to a person" a gate and not one more number in an average?

Show the answer

A caller who needs a person and does not get one is the worst result: they may be stuck, upset or in a hurry. A design with good averages could still fail these callers. So every one of these calls must pass before the other scores count. The IVR fails the gate on one silent caller (I-060), even though its task success is higher than the agent’s.

Recap

  • A cascade keeps text between recognition, the LLM and the voice. A speech-to-speech model goes from audio to audio in one model.
  • Speech-to-speech models may wait less and keep the tone (vendor claims, not measured here). They make logs, redaction, checks in code and step-by-step tests harder.
  • Judge a voice agent as a whole with five measures. They are task success by kind, latency, interruptions handled, wrong actions, and the calls that must go to a person, as a gate.
  • On the 60 recorded test calls: the IVR passed 39 and waited about 1.05 s; the agent passed 25 and waited 2.10 s. The agent passed the gate (13 of 13); the IVR failed it once.
  • A fixed replay is unfair to both designs. Check the decision with callers who react.

You now have a fair way to compare designs. But Omar does not have to choose between the menu and the agent. Next, you run the IVR, the agent and a hybrid of the two on the same calls, add the costs, and make the recommendation.

Next: Menu, agent or both?