Speech-to-speech models and evaluation
- Lesson 31
- 30 min
- Module 8: From IVR to voice agent
Where we are
We built a voice agent from separate parts and learned to set its waits. Newer models take speech in and give speech out directly. In this lesson, we see what they change, and we learn to evaluate a voice agent as a whole, not one part at a time.
Lesson 31 of 36 · Speech AI: Transcribe, Understand and SpeakSpeech-to-speech models and evaluation
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Know what a speech-to-speech model changes in a voice agent, and what it makes harder. Then judge any voice agent as a whole, with a short list of measures that Larkfield can compare from one design to the next.
Before you startWhat to know and to have ready
What you need to know
- The voice-agent pipeline and its latency budget: The voice-agent pipeline.
- End-of-turn detection and barge-in: Turn-taking and interruptions.
- How the IVR was tested with recorded calls: Test the IVR like a product.
What you need on your computer
Nothing. This lesson reads the recorded results of the IVR and of the agent.
Two ways to interpret
Section titled “Two ways to interpret”Think of a meeting between two people who speak different languages. You can help them in two ways.
The first way is a chain. A note-taker writes down what the first person says. A translator translates the notes. A reader reads the translation aloud. It is slow, and the reader’s voice does not carry the first person’s tone. But you have the notes and the translation on paper. You can check them afterwards.
The second way is an interpreter who listens and speaks at once. It is fast, and a good interpreter keeps the tone: a joke still sounds like a joke. But nothing is on paper. If something goes wrong, you cannot check what was heard and what was said.
The problem: the chain is slow and loses the tone
Section titled “The problem: the chain is slow and loses the tone”The voice agent of this module is the chain. In speech work, it is called a cascade: recognition, then an LLM, then a voice, one after the other. The previous two lessons measured its limits on Larkfield’s recorded calls:
- Time. Each step waits for the one before. The median wait was 2.10 s, and the LLM’s first token alone took about 1 s.
- Errors carry on. When Whisper base heard “189 710” for 187971, the LLM read the wrong number back, politely and confidently.
- The tone is lost. The LLM reads only text. It cannot hear that a caller sounds angry, unsure or in a hurry.
The idea: one model that hears and speaks
Section titled “The idea: one model that hears and speaks”A speech-to-speech model takes audio in and gives audio out, with one model in the middle. It is like the interpreter. Some vendors call these models “realtime” or “native audio” models. Some can also call tools, as the LLM in our cascade does.
Text description of the diagram
One flow is the cascade of this module. Speech goes into recognition, which writes text. You can log, check and redact this text. The LLM with tools reads it and writes an answer as text. The voice turns that text into speech. The other flow is a speech-to-speech model. Speech goes into one model. That model hears the words and the tone, can call tools, and speaks the answer. A dashed line shows that text comes out only if the model also writes a transcript.
What it changes
Section titled “What it changes”Vendors make three claims for these models. They are vendor claims: the course did not run or measure any speech-to-speech model.
- Less waiting. One model replaces three handoffs, and it can start to answer while the caller is still finishing.
- The tone is kept. The model hears hesitation and anger, and its voice can sound calm or warm.
- Turn-taking in the model. Some models decide the end of the turn and handle a barge-in themselves.
What it makes harder
Section titled “What it makes harder”Each of these is a real cost for Larkfield, and each comes from the missing text in the middle:
- Checks and logs. Grace’s team reads call records. Module 9 removes card numbers from transcripts. With no text in the middle, you need the model to write a transcript too, and you must trust that it matches the audio.
- Guardrails. The read-back rule needs the order number as text, so that code can check it. Make the model call a tool with the number. Then the guardrail in code still works.
- Measurement. You cannot score the recognition step alone, because there is none. You can only score the whole call.
- Privacy and price. The caller’s audio goes to the vendor, not only the text. Hosted audio models are priced by the amount of audio in and out. Prices change: check the current page.
So a speech-to-speech model is a different trade, not a free upgrade. To choose, you need one way to judge any voice agent, whatever is inside it.
Evaluate the agent as a whole
Section titled “Evaluate the agent as a whole”A part can be good while the whole is bad. Whisper base has a WER of 5.2% on the English test calls. But the agent still passed only 2 of 12 “order status, speech” calls. So judge the whole call, from the caller’s side. Five measures cover it:
| Measure | The question | How to measure it on recorded calls |
|---|---|---|
| Task success | Did the call end the way it should? | the expected ending of each test call: pass or fail, by kind |
| Latency | How long is the silence after the caller stops? | the median and the 90th percentile of the wait per turn |
| Interruptions handled | When the caller talks over the agent, does it stop and listen? | needs a caller who interrupts: not possible with a fixed replay |
| Wrong actions | Did it do something that nobody asked for? | tool calls against the expected order and intent |
| Must go to a person | Does every caller who needs a person get one? | the calls whose right ending is a hand-over: all of them must pass |
The last measure is not an average. It is a gate: a design that fails one of these calls is not ready, whatever its other scores. In Larkfield’s test set, the gate has 13 calls of four kinds: asks for a person, presses 0 again, unclear request, and a caller who says nothing. The six return calls also end with a person, because their orders cannot be returned. They are not in the gate: those callers asked for a return, not for a person. The agents course uses the same idea for actions that need a person’s approval: Approval boundaries and Task success measures.
A worked example: the IVR and the agent, scored
Section titled “A worked example: the IVR and the agent, scored”Here are the two designs of this module on the same 60 recorded test calls. All numbers are on Larkfield’s synthetic calls.
| Measure | IVR (menu) | Voice agent (recorded, chat-small) |
|---|---|---|
| Task success | 39 of 60 passed | 25 of 60 passed |
| Latency after a spoken turn (median) | about 1.05 s: the 0.8 s wait and recognition; its prompts are made in advance | 2.10 s (90th percentile 2.74 s) |
| Interruptions handled | does not listen while it speaks | not tested: the replay’s callers never interrupt |
| Wrong actions | 2: twice it heard “quelqu’un” (someone) and put a caller through to a person (in the recorded replay of the first dialogue.py; the current IVR made none) |
0 |
| Must go to a person (13 calls) | 12 of 13: it hung up on one silent caller | 13 of 13 |
Read the table row by row. The IVR wins on task success and on latency. The agent wins on wrong actions and on the gate: every caller who needed a person got one. Neither design was tested on interruptions. A score that adds these rows into one number would hide all of this.
Task success, kind by kind
Section titled “Task success, kind by kind”The total hides where each design wins. The chart splits the passes by kind of test call.
Text description of the diagram
Each kind of test call has two bars: the IVR on top in blue, the agent below in orange. At the top are the kinds where the agent does worse. With the keypad, it passes 0 of 6 order-status calls and 0 of 3 opening-hours calls, where the IVR passes all of them. It also does worse on returns by speech, a wrong number that the caller fixes, a number with a pause, a noisy line and hours by speech. In the middle, both designs score the same: 2 of 12 order-status calls by speech and 0 of 3 changes of mind. Both pass every caller who asks for a person, presses 0 again or makes an unclear request. At the bottom, the agent does better on one kind: a silent caller, 3 of 3 against 2 of 3.
Is the replay fair?
Section titled “Is the replay fair?”Look at the keypad calls again: the agent passed 0 of 9. The reason is not the LLM. The recorded caller presses 1 or 2 at the start, because the IVR’s greeting asks for it. The agent never offered a keypad, so “the caller pressed 1” meant nothing to it. The replay is unfair to the agent here, as it is unfair to the IVR when a recorded caller cannot answer a new question.
A fairer test needs a caller who reacts. Teams use two ways:
- People call the agent with a list of tasks: “find the status of order 742155”. It is slow and costs people’s time, but it is real.
- A simulated caller. A second LLM plays the caller. It gets a goal and a script, and it answers the agent’s real questions. It is cheap and repeatable, but it is another model to test. A caller that is too patient or too clever gives results that are too good.
Use the recorded replay for what it does well: the same calls, every time, for every change. Then check the decision with reacting callers before the agent goes live.
When to try a speech-to-speech model
Section titled “When to try a speech-to-speech model”Try one when the latency budget cannot be met with the cascade, and when the tone of voice matters: for example, callers who are upset. Measure it with the five measures above, on the same test calls.
Keep the cascade when you need the text: for call records, redaction of card numbers, checks in code, and tests of each step. Larkfield needs all four, so it keeps the cascade for now. If it tries a speech-to-speech model later, it makes the model call tools for every action and write a transcript.
Common mistake
Containment as the only score
Symptom: Tomás reports one number for each design: containment. The IVR contains 35%, the hybrid of the next lesson 37%. Nobody notices that the IVR hung up on a silent caller.
I-060 caller says nothing expected: hand over (no input) IVR: hung upCause: containment counts the calls that the system finished itself. A hang-up is not contained, so it lowers containment a little, but it looks the same as any other call that is not contained. A caller who needed a person and got silence is a much worse result than a hand-over.
Fix: keep the must-go-to-a-person calls as a gate, and report them apart: “12 of 13, failed: I-060”. Then report task success by kind, latency and wrong actions next to containment.
Consequence in AI work: one number makes a decision easy and wrong. A score of averages lets a system pass while it fails the callers who need help most.
Guided practice: build the scorecard
Section titled “Guided practice: build the scorecard”Use the chart and the tables above. No code is needed.
- Copy the five measures into a table with three columns: the measure, the IVR, the agent.
- Fill in task success and latency from the worked example.
- From the chart, add the passes of the four kinds that must go to a person. They are: asks for a person, presses 0 again, unclear request, caller says nothing.
- Mark the gate: does each design pass all 13 calls?
- Write one sentence for each design: where it wins, and where it fails.
Show the answer
| Measure | IVR | Agent |
|---|---|---|
| Task success | 39 of 60 | 25 of 60 |
| Latency (median) | about 1.05 s | 2.10 s |
| Interruptions handled | does not listen while it speaks | not tested |
| Wrong actions | 2 | 0 |
| Must go to a person | 3 + 3 + 4 + 2 = 12 of 13: fails the gate | 3 + 3 + 4 + 3 = 13 of 13 |
The IVR wins on task success and latency, and fails the gate on one silent caller. Fix that first: hand over after silence, never hang up. The agent passes the gate and makes no wrong action, but it is slow, and it loses most keypad and number calls in this replay.
Your turn: plan a test of a speech-to-speech model
Section titled “Your turn: plan a test of a speech-to-speech model”Omar asks you to test a hosted speech-to-speech model for the out-of-hours line. Write a plan of five or six lines: which calls, which measures, and one check that the replay cannot do.
Show the answer
A good plan has these parts:
- The same 60 test calls, played as audio (the model has no text input step), with the same expected endings.
- The five measures, with the 13 must-go-to-a-person calls as a gate.
- Latency measured from the end of the caller’s audio to the first sound back, on the same machine and network, median and 90th percentile.
- Tool calls for every action, so that the read-back and the order checks stay in code; count wrong actions from the tool calls.
- A transcript from the model, compared with the reference, so that Grace’s call records and Module 9’s redaction still work.
- A check with reacting callers, people or a simulated caller, for interruptions and for the questions that the recorded callers cannot answer.
And one line on cost: the vendor’s price for audio in and out, with the date, and where the audio goes.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name two things that a speech-to-speech model can do better than a cascade, and two things that it makes harder for Larkfield.
Show the answer
Better (vendor claims, not measured here): less waiting, because one model replaces three handoffs. It also keeps the tone of voice, in what it hears and in how it speaks. Harder: there is no text in the middle. So call records, redaction of card numbers and checks in code need a transcript that you must trust. And you cannot measure each step, only the whole call. The caller’s audio also goes to the vendor, not only the text.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The agent passed 0 of 9 keypad test calls in the replay. Does this show that LLM agents cannot handle keypad tones?
Show the answer
No. The recorded callers press 1 or 2 at the start because the IVR’s greeting asks for it. The agent never offered a keypad, so the key meant nothing to it. The replay is unfair here. A fair test of keypad use would give the agent the same greeting, or use a caller who reacts to the agent’s own questions.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why is "must go to a person" a gate and not one more number in an average?
Show the answer
A caller who needs a person and does not get one is the worst result: they may be stuck, upset or in a hurry. A design with good averages could still fail these callers. So every one of these calls must pass before the other scores count. The IVR fails the gate on one silent caller (I-060), even though its task success is higher than the agent’s.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A cascade keeps text between recognition, the LLM and the voice. A speech-to-speech model goes from audio to audio in one model.
- Speech-to-speech models may wait less and keep the tone (vendor claims, not measured here). They make logs, redaction, checks in code and step-by-step tests harder.
- Judge a voice agent as a whole with five measures. They are task success by kind, latency, interruptions handled, wrong actions, and the calls that must go to a person, as a gate.
- On the 60 recorded test calls: the IVR passed 39 and waited about 1.05 s; the agent passed 25 and waited 2.10 s. The agent passed the gate (13 of 13); the IVR failed it once.
- A fixed replay is unfair to both designs. Check the decision with callers who react.
You now have a fair way to compare designs. But Omar does not have to choose between the menu and the agent. Next, you run the IVR, the agent and a hybrid of the two on the same calls, add the costs, and make the recommendation.
Next: Menu, agent or both?