The voice-agent pipeline
- Lesson 29
- 40 min
- Module 8: From IVR to voice agent
Where we are
An LLM can handle the calls our menu cannot, but every step adds waiting time. In this lesson, we build the voice-agent pipeline, stream at every step, set a latency budget for one turn, and keep the IVR's guardrails.
Lesson 29 of 36 · Speech AI: Transcribe, Understand and SpeakThe voice-agent pipeline
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Measure how long a caller waits for a voice agent’s answer, split the wait into its parts, and find the part to cut first. Keep the IVR’s safety rules around the LLM while you do it.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It reads the recorded agent turns, adds up the wait of each caller turn, and tries the settings that make the wait shorter. It needs no model download and no account. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Where a menu stops working, and what a voice agent is: Where a menu stops working.
- What a median and a 90th percentile are. The median is the middle value: half of the values are lower. The 90th percentile is the value that 90% of the values are under.
What you need on your computer (only for the local route; the notebook needs none of it)
- Your project folder
larkspeechat the end of Module 7, with the course data inlarkspeech/data.speech ivr --helpworks in it. - The labs repository
nextia-learning-labsnext tolarkspeech. This lesson copies three files from the snapshotend-of-m08. No new package is needed.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.
A relay race
Section titled “A relay race”Think of a relay race with four runners. The team’s time is the sum of the four runners’ times. If one runner is slow, the whole team is slow. And to make the team faster, you first work on the slowest runner. Making a fast runner a little faster changes almost nothing.
A voice agent answers in the same way. Four steps run one after the other, and the caller waits for all of them. The caller does not see the steps. The caller only hears silence, and then the answer.
The problem: 2 seconds of silence
Section titled “The problem: 2 seconds of silence”In a conversation between people, the next person often starts to speak within half a second. After a silence of a second or more, a caller starts to wonder. Some say “hello?”. Others start to talk again, and then the caller and the agent talk at the same time.
On Larkfield’s recorded calls, the voice agent answered a median of 2.10 s after the caller stopped talking. Not one of its 124 answers came in under one second. Omar wants an answer within about one second. Tomás wants to switch on streaming and be done. Priya, the data scientist, wants to measure first.
The idea: a latency budget
Section titled “The idea: a latency budget”Latency is the waiting time between a request and its answer. A latency budget is a target for the whole wait, split into parts, one for each step. It works like a money budget. You decide the total first. Then you see which part spends the most, and you cut there.
For a voice agent, the wait starts when the caller stops talking. It ends when the caller hears the first sound of the answer. Larkfield’s target is 1 second. The pipeline has four steps:
Text description of the diagram
The flow starts when the caller stops talking. Step 1 is the end of turn: the voice activity detector (VAD) waits for 0.8 s of silence, to be sure that the caller has finished. Step 2 is recognition: Whisper base writes the caller’s words. Step 3 is the LLM with tools: chat-small reads the words and streams its first words back. It can call the order system as a tool and get the answer back before it speaks. The LLM has a box of guardrails: read the order number back, hand over after two misses, and never take a card number. Step 4 is the voice: Piper turns the first words into sound. The flow ends with the first sound back.
- End of turn. The voice activity detection (VAD) from Module 3 marks speech and silence. A pause in speech is not always the end of a turn. So the pipeline waits for a fixed time of silence: 0.8 s here, as in the IVR. This wait is a setting. No computer works during it.
- Recognition. Whisper base writes the caller’s words. It works on the whole turn, so it starts after step 1.
- The LLM. chat-small reads the words and the call so far, and starts its answer. Sometimes it calls a tool first, such as the order system. Then the caller waits for the tool call and for a second LLM call.
- The voice. Piper turns the answer into sound. Larkfield plays the sound as soon as the first part is ready.
Streaming
Section titled “Streaming”Streaming means passing on the first part of a result before the whole result is ready. Think of a tap instead of a bucket. With a bucket, you wait until it is full, then you carry it. With a tap, the water flows as soon as it comes.
With streaming, the LLM sends its first words while it still writes the rest. Piper speaks the first sentence while the LLM writes the second one. The caller waits only for the first words and the first sound, not for the whole answer. So in the budget, the LLM part is the time to the first token: the first small piece of text that the LLM sends.
A worked example: one turn
Section titled “A worked example: one turn”Here is one real turn of test call I-001. The caller said the order number 114942. Whisper base wrote “114942.”, and the agent answered “That’s one one four, nine four two. Is that correct?”. These are the times of the four parts, from the recorded agent turns:
| Part | Seconds | Where the number comes from |
|---|---|---|
| End-of-turn wait | 0.800 | the setting |
| Recognition (Whisper base) | 0.238 | measured on the course laptop |
| LLM, first token (chat-small, streamed) | 1.376 | recorded once, replayed |
| Voice, first sound (Piper) | 0.052 | measured on the course laptop |
| Wait for the first sound | 2.466 | the sum |
The LLM is more than half of the wait. Without streaming, the LLM part would be its whole answer, 1.378 s. That is only 0.002 s more: the answer is one short sentence. And Piper needs 0.082 s for all of it instead of 0.052 s for the first sound. So for this turn, streaming saves about 0.03 s.
The voice times in these recorded turns come from Piper loaded once inside our measuring program. The course project runs piper as a separate program, for its licence (see Text-to-speech for prompts). Started again for every sentence, that program would add about 0.8 s to each turn. So the project starts it once and keeps it running. Kept running, its first sentence was ready after about 0.05 s in a separate check. So the budget’s voice part stays about the same. The agent’s turns were not timed again with the separate program.
This turn was slower than most. Now look at all the turns.
Explore the budget
Section titled “Explore the budget”The explorer adds the median of each part over the recorded agent turns. You can change the end-of-turn wait, the recogniser and streaming. The LLM times are recorded answers from chat-small, so you cannot change the LLM here.
Explore
The latency budget of one caller turn
Predict first: Which part do you cut first to get under one second? Write your guess, then move the controls.
- End-of-turn wait: 0.80 s
- Recognition (Whisper base): 0.24 s
- LLM, first words: 1.02 s
- Voice, first sound: 0.05 s
- Total wait
- 2.11 s
- Against the 1 s target
- 1.11 s over
End-of-turn wait 0.8 s, Whisper base, streaming: the parts add up to 2.11 s, 1.11 s over the 1 s target. The largest part is "LLM, first words" (1.02 s). Even with no wait and no recognition at all, the LLM and the voice alone take 1.06 s: more than the target.
| Part | Median | 90th percentile | Range | Source |
|---|---|---|---|---|
| End-of-turn wait | 0.80 s | 0.80 s | fixed | a setting |
| Recognition (Whisper base) | 0.24 s | 0.30 s | 0.23 to 1.01 s | measured on the course laptop |
| LLM, first words | 1.02 s | 1.49 s | 0.69 to 2.45 s | recorded (chat-small) |
| Voice, first sound | 0.05 s | 0.08 s | 0.01 to 0.76 s | measured on the course laptop |
Text description of the explorer
A stacked bar from 0 to 3 seconds, with a dashed line at the 1-second target. With the starting settings (end-of-turn wait 0.8 s, Whisper base, streaming on), the parts are: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first words 1.02 s, voice first sound 0.05 s. They add up to 2.11 s, 1.11 s over the target. The LLM is the largest part. A slider changes the end-of-turn wait from 0.2 s to 1.5 s. Two buttons switch the recogniser between Whisper base and Whisper tiny (0.12 s). Two buttons switch streaming off, which uses the LLM’s whole answer (1.07 s) and the voice’s whole answer (0.09 s). The lowest sum is 1.38 s: a wait of 0.2 s, Whisper tiny and streaming. Even then, the LLM and the voice alone take 1.06 s. A table under the bar gives each part’s median, 90th percentile, range and source.
What did you find? Two parts are small: recognition (0.24 s) and the voice (0.05 s). The end-of-turn wait is a setting, so you can cut it at once. But the LLM’s first words take 1.02 s on their own. That is already over the target. No setting on Larkfield’s side brings the sum under one second. Cut the end-of-turn wait first, because it costs nothing. Then the LLM is what is left.
The explorer adds the medians of the parts: 2.11 s. The project command, in the next section, takes the median of each turn’s whole wait: 2.10 s. These are two different numbers, and they are close here. Report the one you computed, and say how.
All the turns, not only the middle one
Section titled “All the turns, not only the middle one”The median hides the slow turns. The chart shows the wait of every caller turn with the default settings.
Text description of the diagram
Each bar counts the caller turns whose wait is in a range of 0.1 s. The shortest wait is 1.57 s and the longest 3.61 s. Most turns wait between 1.8 and 2.5 s. The dashed red line at 1 s is the target, and no turn is on its left. The median is 2.10 s and the 90th percentile 2.74 s. Of the 12 turns over 3 s, 7 called a tool before the answer. The others had a slow LLM, a slow recognition, or both.
A hosted LLM does not answer in the same time every time. Its first token took from 0.69 s to 2.45 s on these calls. And 10 of the 124 caller turns needed a tool call before the answer. Then the caller waits for two LLM calls, one after the other.
Larkfield: the budget with the project
Section titled “Larkfield: the budget with the project”The snapshot end-of-m08 adds the command speech agent. It replays the recorded agent turns and adds up the budget of each caller turn.
-
In your folder
larkspeech, with the virtual environment active, copy the files of this module from the snapshotend-of-m08:Terminal window S=../nextia-learning-labs/C41/snapshots/end-of-m08cp "$S/larkspeech/agent.py" larkspeech/cp "$S/larkspeech/commands/agent.py" larkspeech/commands/cp "$S/tests/test_agent.py" tests/Terminal window $S = "..\nextia-learning-labs\C41\snapshots\end-of-m08"Copy-Item "$S\larkspeech\agent.py" larkspeech\Copy-Item "$S\larkspeech\commands\agent.py" larkspeech\commands\Copy-Item "$S\tests\test_agent.py" tests\The
speechcommand finds its subcommands by their files, so the new command needs no other change. No new package is needed. -
Run the tests. Four new tests check the budget and the comparison of the next lessons:
Terminal window pytest -q..................................... [100%]37 passed in 1.77sThe time changes from one computer to another.
-
Show the budget with the default settings:
Terminal window speech agent budget124 caller turns answered by the recorded agent; end of turn 0.8 s; Whisper base; streaming (recorded LLM times, chat-small)end of turn (VAD wait) median 0.80 s ################recognition median 0.24 s ####LLM median 1.02 s ####################first audio median 0.05 sfirst sound back: median 2.10 s, 90th percentile 2.74 s; turns under 1 s: 0 of 124Each
#is 0.05 s. The last line is the total for the caller: the median, the 90th percentile and the count of turns under one second. -
Switch streaming off:
Terminal window speech agent budget --no-streaming124 caller turns answered by the recorded agent; end of turn 0.8 s; Whisper base; no streaming (recorded LLM times, chat-small)end of turn (VAD wait) median 0.80 s ################recognition median 0.24 s ####LLM median 1.08 s #####################first audio median 0.09 s #first sound back: median 2.19 s, 90th percentile 2.77 s; turns under 1 s: 0 of 124Streaming saves only 0.09 s at the median. The agent’s answers are one or two short sentences, so the whole answer comes soon after the first word.
-
Try the fastest settings: a wait of 0.3 s and Whisper tiny:
Terminal window speech agent budget --end-of-turn 0.3 --asr tiny124 caller turns answered by the recorded agent; end of turn 0.3 s; Whisper tiny; streaming (recorded LLM times, chat-small)end of turn (VAD wait) median 0.30 s ######recognition median 0.12 s ##LLM median 1.02 s ####################first audio median 0.05 sfirst sound back: median 1.50 s, 90th percentile 2.27 s; turns under 1 s: 0 of 124The median falls from 2.10 s to 1.50 s. Still no turn is under one second. The LLM part did not move.
What is left above one second
Section titled “What is left above one second”With the fastest settings of the command (end of turn 0.3 s, Whisper tiny), the median is 1.50 s, and the LLM part is 1.02 s. For a median under 1 s, the other parts must leave room for the LLM. They take about 0.48 s, so the LLM’s first token would need to come in under about 0.52 s. The hosted model’s first token took 0.69 s even in its fastest turn.
So what would it take? Each option has a price:
- A smaller or closer model. A model on Larkfield’s own server has no internet trip. But it may understand callers less well. Measure it on the same test calls before you choose it.
- A speech-to-speech model, which hears the audio and answers with audio in one model. A later lesson in this module describes them. The course did not measure one.
- A shorter end-of-turn wait. Below 0.3 s, the agent starts to answer when the caller only pauses. The next lesson measures it.
- A short sound for slow turns. When the LLM calls a tool, the voice can say “one moment” at once. This does not make the answer faster. But the caller knows that the line still works.
None of these is free. A team that wants “under one second” must choose which cost it accepts.
Keep the IVR’s guardrails
Section titled “Keep the IVR’s guardrails”A guardrail is a rule that stops a system from doing something harmful, whatever it decides. The IVR has three guardrails from Module 7. It reads the order number back and waits for “yes”. It puts the caller through to a person after two misses. And it never asks for a card number.
The voice agent must keep all three. The course team wrote them into the agent’s system prompt. It says “always read the order number back and get a yes before you act on it”, and “never ask for a card number”. It also says “if you do not understand the caller twice, … transfer them”. But a prompt is a request, not a guarantee. The same prompt said “read order numbers digit by digit”. But on test call I-054, the agent said “six hundred three, one hundred thirty-seven”.
So put each guardrail in code, around the tools:
| Guardrail | In the prompt | In code, around the tools |
|---|---|---|
| Read the number back | “get a yes before you act” | the order tool refuses a number that the caller did not confirm |
| Two misses go to a person | “transfer them” | a counter in the program calls the hand-over tool after two misses |
| No card numbers | “never ask for a card number” | card numbers go through the keypad only, and the recogniser’s text is checked before the LLM sees it |
The agents course teaches this design: Tool contracts checks every tool call in code, and Approval boundaries decides which actions need a yes first. A guardrail in code adds almost no wait. A missing guardrail can start a return that nobody asked for.
Common mistake
"Streaming will fix it"
Symptom: Tomás switches on streaming everywhere and expects the wait to halve. The command shows 2.10 s with streaming against 2.19 s without it.
first sound back: median 2.10 s, 90th percentile 2.74 s; turns under 1 s: 0 of 124 (streaming)first sound back: median 2.19 s, 90th percentile 2.77 s; turns under 1 s: 0 of 124 (no streaming)Cause: streaming removes the time between the first word and the last word. Larkfield’s answers are short, so that time is small. At the median, the first token takes 1.02 s and the whole answer 1.08 s. Streaming does not change the time to the first token, which is most of the wait.
Fix: measure the parts before you change one. Cut the largest part first. Here, that is the time to the first token, which needs another model or a closer one.
Consequence in AI work: a well-known method can be the right method for a different problem. Streaming helps a lot when answers are long, such as a chat answer of several paragraphs. A phone answer is one sentence.
Guided practice: the budget in the notebook
Section titled “Guided practice: the budget in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The setup cell ends with a
Ready:line. - In section 1, run the cell that loads the turns. You see
177 LLM calls in 60 test calls; 127 of them speak. The other 50 calls only ask for a tool. - In section 2, predict the largest part of the first turn. Then run the cell. The LLM part is 1.487 s of a total of 3.31 s.
- In section 3, run the medians with and without streaming. You see a median of 2.10 s with streaming and 2.19 s without it.
- Look at the 90th percentile. The notebook gives 2.79 s, and the project command gives 2.74 s. NumPy computes a percentile between two turns, and the command takes one turn’s value. Both are right; say which one you report.
- In section 4, predict the effect of a wait of 0.5 s instead of 0.8 s. Then run the cell. The median falls by exactly 0.3 s, to 1.80 s.
Your turn: the fastest settings
Section titled “Your turn: the fastest settings”In section 6 of the notebook, try every combination of three things. The end-of-turn wait is 0.8, 0.5 or 0.3 s. Streaming is on or off. The recogniser is Whisper base or Whisper tiny. Find the combination with the lowest median wait. Then find what the LLM part would have to be for a median under one second.
Show the answer
results = {}for end_of_turn in (0.8, 0.5, 0.3): for streaming in (True, False): for recognition in ("asr_base_s", "asr_tiny_s"): results[(end_of_turn, streaming, recognition)] = median_budget(end_of_turn, streaming, recognition)[1]best_settings = min(results, key=results.get)best_median = results[best_settings]print(best_settings, round(best_median, 2))(0.3, True, 'asr_tiny_s') 1.5The best combination is a wait of 0.3 s, streaming on, and Whisper tiny: a median of 1.50 s. No caller turn is under one second. The parts are 0.30 s, 0.12 s, 1.02 s and 0.05 s. For a median under 1 s, the LLM part would have to be under about 0.52 s. Remember also that Whisper tiny makes more recognition errors than base (Module 4). A faster answer to a misheard question is not better.
Knowledge check
Section titled “Knowledge check”Check your understanding
A caller turn has these parts: end-of-turn wait 0.5 s, recognition 0.12 s, LLM first token 0.9 s, voice first sound 0.05 s. What is the wait, and which part do you cut first?
Show the answer
The wait is the sum: 0.5 + 0.12 + 0.9 + 0.05 = 1.57 s. It is 0.57 s over a 1 s target. The LLM part is the largest (0.9 s), but no setting changes it: it needs another model. The end-of-turn wait (0.5 s) is a setting, so it is the cheapest part to cut. Even at 0 s, though, the sum would be 1.07 s. So the target needs a faster LLM too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why did streaming save only 0.09 s at the median on Larkfield's recorded calls, and when would it save more?
Show the answer
Streaming lets the caller hear the first words before the whole answer is written and spoken. It removes the time between the first word and the last word. Larkfield’s agent answers in one or two short sentences, so that time is small. At the median, the first token takes 1.02 s and the whole answer 1.08 s. Streaming saves more when answers are long, for example a spoken explanation of several sentences.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The system prompt says "always read the order number back and get a yes before you act on it". Amira says that this is enough as a guardrail. What do you tell her?
Show the answer
A prompt is a request, not a guarantee. The same prompt asked the agent to read numbers digit by digit. On test call I-054, it said “six hundred three, one hundred thirty-seven”. Keep the rule in the prompt, and also put it in code: the order tool refuses a number that the caller did not confirm. A check in code adds almost no wait, and it works every time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A voice agent is a pipeline: end of turn → recognition → LLM with tools → voice. The caller waits for all four steps.
- A latency budget splits the target wait into parts. Measure each part, then cut the largest one first.
- On Larkfield’s recorded calls: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first token 1.02 s, first sound 0.05 s. Median 2.10 s, 90th percentile 2.74 s, no turn under 1 s.
- Streaming saves little here (2.19 s → 2.10 s), because the answers are short. The fastest settings give 1.50 s. The LLM’s first token alone is about 1 s.
- Keep the IVR’s guardrails around the LLM, and put them in code, not only in the prompt.
The cheapest part to cut is the end-of-turn wait: it is only a setting. But a shorter wait starts to answer when a caller only pauses. Next, you measure how often that happens, and what to do when the caller talks over the agent.