Skip to content
Français

The voice-agent pipeline

  • Lesson 29
  • 40 min
  • Module 8: From IVR to voice agent

Where we are

An LLM can handle the calls our menu cannot, but every step adds waiting time. In this lesson, we build the voice-agent pipeline, stream at every step, set a latency budget for one turn, and keep the IVR's guardrails.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Measure how long a caller waits for a voice agent’s answer, split the wait into its parts, and find the part to cut first. Keep the IVR’s safety rules around the LLM while you do it.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It reads the recorded agent turns, adds up the wait of each caller turn, and tries the settings that make the wait shorter. It needs no model download and no account. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • Where a menu stops working, and what a voice agent is: Where a menu stops working.
  • What a median and a 90th percentile are. The median is the middle value: half of the values are lower. The 90th percentile is the value that 90% of the values are under.

What you need on your computer (only for the local route; the notebook needs none of it)

  • Your project folder larkspeech at the end of Module 7, with the course data in larkspeech/data. speech ivr --help works in it.
  • The labs repository nextia-learning-labs next to larkspeech. This lesson copies three files from the snapshot end-of-m08. No new package is needed.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.

Think of a relay race with four runners. The team’s time is the sum of the four runners’ times. If one runner is slow, the whole team is slow. And to make the team faster, you first work on the slowest runner. Making a fast runner a little faster changes almost nothing.

A voice agent answers in the same way. Four steps run one after the other, and the caller waits for all of them. The caller does not see the steps. The caller only hears silence, and then the answer.

In a conversation between people, the next person often starts to speak within half a second. After a silence of a second or more, a caller starts to wonder. Some say “hello?”. Others start to talk again, and then the caller and the agent talk at the same time.

On Larkfield’s recorded calls, the voice agent answered a median of 2.10 s after the caller stopped talking. Not one of its 124 answers came in under one second. Omar wants an answer within about one second. Tomás wants to switch on streaming and be done. Priya, the data scientist, wants to measure first.

Latency is the waiting time between a request and its answer. A latency budget is a target for the whole wait, split into parts, one for each step. It works like a money budget. You decide the total first. Then you see which part spends the most, and you cut there.

For a voice agent, the wait starts when the caller stops talking. It ends when the caller hears the first sound of the answer. Larkfield’s target is 1 second. The pipeline has four steps:

A flow from left to right. The caller stops talking. Step 1, end of turn: the VAD waits for 0.8 s of silence. Step 2, recognition: Whisper base writes the words. Step 3, LLM with tools: chat-small streams its first words; it can call the order system as a tool and get an answer back. Guardrails are attached to the LLM: read the number back, two misses go to a person, no card numbers. Step 4, voice: Piper makes the first sound. Then the first sound back.
The voice-agent pipeline. The caller waits for all four steps. The guardrails from the IVR stay around the LLM.
Text description of the diagram

The flow starts when the caller stops talking. Step 1 is the end of turn: the voice activity detector (VAD) waits for 0.8 s of silence, to be sure that the caller has finished. Step 2 is recognition: Whisper base writes the caller’s words. Step 3 is the LLM with tools: chat-small reads the words and streams its first words back. It can call the order system as a tool and get the answer back before it speaks. The LLM has a box of guardrails: read the order number back, hand over after two misses, and never take a card number. Step 4 is the voice: Piper turns the first words into sound. The flow ends with the first sound back.

  1. End of turn. The voice activity detection (VAD) from Module 3 marks speech and silence. A pause in speech is not always the end of a turn. So the pipeline waits for a fixed time of silence: 0.8 s here, as in the IVR. This wait is a setting. No computer works during it.
  2. Recognition. Whisper base writes the caller’s words. It works on the whole turn, so it starts after step 1.
  3. The LLM. chat-small reads the words and the call so far, and starts its answer. Sometimes it calls a tool first, such as the order system. Then the caller waits for the tool call and for a second LLM call.
  4. The voice. Piper turns the answer into sound. Larkfield plays the sound as soon as the first part is ready.

Streaming means passing on the first part of a result before the whole result is ready. Think of a tap instead of a bucket. With a bucket, you wait until it is full, then you carry it. With a tap, the water flows as soon as it comes.

With streaming, the LLM sends its first words while it still writes the rest. Piper speaks the first sentence while the LLM writes the second one. The caller waits only for the first words and the first sound, not for the whole answer. So in the budget, the LLM part is the time to the first token: the first small piece of text that the LLM sends.

Here is one real turn of test call I-001. The caller said the order number 114942. Whisper base wrote “114942.”, and the agent answered “That’s one one four, nine four two. Is that correct?”. These are the times of the four parts, from the recorded agent turns:

Part Seconds Where the number comes from
End-of-turn wait 0.800 the setting
Recognition (Whisper base) 0.238 measured on the course laptop
LLM, first token (chat-small, streamed) 1.376 recorded once, replayed
Voice, first sound (Piper) 0.052 measured on the course laptop
Wait for the first sound 2.466 the sum

The LLM is more than half of the wait. Without streaming, the LLM part would be its whole answer, 1.378 s. That is only 0.002 s more: the answer is one short sentence. And Piper needs 0.082 s for all of it instead of 0.052 s for the first sound. So for this turn, streaming saves about 0.03 s.

The voice times in these recorded turns come from Piper loaded once inside our measuring program. The course project runs piper as a separate program, for its licence (see Text-to-speech for prompts). Started again for every sentence, that program would add about 0.8 s to each turn. So the project starts it once and keeps it running. Kept running, its first sentence was ready after about 0.05 s in a separate check. So the budget’s voice part stays about the same. The agent’s turns were not timed again with the separate program.

This turn was slower than most. Now look at all the turns.

The explorer adds the median of each part over the recorded agent turns. You can change the end-of-turn wait, the recogniser and streaming. The LLM times are recorded answers from chat-small, so you cannot change the LLM here.

Explore

The latency budget of one caller turn

  • End-of-turn wait: 0.80 s
  • Recognition (Whisper base): 0.24 s
  • LLM, first words: 1.02 s
  • Voice, first sound: 0.05 s
Total wait
2.11 s
Against the 1 s target
1.11 s over

End-of-turn wait 0.8 s, Whisper base, streaming: the parts add up to 2.11 s, 1.11 s over the 1 s target. The largest part is "LLM, first words" (1.02 s). Even with no wait and no recognition at all, the LLM and the voice alone take 1.06 s: more than the target.

Each part over the 127 recorded agent turns
PartMedian90th percentileRangeSource
End-of-turn wait0.80 s0.80 sfixeda setting
Recognition (Whisper base)0.24 s0.30 s0.23 to 1.01 smeasured on the course laptop
LLM, first words1.02 s1.49 s0.69 to 2.45 srecorded (chat-small)
Voice, first sound0.05 s0.08 s0.01 to 0.76 smeasured on the course laptop
The median of each part over the 127 recorded agent answers that speak (Larkfield's synthetic calls). The histogram below counts caller turns instead: 124 turns got a spoken answer (see step 3 of the guided practice). The end-of-turn wait is a setting; recognition and voice were measured on the course laptop; LLM times are recorded answers from chat-small.
Text description of the explorer

A stacked bar from 0 to 3 seconds, with a dashed line at the 1-second target. With the starting settings (end-of-turn wait 0.8 s, Whisper base, streaming on), the parts are: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first words 1.02 s, voice first sound 0.05 s. They add up to 2.11 s, 1.11 s over the target. The LLM is the largest part. A slider changes the end-of-turn wait from 0.2 s to 1.5 s. Two buttons switch the recogniser between Whisper base and Whisper tiny (0.12 s). Two buttons switch streaming off, which uses the LLM’s whole answer (1.07 s) and the voice’s whole answer (0.09 s). The lowest sum is 1.38 s: a wait of 0.2 s, Whisper tiny and streaming. Even then, the LLM and the voice alone take 1.06 s. A table under the bar gives each part’s median, 90th percentile, range and source.

What did you find? Two parts are small: recognition (0.24 s) and the voice (0.05 s). The end-of-turn wait is a setting, so you can cut it at once. But the LLM’s first words take 1.02 s on their own. That is already over the target. No setting on Larkfield’s side brings the sum under one second. Cut the end-of-turn wait first, because it costs nothing. Then the LLM is what is left.

The explorer adds the medians of the parts: 2.11 s. The project command, in the next section, takes the median of each turn’s whole wait: 2.10 s. These are two different numbers, and they are close here. Report the one you computed, and say how.

The median hides the slow turns. The chart shows the wait of every caller turn with the default settings.

A histogram of 124 caller turns. The waits go from about 1.6 s to 3.6 s, with most between 1.8 and 2.5 s. A dashed red line at 1 s marks the target: no turn is left of it. A line at 2.10 s marks the median, and a line at 2.74 s the 90th percentile. A small group of turns is between 3.0 and 3.6 s.
The wait of each of the 124 caller turns that got a spoken answer (end-of-turn wait 0.8 s, Whisper base, streaming). Median 2.10 s, 90th percentile 2.74 s, and no turn under the 1 s target.
Text description of the diagram

Each bar counts the caller turns whose wait is in a range of 0.1 s. The shortest wait is 1.57 s and the longest 3.61 s. Most turns wait between 1.8 and 2.5 s. The dashed red line at 1 s is the target, and no turn is on its left. The median is 2.10 s and the 90th percentile 2.74 s. Of the 12 turns over 3 s, 7 called a tool before the answer. The others had a slow LLM, a slow recognition, or both.

A hosted LLM does not answer in the same time every time. Its first token took from 0.69 s to 2.45 s on these calls. And 10 of the 124 caller turns needed a tool call before the answer. Then the caller waits for two LLM calls, one after the other.

The snapshot end-of-m08 adds the command speech agent. It replays the recorded agent turns and adds up the budget of each caller turn.

  1. In your folder larkspeech, with the virtual environment active, copy the files of this module from the snapshot end-of-m08:

    Terminal window
    S=../nextia-learning-labs/C41/snapshots/end-of-m08
    cp "$S/larkspeech/agent.py" larkspeech/
    cp "$S/larkspeech/commands/agent.py" larkspeech/commands/
    cp "$S/tests/test_agent.py" tests/

    The speech command finds its subcommands by their files, so the new command needs no other change. No new package is needed.

  2. Run the tests. Four new tests check the budget and the comparison of the next lessons:

    Terminal window
    pytest -q
    ..................................... [100%]
    37 passed in 1.77s

    The time changes from one computer to another.

  3. Show the budget with the default settings:

    Terminal window
    speech agent budget
    124 caller turns answered by the recorded agent; end of turn 0.8 s; Whisper base; streaming (recorded LLM times, chat-small)
    end of turn (VAD wait) median 0.80 s ################
    recognition median 0.24 s ####
    LLM median 1.02 s ####################
    first audio median 0.05 s
    first sound back: median 2.10 s, 90th percentile 2.74 s; turns under 1 s: 0 of 124

    Each # is 0.05 s. The last line is the total for the caller: the median, the 90th percentile and the count of turns under one second.

  4. Switch streaming off:

    Terminal window
    speech agent budget --no-streaming
    124 caller turns answered by the recorded agent; end of turn 0.8 s; Whisper base; no streaming (recorded LLM times, chat-small)
    end of turn (VAD wait) median 0.80 s ################
    recognition median 0.24 s ####
    LLM median 1.08 s #####################
    first audio median 0.09 s #
    first sound back: median 2.19 s, 90th percentile 2.77 s; turns under 1 s: 0 of 124

    Streaming saves only 0.09 s at the median. The agent’s answers are one or two short sentences, so the whole answer comes soon after the first word.

  5. Try the fastest settings: a wait of 0.3 s and Whisper tiny:

    Terminal window
    speech agent budget --end-of-turn 0.3 --asr tiny
    124 caller turns answered by the recorded agent; end of turn 0.3 s; Whisper tiny; streaming (recorded LLM times, chat-small)
    end of turn (VAD wait) median 0.30 s ######
    recognition median 0.12 s ##
    LLM median 1.02 s ####################
    first audio median 0.05 s
    first sound back: median 1.50 s, 90th percentile 2.27 s; turns under 1 s: 0 of 124

    The median falls from 2.10 s to 1.50 s. Still no turn is under one second. The LLM part did not move.

With the fastest settings of the command (end of turn 0.3 s, Whisper tiny), the median is 1.50 s, and the LLM part is 1.02 s. For a median under 1 s, the other parts must leave room for the LLM. They take about 0.48 s, so the LLM’s first token would need to come in under about 0.52 s. The hosted model’s first token took 0.69 s even in its fastest turn.

So what would it take? Each option has a price:

  • A smaller or closer model. A model on Larkfield’s own server has no internet trip. But it may understand callers less well. Measure it on the same test calls before you choose it.
  • A speech-to-speech model, which hears the audio and answers with audio in one model. A later lesson in this module describes them. The course did not measure one.
  • A shorter end-of-turn wait. Below 0.3 s, the agent starts to answer when the caller only pauses. The next lesson measures it.
  • A short sound for slow turns. When the LLM calls a tool, the voice can say “one moment” at once. This does not make the answer faster. But the caller knows that the line still works.

None of these is free. A team that wants “under one second” must choose which cost it accepts.

A guardrail is a rule that stops a system from doing something harmful, whatever it decides. The IVR has three guardrails from Module 7. It reads the order number back and waits for “yes”. It puts the caller through to a person after two misses. And it never asks for a card number.

The voice agent must keep all three. The course team wrote them into the agent’s system prompt. It says “always read the order number back and get a yes before you act on it”, and “never ask for a card number”. It also says “if you do not understand the caller twice, … transfer them”. But a prompt is a request, not a guarantee. The same prompt said “read order numbers digit by digit”. But on test call I-054, the agent said “six hundred three, one hundred thirty-seven”.

So put each guardrail in code, around the tools:

Guardrail In the prompt In code, around the tools
Read the number back “get a yes before you act” the order tool refuses a number that the caller did not confirm
Two misses go to a person “transfer them” a counter in the program calls the hand-over tool after two misses
No card numbers “never ask for a card number” card numbers go through the keypad only, and the recogniser’s text is checked before the LLM sees it

The agents course teaches this design: Tool contracts checks every tool call in code, and Approval boundaries decides which actions need a yes first. A guardrail in code adds almost no wait. A missing guardrail can start a return that nobody asked for.

Common mistake

"Streaming will fix it"

Symptom: Tomás switches on streaming everywhere and expects the wait to halve. The command shows 2.10 s with streaming against 2.19 s without it.

first sound back: median 2.10 s, 90th percentile 2.74 s; turns under 1 s: 0 of 124 (streaming)
first sound back: median 2.19 s, 90th percentile 2.77 s; turns under 1 s: 0 of 124 (no streaming)

Cause: streaming removes the time between the first word and the last word. Larkfield’s answers are short, so that time is small. At the median, the first token takes 1.02 s and the whole answer 1.08 s. Streaming does not change the time to the first token, which is most of the wait.

Fix: measure the parts before you change one. Cut the largest part first. Here, that is the time to the first token, which needs another model or a closer one.

Consequence in AI work: a well-known method can be the right method for a different problem. Streaming helps a lot when answers are long, such as a chat answer of several paragraphs. A phone answer is one sentence.

Guided practice: the budget in the notebook

Section titled “Guided practice: the budget in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The setup cell ends with a Ready: line.
  3. In section 1, run the cell that loads the turns. You see 177 LLM calls in 60 test calls; 127 of them speak. The other 50 calls only ask for a tool.
  4. In section 2, predict the largest part of the first turn. Then run the cell. The LLM part is 1.487 s of a total of 3.31 s.
  5. In section 3, run the medians with and without streaming. You see a median of 2.10 s with streaming and 2.19 s without it.
  6. Look at the 90th percentile. The notebook gives 2.79 s, and the project command gives 2.74 s. NumPy computes a percentile between two turns, and the command takes one turn’s value. Both are right; say which one you report.
  7. In section 4, predict the effect of a wait of 0.5 s instead of 0.8 s. Then run the cell. The median falls by exactly 0.3 s, to 1.80 s.

In section 6 of the notebook, try every combination of three things. The end-of-turn wait is 0.8, 0.5 or 0.3 s. Streaming is on or off. The recogniser is Whisper base or Whisper tiny. Find the combination with the lowest median wait. Then find what the LLM part would have to be for a median under one second.

Show the answer
results = {}
for end_of_turn in (0.8, 0.5, 0.3):
for streaming in (True, False):
for recognition in ("asr_base_s", "asr_tiny_s"):
results[(end_of_turn, streaming, recognition)] = median_budget(end_of_turn, streaming, recognition)[1]
best_settings = min(results, key=results.get)
best_median = results[best_settings]
print(best_settings, round(best_median, 2))
(0.3, True, 'asr_tiny_s') 1.5

The best combination is a wait of 0.3 s, streaming on, and Whisper tiny: a median of 1.50 s. No caller turn is under one second. The parts are 0.30 s, 0.12 s, 1.02 s and 0.05 s. For a median under 1 s, the LLM part would have to be under about 0.52 s. Remember also that Whisper tiny makes more recognition errors than base (Module 4). A faster answer to a misheard question is not better.

Check your understanding

A caller turn has these parts: end-of-turn wait 0.5 s, recognition 0.12 s, LLM first token 0.9 s, voice first sound 0.05 s. What is the wait, and which part do you cut first?

Show the answer

The wait is the sum: 0.5 + 0.12 + 0.9 + 0.05 = 1.57 s. It is 0.57 s over a 1 s target. The LLM part is the largest (0.9 s), but no setting changes it: it needs another model. The end-of-turn wait (0.5 s) is a setting, so it is the cheapest part to cut. Even at 0 s, though, the sum would be 1.07 s. So the target needs a faster LLM too.

Check your understanding

Why did streaming save only 0.09 s at the median on Larkfield's recorded calls, and when would it save more?

Show the answer

Streaming lets the caller hear the first words before the whole answer is written and spoken. It removes the time between the first word and the last word. Larkfield’s agent answers in one or two short sentences, so that time is small. At the median, the first token takes 1.02 s and the whole answer 1.08 s. Streaming saves more when answers are long, for example a spoken explanation of several sentences.

Check your understanding

The system prompt says "always read the order number back and get a yes before you act on it". Amira says that this is enough as a guardrail. What do you tell her?

Show the answer

A prompt is a request, not a guarantee. The same prompt asked the agent to read numbers digit by digit. On test call I-054, it said “six hundred three, one hundred thirty-seven”. Keep the rule in the prompt, and also put it in code: the order tool refuses a number that the caller did not confirm. A check in code adds almost no wait, and it works every time.

Recap

  • A voice agent is a pipeline: end of turn → recognition → LLM with tools → voice. The caller waits for all four steps.
  • A latency budget splits the target wait into parts. Measure each part, then cut the largest one first.
  • On Larkfield’s recorded calls: end-of-turn wait 0.80 s, recognition 0.24 s, LLM first token 1.02 s, first sound 0.05 s. Median 2.10 s, 90th percentile 2.74 s, no turn under 1 s.
  • Streaming saves little here (2.19 s → 2.10 s), because the answers are short. The fastest settings give 1.50 s. The LLM’s first token alone is about 1 s.
  • Keep the IVR’s guardrails around the LLM, and put them in code, not only in the prompt.

The cheapest part to cut is the end-of-turn wait: it is only a setting. But a shorter wait starts to answer when a caller only pauses. Next, you measure how often that happens, and what to do when the caller talks over the agent.

Next: Turn-taking and interruptions