Skip to content
Français

Turn-taking and interruptions

  • Lesson 30
  • 35 min
  • Module 8: From IVR to voice agent

Where we are

Our voice agent now has a latency budget, and its cheapest part to cut is the end-of-turn wait. Real callers do not wait politely for their turn. In this lesson, we look at end-of-turn detection, barge-in when the caller talks over the agent, and the half-spoken answer.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 30 of 36 · Speech AI: Transcribe, Understand and SpeakTurn-taking and interruptions

What you will be able to do

Decide when a caller has finished talking, without cutting them off and without long silences. Then handle the caller who talks while the agent is still speaking.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It finds the pauses inside every recorded caller turn with Silero VAD. It measures what each end-of-turn wait cuts. It also works out what a caller heard before an interruption. It needs no account and no key. The setup installs faster-whisper for its voice detector. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

Nothing for this page. The notebook was tested on macOS with Python 3.12, in Jupyter. Colab and Kaggle were not tested.

Think of a phone call with a friend. You never wait for a fixed number of seconds before you answer. You hear that the sentence is complete. You hear the voice fall at the end of a question. While your friend talks, you say “mm-hm” to show that you listen, and your friend goes on. And when you both start to talk at once, one of you stops.

People do this without thinking. A voice agent must do each part on purpose, with code:

  • know when the caller has finished;
  • stop its own voice when the caller talks over it;
  • ignore a short “mm-hm”;
  • remember what the caller really heard before they interrupted.

The voice-agent pipeline ends the caller’s turn after a fixed silence: the end-of-turn wait. The previous lesson showed that this wait is the cheapest part of the budget to cut. Tomás wants to cut it from 0.8 s to 0.3 s. Then the agent answers 0.5 s sooner.

But some callers pause in the middle of a turn. Listen to the caller of test call I-049. The IVR asked for the order number.

Test call I-049, the caller’s third turn, phone quality (8 kHz, VoIP line). A synthetic voice: Piper, trained on the LibriTTS-R corpus. Transcript: “seven nine eight … zero nine zero” (the order number 798090, with a pause in the middle, planted on purpose).

The caller stops after “seven nine eight”, maybe to read the rest from a paper. If the agent answers in that pause, it has only half of the number.

Deciding when a speaker has finished is called end-of-turn detection. Engineers also call it endpointing: finding the end point of an utterance. There are three ways to do it, from the simplest:

  1. A silence wait. The turn ends after a fixed time of silence. This is what Larkfield’s IVR and agent do. It is simple, but one wait does not fit every caller.
  2. A check on the content. The program knows what kind of answer it expects. An order number has six digits. So after “seven nine eight”, the turn is not complete, and the program keeps listening.
  3. A model that predicts the end. Some systems train a small model on the words and the tone of voice. It gives the chance that the speaker has finished. The course did not measure such a model.

Silero VAD, with the same settings as the IVR’s engine, finds two speech parts in the turn of I-049. “Seven nine eight” runs from 0.58 to 1.89 s. “Zero nine zero” runs from 3.84 to 4.77 s. The pause between them is 1.95 s long.

End-of-turn wait The turn ends at Result
0.3 s 1.89 + 0.3 = 2.19 s cut in the pause: only “seven nine eight”
0.8 s 1.89 + 0.8 = 2.69 s cut in the pause
1.5 s 1.89 + 1.5 = 3.39 s cut in the pause
2.0 s 4.77 + 2.0 = 6.77 s the whole number, but 2 s of silence after every turn
Top: the waveform of the caller's turn of call I-049, 5.1 seconds long. Two shaded speech parts: seven nine eight from 0.58 to 1.89 seconds, and zero nine zero from 3.84 to 4.77 seconds, with a pause of 1.95 seconds between them. Bottom: four rows, one for each end-of-turn wait. The waits of 0.3, 0.8 and 1.5 seconds start at 1.89 seconds and end inside the pause, marked cuts in. The wait of 2.0 seconds starts at 4.77 seconds and ends at 6.77 seconds, marked waits for the end.
Test call I-049: the caller's turn and where each end-of-turn wait ends it. Every wait up to 1.5 s ends the turn in the pause. Only 2.0 s keeps the whole number.
Text description of the diagram

The top chart is the sound of the caller’s turn, drawn from the course file. The shaded parts are the speech that Silero VAD found: “seven nine eight” from 0.58 to 1.89 s and “zero nine zero” from 3.84 to 4.77 s. An arrow marks the pause of 1.95 s between them. The bottom chart has one row for each end-of-turn wait. Each bar starts where the silence starts and ends where the wait is over. The bars for 0.3 s, 0.8 s and 1.5 s all end inside the pause, so the turn is cut. The bar for 2.0 s starts after the second part and ends at 6.77 s, so the turn is complete.

No wait is right for this caller. A short wait cuts the number. A wait long enough for this caller makes every other caller wait 2 seconds before each answer.

Larkfield: the trade-off on every caller turn

Section titled “Larkfield: the trade-off on every caller turn”

One call is an example. Priya ran the same detector on all 141 spoken caller turns of the 60 IVR test calls. 31 of them are order numbers. For each turn, she found the longest pause inside it. A wait cuts the turn when the pause is at least as long as the wait.

A line chart. The x axis is the end-of-turn wait from 0.2 to 1.5 seconds. The y axis is the share of turns cut early. The orange line for the 31 order-number turns starts at 35% at 0.2 seconds, is 29% at 0.3 seconds, falls to 16% at 0.6 seconds and to 6% from 0.7 seconds on. The blue line for all 141 spoken turns starts at 13% at 0.2 seconds, is 9% at 0.3 seconds, 3% from 0.7 to 1.0 seconds and 1% from 1.2 seconds. Dotted lines mark 0.3 seconds, the fastest setting, and 0.8 seconds, the project default.
The share of caller turns that each end-of-turn wait cuts early, on the 60 IVR test calls (Silero VAD, the IVR engine's settings; Larkfield's synthetic calls). Order numbers are cut most, because callers pause between digits.
Text description of the diagram

The chart has two lines. The orange line is the 31 turns where the caller says an order number. The blue line is all 141 spoken turns. At a wait of 0.3 s, 9 of 31 number turns are cut (29%), and 13 of all 141 turns (9%). At 0.8 s, 2 number turns and 4 turns in all (3%) are cut. From 0.7 s on, the lines are almost flat. Even at 1.5 s, 2 turns are still cut: two callers pause for longer than 1.5 s.

Wait Turns cut (of 141) Order-number turns cut (of 31) Time saved per answer, against 0.8 s
0.3 s 13 (9%) 9 (29%) 0.5 s
0.5 s 10 (7%) 6 (19%) 0.3 s
0.7 s 4 (3%) 2 (6%) 0.1 s
0.8 s 4 (3%) 2 (6%) none

So Tomás’s 0.3 s would save half a second on every answer. In exchange, it cuts almost one order number in three. These callers pause because of how our synthetic voices read numbers. The voices leave short gaps between digits, and four calls have a long pause planted on purpose. Real callers pause in their own ways. Measure the pauses on your own calls before you choose a wait.

The IVR has the same problem, and Module 7 gave it a fix. When it hears fewer than six digits, it asks for the rest. Here is its recorded replay of call I-049:

40.08 s say Please say or type your six-digit order number.
45.70 s heard Seven nine eight.
45.94 s say I heard 7 9 8. Please say the rest of the number.
52.15 s heard 090.
52.40 s say Order 7 9 8, 0 9 0. Is that right? Say yes or no, or press 1 for yes and 2 for no.
61.70 s heard Correct.
61.92 s say Your Mistral fan shipped on October 11 and is on its way. Thank you for calling Larkfield. Good
61.92 s end status_given

The notebook prints these lines (cut at 95 characters). The times are the call’s clock in the replay.

The wait of 0.8 s cut the turn, and the content check repaired it. This is one reason why the IVR’s results barely move with the wait. With 0.5 s, it passed 38 of 60 test calls; with 0.8 s and with 1.5 s, it passed 39. A voice agent needs the same check, in code: an LLM that gets “seven nine eight” may simply say “I can’t find that order”.

The opposite problem comes next. The agent speaks, and the caller starts to talk. This is called a barge-in: the caller “barges in” on the voice. It happens often when the answer is long, or wrong. The caller hears “your order number is six hundred…” and wants to say “no!”.

Larkfield’s IVR does not listen while it speaks. Its engine starts to listen after each prompt. A caller who talks during the prompt is not heard. That is simple, but slow for a caller who already knows what to say. A voice agent should listen all the time, and stop its voice when the caller talks.

A sequence diagram with four participants: caller, VAD (always listening), agent and voice. The agent sends an answer to the voice, and the audio starts to play. The caller starts to talk. A note on the VAD says: speech for 0.3 s or more, not mm-hm, is a barge-in. The VAD tells the voice to stop now. The voice tells the agent what was played so far: 2.0 s of 5.0 s, Your order number is six. The agent keeps only the played part in the call history. The caller says: No, it is six zero three one three four. The VAD reports the end of turn, then recognition.
A barge-in, step by step. The answer is a real recorded agent answer (test call I-054); the moment of the interruption is constructed, because the replay's callers never interrupt.
Text description of the diagram

The diagram has four participants, from left to right: the caller, the VAD that always listens, the agent and the voice. The agent sends its answer to the voice: “Your order number is six hundred three, one hundred thirty-seven. Is that correct?”. The audio starts to play. Then the caller starts to talk. The VAD sees speech for 0.3 s or more, and it is not a short “mm-hm”, so it is a barge-in. The VAD tells the voice to stop at once. The voice tells the agent what it played: 2.0 s of the 5.0 s answer, “Your order number is six”. The agent keeps only that part in the call’s history. Then the caller says “No, it is six zero three one three four”. This turn goes through end-of-turn detection and recognition as usual.

  1. Stop fast. Every tenth of a second of voice after the caller starts is speech on top of speech. Stop the audio as soon as the VAD is sure.
  2. Do not stop for back-channels. A back-channel is a short sound that a listener makes to say “go on”: “mm-hm”, “yes”, “okay”, “oui”, “d’accord”. If the agent stops for each one, it never finishes a sentence. A simple rule: stop only after about 0.3 s of speech that is not one of these short words. The rule is a starting point to test, not a measured value.
  3. Keep only what was played. The agent’s call history must hold the words that the caller heard, not the whole answer.

The third rule needs a closer look. The recorded agent answer of call I-054 lasts 5.0 s: “Your order number is six hundred three, one hundred thirty-seven. Is that correct?”. Suppose the caller talks at 2.0 s. At an even speed, the caller heard about “Your order number is six”. The notebook computes this estimate; a real voice gives exact word times.

Now suppose the agent keeps the whole answer in its history. The caller says “no, it’s six zero three one three four”. The LLM reads its own history: it asked “Is that correct?”, and the caller said “no”. So far, so good. But the caller could also have said “yes, go on”. Then the LLM would take “yes” as the answer to a question that the caller never heard. It would look up the wrong order. Keep only the played part, and the LLM knows that the question was never asked.

Method Use it when What it costs
A silence wait of 0.7–0.8 s always, as the base rule about 0.8 s on every answer
A shorter wait (0.3–0.5 s) short yes or no answers callers who pause get cut off
A content check you know the form of the answer: digits, a date, yes or no code for each kind of answer
A model that predicts the end many open questions another model to run and to test; not measured here
Barge-in always for a voice agent echo control, and more tests

The content check is cheap and was measured to work in the IVR. Use it first. Change the silence wait only with the trade-off chart of your own calls in front of you.

Common mistake

The agent interrupts itself

This is a constructed case: the course did not record it. It is a common failure when people test a voice agent on a laptop.

Symptom: Amira adds barge-in to the test phone. She calls it with her laptop’s speakers on. The agent stops after two or three words of every answer, and then waits.

Cause: the agent’s own voice comes out of the speakers and goes back into the microphone. This is echo. The VAD hears speech, so the agent thinks that the caller has barged in. It stops its own voice.

Fix: stop the echo before the VAD hears it. Wear headphones when you test: the project’s README asks for them. Turn on the browser’s echo cancellation: the test phone page already asks for it when it opens the microphone (echoCancellation: true). On a real phone line, the phone system removes most of the echo. Test barge-in with headphones and with speakers, and note which one you used.

Consequence in AI work: a test that ignores the room tests a different system. Write the test conditions next to every result: the microphone, the speakers, the line.

Guided practice: the pauses in the notebook

Section titled “Guided practice: the pauses in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells. The setup installs faster-whisper and downloads 145 small files. It ends with Ready: 145 files in larkfield-calls.
  3. In section 2, predict the length of the pause in call I-049. Then run the cell. You see two speech parts and a pause of 1.95 s.
  4. In section 3, run the end-of-turn rule for four waits. Only the wait of 2.0 s keeps the whole number.
  5. In section 4, run the pauses of all 141 turns. Compare the counts with the table on this page.
  6. In section 5, read the IVR’s recorded call. Find the moment when it asks for the rest of the number.

Larkfield accepts that at most 3% of the spoken caller turns are cut early. In section 7 of the notebook, find the shortest wait that cuts at most 3% of all spoken turns. Before you run it, predict the answer from the chart above.

Show the answer
shortest_wait = min(w for w in waits if cut_all[w] / len(rows) <= 0.03)
print(shortest_wait, cut_all[shortest_wait], "of", len(rows))
0.7 4 of 141

The shortest wait is 0.7 s. It cuts 4 of the 141 turns, the same as 0.8 s, and it saves 0.1 s on every answer. In the latency budget, the median wait would go from 2.10 s to about 2.00 s. That is a real gain, but small. A content check on order numbers would let you go lower for the other questions.

Check your understanding

A caller says "three four one", pauses 1.2 seconds, then says "two two eight". The end-of-turn wait is 0.8 s. What does the agent hear, and what are two ways to fix it?

Show the answer

The pause (1.2 s) is longer than the wait (0.8 s), so the turn ends in the pause. The agent hears only “three four one”. Fix 1: a content check. An order number has six digits, so after three digits the program keeps listening, or asks for the rest, as the IVR does. Fix 2: a longer wait of about 1.3 s, but then every caller waits longer after every turn. The content check is better: it adds no wait to other turns.

Check your understanding

What is a back-channel, and why must barge-in ignore it?

Show the answer

A back-channel is a short sound that a listener makes to say “I am listening, go on”: “mm-hm”, “yes”, “okay”, “oui”. The caller does not want to interrupt. If barge-in stops the voice for every back-channel, the agent never finishes a sentence. So stop only after a longer stretch of speech, about 0.3 s (a value to test). Or stop when the words are not a back-channel.

Check your understanding

The voice stops after 2.0 s of a 5.0 s answer that ends with "Is that correct?". Why must the agent's history keep only the played part?

Show the answer

The caller heard only the first part of the answer, about “Your order number is six”. The question “Is that correct?” was never heard. If the history keeps the whole answer, the LLM thinks that it asked the question. Then it may take the caller’s next words, such as “yes”, as the answer. Then it acts on a number that the caller never confirmed. With only the played part, the LLM knows that the question is still open.

Recap

  • End-of-turn detection decides when the caller has finished: a silence wait, a check on the content, or a model that predicts the end.
  • A shorter wait answers sooner, but cuts callers who pause. On Larkfield’s 141 spoken caller turns, a wait of 0.3 s cuts 9% of turns and 29% of order numbers. A wait of 0.7–0.8 s cuts 3% and 6%.
  • A content check (“six digits, or keep listening”) repaired the cut number in the IVR, so its results barely change with the wait.
  • Barge-in: stop the voice fast when the caller talks over it. Ignore back-channels such as “mm-hm”, and stop the echo of the agent’s own voice.
  • Keep only the played part of an interrupted answer in the call’s history.

You can now build each part of a voice agent and set its waits. Some newer models replace the whole pipeline with one model that hears and speaks. Next, you see what they change, and how to judge a voice agent as a whole.

Next: Speech-to-speech models and evaluation