Turn-taking and interruptions
- Lesson 30
- 35 min
- Module 8: From IVR to voice agent
Where we are
Our voice agent now has a latency budget, and its cheapest part to cut is the end-of-turn wait. Real callers do not wait politely for their turn. In this lesson, we look at end-of-turn detection, barge-in when the caller talks over the agent, and the half-spoken answer.
Lesson 30 of 36 · Speech AI: Transcribe, Understand and SpeakTurn-taking and interruptions
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Decide when a caller has finished talking, without cutting them off and without long silences. Then handle the caller who talks while the agent is still speaking.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It finds the pauses inside every recorded caller turn with Silero VAD. It measures what each end-of-turn wait cuts. It also works out what a caller heard before an interruption. It needs no account and no key. The setup installs faster-whisper for its voice detector. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The latency budget of a voice agent, and why the end-of-turn wait is its cheapest part: The voice-agent pipeline.
- What voice activity detection (VAD) does: it marks speech and silence in a recording. See WebRTC VAD and Silero VAD.
What you need on your computer
Nothing for this page. The notebook was tested on macOS with Python 3.12, in Jupyter. Colab and Kaggle were not tested.
How people take turns
Section titled “How people take turns”Think of a phone call with a friend. You never wait for a fixed number of seconds before you answer. You hear that the sentence is complete. You hear the voice fall at the end of a question. While your friend talks, you say “mm-hm” to show that you listen, and your friend goes on. And when you both start to talk at once, one of you stops.
People do this without thinking. A voice agent must do each part on purpose, with code:
- know when the caller has finished;
- stop its own voice when the caller talks over it;
- ignore a short “mm-hm”;
- remember what the caller really heard before they interrupted.
The problem: a pause is not the end
Section titled “The problem: a pause is not the end”The voice-agent pipeline ends the caller’s turn after a fixed silence: the end-of-turn wait. The previous lesson showed that this wait is the cheapest part of the budget to cut. Tomás wants to cut it from 0.8 s to 0.3 s. Then the agent answers 0.5 s sooner.
But some callers pause in the middle of a turn. Listen to the caller of test call I-049. The IVR asked for the order number.
The caller stops after “seven nine eight”, maybe to read the rest from a paper. If the agent answers in that pause, it has only half of the number.
The idea: end-of-turn detection
Section titled “The idea: end-of-turn detection”Deciding when a speaker has finished is called end-of-turn detection. Engineers also call it endpointing: finding the end point of an utterance. There are three ways to do it, from the simplest:
- A silence wait. The turn ends after a fixed time of silence. This is what Larkfield’s IVR and agent do. It is simple, but one wait does not fit every caller.
- A check on the content. The program knows what kind of answer it expects. An order number has six digits. So after “seven nine eight”, the turn is not complete, and the program keeps listening.
- A model that predicts the end. Some systems train a small model on the words and the tone of voice. It gives the chance that the speaker has finished. The course did not measure such a model.
A worked example: one turn, four waits
Section titled “A worked example: one turn, four waits”Silero VAD, with the same settings as the IVR’s engine, finds two speech parts in the turn of I-049. “Seven nine eight” runs from 0.58 to 1.89 s. “Zero nine zero” runs from 3.84 to 4.77 s. The pause between them is 1.95 s long.
| End-of-turn wait | The turn ends at | Result |
|---|---|---|
| 0.3 s | 1.89 + 0.3 = 2.19 s | cut in the pause: only “seven nine eight” |
| 0.8 s | 1.89 + 0.8 = 2.69 s | cut in the pause |
| 1.5 s | 1.89 + 1.5 = 3.39 s | cut in the pause |
| 2.0 s | 4.77 + 2.0 = 6.77 s | the whole number, but 2 s of silence after every turn |
Text description of the diagram
The top chart is the sound of the caller’s turn, drawn from the course file. The shaded parts are the speech that Silero VAD found: “seven nine eight” from 0.58 to 1.89 s and “zero nine zero” from 3.84 to 4.77 s. An arrow marks the pause of 1.95 s between them. The bottom chart has one row for each end-of-turn wait. Each bar starts where the silence starts and ends where the wait is over. The bars for 0.3 s, 0.8 s and 1.5 s all end inside the pause, so the turn is cut. The bar for 2.0 s starts after the second part and ends at 6.77 s, so the turn is complete.
No wait is right for this caller. A short wait cuts the number. A wait long enough for this caller makes every other caller wait 2 seconds before each answer.
Larkfield: the trade-off on every caller turn
Section titled “Larkfield: the trade-off on every caller turn”One call is an example. Priya ran the same detector on all 141 spoken caller turns of the 60 IVR test calls. 31 of them are order numbers. For each turn, she found the longest pause inside it. A wait cuts the turn when the pause is at least as long as the wait.
Text description of the diagram
The chart has two lines. The orange line is the 31 turns where the caller says an order number. The blue line is all 141 spoken turns. At a wait of 0.3 s, 9 of 31 number turns are cut (29%), and 13 of all 141 turns (9%). At 0.8 s, 2 number turns and 4 turns in all (3%) are cut. From 0.7 s on, the lines are almost flat. Even at 1.5 s, 2 turns are still cut: two callers pause for longer than 1.5 s.
| Wait | Turns cut (of 141) | Order-number turns cut (of 31) | Time saved per answer, against 0.8 s |
|---|---|---|---|
| 0.3 s | 13 (9%) | 9 (29%) | 0.5 s |
| 0.5 s | 10 (7%) | 6 (19%) | 0.3 s |
| 0.7 s | 4 (3%) | 2 (6%) | 0.1 s |
| 0.8 s | 4 (3%) | 2 (6%) | none |
So Tomás’s 0.3 s would save half a second on every answer. In exchange, it cuts almost one order number in three. These callers pause because of how our synthetic voices read numbers. The voices leave short gaps between digits, and four calls have a long pause planted on purpose. Real callers pause in their own ways. Measure the pauses on your own calls before you choose a wait.
The IVR’s fix: check the content
Section titled “The IVR’s fix: check the content”The IVR has the same problem, and Module 7 gave it a fix. When it hears fewer than six digits, it asks for the rest. Here is its recorded replay of call I-049:
40.08 s say Please say or type your six-digit order number. 45.70 s heard Seven nine eight. 45.94 s say I heard 7 9 8. Please say the rest of the number. 52.15 s heard 090. 52.40 s say Order 7 9 8, 0 9 0. Is that right? Say yes or no, or press 1 for yes and 2 for no. 61.70 s heard Correct. 61.92 s say Your Mistral fan shipped on October 11 and is on its way. Thank you for calling Larkfield. Good 61.92 s end status_givenThe notebook prints these lines (cut at 95 characters). The times are the call’s clock in the replay.
The wait of 0.8 s cut the turn, and the content check repaired it. This is one reason why the IVR’s results barely move with the wait. With 0.5 s, it passed 38 of 60 test calls; with 0.8 s and with 1.5 s, it passed 39. A voice agent needs the same check, in code: an LLM that gets “seven nine eight” may simply say “I can’t find that order”.
Barge-in: the caller talks over the agent
Section titled “Barge-in: the caller talks over the agent”The opposite problem comes next. The agent speaks, and the caller starts to talk. This is called a barge-in: the caller “barges in” on the voice. It happens often when the answer is long, or wrong. The caller hears “your order number is six hundred…” and wants to say “no!”.
Larkfield’s IVR does not listen while it speaks. Its engine starts to listen after each prompt. A caller who talks during the prompt is not heard. That is simple, but slow for a caller who already knows what to say. A voice agent should listen all the time, and stop its voice when the caller talks.
Text description of the diagram
The diagram has four participants, from left to right: the caller, the VAD that always listens, the agent and the voice. The agent sends its answer to the voice: “Your order number is six hundred three, one hundred thirty-seven. Is that correct?”. The audio starts to play. Then the caller starts to talk. The VAD sees speech for 0.3 s or more, and it is not a short “mm-hm”, so it is a barge-in. The VAD tells the voice to stop at once. The voice tells the agent what it played: 2.0 s of the 5.0 s answer, “Your order number is six”. The agent keeps only that part in the call’s history. Then the caller says “No, it is six zero three one three four”. This turn goes through end-of-turn detection and recognition as usual.
Three rules for a barge-in
Section titled “Three rules for a barge-in”- Stop fast. Every tenth of a second of voice after the caller starts is speech on top of speech. Stop the audio as soon as the VAD is sure.
- Do not stop for back-channels. A back-channel is a short sound that a listener makes to say “go on”: “mm-hm”, “yes”, “okay”, “oui”, “d’accord”. If the agent stops for each one, it never finishes a sentence. A simple rule: stop only after about 0.3 s of speech that is not one of these short words. The rule is a starting point to test, not a measured value.
- Keep only what was played. The agent’s call history must hold the words that the caller heard, not the whole answer.
The half-spoken answer
Section titled “The half-spoken answer”The third rule needs a closer look. The recorded agent answer of call I-054 lasts 5.0 s: “Your order number is six hundred three, one hundred thirty-seven. Is that correct?”. Suppose the caller talks at 2.0 s. At an even speed, the caller heard about “Your order number is six”. The notebook computes this estimate; a real voice gives exact word times.
Now suppose the agent keeps the whole answer in its history. The caller says “no, it’s six zero three one three four”. The LLM reads its own history: it asked “Is that correct?”, and the caller said “no”. So far, so good. But the caller could also have said “yes, go on”. Then the LLM would take “yes” as the answer to a question that the caller never heard. It would look up the wrong order. Keep only the played part, and the LLM knows that the question was never asked.
When to use each, and what it costs
Section titled “When to use each, and what it costs”| Method | Use it when | What it costs |
|---|---|---|
| A silence wait of 0.7–0.8 s | always, as the base rule | about 0.8 s on every answer |
| A shorter wait (0.3–0.5 s) | short yes or no answers | callers who pause get cut off |
| A content check | you know the form of the answer: digits, a date, yes or no | code for each kind of answer |
| A model that predicts the end | many open questions | another model to run and to test; not measured here |
| Barge-in | always for a voice agent | echo control, and more tests |
The content check is cheap and was measured to work in the IVR. Use it first. Change the silence wait only with the trade-off chart of your own calls in front of you.
Common mistake
The agent interrupts itself
This is a constructed case: the course did not record it. It is a common failure when people test a voice agent on a laptop.
Symptom: Amira adds barge-in to the test phone. She calls it with her laptop’s speakers on. The agent stops after two or three words of every answer, and then waits.
Cause: the agent’s own voice comes out of the speakers and goes back into the microphone. This is echo. The VAD hears speech, so the agent thinks that the caller has barged in. It stops its own voice.
Fix: stop the echo before the VAD hears it. Wear headphones when you test: the project’s README asks for them. Turn on the browser’s echo cancellation: the test phone page already asks for it when it opens the microphone (echoCancellation: true). On a real phone line, the phone system removes most of the echo. Test barge-in with headphones and with speakers, and note which one you used.
Consequence in AI work: a test that ignores the room tests a different system. Write the test conditions next to every result: the microphone, the speakers, the line.
Guided practice: the pauses in the notebook
Section titled “Guided practice: the pauses in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells. The setup installs faster-whisper and downloads 145 small files. It ends with
Ready: 145 files in larkfield-calls. - In section 2, predict the length of the pause in call I-049. Then run the cell. You see two speech parts and a pause of 1.95 s.
- In section 3, run the end-of-turn rule for four waits. Only the wait of 2.0 s keeps the whole number.
- In section 4, run the pauses of all 141 turns. Compare the counts with the table on this page.
- In section 5, read the IVR’s recorded call. Find the moment when it asks for the rest of the number.
Your turn: the shortest safe wait
Section titled “Your turn: the shortest safe wait”Larkfield accepts that at most 3% of the spoken caller turns are cut early. In section 7 of the notebook, find the shortest wait that cuts at most 3% of all spoken turns. Before you run it, predict the answer from the chart above.
Show the answer
shortest_wait = min(w for w in waits if cut_all[w] / len(rows) <= 0.03)print(shortest_wait, cut_all[shortest_wait], "of", len(rows))0.7 4 of 141The shortest wait is 0.7 s. It cuts 4 of the 141 turns, the same as 0.8 s, and it saves 0.1 s on every answer. In the latency budget, the median wait would go from 2.10 s to about 2.00 s. That is a real gain, but small. A content check on order numbers would let you go lower for the other questions.
Knowledge check
Section titled “Knowledge check”Check your understanding
A caller says "three four one", pauses 1.2 seconds, then says "two two eight". The end-of-turn wait is 0.8 s. What does the agent hear, and what are two ways to fix it?
Show the answer
The pause (1.2 s) is longer than the wait (0.8 s), so the turn ends in the pause. The agent hears only “three four one”. Fix 1: a content check. An order number has six digits, so after three digits the program keeps listening, or asks for the rest, as the IVR does. Fix 2: a longer wait of about 1.3 s, but then every caller waits longer after every turn. The content check is better: it adds no wait to other turns.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
What is a back-channel, and why must barge-in ignore it?
Show the answer
A back-channel is a short sound that a listener makes to say “I am listening, go on”: “mm-hm”, “yes”, “okay”, “oui”. The caller does not want to interrupt. If barge-in stops the voice for every back-channel, the agent never finishes a sentence. So stop only after a longer stretch of speech, about 0.3 s (a value to test). Or stop when the words are not a back-channel.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The voice stops after 2.0 s of a 5.0 s answer that ends with "Is that correct?". Why must the agent's history keep only the played part?
Show the answer
The caller heard only the first part of the answer, about “Your order number is six”. The question “Is that correct?” was never heard. If the history keeps the whole answer, the LLM thinks that it asked the question. Then it may take the caller’s next words, such as “yes”, as the answer. Then it acts on a number that the caller never confirmed. With only the played part, the LLM knows that the question is still open.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- End-of-turn detection decides when the caller has finished: a silence wait, a check on the content, or a model that predicts the end.
- A shorter wait answers sooner, but cuts callers who pause. On Larkfield’s 141 spoken caller turns, a wait of 0.3 s cuts 9% of turns and 29% of order numbers. A wait of 0.7–0.8 s cuts 3% and 6%.
- A content check (“six digits, or keep listening”) repaired the cut number in the IVR, so its results barely change with the wait.
- Barge-in: stop the voice fast when the caller talks over it. Ignore back-channels such as “mm-hm”, and stop the echo of the agent’s own voice.
- Keep only the played part of an interrupted answer in the call’s history.
You can now build each part of a voice agent and set its waits. Some newer models replace the whole pipeline with one model that hears and speaks. Next, you see what they change, and how to judge a voice agent as a whole.