Why VAD makes recognition better
- Lesson 9
- 40 min
- Module 3: Find the speech: voice activity detection
Where we are
We can now find the speech in a call. Does that help the recogniser? In this lesson, we cut long silences before recognition, see Whisper stop inventing text, and measure the change in error and in cost per audio hour.
Lesson 9 of 36 · Speech AI: Transcribe, Understand and SpeakWhy VAD makes recognition better
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Put voice activity detection in front of Whisper, and measure what it really changes. Look at the text that Whisper invents in silences, the word error rate and the cost of an hour of audio. Then tune the detector, so that it removes the invented text without throwing away the caller’s words.
NotebookDo this lesson in a notebook
The notebook finds the invented sentences in the recorded transcripts and scores Whisper small with and without VAD on the 60 test calls. Then it runs Whisper tiny live on eight calls, so that you can tune the threshold yourself. It downloads Whisper tiny (about 75 MB) once. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Silero VAD, its threshold and missed speech, from WebRTC VAD and Silero VAD.
- Substitutions, deletions and insertions in the word error rate, from Word error rate, step by step.
- The real-time factor (RTF): compute time divided by audio time, from Read the result in Set up the speech project. An RTF of 0.05 means 3 seconds of compute for a minute of audio.
What you need on your computer (only for the guided practice)
- Your project folder
larkspeechwith the files of this module:pytest -qprints18 passed.
Tested on macOS with Python 3.12.13.
A helper who taps the note-taker’s shoulder
Section titled “A helper who taps the note-taker’s shoulder”Go back to the note-taker of the first lesson of this module, the one who cannot put her pen down. Give her a helper. The helper listens too, and taps her shoulder when nobody speaks: “Stop writing now.” During the hold music, she stops. Her notes get shorter and cleaner, and she finishes sooner.
But the helper must tap at the right moments. If he taps whenever the caller speaks softly, she stops writing in the middle of a real answer. Her notes are now clean and wrong: they miss what the caller said. A helper is only as good as his timing.
The problem: Whisper writes words that nobody said
Section titled “The problem: Whisper writes words that nobody said”In call C-0251, the caller wants to stop an order. The agent asks for the order number. The caller says “Hold on, let me find it.” Then the caller is silent for 21.4 seconds. Listen to 8 seconds from the middle of that wait:
Transcript. No words from the agent or the caller. Quiet background talk: several voices far away, as in a busy room.
Whisper small transcribed this call without a VAD. It is the most accurate recogniser that this module uses. Here is what it wrote for the whole wait, between “Hold on, let me find it.” and the caller’s “Sorry about that”:
Hello there, could you please explain how do I travel last year? Hi, I’m good morning. Could you please tell me how to train you to buy a pair of shoes? I’m a fan of my iRomite, so I can buy a pair of shoes. I appreciate your help with my car.
Nobody said this to Larkfield. Whisper took a murmur in the background and wrote confident, complete sentences. They sit in the middle of the transcript as if the caller said them. A search for “pair of shoes” now finds this call. A summary model, in Module 6, could read them as the caller’s words.
This is the second note-taker of the first lesson: asked to write something, Whisper writes something. Whisper learned from a very large amount of audio with transcripts. For every stretch of audio, it produces the text that a transcript would most likely have there. Where there is no clear speech, it can still produce likely-looking text. People call this hallucination; this course says invented text. Other examples from our calls:
| Recogniser | Call | What fills the wait | What it wrote there |
|---|---|---|---|
| Whisper small | C-0104 (French) | Music on hold | “Merci. Merci.” |
| Whisper small | C-0230 (French) | Line hiss only | “Bonjour.” |
| Whisper base | C-0251 | Background talk | “I just forgot.” |
| Whisper tiny | C-0230 (French) | Line hiss only | “Les olympiques,” |
On the 15 validation and test calls with a long planted wait, Whisper small wrote 65 words inside the waits. Whisper tiny and base wrote fewer, 3 each.
The idea: send Whisper only the speech
Section titled “The idea: send Whisper only the speech”The helper’s job has a simple form in code. faster-whisper, the library that runs Whisper in this course, has an option vad_filter=True. With it, faster-whisper does four things:
- It runs Silero VAD on the whole call, with its own default settings. The threshold is 0.5. A region ends after 2 seconds below the threshold. Each region gets 0.4 seconds of padding on each side.
- It keeps only the speech regions, and joins them end to end into a shorter recording.
- It transcribes the shorter recording.
- It moves every time back to its place in the original call, so word times stay right.
A tiny example: one wait of 20 seconds
Section titled “A tiny example: one wait of 20 seconds”Take a made-up call of 60 seconds. Somebody speaks for the first 18 seconds and the last 22 seconds. In between, the caller looks for an order number for 20 seconds.
Text description of the chart
Three rows on an axis from 0 to 60 seconds, with the wait from 18 to 38 seconds shaded. The call row has two bars: speech from 0.5 to 18 seconds, and from 38 to 59.5 seconds. The VAD row has two slightly wider bars, from 0.1 to 18.4 and from 37.6 to 59.9 seconds, because of the padding. The last row shows what Whisper gets: the two regions joined end to end, with no wait between them. They run from 0 to 18.3 and from 18.3 to 40.6 seconds.
Now the cost, with every step visible. Say Whisper’s real-time factor is 0.09 on this computer. Silero needs 90 ms of compute for a minute of audio (the last lesson measured 84 to 93 ms).
| Step | Without VAD | With VAD |
|---|---|---|
| Audio that Whisper transcribes | 60 s | 18.3 + 22.3 = 40.6 s |
| Whisper’s compute (× 0.09) | 5.4 s | 3.65 s |
| Silero’s compute (1 minute × 90 ms) | none | 0.09 s |
| Total compute | 5.4 s | 3.74 s |
| Real-time factor (total ÷ 60 s) | 0.090 | 0.062 |
| Cost per audio hour (RTF × US$0.17) | US$0.0153 | US$0.0105 |
The last row uses the course’s assumption for a CPU server: US$0.17 an hour for 4 virtual CPUs (costs.json). One hour of audio needs RTF hours of compute, so the cost per audio hour is the RTF times US$0.17. Here the VAD saves about 31% of the compute, because a third of the call was a wait.
Larkfield: the measurement
Section titled “Larkfield: the measurement”Priya measures it for real, with Whisper small and faster-whisper’s VAD at its default settings.
On the calls with a long wait
Section titled “On the calls with a long wait”First, the 15 validation and test calls with a planted long wait (music on hold or line hiss):
| Whisper small, 15 calls with a long wait | Without VAD | With VAD (default) |
|---|---|---|
| Words written inside the waits | 65 | 1 |
| Inserted words (all errors of this kind) | 99 | 22 |
| Real-time factor | 0.092 | 0.053 |
| Cost per audio hour (× US$0.17) | US$0.0156 | US$0.0090 |
| Word error rate | 13.2% | 16.1% |
The VAD did what the helper should do. The invented words nearly vanish: 65 become 1. Compute falls by 42%. And yet the word error rate goes up, from 13.2% to 16.1%. Something else got worse.
On all 60 test calls
Section titled “On all 60 test calls”Priya checks the whole test set, with the same recogniser and the same normalisation guide:
- Without VAD (WER 8.6%)
- With VAD, default settings (WER 13.9%)
Text description of the chart
A column chart with three pairs of columns. Substitutions: 371 without VAD, 371 with VAD. Deletions: 115 without VAD, 511 with VAD. Insertions: 142 without VAD, 138 with VAD. Only the deletions change much: they are more than four times higher with the VAD.
The word error rate goes from 8.6% to 13.9%. Almost all the extra errors are deletions: words that the caller or the agent said and that are missing from the transcript. On these calls, most waits are short, so insertions hardly change. And the real-time factor barely moves, from 0.075 to 0.073, because the VAD keeps 85% of the audio. On normal calls, there is little silence to cut.
Tomás is surprised: “VAD should make recognition better. It is in the name of this lesson.” Priya: “It removed the invented text. It also removed real words. We count both.”
Where the words went
Section titled “Where the words went”The deletions come from the problem of the last lesson: Silero is unsure about phone speech. At its default threshold of 0.5, faster-whisper’s VAD missed 9.0% of the speech time in the 60 test calls. (It misses less than the project’s setting of the last lesson, 18.8%, because faster-whisper pads each region with 0.4 seconds.) Words in the missed stretches never reach Whisper.
The call that lost the most words is C-0194, a street-noise call on a landline: 63 more words deleted with the VAD. The chart shows why.
Text description of the chart
Three rows over 44 seconds. True speech has nine bars, one per turn, covering most of the call. The VAD at 0.5 has six regions: one for the first 8.6 seconds, then only short pieces between 22 and 41 seconds. From 8.6 to 22.8 seconds, a stretch with three turns, it finds no speech at all: missed 55.1%. The VAD at 0.3 has two long regions that cover almost the whole call: missed 1.5%.
At 0.5, the VAD drops 14 seconds in the middle of this call, three whole turns. At 0.3, it keeps almost everything. So the fix is the one that the last lesson suggested: lower the threshold for the phone line. But a lower threshold also keeps more of the waits, and the waits are where the invented text comes from. Is there a setting that keeps both wins?
Tune the detector for the phone line
Section titled “Tune the detector for the phone line”Priya tries five settings with Whisper small. She chooses on the 50 validation calls. Then she checks the chosen setting once on the test calls. Train, validation and test sets in the machine learning course explains why. The invented words are counted on the 15 calls with a planted wait (validation and test calls together): a special check, not a score.
The five settings change two things. The threshold sets how sure Silero must be. The minimum silence sets how long a stretch below the threshold must last before the VAD cuts it. faster-whisper’s default is 2 seconds. At 4 seconds, the VAD cuts only long waits and leaves the short pauses inside a turn alone. That is exactly the lesson’s aim: cut long silences, not quiet speech.
| Setting | WER (validation) | Deletions | Insertions | Order numbers missed | Words in the waits (15 calls) | RTF |
|---|---|---|---|---|---|---|
| No VAD | 7.2% | 41 | 158 | 5 of 38 | 72 | 0.084 |
| Threshold 0.5 (default) | 11.8% | 378 | 76 | 10 of 38 | 1 | 0.077 |
| Threshold 0.3 | 8.2% | 88 | 174 | 6 of 38 | 97 | 0.073 |
| Threshold 0.2 | 7.2% | 47 | 147 | 6 of 38 | 71 | 0.086 |
| Threshold 0.3, minimum silence 4 s | 7.9% | 65 | 177 | 6 of 38 | 43 | 0.079 |
These runs come from one session of the course’s computer, a busier one than in the first measurement. Here, the run without VAD wrote 72 words in the waits, not 65, and its RTFs are higher. Compare the rows with each other, not with the earlier tables.
- No VAD
- Threshold 0.5 (default)
- Threshold 0.3
- Threshold 0.2
- 0.3 and 4 s minimum silence
Text description of the chart
A scatter plot with five dots. No VAD: 72 invented words, WER 7.2%, low and on the right. Threshold 0.2: 71 words, 7.2%, next to it. Threshold 0.3: 97 words, 8.2%, the furthest right. Threshold 0.3 with a 4-second minimum silence: 43 words, 7.9%. Threshold 0.5, the default: 1 word, 11.8%, at the top left. Without threshold 0.3, the dots form a curve: less invented text costs a higher WER.
Read the chart as a trade-off, as with every threshold in this module:
- No setting beats “no VAD” on the WER. On validation, Whisper small without VAD has the fewest errors, 7.2%. Threshold 0.2 comes close, with only 3 more errors. Every VAD setting drops some speech.
- The default removes the invented text and pays 4.6 points of WER. That is 337 more deleted words on 50 calls, and 5 more missed order numbers.
- Threshold 0.3 with a 4-second minimum silence keeps the WER within 1 point of no VAD (7.9%). It cuts the invented words from 72 to 43.
- Threshold 0.2 brings almost all the invented text back (71 words): it keeps nearly every wait, so it does almost nothing.
- Threshold 0.3 alone wrote even more than no VAD (97 words). 94 of them are in one call, C-0251, where Whisper wrote the same invented sentences again and again.
Priya chooses threshold 0.3 with a 4-second minimum silence for Larkfield’s batch transcripts. Grace agrees: a missed order number costs a follow-up call, and an invented sentence can mislead a summary. Less than one point of WER for 40% less invented text is a fair trade here.
The check on the test calls
Section titled “The check on the test calls”Priya runs the chosen setting once on the 60 test calls, with the other settings of the same session for comparison:
| Test calls, same session | No VAD | Default (0.5) | Chosen (0.3, 4 s) |
|---|---|---|---|
| WER | 8.5% | 14.0% | 7.9% |
| Deletions | 115 | 508 | 83 |
| Order numbers missed | 8 of 52 | 12 of 52 | 5 of 52 |
| Negations missed (“not”, “pas”) | 1 of 29 | 3 of 29 | 0 of 29 |
| RTF | 0.087 | 0.081 | 0.073 |
| Cost per audio hour (× US$0.17) | US$0.0147 | US$0.0138 | US$0.0124 |
On the test calls, the chosen setting is even a little better than no VAD. The difference of 0.6 points is within the noise level of these 60 calls. The honest summary is: the tuned VAD costs no clear accuracy, removes 40% of the invented text, and saves about 16% of compute. The default VAD costs 5.5 points.
What a VAD cannot fix
Section titled “What a VAD cannot fix”Even the chosen setting leaves 43 invented words. Most of them are in one call, C-0251, the call with background talk in the wait. At threshold 0.3, Whisper wrote there: “Could you please tell me what you’re finding on my diorhobitarracotoraphera?” The VAD keeps that wait because background talk is speech. No threshold can remove it without removing the caller too. That call needs another tool: the caller’s own channel, and who speaks when, in Module 5.
Guided practice: tune it yourself
Section titled “Guided practice: tune it yourself”The project gets the --vad option of speech transcribe in the next module. For now, do the tuning in the notebook of this lesson, with Whisper tiny on eight validation calls. Its key cell runs the recogniser with and without the VAD:
segments, _ = model.transcribe(audio16[call_id], language=c["language"], beam_size=1, vad_filter=threshold is not None, vad_parameters=None if threshold is None else VadOptions(threshold=threshold), condition_on_previous_text=False)The notebook runs it without VAD and then at four thresholds, on about 7 minutes of audio. The output of the solution notebook on the course’s computer (your rtf will differ; the words were the same on every run):
no VAD: {'wer': 0.099, 'deletions': 16, 'insertions': 20, 'rtf': 0.015}0.5 {'wer': 0.238, 'deletions': 160, 'insertions': 12, 'rtf': 0.013}0.4 {'wer': 0.155, 'deletions': 68, 'insertions': 19, 'rtf': 0.015}0.3 {'wer': 0.119, 'deletions': 34, 'insertions': 20, 'rtf': 0.017}0.2 {'wer': 0.097, 'deletions': 15, 'insertions': 18, 'rtf': 0.017}It is the same pattern as with Whisper small, only stronger. At the default 0.5, Whisper tiny loses 160 words on eight calls, and the WER more than doubles. Each lower threshold gives words back. Tiny invents little text in waits, so on these calls the lowest threshold wins.
In your project, compare Silero’s missed speech at the default threshold and at 0.3, on all 50 validation calls. This uses the project’s own settings of the last lesson, so the numbers differ from faster-whisper’s padded VAD:
speech vad --method silerospeech vad --method silero --threshold 0.3The output on the course’s computer:
silero on 50 validation calls (phone audio)noise calls missed speech false speech ms per minutebabble 13 31.3% 24.8% 98.6quiet 23 21.7% 3.8% 99.1street 14 11.8% 4.6% 100.7silero {'threshold': 0.3} on 50 validation calls (phone audio)noise calls missed speech false speech ms per minutebabble 13 11.8% 56.5% 100.9quiet 23 10.0% 11.0% 101.1street 14 7.3% 13.1% 101.3At 0.3, missed speech falls by about half in every group. That is the speech that comes back to the recogniser.
Common mistakes
Section titled “Common mistakes”Common mistake
'Turn VAD on' as a fix, without a measurement
Symptom: a team adds vad_filter=True because “it stops hallucinations”. The invented text goes away, and nobody notices that transcripts now miss words.
Cause: the makers of the VAD chose its default settings on other audio. On Larkfield’s phone line, the default threshold of 0.5 drops quiet speech: deletions grew from 115 to 511 on the test calls.
Diagnose: score the recogniser with and without the VAD on the same calls. Look at deletions and insertions apart, not only at the WER. Find the calls that lost the most words, and draw their timelines.
Fix: tune the threshold on validation calls, check the chosen setting once on the test calls, and keep the old transcripts for comparison.
Common mistake
One run says the invented text is gone
Symptom: you run Whisper once on a few calls and see no invented text. You conclude that the problem has gone.
Cause: invented text is unstable. In the course’s repeat run of Whisper small without VAD, call C-0195 got 8 words in the music on hold: “The call is reached. I appreciate your help.” The first run had written none there. In call C-0251, the VAD kept most of the wait (it hears the background voices), and whether Whisper writes words there changes from run to run.
Fix: count invented words on many calls with known waits, and on more than one run. Our calls have the waits planted on purpose, so the count is exact. On real calls, listen to a sample of the waits.
When to use a VAD before the recogniser
Section titled “When to use a VAD before the recogniser”Use it when your calls have long stretches without speech: music on hold, a caller who looks for a document, a line left open. It removes most invented text there, and it cuts compute in proportion to the silence. Tune its threshold on your own audio first, and measure deletions.
Think twice when calls are mostly speech, as most Larkfield calls are. The compute saving is small (RTF 0.075 to 0.073 on the test calls), and every missed stretch is lost words. A VAD also cannot remove background talk, because it is speech.
What it costs: Silero adds about 90 ms of compute per audio minute, a few minutes a month for all of Larkfield’s calls. The real cost is the risk of lost words, and the time to measure it.
Knowledge check
Section titled “Knowledge check”Check your understanding
Show one passage of invented text that VAD removes, with the numbers.
Show the answer
In call C-0251, the caller is silent for 21.4 seconds while background voices talk quietly. Without VAD, Whisper small wrote there “Could you please tell me how to train you to buy a pair of shoes? I’m a fan of my iRomite, so I can buy a pair of shoes.” Nobody in the call said it. With faster-whisper’s VAD, it wrote nothing there. On the 15 calls with a planted wait, words inside the waits went from 65 to 1, and insertions from 99 to 22. The RTF went from 0.092 to 0.053: about US$0.0156 to US$0.0090 per audio hour at US$0.17 per server hour. But the WER on those calls rose from 13.2% to 16.1%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'VAD removes hallucinations, so turn it on everywhere.' What happens on Larkfield's test calls with the default settings, and why?
Show the answer
The WER of Whisper small rises from 8.6% to 13.9%. Deletions grow from 115 to 511, while substitutions and insertions hardly change. The cause is Silero at its default threshold of 0.5. It is unsure about phone speech and missed 9.0% of the speech time, so those words never reach Whisper. On C-0194, it dropped three whole turns. The compute saving is small on normal calls (RTF 0.075 to 0.073), because they are mostly speech.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: choose a VAD setting for Larkfield's batch transcripts. Give the evidence on the word error rate, the invented words and the compute, and say how you chose it.
Show the answer
Threshold 0.3 with a 4-second minimum silence. Choose it on the 50 validation calls. There, its WER is 7.9%, against 7.2% without VAD and 11.8% with the default. Its invented words in the waits of 15 calls: 43, against 72 without VAD. Check it once on the test calls: WER 7.9% (no VAD 8.5% in the same session, default 14.0%) and order numbers missed 5 of 52 (default 12). Its RTF is 0.073 against 0.087: about US$0.0124 against US$0.0147 per audio hour at US$0.17 a server hour. It does not remove the invented text in C-0251, because the background talk there is speech.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Whisper can write invented text in long stretches without the caller’s speech. It wrote 65 words in the waits of 15 calls, such as “how to train you to buy a pair of shoes”.
- faster-whisper’s
vad_filter=Trueruns Silero, keeps the speech regions, joins them and maps the times back. Cost per audio hour is the RTF times the server’s price per hour. - With the default settings, the VAD removed the invented words (65 to 1) and cut compute on those calls (RTF 0.092 to 0.053). But it raised the WER on the test calls from 8.6% to 13.9%, because deletions grew from 115 to 511.
- Missed phone speech causes it. Tune the threshold on validation calls, check once on the test calls, and count deletions, insertions and invented words apart.
- Invented text is unstable: measure it on many calls, more than once.
Module 3 in one paragraph. A voice activity detector finds the speech in a call. The energy VAD is nearly free but calls music and background talk speech. WebRTC VAD is fast and misses little speech, but it calls music speech too. Silero VAD rejects music and street noise, but on Larkfield’s phone line it is unsure about speech, so its threshold must come down. In front of Whisper, a VAD removes invented text in long waits, and a badly set VAD removes real words. You measure both, by time and by words, on Larkfield’s synthetic calls.
You can now find the speech and send only that to a recogniser. But which recogniser? Whisper comes in sizes from tiny to large, and older systems still answer many phone lines. Next, we look at how a recogniser turns sound into text, so we can compare the families fairly.
Next: From sound to text