Skip to content
Français

WebRTC VAD and Silero VAD

  • Lesson 8
  • 35 min
  • Module 3: Find the speech: voice activity detection

Where we are

The energy threshold calls music on hold "speech". Better detectors exist and run on a laptop. In this lesson, we compare WebRTC VAD and Silero VAD on the noise set by missed speech, false speech and speed.

What you will be able to do

Compare two voice activity detectors that look at the shape of the sound: WebRTC VAD and Silero VAD. Compare them with the energy threshold of the last lesson. Measure what each one misses, what noise each one keeps and how fast each one is. Then choose a detector and a setting for Larkfield’s phone line.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook runs the three detectors on the call with music on hold and on the noise set. It prints Silero’s probabilities, times each detector and shows the sample-rate mistake of this lesson. Then it asks you to choose a Silero threshold for six calls. It installs WebRTC VAD and faster-whisper (which brings Silero VAD) and downloads no recogniser. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • The energy VAD, the hangover, missed speech and false speech, from Speech or silence?.
  • Sample rates: phone audio has 8,000 samples a second, wideband audio 16,000, from Audio as data.

What you need on your computer (only for the guided practice)

  • Your project folder larkspeech with the files of the last lesson: pytest -q prints 18 passed, and speech vad --method energy --limit 20 works.

Tested on macOS with Python 3.12.13.

You sit in a busy café with a friend. The radio plays a song. A cup falls on the floor. People talk at the next table. Your friend starts a sentence, and you notice at once. You do not listen for the loudest sound: the song and the cup are louder than your friend. You listen for the shape of a voice.

A voice has a shape that music and noise do not have. Its pitch moves up and down. It comes in syllables, a few each second. Its sound changes all the time, from a vowel to a consonant and back. A note of music stays on one pitch. A falling cup is one short burst. Your ear learned these shapes over many years.

The energy VAD of the last lesson has no ear for shape. It measures loudness only. On the noise set, it called 100% of the music on hold speech, 97% of the background talk and 33% of the street noise. On call C-0022, it called 17.1 of the 17.7 seconds of music speech. And a quieter phone line made it miss three quarters of the speech.

Priya needs a detector that hears the difference between a voice and a tune. Two free detectors run on a laptop and are used in many products. This lesson compares them with the energy VAD, on the same calls and with the same two numbers.

The idea: detectors that look at the shape

Section titled “The idea: detectors that look at the shape”

WebRTC VAD comes from the WebRTC project, the open-source code that web browsers use for voice and video calls. Google first wrote it, so it had to be very small and very fast. For each frame of 10, 20 or 30 ms, it measures the energy in six frequency bands, from low to high. It compares this pattern with a simple model of speech and a simple model of noise. It keeps updating the noise model as the call goes on. It answers yes or no. Its mode, from 0 to 3, sets how strict it is: in mode 3, it says “speech” least often.

Silero VAD is a small neural network: a model made of many numbers that learned from examples, here many hours of speech and of noise. (The deep learning course explains how such a model learns.) The model file is only 1.2 MB, and it is free (MIT licence). For each chunk of 32 ms of 16 kHz audio, it gives a probability of speech: a number between 0 and 1. A threshold turns the probability into yes or no. faster-whisper is the recogniser library of this course. It ships a copy of Silero, so your project already has it.

Both detectors still need the extra rules of the last lesson. A region starts when a chunk passes the threshold. It ends only after a short run of chunks below it: 0.3 seconds in the project. Silero also uses a lower second threshold, 0.15 below the first, to decide that speech has stopped.

Call C-0022 is the call with music on hold from the last lesson. From 21.9 to 24.9 seconds, the agent says “Thank you. Let me check that for you. Please hold.” Here are Silero’s probabilities for 10 chunks in the middle of that sentence, a third of a second in all. These are measured numbers from the call:

Chunk starts at (s) 22.75 22.78 22.82 22.85 22.88 22.91 22.94 22.98 23.01 23.04
Probability 0.476 0.506 0.501 0.485 0.446 0.337 0.216 0.264 0.496 0.469
Speech at 0.5? no yes yes no no no no no no no
Speech at 0.3? yes yes yes yes yes yes no no yes yes

The agent speaks in all 10 chunks. There is only a short pause between two words, at 22.94 and 22.98 seconds. Yet at the usual threshold of 0.5, only 2 of the 10 chunks are “speech”. At 0.3, 8 of the 10 are. Silero is unsure about this voice: its probabilities stay close to 0.5 instead of near 1. The chart below shows that this is true for the whole stretch, not only for these 10 chunks.

Silero's probability for each 32 ms chunk of C-0022, from 18.4 to 26.0 seconds (measured, phone audio). During speech, it stays mostly between 0.3 and 0.6. At 0.5, Silero misses almost half of the agent's sentence; at 0.3, it finds nearly all of it.
Text description of the chart

The top panel shows a line that is near 0 between turns and rises to about 0.3 to 0.6 during speech. Two dashed lines mark the thresholds 0.5 and 0.3. During the caller’s turn (19.1 to 21.1 s) and the agent’s turn (21.9 to 24.9 s), the line goes above 0.5 only for short moments. It stays above 0.3 most of the time. At 25.4 s, the music starts: the line jumps to about 0.4 and then slowly falls. Below are three rows. True speech has two bars, one for each turn. Silero at 0.5 has three bars. One covers the caller’s turn. Two cover only pieces of the agent’s turn, with a gap of almost a second in the middle. Silero at 0.3 covers both turns almost completely. It also has a short bar from 25.3 to 26.2 seconds, where the music starts.

Why is Silero unsure? A likely reason is the phone line. The line removes every sound above 3,400 Hz, and much of the “shape” of a voice lives there. To test this, the course ran Silero at 0.5 on the wideband copies of the same 50 validation calls. They have 16 kHz audio, without the phone line’s cut. It missed 4.4% of the speech there, against 18.8% on the phone audio. The same detector, with the same voices, loses four times more speech on the phone line.

Priya runs the three detectors on call C-0022 with the project’s default settings. They are energy at −40 dBFS, WebRTC VAD in mode 2 and Silero at 0.5.

Call C-0022 (70.4 s, synthetic): energy and WebRTC VAD call the music on hold speech; Silero VAD does not, but it misses 41% of the speech.
Text description of the chart

Four rows of bars over 70 seconds, with the music on hold shaded from 25.1 to 42.8 seconds. True speech has 11 bars, one per turn. The energy VAD covers the speech and the whole music on hold: missed 19.0%, false 62.6%. WebRTC VAD looks almost the same: it covers the speech and the music, missed 10.6%, false 63.8%. Silero VAD at 0.5 has nothing during the music, but its bars cover only parts of each turn: missed 41.1%, false 1.6%.

Energy and WebRTC VAD call the music speech: 17.1 and 17.2 of the 17.7 seconds. Silero calls none of it speech. It is the only detector here that hears that a tune is not a voice. But on this call it pays for that with 41.1% missed speech, the worst of the three. At 0.3, Silero misses 12.0% of the speech in C-0022, and it still calls only 0.9 seconds of the music speech.

Tomás likes the result: “Silero is a neural network. It wins.” Priya answers: “It wins on music. At its default threshold, it loses the most words. Let’s measure it on more than one call.”

The noise set gives 20 seconds of each sound that is not speech, through the phone line. The share of each that the detectors call speech (measured):

Noise Energy (−40 dBFS) WebRTC VAD (mode 2) Silero VAD (0.5)
Music on hold 100% 100% 3%
Background talk 97% 100% 100%
Street 33% 25% 0%
Line hiss 0% 0% 0%

Silero rejects the music and the street. WebRTC VAD does a little better than energy on the street, and no better on music. And every detector calls background talk speech, because it is speech. A voice detector cannot know that the voice belongs to someone in the caller’s room, not to the caller.

The course measured each detector on the 50 validation calls, by the noise on the caller’s side. The chart puts the two numbers side by side for each detector. Missed speech grows to the left: those are lost words. False speech grows to the right: that is noise that the recogniser must still process.

Missed speech (left) and false speech (right) on the 50 validation calls, by the noise on the caller's side (measured, phone audio, default settings). Silero keeps the least noise, and on these phone calls it misses the most speech.
Text description of the chart

Three groups of three rows. Quiet line, 23 calls. Energy misses 15.2% and keeps 3.7% false speech. WebRTC VAD: 9.1% and 10.2%. Silero: 19.0% and 10.4%. Background talk, 13 calls. Energy: 2.9% and 75.7%. WebRTC VAD: 0.3% and 95.2%. Silero: 29.3% and 27.2%. Street noise, 14 calls. Energy: 10.5% and 17.0%. WebRTC VAD: 7.5% and 19.9%. Silero: 8.8% and 12.7%. In each group, Silero’s bar to the right is short or the shortest, and its bar to the left is long. Background talk gives the longest bars to the right for energy and WebRTC VAD.

Three things to read in the chart:

  1. Background talk is the hard case for everyone. WebRTC VAD calls 95% of the non-speech time speech, energy 76%. Silero keeps less of it, 27%, but it also misses 29% of the speech on those calls.
  2. On quiet lines, all three are close. The simple energy VAD is not worse there.
  3. Silero at 0.5 misses the most speech on the phone line: 18.8% over all 50 calls.

The setting moves each detector along the same trade-off. The table also gives the speed: milliseconds (ms) of compute for one minute of audio, on 4 CPU threads of the course’s laptop.

Detector and setting Missed speech False speech ms per audio minute
Energy, −50 dBFS 5.3% 51.6% 0.15
Energy, −40 dBFS 10.7% 26.1% 0.15
Energy, −30 dBFS 28.9% 3.8% 0.15
WebRTC VAD, mode 0 4.5% 45.3% 3.2 to 12
WebRTC VAD, mode 2 6.4% 35.0% 3.2
WebRTC VAD, mode 3 8.1% 28.9% 3.1
Silero VAD, 0.3 7.2% 30.0% 93
Silero VAD, 0.5 18.8% 15.4% 84
Silero VAD, 0.7 46.6% 6.9% 85

Silero is about 25 times slower than WebRTC VAD and 500 times slower than energy. That sounds bad, but look at the size. Larkfield takes about 4,000 calls a month of about 44 seconds each: about 2,900 minutes of audio. At 90 ms a minute, Silero needs about 4 minutes of compute a month. Speed does not decide here. The two error numbers do.

Which error matters more depends on what the VAD is for. Ask what happens after the VAD:

  • Before the recogniser, missed speech is the expensive error. A word that the VAD drops never reaches the recogniser, and nothing can bring it back. False speech only costs compute, and the recogniser often writes nothing for noise. So keep missed speech low and accept some false speech.
  • For a quick count, for example “how many seconds did the caller wait in silence?”, false speech matters as much as missed speech.

For Larkfield’s transcripts, the first case applies. So Priya does not use Silero at its default 0.5 on the phone line. At 0.3, Silero misses 7.2% of the speech, close to WebRTC VAD in mode 2 (6.4%). And it still rejects music on hold. WebRTC VAD is the fast choice if music is rare. Malik adds one fact from the phone system: music on hold plays in many Larkfield calls. That makes Silero at a lower threshold the better candidate.

This is a choice on 50 synthetic validation calls, with missed speech counted by time. The next lesson checks it where it matters: the recogniser’s errors and its compute time.

Guided practice: WebRTC VAD and Silero VAD

Section titled “Guided practice: WebRTC VAD and Silero VAD”

You added vad.py in the last lesson, so both detectors are already in your project.

  1. In your project folder larkspeech, with .venv active, run WebRTC VAD on the first 20 validation calls:

    Terminal window
    speech vad --method webrtc --limit 20

    The output on the course’s computer:

    webrtc on 20 validation calls (phone audio)
    noise calls missed speech false speech ms per minute
    babble 7 0.6% 94.1% 3.3
    quiet 9 11.3% 12.8% 3.6
    street 4 11.7% 24.0% 3.9
  2. Run Silero VAD on the same calls. The first run takes about 15 seconds, most of it to start Python and load the model.

    Terminal window
    speech vad --method silero --limit 20
    silero on 20 validation calls (phone audio)
    noise calls missed speech false speech ms per minute
    babble 7 33.5% 20.0% 85.3
    quiet 9 18.1% 5.1% 85.4
    street 4 12.7% 6.5% 85.2
  3. Run Silero again with a lower threshold:

    Terminal window
    speech vad --method silero --threshold 0.3 --limit 20
    silero {'threshold': 0.3} on 20 validation calls (phone audio)
    noise calls missed speech false speech ms per minute
    babble 7 13.9% 50.6% 96.5
    quiet 9 8.4% 15.3% 101.6
    street 4 7.7% 15.0% 100.0

    Missed speech falls in every group, and false speech rises. Your milliseconds will differ.

Change one setting. Run WebRTC VAD in mode 3, the strictest. Before you run it, predict: on the background talk calls, does false speech fall a little or a lot? And what happens to missed speech?

Terminal window
speech vad --method webrtc --mode 3 --limit 20
Show the answer
webrtc {'mode': 3} on 20 validation calls (phone audio)
noise calls missed speech false speech ms per minute
babble 7 2.8% 77.2% 3.4
quiet 9 13.2% 11.1% 3.5
street 4 14.2% 20.2% 3.5

False speech on the background talk calls falls from 94.1% to 77.2%: still most of the non-speech time. Missed speech rises in every group. A stricter mode cannot remove background voices, because they are voices.

Common mistake

Silero gets 8 kHz audio, and every time is half the real time

Symptom: Silero’s regions all end in the first half of the call, and missed speech is very high. No error message appears.

Cause: faster-whisper’s copy of Silero expects 16,000 samples a second. A phone call has 8,000. Given 8 kHz audio, the code counts 16,000 samples as one second, so every time comes out at half its real value. In the notebook, the last region of C-0022 ends at 24.2 seconds, but the call lasts 70.4 seconds. On the 50 validation calls, missed speech jumps from 18.8% to 82.0%.

Diagnose: compare the end of the last region with the length of the call. If it is about half, the sample rate is wrong.

Fix: resample to 16 kHz before Silero. The project’s silero() in vad.py does it for you: if sr != 16000: x = resample(x, sr, 16000).

Common mistake

A better detector, and background talk is still 'speech'

Symptom: you switch from energy to WebRTC VAD or Silero, and false speech on the background talk calls stays high.

Cause: background talk is real speech. A VAD answers “is someone speaking?”, not “is the caller speaking?”.

Fix: use the call’s structure. Many Larkfield calls record the agent and the caller on separate channels. Module 5 uses those channels to find who speaks.

Detector Good at Weak at Cost
Energy Quiet lines with a known loudness Music, background talk, a quieter line Almost nothing (0.15 ms a minute)
WebRTC VAD Fast; misses little speech on the phone line Music on hold, background talk 3 ms a minute
Silero VAD Music and street noise Phone speech at its default 0.5: lower the threshold About 90 ms a minute, still small

Do not choose by name or by reputation. Choose on your own audio, with missed and false speech counted separately, and with the use after the VAD in mind.

Check your understanding

Tomás says: 'Silero is a neural network, so it is better at everything.' Use two measurements from this lesson to answer him.

Show the answer

Silero is better at rejecting noise that is not a voice. It called 3% of the music on hold speech, against 100% for energy and WebRTC VAD, and none of the street noise. But Larkfield’s phone audio is hard for it at its default threshold of 0.5. It missed 18.8% of the speech on the 50 validation calls, the most of the three (energy 10.7%, WebRTC VAD 6.4%). On C-0022, it missed 41.1%. It also calls background talk speech, like the others. On wideband audio it missed only 4.4%, so the phone line is part of the problem.

Check your understanding

Predict: you raise Silero's threshold from 0.5 to 0.7. What happens to missed and false speech, and is it a good idea before the recogniser?

Show the answer

Fewer chunks pass 0.7, so missed speech rises and false speech falls. On the 50 validation calls, missed speech went from 18.8% to 46.6%, and false speech from 15.4% to 6.9%. Before the recogniser it is a bad idea: nearly half the speech would never be transcribed. Missed words cannot be recovered; false speech only costs some compute.

Check your understanding

Practical check: run speech vad --method silero --threshold 0.3 --limit 20 and compare it with the default 0.5. Which threshold do you choose for Larkfield's transcripts, and why?

Show the answer

At 0.3, missed speech falls in every group: babble 33.5% to 13.9%, quiet 18.1% to 8.4%, street 12.7% to 7.7%. False speech rises, for example babble 20.0% to 50.6%. For transcripts, 0.3 is the better choice. Missed speech loses words, and false speech only costs compute. Silero at 0.3 still rejects music on hold, which WebRTC VAD does not. The choice must be checked on the recogniser’s own errors, not only on time shares.

Recap

  • WebRTC VAD compares the energy in six frequency bands with simple models of speech and noise. Its mode (0 to 3) sets how strict it is. It is fast (3 ms a minute) and calls music speech.
  • Silero VAD is a small neural network. It gives a probability of speech for each 32 ms chunk, and a threshold turns it into yes or no. It rejects music (3%) and street noise (0%).
  • Every detector calls background talk speech, because it is speech.
  • On Larkfield’s phone audio, Silero at 0.5 is unsure: it missed 18.8% of the speech, against 4.4% on the wideband copies. At 0.3, it misses 7.2%.
  • Give Silero 16 kHz audio. With 8 kHz audio, every time halves and missed speech jumps to 82.0%.

You can now choose a detector and a threshold by missed and false speech. But a share of seconds is not yet a share of words. Next, we put the VAD in front of Whisper. We measure what it does to invented text, to the word error rate and to the cost of an hour of audio.

Next: Why VAD makes recognition better