Speech or silence?
- Lesson 7
- 35 min
- Module 3: Find the speech: voice activity detection
Where we are
Our test set shows where the recogniser makes mistakes. Some mistakes come from parts of the call where nobody speaks. In this lesson, we find speech with the simplest detector, an energy threshold, and we see where it fails.
Lesson 7 of 36 · Speech AI: Transcribe, Understand and SpeakSpeech or silence?
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Find the parts of a call where somebody speaks, with the simplest possible detector: a loudness threshold. Then you can measure what it gets wrong. The recogniser of the next modules should not spend time, or write words, on music and silence.
NotebookDo this lesson in a notebook
The notebook builds the energy detector of this lesson and scores it on the call with music on hold. Then it runs it on the noise set and asks you to choose a threshold for six calls. It downloads only the files it needs and no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Sample rate, frames and loudness in dBFS, from Loudness and silence in Audio as data.
- The test set and word error rate, from Word error rate, step by step.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your project folder
larkspeechfrom Set up the speech project, with the changes of Module 2. In a terminal, in that folder, with.venvactive,pytest -qprints15 passed. - The labs folder
nextia-learning-labsnext tolarkspeech, and the course data inlarkspeech/data, as in that lesson. (If you keep the data elsewhere, setLARKSPEECH_DATAto its folder.)
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands were not run.
A note-taker on the phone
Section titled “A note-taker on the phone”Think of a call-centre agent who takes notes during a phone call. She writes when the caller speaks. When the caller says “Hold on, let me find it”, she puts her pen down and waits. When the hold music plays, she does not write the music. She only writes speech.
Now give the same call to a note-taker who cannot put the pen down. She must write something for every second of the call. During 20 seconds of music, she guesses words. During a long silence, she writes what she expects to hear. Her notes are longer, they took more time, and some of the words were never said.
The problem: the recogniser hears everything
Section titled “The problem: the recogniser hears everything”A speech recogniser, such as Whisper in the next module, is the second note-taker. It gets the whole recording, second by second. Larkfield’s calls have stretches with no speech: music on hold, a caller who looks for an order number, background noise.
Listen to 10 seconds of call C-0022. The agent asks the caller to hold, and the music starts.
Transcript. Agent: “Thank you. Let me check that for you. Please hold.” Then music on hold (a simple tune) until the end of the clip.
In this call, the music lasts 17.7 seconds of a 70.4-second call. The recogniser spends compute time on those seconds. And, as you will see in Why VAD makes recognition better, it sometimes writes words in such stretches. Across the 110 validation and test calls, 19% of the time has no speech. It is 8% in the busiest call and 48% in the call with the longest wait. Larkfield takes about 4,000 calls a month, so that is a lot of audio with nothing to write.
Tomás, the AI engineer, does not see a problem: “Whisper is very good. Send it the whole call.” Priya, the data scientist, wants a number first. How much of a call is speech, and can a program find it?
The idea: find the speech first
Section titled “The idea: find the speech first”The note-taker’s skill has a name. Voice activity detection, or VAD, finds the parts of a recording where somebody speaks. “Voice activity” means that a voice is active, that is, someone is talking. A VAD does not understand words. It answers one question for each short moment of audio: speech or not speech?
The output of a VAD is a list of speech regions: a start time and an end time for each stretch of speech. For example, [(0.98, 8.55), (9.36, 14.12)] means speech from 0.98 to 8.55 seconds, and again from 9.36 to 14.12 seconds. Later steps then use only those regions.
The simplest detector: loud is speech
Section titled “The simplest detector: loud is speech”Speech is louder than a quiet line. So the simplest VAD works like this:
- Cut the audio into short frames of 30 milliseconds (ms).
- Measure the loudness of each frame in dBFS, as in Audio as data. 0 dBFS is the loudest possible sound, and a quiet phone line is near −60 dBFS.
- Call a frame speech when it is louder than a threshold, for example −40 dBFS.
- Join neighbouring speech frames into regions.
This is an energy VAD: “energy” is the technical word for the loudness of a signal. Two small settings make its regions usable:
- The hangover keeps a region open through a short pause. People pause between words for a fraction of a second. Without a hangover, one sentence becomes ten small regions. The project uses 0.3 seconds.
- The minimum speech length drops regions shorter than 0.1 seconds, such as a click on the line.
A tiny example with 12 frames
Section titled “A tiny example with 12 frames”Here are 12 frames: 0.36 seconds of a made-up recording, so that every number fits on the page. Someone says one short word in frames 3 to 8, with a tiny pause in frame 6. In frames 10 and 11, the caller coughs: loud, but not speech.
| Frame | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Loudness (dBFS) | −61 | −58 | −34 | −29 | −31 | −52 | −30 | −27 | −59 | −33 | −32 | −60 |
| True speech | no | no | yes | yes | yes | yes | yes | yes | no | no | no | no |
| Louder than −40? | no | no | yes | yes | yes | no | yes | yes | no | yes | yes | no |
Score the detector the way the next sections do. Six frames are true speech (3 to 8). The detector missed one of them, frame 6, in the pause. So it missed 1 of 6 speech frames: 17% missed speech. Six frames are not speech (1, 2 and 9 to 12). The detector called two of them speech, frames 10 and 11, the cough. So 33% false speech.
Now add a hangover of one frame (30 ms), so a gap of one quiet frame stays inside a region. Frame 6 is now speech, and missed speech falls to 0%. But frame 9 is also a one-frame gap, between the word and the cough. The region now runs from frame 3 to frame 11, and false speech rises to 3 of 6: 50%. One setting fixed one error and made another one bigger.
Text description of the chart
The top panel shows the loudness of 12 frames as steps, with a dashed line at −40 dBFS. Frames 3 to 5, 7 and 8, and 10 and 11 are above the line. Below are three rows of bars. True speech covers frames 3 to 8, from 0.06 to 0.24 seconds. “Louder than −40” has three bars: frames 3 to 5, frames 7 and 8, and frames 10 and 11. “With a 30 ms hangover” has one bar from frame 3 to frame 11, from 0.06 to 0.33 seconds. It fills the pause in frame 6 and also the gap in frame 9, so it joins the word and the cough.
Two numbers that score a VAD
Section titled “Two numbers that score a VAD”The tiny example used the two numbers that score every detector in this module. Both are shares of time, counted every 10 ms against the true turns of the call:
- Missed speech is the share of true speech that the VAD did not find. Missed speech never reaches the recogniser, so its words are lost.
- False speech is the share of non-speech that the VAD called speech. False speech costs compute, and it gives the recogniser a chance to write words that nobody said.
If you know the machine learning course: missed speech is one minus the recall for speech, and false speech is the false-alarm rate. A good detector keeps both low. A threshold moves one up and the other down, so you choose where to stand. Our calls make this measurement possible: the voices’ own word times give the exact true speech, to a few milliseconds. Real recordings rarely have such labels.
Larkfield: the call with music on hold
Section titled “Larkfield: the call with music on hold”Priya runs the energy VAD on call C-0022 at −40 dBFS, with the project’s 0.3-second hangover. The audio is the phone recording at 8 kHz, with both sides of the call in one channel.
Text description of the chart
The top panel shows the loudness of the call over 70 seconds, with a dashed threshold line at −40 dBFS. During speech, the loudest frames are mostly between −30 and −15 dBFS. Between turns, the loudness falls to about −55 dBFS. From 25.1 to 42.8 seconds, the music on hold is shaded: its loudness stays near −25 dBFS the whole time, above the threshold. Below are two rows. True speech has 11 bars, one for each turn, with a long gap during the music. The energy VAD has 22 regions. It covers almost all the true speech, and it also has one long region over the whole music on hold.
The chart shows the problem in one look. The music is as loud as the voices, so the energy VAD calls it speech. The notebook counts it: 22 regions, 19.0% missed speech and 62.6% false speech. Of the 17.7 seconds of music, 17.1 are “speech”. The missed speech comes from the quiet starts and ends of words, which fall under −40 dBFS.
Malik, the telephony engineer, is not surprised. “Hold music must be loud. Callers hang up when they think the line is dead.” A loudness threshold cannot tell a loud tune from a loud voice.
The noise set
Section titled “The noise set”The course data has a noise set: 20 seconds of each kind of sound that is not speech, put through our phone line. How much of each does the energy VAD call speech?
Text description of the chart
A column chart with four columns on an axis from 0 to 100%. Music (music on hold): 100%. Talk (background talk): 97%. Street: 33%. Hiss (line hiss): 0%. The first three are sounds that are not the caller speaking, but the detector calls most of them speech, because they are loud.
Only the quiet line hiss passes the test. Background talk is the hardest case of all, because it is speech, only not the caller’s. No detector in this module can tell whose voice it hears. Module 5 deals with “who spoke”.
All 50 validation calls
Section titled “All 50 validation calls”The course measured the energy VAD on the 50 validation calls, grouped by the noise on the caller’s side. At −40 dBFS:
| Noise | Calls | Missed speech | False speech |
|---|---|---|---|
| Quiet | 23 | 15.2% | 3.7% |
| Background talk | 13 | 2.9% | 75.7% |
| Street | 14 | 10.5% | 17.0% |
| All 50 | 50 | 10.7% | 26.1% |
On quiet calls, the energy VAD works well enough: it keeps most speech and little else. With background talk, three quarters of the non-speech time is “speech”. It costs almost nothing: 0.15 ms of compute for a minute of audio. These numbers are on Larkfield’s synthetic calls; real calls will differ.
Guided practice: add VAD to the project
Section titled “Guided practice: add VAD to the project”Now add the detectors of this module to your project. The file vad.py has three detectors: the energy VAD of this lesson, and two that the next lesson compares. The command speech vad runs one of them on the calls and scores it.
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Copy the module’s three files from the snapshot
end-of-m03.Terminal window cp ../nextia-learning-labs/C41/snapshots/end-of-m03/larkspeech/vad.py larkspeech/cp ../nextia-learning-labs/C41/snapshots/end-of-m03/larkspeech/commands/vad.py larkspeech/commands/cp ../nextia-learning-labs/C41/snapshots/end-of-m03/tests/test_vad.py tests/Terminal window Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m03\larkspeech\vad.py larkspeech\Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m03\larkspeech\commands\vad.py larkspeech\commands\Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m03\tests\test_vad.py tests\ -
Copy the new
requirements.txtfrom the same snapshot (cporCopy-Item, as above). It adds one package, WebRTC VAD, for the next lesson. -
Install the new package:
Terminal window pip install -r requirements.txtThe last line should be
Successfully installed webrtcvad-wheels-2.0.14.post1. -
Run the tests:
Terminal window pytest -qYou should see
18 passed: the 15 tests of Module 2 and 3 new tests for the detectors. -
Run the energy VAD on the first 20 validation calls:
Terminal window speech vad --method energy --limit 20The output on the course’s computer:
energy on 20 validation calls (phone audio)noise calls missed speech false speech ms per minutebabble 7 4.8% 70.7% 0.2quiet 9 17.8% 8.5% 0.2street 4 15.9% 21.0% 0.2“babble” is the background talk. The milliseconds per minute will differ on your computer.
These are 20 calls, not 50, so the numbers differ from the table above. The command also counts each turn from its start to its end, with the short silences at both ends. The table counts from the first word to the last word. The pattern is the same: background talk fills the false speech column.
Your turn: move the threshold
Section titled “Your turn: move the threshold”Change one setting: the threshold. Before you run the two commands below, predict for each one: does missed speech go up or down? And false speech?
speech vad --method energy --threshold -30 --limit 20speech vad --method energy --threshold -50 --limit 20Show the answer
At −30 dBFS, fewer frames are loud enough. Missed speech goes up, and false speech goes down:
energy {'threshold_db': -30.0} on 20 validation calls (phone audio)noise calls missed speech false speech ms per minutebabble 7 39.7% 0.0% 0.2quiet 9 31.3% 8.4% 0.2street 4 33.3% 16.8% 0.2At −50 dBFS, almost everything is loud enough. Missed speech goes down, and false speech goes up. With background talk, 99.1% of the non-speech time is now “speech”:
energy {'threshold_db': -50.0} on 20 validation calls (phone audio)noise calls missed speech false speech ms per minutebabble 7 0.0% 99.1% 0.2quiet 9 12.7% 10.7% 0.2street 4 2.6% 81.5% 0.2No threshold makes both columns small on the noisy calls. In the notebook, −30 dBFS gives the lowest sum of the two numbers on six calls. Even there, the sum is 0.47.
Common mistakes
Section titled “Common mistakes”Common mistake
The same threshold fails on a quieter line
Symptom: the energy VAD worked on last week’s calls. This week, missed speech is very high, although nothing changed in the code.
Cause: a threshold in decibels assumes a loudness. Another phone line, another headset or a lower recording gain makes every call quieter. In the notebook, call C-0022 made 20 dB quieter loses 75.4% of its speech at −40 dBFS, against 19.0% before.
Diagnose: check the loudness of the new calls with speech inspect, from Set up the speech project. Compare it with the calls you chose the threshold on.
Fix: set the threshold from each call itself. For example, take a number of decibels above the loudness of its quietest frames. Or use a detector that does not depend on loudness, as in the next lesson.
Common mistake
One sentence becomes many small regions
Symptom: the VAD finds hundreds of regions in a short call, and missed speech is high.
Cause: the hangover is 0 or very small, so every short pause between words ends a region. The quiet ends of words fall under the threshold too.
Fix: keep a hangover of about 0.3 seconds. The notebook shows the trade: a long hangover of 1 second joins everything, and false speech goes up.
When to use an energy VAD
Section titled “When to use an energy VAD”Use it when the line is quiet and its loudness is known. Examples are a recording studio, a headset in a quiet office, or a first filter that removes dead silence. It costs almost nothing, and you can explain every decision it makes.
Do not use it alone when calls have music on hold, background talk or street noise, which is most of Larkfield’s calls. It calls those sounds speech. The time and the invented words that VAD should save come back.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'Whisper is very good. Send it the whole call.' Use call C-0022 to explain what a VAD is for.
Show the answer
C-0022 has 17.7 seconds of music on hold in a 70.4-second call: a quarter of the audio has no speech. Sent whole, the recogniser spends compute on those seconds, and it can write words where nobody spoke. A VAD finds the speech regions first, so the recogniser gets only the parts where somebody talks. The VAD must be good, though: on C-0022, the energy VAD called 17.1 of the 17.7 seconds of music speech.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you lower the energy threshold from −40 to −50 dBFS. What happens to missed speech and to false speech, and why?
Show the answer
More frames are louder than −50 dBFS, so more of the call is called speech. Missed speech goes down, because the quiet starts and ends of words now pass. False speech goes up, because more noise also passes. On the 20 validation calls, the calls with background talk show it best. Their false speech went from 70.7% to 99.1%, and their missed speech from 4.8% to 0.0%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run speech vad --method energy on the 50 validation calls. Which noise group has the worst false speech, and why can a louder or quieter threshold not fix it?
Show the answer
speech vad --method energy prints babble (background talk) 4.1% missed and 73.7% false speech, quiet 19.6% and 3.5%, street 14.5% and 16.1%. Background talk has the worst false speech. It is people speaking, as loud as the caller, so any threshold that keeps the caller’s voice also keeps the background voices. Raising the threshold to −30 dBFS removes the background talk, but then the detector misses about 40% of the caller’s speech too.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Voice activity detection (VAD) finds the speech regions of a recording, so later steps can skip music and silence.
- The energy VAD calls a 30 ms frame speech when it is louder than a threshold. A hangover of 0.3 s keeps a sentence in one region.
- Two numbers score a VAD against the true turns: missed speech (words lost) and false speech (noise kept). A threshold moves one up and the other down.
- On call C-0022, the energy VAD at −40 dBFS called 17.1 of 17.7 seconds of music speech. On the noise set, it called 100% of the music, 97% of the background talk and 33% of the street noise speech.
- Your project now has
vad.pyandspeech vad, with 18 tests.
You can now find speech by its loudness and measure the result. But loud music and background voices pass as speech, and a quieter line breaks the threshold. Next, we try two detectors that look at the shape of the sound, not only at its loudness.