Skip to content
Français

Spot commands on a small device

  • Case study 3
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we searched transcripts for important words. In this case study, we spot short spoken commands directly in the audio, compare a small CNN with Whisper tiny, and count false alarms per hour.

Case study 3 of 4 · Speech AI: Transcribe, Understand and SpeakSpot commands on a small device

What you will be able to do

Choose a keyword spotter for a small device, from real recordings of real people. You will compare a simple template baseline, a small convolutional network, Whisper tiny with a word list and Vosk with a grammar. You will judge them the way the product team does: commands caught, false alarms per hour, speed and size. Then you choose a threshold on validation, test once, and run the winner on a new recording.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Log-mel spectrogram
  • Template baseline
  • Small CNN in PyTorch
  • Whisper with a word list
  • Vosk with a grammar
  • False alarms per hour
  • Threshold for a budget
  • Latency and model size

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page with the same code. It downloads about 36 MB: the clips as log-mel pictures, 24 clips as audio, and the recorded outputs of Whisper tiny and Vosk. It trains the CNN on your computer in under a minute. An optional cell runs Whisper tiny and Vosk live on six clips. It needs no account and no key.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses ideas from across the course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • A virtual environment with NumPy, SciPy, soundfile, matplotlib and PyTorch 2.15.0 (CPU). For the optional live cell: faster-whisper 1.2.1 and Vosk 0.3.44. The larkspeech project’s environment from Module 4 works after you add PyTorch and Vosk; this case study does not change the project’s files.
  • The notebook, downloaded from the link above. Its setup cell makes a new folder speech-commands. To start again, delete that folder.

Tested on macOS with Python 3.12. Windows, Linux, Colab and Kaggle were not tested.

The Mistral fan is a product of Larkfield Basics, Larkfield’s own brand. Omar, the product manager, wants the next model to obey spoken commands: “on”, “off”, “up” and “down” for the speed, “left” and “right” to turn, “stop” and “go” for the swing, and “yes” and “no” to confirm. The fan has a small chip and no internet connection. So the spotter must run on the fan, and no audio may leave the room.

Omar names the two errors that matter:

  • A false alarm. The fan acts when nobody gave a command. People return a fan that starts at night, or turns off during a call. Omar sets a budget: at most 10 false alarms per hour of speech and noise that is not a command. That is still one every six minutes, so it is a first target, not a final one.
  • A missed command. The person says “off” and nothing happens. They say it again, a little annoyed. That is bad, but not as bad as a fan that acts by itself.

Tomás has an answer ready: “Put Whisper on it. It understands everything.” Priya, the course’s data scientist, wants to measure first. Her questions are: which method keeps the budget, how many commands it catches, and whether it fits on the chip.

A keyword spotter is a loop. It listens to the last second of sound, again and again, and decides each time: a command, or nothing?

A flow from left to right. Microphone, 16,000 numbers a second. The last second, taken again every 100 ms. Log-mel picture, 40 bands by 98 frames. Spotter, a word and a score that says how sure it is. A question: a command, and score at least the threshold? Yes: act, then wait 1 second. No: do nothing; the audio is not kept.
The spotter's loop on the device. Each method of this case study fills the 'Spotter' box; the threshold is chosen for the false-alarm budget.
Text description of the diagram

The microphone gives 16,000 numbers a second. Every 100 milliseconds, the device takes the last second of sound and turns it into a log-mel picture of 40 bands by 98 frames. The spotter reads the picture and gives a word and a score that says how sure it is. If the word is a command and the score is at least the threshold, the device acts and then waits one second. Otherwise it does nothing, and the audio is not kept.

You will try four spotters in the middle box, on the same clips and with the same measures:

  1. Energy and templates (the baseline): a loudness check, then the nearest average “word shape”.
  2. A small CNN trained on log-mel pictures: 45,132 weights, small enough for a chip.
  3. Whisper tiny with a word list: the recogniser of the course, with the ten commands as its prompt.
  4. Vosk with a grammar: the older recogniser of the course, allowed to answer only with the ten commands or “unknown”.

Then you choose each one’s threshold for the budget on validation, and test once. Last, you compare speed and size, and run the winner on a new recording.

The data is Speech Commands v0.02. It has 105,829 one-second clips of 35 English words. 2,618 volunteers recorded them on their own phones and laptops, in their own rooms. The volunteers recorded the words for this purpose, through a web page. There are also six longer recordings of background noise.

This is real speech. The Larkfield calls of the course are synthetic: clear voices that read scripts. Here, people mumble, speak far from the microphone or cut the word short. Expect it to be harder than Larkfield’s calls.

Listen to two clips from the test split.

A command: “yes” (test clip yes/43fc47a7_nohash_0). A volunteer of Speech Commands, CC BY 4.0. One second, 16 kHz.

Transcript. “Yes.”

Another word: “bed” (test clip bed/1b4c9b89_nohash_0). A volunteer of Speech Commands, CC BY 4.0. One second, 16 kHz.

Transcript. “Bed.” It is not a command. A spotter that acts on it makes a false alarm.

Part What it is Use
The ten commands yes, no, up, down, left, right, on, off, stop, go What the fan must catch
Other words the other 25 words, such as bed, happy, follow and the digits Speech that is not a command: false alarms
Noise six recordings: washing dishes, a cat, an exercise bike, a running tap, pink and white noise Sound with no speech: false alarms
speaker a random ID for each volunteer To keep each person in one split, and to find the worst speakers

The dataset kept no age, gender, accent or place of the volunteers. So this case study can compare words and speakers, but not groups of people.

The dataset has its own split rule. A hash of the speaker’s ID (a fixed scramble of a text into a number) puts every clip of one person in the same split. Priya keeps that rule, and takes a fixed sample to keep the download small:

Split Commands Other words Noise windows Speakers
Train 2,985 1,600 1,416 1,522
Validation 600 1,200 152 240
Test 1,000 1,800 152 242

The noise windows are one second long, one every half second. Each noise file is cut in time: the first 60% for training, the next 20% for validation and the last 20% for testing. The dataset’s own test set does not do this: its “silence” clips are cut at random from the same noise files that training uses. We do not use them.

The notebook checks the split:

people = {s: set(speaker[(split == s) & (label != "noise")]) for s in ["train", "valid", "test"]}
shared = len(people["train"] & people["test"]) + len(people["train"] & people["valid"]) + len(people["valid"] & people["test"])
print("speakers in two splits:", shared)
speakers in two splits: 0

What this tells us. No volunteer is in two splits, so the test has only new voices, as a real fan does.

  • The unit: one second of sound, every 100 ms on the device. In the measurements, one clip or one noise window.
  • The decision: act on a command, or do nothing.
  • Caught: the share of command clips where the spotter acts with the right word. A wrong word (“up” heard as “off”) is worse than a miss, so it is counted apart.
  • False alarms per hour: the number of times the spotter acts on other words or noise, divided by the hours of that audio. An other-word clip counts as one second; a noise window adds half a second.
  • The rule: choose each method’s threshold on validation, as the lowest threshold that keeps within 10 false alarms per hour. Then measure once on test.

The validation split has 1,200 other-word clips and 152 noise windows: 1,276 seconds, about 21 minutes. One false alarm there is already 2.8 per hour. So the budget of 10 allows 3 false alarms in validation.

That is the hard part of false alarms. A rule of thumb, the rule of three: if you see no false alarm in N hours, the true rate can still be up to about 3 ÷ N per hour (with 95% confidence). With our 21 minutes, “no false alarm” still allows about 8.5 per hour. We can test a budget of 10, but not “one a day”: that needs days of audio with no command in it.

Sound as a picture: the log-mel spectrogram

Section titled “Sound as a picture: the log-mel spectrogram”

In Audio as data, a spectrogram showed time across and pitch up. A log-mel spectrogram is the same picture with two changes that copy the human ear. First, it groups the pitches into 40 mel bands. The bands are narrow at low pitch and wide at high pitch, because the ear separates low pitches better. Second, the loudness is in decibels (a log scale), because the ear hears a sound ten times stronger as only “a bit louder”. One second becomes a small picture: 40 bands by 98 frames of 10 ms. Every spotter in this case study reads this picture.

Three log-mel pictures side by side, each one second long with 40 mel bands from low to high. Yes: a bright shape from about 0.4 to 0.85 seconds, bright at the bottom first, then a bright patch in the top bands at the end, the s sound. Bed (another word): a bright shape from about 0.5 to 0.9 seconds, brightest in the lower bands. Running tap (noise): an even grey-orange texture over the whole second and all bands, with no shape.
Log-mel pictures of a command, another word and noise, drawn from the case study's data. Bright is loud. A word is a shape in time; noise is a texture.
Text description of the diagram

The three pictures share the same axes: time from 0 to 1 second across, and 40 mel bands from low to high pitch up. “Yes” is a bright shape in the middle of the second. Its lower bands are bright through the vowel, and the top bands light up at the end, for the hissing “s”. “Bed” is a shorter shape, mostly in the lower bands. The running tap fills the whole picture with an even texture, in every band, with no beginning and no end.

The notebook computes the picture with about 15 lines of NumPy (log_mel), and checks it against the stored picture (within 0.25 dB).

What this tells us. A word has a beginning, a shape and an end; noise has none. But two words can have similar shapes, so the spotter must tell shapes apart, not only sound from silence.

Before any model, Priya tries the oldest trick. It has two steps:

  1. Energy: add the loudness of all bands in each frame, as the energy detector of Speech or silence? did. If the loudest frame is under 10 dB, there is no speech: do nothing.
  2. Templates: cut out the loud part of the clip, and stretch it to 32 frames. Compare it with a template of each command: the average shape of that word in the training clips. There is also a template for “other words”. The nearest command template wins. The score is how much closer the clip is to that command’s template than to the “other words” template.
templates = {}
for c in COMMANDS + ["other"]: # the mean shape of each word in the training clips
templates[c] = np.mean([s for s, s_split, s_label in zip(shapes, split, label)
if s is not None and s_split == "train" and s_label == c], axis=0)

On validation, with no threshold at all:

{'right': 0.502, 'wrong word': 0.428, 'missed': 0.07, 'false alarms (other)': 1094, 'per hour (other)': 3282.0,
'false alarms (noise)': 131, 'per hour (noise)': 6205.3, 'false alarms per hour': 3456.1}

What this tells us. The baseline gets half the commands right. But it acts on 1,094 of the 1,200 other words and on 131 of the 152 noise windows. That is about 3,456 false alarms an hour. One average shape per word cannot hold thousands of voices, and loud noise passes the energy check. This is the floor every other method must beat.

The deep learning course built a convolutional network for photos in Classify clothing images. The idea is a small filter: a set of weights that slides over the input and gives a high number where its pattern is. Here the filter covers all 40 bands and 5 frames (50 ms), and it slides along time only. A filter that learned “a vowel starts here” fires at the start of the vowel in “yes”, wherever the word sits in the second.

Top: the log-mel picture of yes, with a white box 5 frames wide and as tall as all 40 bands, and an arrow to the right: the filter slides along time. Bottom: a line of the filter's response for each of the 98 frames. It is near zero almost everywhere, and has one clear peak where the box is, at frame 43, where the vowel of yes starts.
One of the 48 filters in the trained CNN's first layer, on the 'yes' clip. It responds most where its learned pattern is. Drawn from the trained network.
Text description of the diagram

The top part shows the log-mel picture of “yes” with a white box. The box is 5 frames wide and covers all 40 bands. An arrow shows that the box slides along time. The bottom part shows the filter’s response at each of the 98 frames. It is almost zero everywhere. It has one clear peak where the box is drawn, at frame 43, where the vowel of “yes” starts.

The network has four such layers of 48 filters. After each layer, it keeps the larger of every two frames, so later filters see longer pieces of the word. At the end, it keeps the strongest response of each filter in the whole second, and a last layer turns those 48 numbers into 12 class probabilities: the ten commands, “other” and “noise”. It has 45,132 weights. Each picture is first scaled to its own loudest point, so a quiet voice and a loud voice look alike.

cnn = nn.Sequential(*block(40, 48), *block(48, 48), *block(48, 48), *block(48, 48),
nn.AdaptiveMaxPool1d(1), nn.Flatten(), nn.Dropout(0.2), nn.Linear(48, len(CLASSES)))

Training takes 12 epochs over the 6,001 training items, with each word moved in time by up to 60 ms at random. On the course’s laptop, it took about 30 seconds on two CPU threads. The spotter’s word is the most likely class; its score is that class’s probability. On validation, with no threshold:

{'right': 0.902, 'wrong word': 0.047, 'missed': 0.052, 'false alarms (other)': 72, 'per hour (other)': 216.0,
'false alarms (noise)': 0, 'per hour (noise)': 0.0, 'false alarms per hour': 203.1}

What this tells us. 90.2% right, and no false alarm on noise: the “noise” class works. But it still acts on 72 of 1,200 other words, mostly words that sound like a command.

Tomás’s idea, made cheap: the smallest Whisper of Whisper sizes on a laptop, told the language (English) and given the ten commands as its prompt, as in Help the model with your words. Whisper writes text, so the spotter needs a rule: take the first word after the course’s normalisation guide; if it is a command, that is the word heard. The score is Whisper’s average word probability.

Running Whisper on 4,904 items takes time, so the course team ran it once and recorded every output. The notebook replays them, and an optional cell runs Whisper live on six clips. What it wrote for some test items:

What was said What Whisper wrote (most common first)
“on” “on.” 52, “On.” 17, “on” 13, “Come on.” 2
“off” “off.” 79, “up.” 5, “Off.” 3, “Oh.” 2
digits (other words) “7” 81, “9” 68, “Zero.” 57, “6.” 56
noise “you” 142, nothing 9, “yes, no, up, down.” 1

What this tells us. Whisper hears the words well, and it writes other words as themselves, so they are easy to ignore. On noise it writes “you”: the invented text of Why VAD makes recognition better. Once, it wrote its own prompt back: a prompt can leak into the output.

The older recogniser of the course, Vosk small, can take a grammar: the list of the only words it may answer. Here: the ten commands and [unk] (unknown). In Compliance phrases and keyword spotting, this made Vosk a fast spotter. The score is Vosk’s confidence in the word. It was recorded the same way.

The four methods on validation, before any threshold

Section titled “The four methods on validation, before any threshold”
Method Right Wrong word Missed False alarms per hour
Energy + template 50.2% 42.8% 7.0% 3,456
Small CNN 90.2% 4.7% 5.2% 203
Whisper tiny + word list 91.2% 2.5% 6.3% 51
Vosk + grammar 95.2% 1.2% 3.7% 344

Vosk catches the most commands, and Whisper makes the fewest false alarms. But nobody would ship any of them like this: the budget is 10 an hour.

A fair comparison: one budget, one threshold rule

Section titled “A fair comparison: one budget, one threshold rule”

Each method has a score. A higher threshold means fewer false alarms and more misses. The chart shows the whole trade-off on validation: each point is one threshold.

  • Small CNN
  • Whisper tiny
  • Vosk
  • Energy + template
  • Budget: 10 an hour
Commands caught against false alarms per hour on validation, as each method's threshold moves (measured; Whisper tiny and Vosk recorded). Up and to the left is better. Methods must stay left of the dashed budget line.
Text description of the chart

A line chart. Across: false alarms per hour on validation, from 0 to 200. Up: the share of commands caught, from 0 to 100%. A dashed vertical line marks the budget at 10 an hour. Whisper tiny’s line rises fastest: at the budget it catches 78.3%. The CNN’s line is a little lower near the budget, 71.2%, and climbs above 85% when more false alarms are allowed. Vosk appears only on the right: even at its highest confidence it makes about 135 false alarms an hour, while catching about 91%. The template baseline stays near the bottom: about 4% at the budget.

Priya’s thresholds, chosen on validation only:

Method Threshold Right (validation) False alarms per hour (validation)
Energy + template 136.0 4.3% 8.5
Small CNN 0.947 71.2% 8.5
Whisper tiny + word list 0.613 78.3% 8.5
Vosk + grammar 1.000 (its highest) 90.8% 135.4

What this tells us. Three methods keep the budget (3 false alarms in 21 minutes). Vosk cannot: most of its answers, right and wrong, have a confidence of exactly 1.0. A score that does not separate gives you no threshold to move.

The test split has not been used for any choice. Priya runs each method once, at its validation threshold.

for name, (h, s) in methods.items():
r = evaluate(h, s, thresholds[name], "test") # the test split, once, at the validation threshold
Method Right Wrong word Missed False alarms per hour: other words noise all
Energy + template 3.7% 0.0% 96.3% 2.0 0.0 1.9
Small CNN 67.7% 0.1% 32.2% 14.0 0.0 13.4
Whisper tiny + word list 76.0% 0.6% 23.4% 10.0 0.0 9.6
Vosk + grammar (no threshold keeps the budget) 93.7% 0.2% 6.1% 152.0 0.0 145.8

What this tells us. The test agrees with validation, with one honest surprise: the CNN’s false alarms rose from 8.5 to 13.4 an hour, a little over the budget. That is 7 false alarms in 31 minutes instead of 3 in 21: with so few events, the difference is within chance. Whisper tiny catches about 3 in 4 commands, and the CNN about 2 in 3. Wrong words almost disappear: most wrong answers were unsure, and the threshold removes them.

More data: the same network on every training clip

Section titled “More data: the same network on every training clip”

The notebook’s CNN learned from 2,985 command clips and 1,600 other words. That keeps the training to half a minute. The dataset’s training split has 84,843 clips. Priya trains the same network with the same recipe once on all of them (about seven minutes on the course’s laptop). Only the training data changes: the same validation clips choose the threshold, and the same test clips measure it.

Small CNN Training clips Threshold Right (test) False alarms per hour (test)
The notebook’s sample 4,585 + noise 0.947 67.7% 13.4
All the training clips (run once by the course team) 84,843 + noise 0.838 90.3% 3.8
Commands caught on the 1,000 test commands at each method's validation threshold (measured once). Vosk's bar breaks the budget (145.8 false alarms an hour); the others keep it on validation.
Text description of the chart

A column chart with five columns, from 0 to 100%. Template 3.7%. CNN on the sample 67.7%. Whisper tiny 76.0%. Vosk 93.7%, but at 145.8 false alarms an hour, far above the budget of 10. The same CNN trained on all the training clips 90.3%, at 3.8 false alarms an hour.

What this tells us. With 18 times more training clips, the same 45,132 weights catch 9 commands in 10 and make only 2 false alarms in 31 minutes. Every word is now at 83% or more, and the median speaker gets every command. The notebook’s numbers are about the method; the shipping model needs all the data. For the full model, the worst tenth of speakers is still caught only 43.7% of the time.

An average hides who has the problem. Here is each command, on test:

Word Template CNN Whisper tiny Vosk
yes 3% 81% 94% 98%
no 2% 50% 74% 98%
up 0% 67% 80% 82%
down 0% 59% 73% 93%
left 1% 74% 82% 91%
right 9% 70% 83% 96%
on 0% 68% 36% 93%
off 0% 78% 66% 93%
stop 22% 85% 91% 96%
go 0% 45% 81% 97%

What this tells us. Each method has a different weak word. Whisper tiny catches “on” only 36% of the time, although it writes “on” in 84 of 100 clips. A one-syllable word gets a low average probability, and one threshold for all words cuts it. The CNN is weakest on “go” (45%) and “no” (50%): short words with a single vowel, close to other words of the set such as “dog” and “four”.

And by speaker: the test split has 122 volunteers with at least three command clips.

Method Median speaker Worst tenth of speakers Worst speaker
Small CNN 74.2% 6.0% 0%
Whisper tiny + word list 80.0% 24.9% 0%
Vosk (over budget) 100% 51.8% 0%

What this tells us. For one person in ten, the CNN almost never acts. Their voice, microphone or room is far from most training clips. With no age, gender or accent in the data, we cannot say who they are; a pilot with real buyers must ask.

A fan’s chip is small. The course measured each method’s time for one decision (one second of audio, the log-mel picture included) on one CPU thread of the course’s laptop, an Apple M4:

Method Time per decision Model size Fits a small chip?
Energy + template 0.25 ms 0.06 MB yes
Small CNN 0.31 ms 0.18 MB (45,132 weights × 4 bytes; about 0.05 MB at 8 bits) yes
Vosk small + grammar 4.4 ms 70.9 MB no: needs a small computer, such as a single-board computer
Whisper tiny 275.7 ms 78.2 MB no

The device decides every 100 ms. Whisper tiny needs 276 ms for one decision on a laptop core, so it cannot keep up even there. A small chip is much slower than a laptop, so the CNN’s 0.31 ms will grow too. The course did not measure a chip: measure on the real one before you promise a speed.

And a server? Tomás suggests: “Then send the audio to a server with Whisper.” Take the course’s assumption: a CPU server costs US$0.17 an hour for 4 vCPUs, about US$0.0425 per core-hour. If Whisper decides every half second, one core works 2 × 0.276 = 0.55 seconds per second of audio. That is about US$0.023 per hour of listening for one fan. A fan that listens all day costs about US$205 a year. The CNN on the fan costs nothing per hour: the chip is in the fan anyway. And the server route sends every second of the room’s sound out of the house, which breaks Omar’s first rule.

Priya writes to Omar in his words:

  • Use the small CNN on the fan’s chip. It is the only method that is both accurate enough and small enough. It is 0.18 MB and takes well under a millisecond per decision on a laptop. Whisper tiny is 430 times larger and about 900 times slower. Vosk cannot keep the false-alarm budget, and it is too large too.
  • Train it on all the training clips, with threshold 0.838. On test, that model caught 90.3% of commands. It made 3.8 false alarms an hour on isolated other words, and none on noise. The notebook’s smaller model (67.7%) shows the method, not the product.
  • Wait until the word ends before acting again. In the kitchen recording below, the full model acted twice on one “yes”. The rule “wait one second” is too short.
  • Add a confirmation for risky commands. For “off” and “stop”, act at once; they are safe. For “on” at night, a short beep and a second “yes” costs one second and removes most of the harm of a false alarm.
  • What a person still checks: a pilot in real homes, with the owners’ consent. The fan logs only the times and words it acted on, never audio. A person reads the log and asks the owner which actions were false.

And what not to use it for: anything that needs to know who speaks, anything safety-critical, or another language without new data.

End to end on new input: the kitchen recording

Section titled “End to end on new input: the kitchen recording”

A test clip is one clean word. A real room has a word somewhere in a stream of noise. Priya builds a new recording from parts the model has never seen. She takes the last 20% of the dishes noise (the test part) and places six test clips in it. It is a constructed example, made from real recordings.

The kitchen recording: 19 seconds of washing dishes (from Speech Commands’ background noise, at half volume) with six test clips of different volunteers added. Constructed by Nextia Learning from CC BY 4.0 recordings.

Transcript. At 2.0 s: “yes”. At 5.0 s: “bed”. At 8.0 s: “stop”. At 11.0 s: “go”. At 14.0 s: “seven”. At 16.5 s: “off”. Dishes all the time.

The CNN looks at the last second every 100 ms, acts when its score is at least its threshold, and then waits one second. Each row is the start of the second it acted on, the word and the score. First the notebook’s CNN (threshold 0.947):

said: [(2.0, 'yes'), (5.0, 'bed'), (8.0, 'stop'), (11.0, 'go'), (14.0, 'seven'), (16.5, 'off')]
acted: [(1.7, 'yes', 0.982), (7.5, 'stop', 0.978)]

Then the same network trained on all the clips (threshold 0.838):

acted: [(1.4, 'yes', 0.911), (2.4, 'yes', 0.965), (7.4, 'stop', 0.873), (8.4, 'up', 0.891), (10.5, 'go', 0.841), (16.4, 'off', 0.985)]
A waveform of the 19-second kitchen recording, with six shaded boxes where the words are. Under it, the words said: yes at 2 s, bed at 5 s, stop at 8 s, go at 11 s, seven at 14 s, off at 16.5 s. Orange dashed lines and labels: the sample CNN acted on yes and stop. Green dashed lines and labels: the CNN trained on all data acted on yes twice, stop, up, go and off. Neither acted on bed or seven.
The kitchen recording, the words said, and where each CNN acted (each line marks the middle of the second it acted on). Drawn from the case study's data.
Text description of the diagram

The waveform shows the dishes noise all the way through, with six shaded boxes where the words are. Under it are the words said: yes, bed, stop, go, seven and off. In orange, the notebook’s CNN acted twice: on “yes” and on “stop”. In green, the CNN trained on all the clips acted six times: twice on “yes”, on “stop”, then “up” while “stop” was still ending, then on “go” and on “off”. Neither model acted on “bed” or “seven”.

What this tells us. Both models ignored the noise and the two other words. The notebook’s model caught two of four commands: “go” was already its weakest word, and in noise its score fell below the threshold. The full model caught all four, but it acted twice on one “yes”, and it heard “up” in the end of “stop”. Both extra actions come from the loop, not from the clip tests: a word can fill two windows, so “wait one second” is too short. A better rule waits until the score falls below the threshold again. The training pictures also had noise or speech, never both together; mixing training words with noise is the next change to try.

  • False alarms decide the threshold, and the threshold decides the method. Before any threshold, all three alternatives catch more than 90% of commands. At 10 false alarms an hour, the sample CNN catches 68% and Whisper tiny 76%. Vosk cannot get there at all.
  • Data beat model size. The same small CNN went from 67.7% to 90.3% with all the training clips, past Whisper tiny, which is 430 times larger.
  • The loop around the model matters. Clip tests do not show double actions; a recording with words in noise does.
  • A good recogniser is not a good spotter by default. Whisper writes “on” correctly, but its probability is low for a short word. Vosk’s grammar gives full confidence to wrong words. A spotter needs a score that separates.
  • Real speech is harder than Larkfield’s synthetic calls. On Larkfield’s calls, the Vosk spotter found 56% of the recording notices with no false alarm, on only 3 calls without the notice. Here, with 1,800 other words, its false alarms are measurable, and too many.
  • Limits. Isolated words are not a living room with sentences, television and music. 31 minutes of test negatives cannot show a rate like “one a day”. Speeds are from one laptop core. And the data has no groups of people, only speakers and words.
  • No audio leaves the device, and none is kept. That is the main reason to run the spotter on the fan. Write it in the product’s privacy notice, and keep it true in every update.
  • Tell people the fan listens. A light that shows “listening”, and a switch that turns the microphone off.
  • Test with the people who will use it. Even the full model catches the worst tenth of speakers less than half the time. Recruit testers of different ages, accents and voices, with their consent, before launch.
  • The volunteers’ voices. Real people gave these clips for this kind of research, under CC BY 4.0. Credit the dataset, use the clips for spotting words, and never try to identify or imitate a speaker. No clip of this case study was sent to a hosted model.

This case study shows that you can take a speech model from data to a device decision, with a product team’s measures. Make it yours with one change. Mix noise into the training words, add per-word thresholds, or fix the double action in the loop. Or try another language from the Multilingual Spoken Words Corpus (CC BY 4.0). Then make it public, easy to read in five minutes.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • results.csv: one row per method, with the threshold, right, wrong word, missed and false alarms per hour on validation and test.
  • MODEL_CARD.md: intended use (ten English commands, on the device) and what it is not for. Add the data, the budget, the per-word and per-speaker results, and the limits.
  • The data source and licence with the attribution text from the box above.

A README template:

# A keyword spotter for a small device
## Problem
A fan obeys ten spoken commands, on the device, with no internet.
Budget: at most 10 false alarms per hour of speech and noise that is not a command.
## Data and licence
Speech Commands v0.02 (Warden, 2018), CC BY 4.0, https://arxiv.org/abs/1804.03209.
Fixed sample: train 2,985 commands + 1,600 other words + noise; validation and test by the
dataset's speaker split. Noise files cut in time so test noise is new.
## Approach
Log-mel pictures (40 x 98). Four spotters: energy + templates, a 45k-weight CNN,
Whisper tiny with the commands as prompt, Vosk with a grammar. Thresholds chosen on
validation for the budget; test used once.
## Results (test, once)
CNN trained on all 84,843 training clips: 90.3% caught at 3.8 false alarms/hour,
0.3 ms per decision on a laptop core, 0.18 MB. The same CNN on a 4,585-clip sample:
67.7% at 13.4/hour. Whisper tiny 76.0% at 9.6/hour, but 276 ms and 78 MB.
Vosk 93.7% but 146 false alarms/hour at its strictest.
## Limits
Isolated words, 31 minutes of test negatives, no speaker groups, speed measured on a laptop.

How to describe it in an interview, in two sentences: “I compared four keyword spotters for a device with no internet, with thresholds chosen on validation for 10 false alarms an hour. On new speakers, a 45,000-weight CNN caught 90% of commands at under 4 false alarms an hour; Whisper tiny caught 76% and was 900 times slower.”

Go further:

  • Mix the training words with the training noise, retrain, and run the kitchen recording again. With pictures, add the power of the two (10 ** (db / 10)) at a random volume, then take the log again. Predict first whether “go” comes back.
  • Give each word its own threshold on validation, for the same total budget. Which words gain?
  • Add a two-step rule: act only when two windows in a row agree. Measure what it does to false alarms and to the delay.
  • Quantise the CNN to 8 bits and measure the change in accuracy.

Check your understanding

Before any threshold, Vosk with a grammar catches 95.2% of validation commands, the most of the four methods. Omar asks why it is not the choice. What do you answer?

Show the answer

Because the fan’s main risk is acting by itself, and Vosk cannot keep the budget. Most of its answers have a confidence of exactly 1.0, right and wrong alike. So even at its highest threshold it makes about 135 false alarms an hour on validation (145.8 on test), against a budget of 10. A threshold only helps when the score separates right answers from wrong ones. Vosk is also about 71 MB, too large for the fan’s chip.

Check your understanding

The validation split has about 21 minutes of other words and noise. The CNN made no false alarm on noise. Tomás says: 'So noise is solved.' What can you really say, and what would you need to say more?

Show the answer

The noise part of validation is only 76 seconds of new noise, so “zero false alarms” is weak evidence. By the rule of three, zero events in N hours still allows a rate up to about 3 ÷ N per hour. For 76 seconds, that is over 140 an hour. You can say that no false alarm happened on these six kinds of noise. To say more, you need many hours of real home sound with no command, such as television, music and talk. Then you count the false alarms there.

Check your understanding

Whisper tiny writes 'on' correctly in 84 of 100 test clips, but at its threshold it catches only 36% of them. Explain why, and suggest a fix that does not change the model.

Show the answer

The score is Whisper’s average word probability, and a single short word like “on” often gets a low probability even when it is right. One threshold for all ten words is set by the false alarms of all words together, and it cuts many right “on” answers. A fix: choose a threshold per word on validation (a lower one for “on”), keeping the total false alarms within the budget. Then check on test once that “on” improves and the false alarms stay near 10 an hour.

Recap

  • A spotter acts on a command or does nothing, every 100 ms. Score it with commands caught, wrong words and false alarms per hour, on other words and on noise apart.
  • A log-mel picture is one second of sound as 40 mel bands by 98 frames in dB: words are shapes, noise is texture.
  • Choose the threshold on validation for the false-alarm budget, then test once. At 10 an hour, the sample CNN caught 67.7% of test commands and Whisper tiny 76.0%. Vosk could not keep the budget, and the template baseline caught 3.7%.
  • Only the small CNN fits the fan: 45,132 weights, 0.18 MB, 0.31 ms per decision on a laptop core. Trained on all 84,843 training clips, it caught 90.3% at 3.8 false alarms an hour. Whisper tiny is 900 times slower.
  • Check the worst words and speakers: Whisper missed “on”, the CNN missed “go”, and for one speaker in ten the CNN almost never acts.
  • Few hours of negatives limit what you can claim: no false alarm in N hours still allows about 3 ÷ N an hour.

Next, the last case study: an IVR for a bank’s spoken requests, tested by speaker group.

Next: An IVR for spoken intents