Spot commands on a small device
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we searched transcripts for important words. In this case study, we spot short spoken commands directly in the audio, compare a small CNN with Whisper tiny, and count false alarms per hour.
Case study 3 of 4 · Speech AI: Transcribe, Understand and SpeakSpot commands on a small device
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Choose a keyword spotter for a small device, from real recordings of real people. You will compare a simple template baseline, a small convolutional network, Whisper tiny with a word list and Vosk with a grammar. You will judge them the way the product team does: commands caught, false alarms per hour, speed and size. Then you choose a threshold on validation, test once, and run the winner on a new recording.
The notebook runs every step of this page with the same code. It downloads about 36 MB: the clips as log-mel pictures, 24 clips as audio, and the recorded outputs of Whisper tiny and Vosk. It trains the CNN on your computer in under a minute. An optional cell runs Whisper tiny and Vosk live on six clips. It needs no account and no key.
Before you startWhat to know and to have ready
What you need to know
This case study uses ideas from across the course. These lessons matter most:
- Keyword spotting, recall and false alarms, from Compliance phrases and keyword spotting.
- The spectrogram, from Audio as data, and the energy detector, from Speech or silence?.
- A prompt with your own words, from Help the model with your words, and the older Vosk pipeline, from From sound to text.
- A threshold chosen on validation and the test set used once, from the machine learning course: Thresholds and calibration and Train, validation and test sets.
- The training loop, from the deep learning course: Batches, epochs and validation.
What you need on your computer (only for the local route; the notebook needs none of it)
- A virtual environment with NumPy, SciPy, soundfile, matplotlib and PyTorch 2.15.0 (CPU). For the optional live cell: faster-whisper 1.2.1 and Vosk 0.3.44. The
larkspeechproject’s environment from Module 4 works after you add PyTorch and Vosk; this case study does not change the project’s files. - The notebook, downloaded from the link above. Its setup cell makes a new folder
speech-commands. To start again, delete that folder.
Tested on macOS with Python 3.12. Windows, Linux, Colab and Kaggle were not tested.
The situation
Section titled “The situation”The Mistral fan is a product of Larkfield Basics, Larkfield’s own brand. Omar, the product manager, wants the next model to obey spoken commands: “on”, “off”, “up” and “down” for the speed, “left” and “right” to turn, “stop” and “go” for the swing, and “yes” and “no” to confirm. The fan has a small chip and no internet connection. So the spotter must run on the fan, and no audio may leave the room.
Omar names the two errors that matter:
- A false alarm. The fan acts when nobody gave a command. People return a fan that starts at night, or turns off during a call. Omar sets a budget: at most 10 false alarms per hour of speech and noise that is not a command. That is still one every six minutes, so it is a first target, not a final one.
- A missed command. The person says “off” and nothing happens. They say it again, a little annoyed. That is bad, but not as bad as a fan that acts by itself.
Tomás has an answer ready: “Put Whisper on it. It understands everything.” Priya, the course’s data scientist, wants to measure first. Her questions are: which method keeps the budget, how many commands it catches, and whether it fits on the chip.
What you will build
Section titled “What you will build”A keyword spotter is a loop. It listens to the last second of sound, again and again, and decides each time: a command, or nothing?
Text description of the diagram
The microphone gives 16,000 numbers a second. Every 100 milliseconds, the device takes the last second of sound and turns it into a log-mel picture of 40 bands by 98 frames. The spotter reads the picture and gives a word and a score that says how sure it is. If the word is a command and the score is at least the threshold, the device acts and then waits one second. Otherwise it does nothing, and the audio is not kept.
You will try four spotters in the middle box, on the same clips and with the same measures:
- Energy and templates (the baseline): a loudness check, then the nearest average “word shape”.
- A small CNN trained on log-mel pictures: 45,132 weights, small enough for a chip.
- Whisper tiny with a word list: the recogniser of the course, with the ten commands as its prompt.
- Vosk with a grammar: the older recogniser of the course, allowed to answer only with the ten commands or “unknown”.
Then you choose each one’s threshold for the budget on validation, and test once. Last, you compare speed and size, and run the winner on a new recording.
The data
Section titled “The data”The data is Speech Commands v0.02. It has 105,829 one-second clips of 35 English words. 2,618 volunteers recorded them on their own phones and laptops, in their own rooms. The volunteers recorded the words for this purpose, through a web page. There are also six longer recordings of background noise.
This is real speech. The Larkfield calls of the course are synthetic: clear voices that read scripts. Here, people mumble, speak far from the microphone or cut the word short. Expect it to be harder than Larkfield’s calls.
Listen to two clips from the test split.
Transcript. “Yes.”
Transcript. “Bed.” It is not a command. A spotter that acts on it makes a false alarm.
What we use
Section titled “What we use”| Part | What it is | Use |
|---|---|---|
| The ten commands | yes, no, up, down, left, right, on, off, stop, go | What the fan must catch |
| Other words | the other 25 words, such as bed, happy, follow and the digits | Speech that is not a command: false alarms |
| Noise | six recordings: washing dishes, a cat, an exercise bike, a running tap, pink and white noise | Sound with no speech: false alarms |
speaker |
a random ID for each volunteer | To keep each person in one split, and to find the worst speakers |
The dataset kept no age, gender, accent or place of the volunteers. So this case study can compare words and speakers, but not groups of people.
The split, and the sample
Section titled “The split, and the sample”The dataset has its own split rule. A hash of the speaker’s ID (a fixed scramble of a text into a number) puts every clip of one person in the same split. Priya keeps that rule, and takes a fixed sample to keep the download small:
| Split | Commands | Other words | Noise windows | Speakers |
|---|---|---|---|---|
| Train | 2,985 | 1,600 | 1,416 | 1,522 |
| Validation | 600 | 1,200 | 152 | 240 |
| Test | 1,000 | 1,800 | 152 | 242 |
The noise windows are one second long, one every half second. Each noise file is cut in time: the first 60% for training, the next 20% for validation and the last 20% for testing. The dataset’s own test set does not do this: its “silence” clips are cut at random from the same noise files that training uses. We do not use them.
The notebook checks the split:
people = {s: set(speaker[(split == s) & (label != "noise")]) for s in ["train", "valid", "test"]}shared = len(people["train"] & people["test"]) + len(people["train"] & people["valid"]) + len(people["valid"] & people["test"])print("speakers in two splits:", shared)speakers in two splits: 0What this tells us. No volunteer is in two splits, so the test has only new voices, as a real fan does.
Frame the problem
Section titled “Frame the problem”- The unit: one second of sound, every 100 ms on the device. In the measurements, one clip or one noise window.
- The decision: act on a command, or do nothing.
- Caught: the share of command clips where the spotter acts with the right word. A wrong word (“up” heard as “off”) is worse than a miss, so it is counted apart.
- False alarms per hour: the number of times the spotter acts on other words or noise, divided by the hours of that audio. An other-word clip counts as one second; a noise window adds half a second.
- The rule: choose each method’s threshold on validation, as the lowest threshold that keeps within 10 false alarms per hour. Then measure once on test.
How many hours can we measure?
Section titled “How many hours can we measure?”The validation split has 1,200 other-word clips and 152 noise windows: 1,276 seconds, about 21 minutes. One false alarm there is already 2.8 per hour. So the budget of 10 allows 3 false alarms in validation.
That is the hard part of false alarms. A rule of thumb, the rule of three: if you see no false alarm in N hours, the true rate can still be up to about 3 ÷ N per hour (with 95% confidence). With our 21 minutes, “no false alarm” still allows about 8.5 per hour. We can test a budget of 10, but not “one a day”: that needs days of audio with no command in it.
Explore what the decision needs
Section titled “Explore what the decision needs”Sound as a picture: the log-mel spectrogram
Section titled “Sound as a picture: the log-mel spectrogram”In Audio as data, a spectrogram showed time across and pitch up. A log-mel spectrogram is the same picture with two changes that copy the human ear. First, it groups the pitches into 40 mel bands. The bands are narrow at low pitch and wide at high pitch, because the ear separates low pitches better. Second, the loudness is in decibels (a log scale), because the ear hears a sound ten times stronger as only “a bit louder”. One second becomes a small picture: 40 bands by 98 frames of 10 ms. Every spotter in this case study reads this picture.

Text description of the diagram
The three pictures share the same axes: time from 0 to 1 second across, and 40 mel bands from low to high pitch up. “Yes” is a bright shape in the middle of the second. Its lower bands are bright through the vowel, and the top bands light up at the end, for the hissing “s”. “Bed” is a shorter shape, mostly in the lower bands. The running tap fills the whole picture with an even texture, in every band, with no beginning and no end.
The notebook computes the picture with about 15 lines of NumPy (log_mel), and checks it against the stored picture (within 0.25 dB).
What this tells us. A word has a beginning, a shape and an end; noise has none. But two words can have similar shapes, so the spotter must tell shapes apart, not only sound from silence.
A baseline: energy and templates
Section titled “A baseline: energy and templates”Before any model, Priya tries the oldest trick. It has two steps:
- Energy: add the loudness of all bands in each frame, as the energy detector of Speech or silence? did. If the loudest frame is under 10 dB, there is no speech: do nothing.
- Templates: cut out the loud part of the clip, and stretch it to 32 frames. Compare it with a template of each command: the average shape of that word in the training clips. There is also a template for “other words”. The nearest command template wins. The score is how much closer the clip is to that command’s template than to the “other words” template.
templates = {}for c in COMMANDS + ["other"]: # the mean shape of each word in the training clips templates[c] = np.mean([s for s, s_split, s_label in zip(shapes, split, label) if s is not None and s_split == "train" and s_label == c], axis=0)On validation, with no threshold at all:
{'right': 0.502, 'wrong word': 0.428, 'missed': 0.07, 'false alarms (other)': 1094, 'per hour (other)': 3282.0, 'false alarms (noise)': 131, 'per hour (noise)': 6205.3, 'false alarms per hour': 3456.1}What this tells us. The baseline gets half the commands right. But it acts on 1,094 of the 1,200 other words and on 131 of the 152 noise windows. That is about 3,456 false alarms an hour. One average shape per word cannot hold thousands of voices, and loud noise passes the energy check. This is the floor every other method must beat.
Three alternatives
Section titled “Three alternatives”A small CNN on the pictures
Section titled “A small CNN on the pictures”The deep learning course built a convolutional network for photos in Classify clothing images. The idea is a small filter: a set of weights that slides over the input and gives a high number where its pattern is. Here the filter covers all 40 bands and 5 frames (50 ms), and it slides along time only. A filter that learned “a vowel starts here” fires at the start of the vowel in “yes”, wherever the word sits in the second.

Text description of the diagram
The top part shows the log-mel picture of “yes” with a white box. The box is 5 frames wide and covers all 40 bands. An arrow shows that the box slides along time. The bottom part shows the filter’s response at each of the 98 frames. It is almost zero everywhere. It has one clear peak where the box is drawn, at frame 43, where the vowel of “yes” starts.
The network has four such layers of 48 filters. After each layer, it keeps the larger of every two frames, so later filters see longer pieces of the word. At the end, it keeps the strongest response of each filter in the whole second, and a last layer turns those 48 numbers into 12 class probabilities: the ten commands, “other” and “noise”. It has 45,132 weights. Each picture is first scaled to its own loudest point, so a quiet voice and a loud voice look alike.
cnn = nn.Sequential(*block(40, 48), *block(48, 48), *block(48, 48), *block(48, 48), nn.AdaptiveMaxPool1d(1), nn.Flatten(), nn.Dropout(0.2), nn.Linear(48, len(CLASSES)))Training takes 12 epochs over the 6,001 training items, with each word moved in time by up to 60 ms at random. On the course’s laptop, it took about 30 seconds on two CPU threads. The spotter’s word is the most likely class; its score is that class’s probability. On validation, with no threshold:
{'right': 0.902, 'wrong word': 0.047, 'missed': 0.052, 'false alarms (other)': 72, 'per hour (other)': 216.0, 'false alarms (noise)': 0, 'per hour (noise)': 0.0, 'false alarms per hour': 203.1}What this tells us. 90.2% right, and no false alarm on noise: the “noise” class works. But it still acts on 72 of 1,200 other words, mostly words that sound like a command.
Whisper tiny with a word list
Section titled “Whisper tiny with a word list”Tomás’s idea, made cheap: the smallest Whisper of Whisper sizes on a laptop, told the language (English) and given the ten commands as its prompt, as in Help the model with your words. Whisper writes text, so the spotter needs a rule: take the first word after the course’s normalisation guide; if it is a command, that is the word heard. The score is Whisper’s average word probability.
Running Whisper on 4,904 items takes time, so the course team ran it once and recorded every output. The notebook replays them, and an optional cell runs Whisper live on six clips. What it wrote for some test items:
| What was said | What Whisper wrote (most common first) |
|---|---|
| “on” | “on.” 52, “On.” 17, “on” 13, “Come on.” 2 |
| “off” | “off.” 79, “up.” 5, “Off.” 3, “Oh.” 2 |
| digits (other words) | “7” 81, “9” 68, “Zero.” 57, “6.” 56 |
| noise | “you” 142, nothing 9, “yes, no, up, down.” 1 |
What this tells us. Whisper hears the words well, and it writes other words as themselves, so they are easy to ignore. On noise it writes “you”: the invented text of Why VAD makes recognition better. Once, it wrote its own prompt back: a prompt can leak into the output.
Vosk with a grammar
Section titled “Vosk with a grammar”The older recogniser of the course, Vosk small, can take a grammar: the list of the only words it may answer. Here: the ten commands and [unk] (unknown). In Compliance phrases and keyword spotting, this made Vosk a fast spotter. The score is Vosk’s confidence in the word. It was recorded the same way.
The four methods on validation, before any threshold
Section titled “The four methods on validation, before any threshold”| Method | Right | Wrong word | Missed | False alarms per hour |
|---|---|---|---|---|
| Energy + template | 50.2% | 42.8% | 7.0% | 3,456 |
| Small CNN | 90.2% | 4.7% | 5.2% | 203 |
| Whisper tiny + word list | 91.2% | 2.5% | 6.3% | 51 |
| Vosk + grammar | 95.2% | 1.2% | 3.7% | 344 |
Vosk catches the most commands, and Whisper makes the fewest false alarms. But nobody would ship any of them like this: the budget is 10 an hour.
A fair comparison: one budget, one threshold rule
Section titled “A fair comparison: one budget, one threshold rule”Each method has a score. A higher threshold means fewer false alarms and more misses. The chart shows the whole trade-off on validation: each point is one threshold.
- Small CNN
- Whisper tiny
- Vosk
- Energy + template
- Budget: 10 an hour
Text description of the chart
A line chart. Across: false alarms per hour on validation, from 0 to 200. Up: the share of commands caught, from 0 to 100%. A dashed vertical line marks the budget at 10 an hour. Whisper tiny’s line rises fastest: at the budget it catches 78.3%. The CNN’s line is a little lower near the budget, 71.2%, and climbs above 85% when more false alarms are allowed. Vosk appears only on the right: even at its highest confidence it makes about 135 false alarms an hour, while catching about 91%. The template baseline stays near the bottom: about 4% at the budget.
Priya’s thresholds, chosen on validation only:
| Method | Threshold | Right (validation) | False alarms per hour (validation) |
|---|---|---|---|
| Energy + template | 136.0 | 4.3% | 8.5 |
| Small CNN | 0.947 | 71.2% | 8.5 |
| Whisper tiny + word list | 0.613 | 78.3% | 8.5 |
| Vosk + grammar | 1.000 (its highest) | 90.8% | 135.4 |
What this tells us. Three methods keep the budget (3 false alarms in 21 minutes). Vosk cannot: most of its answers, right and wrong, have a confidence of exactly 1.0. A score that does not separate gives you no threshold to move.
The final test, once
Section titled “The final test, once”The test split has not been used for any choice. Priya runs each method once, at its validation threshold.
for name, (h, s) in methods.items(): r = evaluate(h, s, thresholds[name], "test") # the test split, once, at the validation threshold| Method | Right | Wrong word | Missed | False alarms per hour: other words | noise | all |
|---|---|---|---|---|---|---|
| Energy + template | 3.7% | 0.0% | 96.3% | 2.0 | 0.0 | 1.9 |
| Small CNN | 67.7% | 0.1% | 32.2% | 14.0 | 0.0 | 13.4 |
| Whisper tiny + word list | 76.0% | 0.6% | 23.4% | 10.0 | 0.0 | 9.6 |
| Vosk + grammar (no threshold keeps the budget) | 93.7% | 0.2% | 6.1% | 152.0 | 0.0 | 145.8 |
What this tells us. The test agrees with validation, with one honest surprise: the CNN’s false alarms rose from 8.5 to 13.4 an hour, a little over the budget. That is 7 false alarms in 31 minutes instead of 3 in 21: with so few events, the difference is within chance. Whisper tiny catches about 3 in 4 commands, and the CNN about 2 in 3. Wrong words almost disappear: most wrong answers were unsure, and the threshold removes them.
More data: the same network on every training clip
Section titled “More data: the same network on every training clip”The notebook’s CNN learned from 2,985 command clips and 1,600 other words. That keeps the training to half a minute. The dataset’s training split has 84,843 clips. Priya trains the same network with the same recipe once on all of them (about seven minutes on the course’s laptop). Only the training data changes: the same validation clips choose the threshold, and the same test clips measure it.
| Small CNN | Training clips | Threshold | Right (test) | False alarms per hour (test) |
|---|---|---|---|---|
| The notebook’s sample | 4,585 + noise | 0.947 | 67.7% | 13.4 |
| All the training clips (run once by the course team) | 84,843 + noise | 0.838 | 90.3% | 3.8 |
Text description of the chart
A column chart with five columns, from 0 to 100%. Template 3.7%. CNN on the sample 67.7%. Whisper tiny 76.0%. Vosk 93.7%, but at 145.8 false alarms an hour, far above the budget of 10. The same CNN trained on all the training clips 90.3%, at 3.8 false alarms an hour.
What this tells us. With 18 times more training clips, the same 45,132 weights catch 9 commands in 10 and make only 2 false alarms in 31 minutes. Every word is now at 83% or more, and the median speaker gets every command. The notebook’s numbers are about the method; the shipping model needs all the data. For the full model, the worst tenth of speakers is still caught only 43.7% of the time.
The worst cases: words and speakers
Section titled “The worst cases: words and speakers”An average hides who has the problem. Here is each command, on test:
| Word | Template | CNN | Whisper tiny | Vosk |
|---|---|---|---|---|
| yes | 3% | 81% | 94% | 98% |
| no | 2% | 50% | 74% | 98% |
| up | 0% | 67% | 80% | 82% |
| down | 0% | 59% | 73% | 93% |
| left | 1% | 74% | 82% | 91% |
| right | 9% | 70% | 83% | 96% |
| on | 0% | 68% | 36% | 93% |
| off | 0% | 78% | 66% | 93% |
| stop | 22% | 85% | 91% | 96% |
| go | 0% | 45% | 81% | 97% |
What this tells us. Each method has a different weak word. Whisper tiny catches “on” only 36% of the time, although it writes “on” in 84 of 100 clips. A one-syllable word gets a low average probability, and one threshold for all words cuts it. The CNN is weakest on “go” (45%) and “no” (50%): short words with a single vowel, close to other words of the set such as “dog” and “four”.
And by speaker: the test split has 122 volunteers with at least three command clips.
| Method | Median speaker | Worst tenth of speakers | Worst speaker |
|---|---|---|---|
| Small CNN | 74.2% | 6.0% | 0% |
| Whisper tiny + word list | 80.0% | 24.9% | 0% |
| Vosk (over budget) | 100% | 51.8% | 0% |
What this tells us. For one person in ten, the CNN almost never acts. Their voice, microphone or room is far from most training clips. With no age, gender or accent in the data, we cannot say who they are; a pilot with real buyers must ask.
Speed, size and cost
Section titled “Speed, size and cost”A fan’s chip is small. The course measured each method’s time for one decision (one second of audio, the log-mel picture included) on one CPU thread of the course’s laptop, an Apple M4:
| Method | Time per decision | Model size | Fits a small chip? |
|---|---|---|---|
| Energy + template | 0.25 ms | 0.06 MB | yes |
| Small CNN | 0.31 ms | 0.18 MB (45,132 weights × 4 bytes; about 0.05 MB at 8 bits) | yes |
| Vosk small + grammar | 4.4 ms | 70.9 MB | no: needs a small computer, such as a single-board computer |
| Whisper tiny | 275.7 ms | 78.2 MB | no |
The device decides every 100 ms. Whisper tiny needs 276 ms for one decision on a laptop core, so it cannot keep up even there. A small chip is much slower than a laptop, so the CNN’s 0.31 ms will grow too. The course did not measure a chip: measure on the real one before you promise a speed.
And a server? Tomás suggests: “Then send the audio to a server with Whisper.” Take the course’s assumption: a CPU server costs US$0.17 an hour for 4 vCPUs, about US$0.0425 per core-hour. If Whisper decides every half second, one core works 2 × 0.276 = 0.55 seconds per second of audio. That is about US$0.023 per hour of listening for one fan. A fan that listens all day costs about US$205 a year. The CNN on the fan costs nothing per hour: the chip is in the fan anyway. And the server route sends every second of the room’s sound out of the house, which breaks Omar’s first rule.
The recommendation
Section titled “The recommendation”Priya writes to Omar in his words:
- Use the small CNN on the fan’s chip. It is the only method that is both accurate enough and small enough. It is 0.18 MB and takes well under a millisecond per decision on a laptop. Whisper tiny is 430 times larger and about 900 times slower. Vosk cannot keep the false-alarm budget, and it is too large too.
- Train it on all the training clips, with threshold 0.838. On test, that model caught 90.3% of commands. It made 3.8 false alarms an hour on isolated other words, and none on noise. The notebook’s smaller model (67.7%) shows the method, not the product.
- Wait until the word ends before acting again. In the kitchen recording below, the full model acted twice on one “yes”. The rule “wait one second” is too short.
- Add a confirmation for risky commands. For “off” and “stop”, act at once; they are safe. For “on” at night, a short beep and a second “yes” costs one second and removes most of the harm of a false alarm.
- What a person still checks: a pilot in real homes, with the owners’ consent. The fan logs only the times and words it acted on, never audio. A person reads the log and asks the owner which actions were false.
And what not to use it for: anything that needs to know who speaks, anything safety-critical, or another language without new data.
End to end on new input: the kitchen recording
Section titled “End to end on new input: the kitchen recording”A test clip is one clean word. A real room has a word somewhere in a stream of noise. Priya builds a new recording from parts the model has never seen. She takes the last 20% of the dishes noise (the test part) and places six test clips in it. It is a constructed example, made from real recordings.
Transcript. At 2.0 s: “yes”. At 5.0 s: “bed”. At 8.0 s: “stop”. At 11.0 s: “go”. At 14.0 s: “seven”. At 16.5 s: “off”. Dishes all the time.
The CNN looks at the last second every 100 ms, acts when its score is at least its threshold, and then waits one second. Each row is the start of the second it acted on, the word and the score. First the notebook’s CNN (threshold 0.947):
said: [(2.0, 'yes'), (5.0, 'bed'), (8.0, 'stop'), (11.0, 'go'), (14.0, 'seven'), (16.5, 'off')]acted: [(1.7, 'yes', 0.982), (7.5, 'stop', 0.978)]Then the same network trained on all the clips (threshold 0.838):
acted: [(1.4, 'yes', 0.911), (2.4, 'yes', 0.965), (7.4, 'stop', 0.873), (8.4, 'up', 0.891), (10.5, 'go', 0.841), (16.4, 'off', 0.985)]Text description of the diagram
The waveform shows the dishes noise all the way through, with six shaded boxes where the words are. Under it are the words said: yes, bed, stop, go, seven and off. In orange, the notebook’s CNN acted twice: on “yes” and on “stop”. In green, the CNN trained on all the clips acted six times: twice on “yes”, on “stop”, then “up” while “stop” was still ending, then on “go” and on “off”. Neither model acted on “bed” or “seven”.
What this tells us. Both models ignored the noise and the two other words. The notebook’s model caught two of four commands: “go” was already its weakest word, and in noise its score fell below the threshold. The full model caught all four, but it acted twice on one “yes”, and it heard “up” in the end of “stop”. Both extra actions come from the loop, not from the clip tests: a word can fill two windows, so “wait one second” is too short. A better rule waits until the score falls below the threshold again. The training pictures also had noise or speech, never both together; mixing training words with noise is the next change to try.
Insights and limits
Section titled “Insights and limits”- False alarms decide the threshold, and the threshold decides the method. Before any threshold, all three alternatives catch more than 90% of commands. At 10 false alarms an hour, the sample CNN catches 68% and Whisper tiny 76%. Vosk cannot get there at all.
- Data beat model size. The same small CNN went from 67.7% to 90.3% with all the training clips, past Whisper tiny, which is 430 times larger.
- The loop around the model matters. Clip tests do not show double actions; a recording with words in noise does.
- A good recogniser is not a good spotter by default. Whisper writes “on” correctly, but its probability is low for a short word. Vosk’s grammar gives full confidence to wrong words. A spotter needs a score that separates.
- Real speech is harder than Larkfield’s synthetic calls. On Larkfield’s calls, the Vosk spotter found 56% of the recording notices with no false alarm, on only 3 calls without the notice. Here, with 1,800 other words, its false alarms are measurable, and too many.
- Limits. Isolated words are not a living room with sentences, television and music. 31 minutes of test negatives cannot show a rate like “one a day”. Speeds are from one laptop core. And the data has no groups of people, only speakers and words.
Responsible use
Section titled “Responsible use”- No audio leaves the device, and none is kept. That is the main reason to run the spotter on the fan. Write it in the product’s privacy notice, and keep it true in every update.
- Tell people the fan listens. A light that shows “listening”, and a switch that turns the microphone off.
- Test with the people who will use it. Even the full model catches the worst tenth of speakers less than half the time. Recruit testers of different ages, accents and voices, with their consent, before launch.
- The volunteers’ voices. Real people gave these clips for this kind of research, under CC BY 4.0. Credit the dataset, use the clips for spotting words, and never try to identify or imitate a speaker. No clip of this case study was sent to a hosted model.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can take a speech model from data to a device decision, with a product team’s measures. Make it yours with one change. Mix noise into the training words, add per-word thresholds, or fix the double action in the loop. Or try another language from the Multilingual Spoken Words Corpus (CC BY 4.0). Then make it public, easy to read in five minutes.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
results.csv: one row per method, with the threshold, right, wrong word, missed and false alarms per hour on validation and test.MODEL_CARD.md: intended use (ten English commands, on the device) and what it is not for. Add the data, the budget, the per-word and per-speaker results, and the limits.- The data source and licence with the attribution text from the box above.
A README template:
# A keyword spotter for a small device
## ProblemA fan obeys ten spoken commands, on the device, with no internet.Budget: at most 10 false alarms per hour of speech and noise that is not a command.
## Data and licenceSpeech Commands v0.02 (Warden, 2018), CC BY 4.0, https://arxiv.org/abs/1804.03209.Fixed sample: train 2,985 commands + 1,600 other words + noise; validation and test by thedataset's speaker split. Noise files cut in time so test noise is new.
## ApproachLog-mel pictures (40 x 98). Four spotters: energy + templates, a 45k-weight CNN,Whisper tiny with the commands as prompt, Vosk with a grammar. Thresholds chosen onvalidation for the budget; test used once.
## Results (test, once)CNN trained on all 84,843 training clips: 90.3% caught at 3.8 false alarms/hour,0.3 ms per decision on a laptop core, 0.18 MB. The same CNN on a 4,585-clip sample:67.7% at 13.4/hour. Whisper tiny 76.0% at 9.6/hour, but 276 ms and 78 MB.Vosk 93.7% but 146 false alarms/hour at its strictest.
## LimitsIsolated words, 31 minutes of test negatives, no speaker groups, speed measured on a laptop.How to describe it in an interview, in two sentences: “I compared four keyword spotters for a device with no internet, with thresholds chosen on validation for 10 false alarms an hour. On new speakers, a 45,000-weight CNN caught 90% of commands at under 4 false alarms an hour; Whisper tiny caught 76% and was 900 times slower.”
Go further:
- Mix the training words with the training noise, retrain, and run the kitchen recording again. With pictures, add the power of the two (
10 ** (db / 10)) at a random volume, then take the log again. Predict first whether “go” comes back. - Give each word its own threshold on validation, for the same total budget. Which words gain?
- Add a two-step rule: act only when two windows in a row agree. Measure what it does to false alarms and to the delay.
- Quantise the CNN to 8 bits and measure the change in accuracy.
Knowledge check
Section titled “Knowledge check”Check your understanding
Before any threshold, Vosk with a grammar catches 95.2% of validation commands, the most of the four methods. Omar asks why it is not the choice. What do you answer?
Show the answer
Because the fan’s main risk is acting by itself, and Vosk cannot keep the budget. Most of its answers have a confidence of exactly 1.0, right and wrong alike. So even at its highest threshold it makes about 135 false alarms an hour on validation (145.8 on test), against a budget of 10. A threshold only helps when the score separates right answers from wrong ones. Vosk is also about 71 MB, too large for the fan’s chip.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The validation split has about 21 minutes of other words and noise. The CNN made no false alarm on noise. Tomás says: 'So noise is solved.' What can you really say, and what would you need to say more?
Show the answer
The noise part of validation is only 76 seconds of new noise, so “zero false alarms” is weak evidence. By the rule of three, zero events in N hours still allows a rate up to about 3 ÷ N per hour. For 76 seconds, that is over 140 an hour. You can say that no false alarm happened on these six kinds of noise. To say more, you need many hours of real home sound with no command, such as television, music and talk. Then you count the false alarms there.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Whisper tiny writes 'on' correctly in 84 of 100 test clips, but at its threshold it catches only 36% of them. Explain why, and suggest a fix that does not change the model.
Show the answer
The score is Whisper’s average word probability, and a single short word like “on” often gets a low probability even when it is right. One threshold for all ten words is set by the false alarms of all words together, and it cuts many right “on” answers. A fix: choose a threshold per word on validation (a lower one for “on”), keeping the total false alarms within the budget. Then check on test once that “on” improves and the false alarms stay near 10 an hour.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A spotter acts on a command or does nothing, every 100 ms. Score it with commands caught, wrong words and false alarms per hour, on other words and on noise apart.
- A log-mel picture is one second of sound as 40 mel bands by 98 frames in dB: words are shapes, noise is texture.
- Choose the threshold on validation for the false-alarm budget, then test once. At 10 an hour, the sample CNN caught 67.7% of test commands and Whisper tiny 76.0%. Vosk could not keep the budget, and the template baseline caught 3.7%.
- Only the small CNN fits the fan: 45,132 weights, 0.18 MB, 0.31 ms per decision on a laptop core. Trained on all 84,843 training clips, it caught 90.3% at 3.8 false alarms an hour. Whisper tiny is 900 times slower.
- Check the worst words and speakers: Whisper missed “on”, the CNN missed “go”, and for one speaker in ten the CNN almost never acts.
- Few hours of negatives limit what you can claim: no false alarm in N hours still allows about 3 ÷ N an hour.
Next, the last case study: an IVR for a bank’s spoken requests, tested by speaker group.