Skip to content
Français

From sound to text

  • Lesson 10
  • 35 min
  • Module 4: Speech recognition: compare the families

Where we are

So far, we used recognisers as black boxes. To choose between recognisers, we need to know roughly how they work. In this lesson, we compare the older pipeline of Vosk with the end-to-end encoder and decoder of Whisper. First, think of a person who takes notes on a phone call.

What you will be able to do

Know what happens inside a speech recogniser, so that you can predict its mistakes. Then you can choose between the two families of recognisers for a job, and read their errors without surprise.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

  • Your larkspeech project folder at the end of Module 3, with its .venv environment, and the course data in larkspeech/data. If your copy is broken, copy labs C41/snapshots/end-of-m03 again, as in Set up the speech project.
  • Headphones or speakers for one short clip. Every clip has its transcript on the page.
  • No new package and no download: this lesson reads recorded transcripts.

Imagine a new help-desk agent who takes notes during calls. On her first day, she knows the common English words, but not the shop. A caller says “my Fern wall planter”. She hears the sounds clearly. But “Fern wall planter” is not in her head as a name, so she writes “phone will come to”. The words she wrote are real English words, and the sentence almost makes sense. It is still wrong.

A second note-taker works another way. He has a printed list of every word he is allowed to write. For each sound, he looks for the closest word on his list. “Larkfield” is not on his list, so he can never write it. He writes “luck field” instead: two words that are on the list and sound almost the same.

These two note-takers are the two families of speech recognisers: programs that turn speech into text (ASR). In this lesson, we look inside both families, just enough to predict their mistakes.

The problem: same calls, very different mistakes

Section titled “The problem: same calls, very different mistakes”

Larkfield has measured two recognisers on the same 60 test calls, with the same normalisation guide. These are the scores from Module 2’s speech score command:

Recogniser WER Order numbers wrong Product names wrong
Vosk small 31.2% 35 of 52 51 of 60
Whisper small 8.6% 8 of 52 41 of 60

The WER says that Whisper is better. But it does not say why, or which mistakes each one makes. Look at one call. Call C-0056 is from a caller whose Fern wall planter arrived damaged. Listen to the start of it:

Call C-0056 on the phone line, turns 2 to 4 (8.27 to 19.25 s). Both voices are synthetic. The agent, Freya, is Kokoro (voice bf_emma). The caller is Piper, trained on the VCTK corpus (speaker p254).

Transcript. Caller: “Hi. My Fern wall planter arrived damaged, and I need help quickly before I head out.” Agent: “I can help with that. Can I have your order number, please?” Caller: “Yes, it’s thirty-eight, forty-five, twenty-nine.” (The script writes the number as 384529.)

Now see what each recogniser wrote for the caller’s three turns. The picture marks every error. A wrong word is orange, an extra word is blue, and a missing word is red and crossed out.

The three caller turns of call C-0056, each with three lines: the script, Vosk small and Whisper small, after the normalisation guide. Turn 2: Vosk wrote i am a male on my boy was i move on to write done of a neat how quickly the florida; only my is right. Whisper wrote hi my phone will come to arrive damaged and i need help quickly before i head out; only the product name and arrived are wrong. Turn 4: Vosk and Whisper both wrote yes it is 384529, all right. Turn 6: the script is i do not want a refund i want a new 1. Vosk wrote we are is i do not want to be funny was me what minority or. Whisper wrote i do not want to refund i want a new 1.
The caller's three turns of call C-0056, after the normalisation guide. Orange: a wrong word. Blue: an extra word. Red, crossed out: a missing word. Recorded transcripts of Vosk small and Whisper small on the phone file.
Text description of the diagram

Turn 2 has 16 words in the script. Vosk small got 1 of them right (“my”); the rest are 15 wrong words and 5 extra words. Whisper small got everything right except the product name and “arrived”: it wrote “phone will come to arrive”. Turn 4 is the order number, said in pairs. Vosk wrote “yes it’s thirty eight forty five twenty nine”, and the guide turned the number words into “384529”. Whisper wrote “yes it is 384529”. So both got turn 4 all right. Turn 6 is “I do not want a refund, I want a new one.” Whisper wrote “I do not want to refund”, so it kept “not” and “refund”, and only “a” became “to”. You check Vosk’s turn 6 yourself at the end of this lesson. Note: an alignment pairs words by the smallest number of changes, so it sometimes pairs words in a way a person would not. The counts are still right.

Two things stand out. First, Vosk’s mistakes look like nonsense (“a neat how quickly the florida”). Whisper’s mistakes look like good English (“phone will come to arrive”). Second, Vosk writes numbers as words (“thirty eight forty five twenty nine”). On this call, the guide turned them into the right digits. But on the 60 test calls, Vosk got 35 of the 52 order numbers wrong. To understand why, we look at how each family works.

Both families start the same way. A recogniser does not read the waveform directly. It first turns the sound into a picture of pitch over time, close to the spectrogram of Audio as data.

Whisper uses a log-mel spectrogram. “Mel” is a pitch scale that follows the human ear. We hear a step from 200 to 300 Hz as a big change, and a step from 3,200 to 3,300 Hz as a small one. So the mel scale has many narrow bands at low pitches and fewer, wider bands at high pitches. “Log” means the loudness is on a logarithmic scale, like decibels. Whisper measures 80 mel bands every 10 milliseconds, in a window of 25 milliseconds.

Here is turn 4 of call C-0056, “Yes, it’s thirty-eight, forty-five, twenty-nine”, as Whisper’s own feature code computes it:

Step Value
Length of the turn 2.73 s
Samples at 16,000 a second (what Whisper expects) 43,616
Frames, one every 10 ms 272
Numbers in each frame (mel bands) 80
Numbers in the picture 272 × 80 = 21,760

So the picture has about half as many numbers as the sound. More important, its numbers are arranged by pitch, which is how speech sounds differ.

Two charts that share a time axis from 0 to 2.7 seconds. Top: the waveform of the caller saying Yes, it's thirty-eight, forty-five, twenty-nine, with each word written above its burst of sound. Bottom: a log-mel picture with 80 bands from low pitch at the bottom to high pitch at the top. Bright colour in the low bands for each word, and thinner stripes higher up. Above band 65 the picture is black for the whole turn.
Turn 4 of call C-0056 (phone line, resampled to 16,000 samples a second). Top: the sound. Bottom: the 80-band log-mel picture that Whisper reads, from faster-whisper's feature code.
Text description of the diagram

The top chart shows the sound, with each spoken word above its burst. The bottom chart is the log-mel picture: time across, 80 mel bands up, and brightness for loudness. Each word makes bright patches in the low bands, with stripes higher up. The top bands, above about band 65, are black for the whole turn. These bands hold pitches above 4,000 Hz, and a phone file at 8,000 samples a second has nothing there. So on every phone call, Whisper reads a picture with an empty top.

After this first step, the two families go very different ways.

A flow chart. Sound goes into two paths. The older pipeline, Vosk: features, a short slice every 10 milliseconds; acoustic model, which speech sound is in this slice; pronunciation list, which words are made of these sounds; language model, which word order is likely; then text with only words from the list. End to end, Whisper: log-mel picture, 80 numbers every 10 milliseconds; encoder, reads the whole picture of 30 seconds; decoder, writes the text one piece at a time; then text with any word it can spell, with punctuation.
The two families of recognisers. The older pipeline has separate parts that each answer one question. The end-to-end model learns the whole path from picture to text.
Text description of the diagram

The older pipeline, used by Vosk, has four parts in a row. The features describe a short slice of sound every 10 milliseconds. The acoustic model says which speech sound is in each slice. The pronunciation list says which words are made of those sounds. The language model says which order of words is likely. The output can only contain words from the list. The end-to-end path, used by Whisper, has three parts. The log-mel picture comes first. An encoder reads the whole picture, and a decoder writes the text one piece at a time. Its output can contain any word that it can spell, with capital letters and punctuation.

Vosk is built on Kaldi, a toolkit that speech researchers used for many years. Its models are small: about 40 MB for each language in the “small” models that this course uses. It runs on a laptop CPU, and even on a phone. The pipeline has three trained parts after the features.

Speech is made of about 40 basic sounds in English. These sounds are called phonemes. “Field” has four of them: f, ee, l, d. The acoustic model looks at each 10-millisecond slice and gives a score to each phoneme: “this slice is probably an f, maybe a th”. It knows sounds, not words.

The pronunciation list (also called a lexicon) gives the phonemes of every word that the recogniser knows. For example, “field” is f-ee-l-d and “feel” is f-ee-l. A word that is not in the list can never be in the output. This is called a closed vocabulary.

Many word sequences match the same sounds. “Recognise speech” and “wreck a nice beach” sound almost the same. The language model gives a score to each word sequence: how often do these words follow each other in a large amount of text? A decoder then searches for the words with the best total score: sounds that match, and words that often go together.

“Larkfield” is not in the pronunciation list of Vosk’s small English model. So Vosk can never write it. On the 60 test calls, the scripts say “Larkfield” 120 times. Vosk wrote it 0 times. After “calling”, it wrote these words instead:

Vosk wrote Times
luck field 16
lark feel 9
work field 7
luck feels 5
large field 5

Each one is two listed words that sound like “Larkfield”. The same happens to product names: “my Fern wall planter arrived” became “my boy was I move on”. It also explains many order numbers. Vosk’s language model learned from ordinary text, where a row of single numbers is rare. So the search often prefers common words that sound similar. On test call C-0288, the caller says “two six zero, zero two six”, and Vosk wrote “to six there sarah to sick”.

The good news: the parts are separate, so you can change one. Vosk can take a list of allowed phrases for a short command, and its lexicon and language model can be rebuilt with new words. That takes skill and time, and the course does not do it. Module 6 uses the phrase list to listen for the recording notice.

Whisper is a model that OpenAI trained and published under the MIT licence. Its authors say that they trained it on about 680,000 hours of audio from the web, with the text that went with it, in many languages. It comes in sizes: tiny (39 million parameters), base (74 million), small (244 million), medium (769 million) and larger. These are the authors’ numbers, not ours. A parameter is one number that the model learned in training.

Whisper is end to end: one neural network goes from the log-mel picture to the text. It has no phonemes, no separate pronunciation list and no separate language model. It learned all of them together from examples. It has two halves.

The encoder reads a log-mel picture of up to 30 seconds at once. It turns it into a list of vectors (lists of numbers), one for each short moment of the audio. Each vector describes that moment in its context: the sounds before and after it. A longer call is cut into 30-second windows.

The decoder writes the text, one piece at a time

Section titled “The decoder writes the text, one piece at a time”

The decoder writes the text in small pieces called tokens. A token is a common word or a part of a word: “Lark” and “field” could be two tokens. At each step, the decoder looks at two things: the encoder’s vectors, and the tokens that it wrote so far. Then it chooses the most likely next token. It stops when it writes a special “end” token.

This is the same idea as a large language model that writes text, with the sound as extra input. The deep learning course explains encoders and decoders in detail, in The original transformer, step by step. For this course, you need only the idea: the decoder writes what is likely, given the sound and the words before.

Whisper’s vocabulary is open: it can spell any word from its tokens. So it can write “Larkfield”, and it often does: Whisper small wrote it 55 times of 120. But it writes what is likely. “Fern wall planter” is a rare name, and “phone will come to” is likely English, so it wrote that. Its mistakes are fluent, and fluent mistakes are hard to see when you read a transcript.

The same strength writes numbers well. In its training text, people write “384529”, not “three eight four five two nine”. So Whisper writes digits, which Larkfield’s order system needs.

The decoder has one more risk: it always writes something likely. When the audio is music or silence, it can invent a sentence. You saw this in Module 3: in long silences, Whisper small wrote 65 words that nobody said.

Guided practice: read the two families’ errors

Section titled “Guided practice: read the two families’ errors”

You read the recorded transcripts of both families with the score command of Module 2. No model runs, so each command takes a few seconds.

  1. Open a terminal in your larkspeech folder and activate the environment: source .venv/bin/activate on macOS and Linux, or .venv\Scripts\Activate.ps1 in Windows PowerShell.

  2. Score Vosk small on the test calls:

    Terminal window
    speech score --method vosk-small
    vosk-small, test calls, phone audio, normalisation full
    slice calls words WER CER order nos wrong products wrong
    all 60 7334 31.2% 20.1% 35 of 52 51 of 60
  3. Score Whisper small the same way:

    Terminal window
    speech score --method whisper-small
    whisper-small, test calls, phone audio, normalisation full
    slice calls words WER CER order nos wrong products wrong
    all 60 7334 8.6% 4.2% 8 of 52 41 of 60
  4. Now align one caller turn with speech wer. Use the script’s words and Vosk’s words for turn 2 of call C-0056:

    Terminal window
    speech wer "Hi. My Fern wall planter arrived damaged, and I need help quickly before I head out." "i'm a male on my boy was i move on to write done of a neat how quickly the florida"
    REF: * ** * **** hi my *** fern wall planter arrived damaged and i need help quickly before i head out
    HYP: i am a male on my boy was i move on to write done of a neat how quickly the florida
    I I I I S I S S S S S S S S S S S S S S
    15 substitutions, 0 deletions, 5 insertions, 16 reference words: WER 125.0%, CER 74.1% (normalisation: full)

    The guide turned Vosk’s “i’m” into “i am”. The script has 16 words, and Vosk wrote 21 words for them. That is why the WER of one turn can be above 100%.

On Windows, the commands are the same. They were tested on macOS only.

Turn 6 of call C-0056 is “I do not want a refund, I want a new one.” Vosk small wrote “we are is i do not want to be funny was me what minority or”. Before you run it, predict: how many substitutions, deletions and insertions? Is the negation (“not”) kept? Then check with speech wer.

Check your answer
REF: ** *** ** i do not want ** a refund i want a new 1
HYP: we are is i do not want to be funny was me what minority or
I I I I S S S S S S S
7 substitutions, 0 deletions, 4 insertions, 11 reference words: WER 100.0%, CER 89.2% (normalisation: full)

“Not” is kept, so the caller still refuses something. But “refund” is lost (“be funny”), so a reader cannot tell what the caller refuses. On this turn, Whisper small kept both words: it wrote “I do not want to refund”, with one wrong word.

Older pipeline (Vosk small) End to end (Whisper small)
WER on Larkfield’s phone test calls 31.2% 8.6%
Speed (real-time factor, 4 CPU threads) 0.085 0.075
Memory 566 MB 1,267 MB
New words rebuild the lexicon and language model, or a phrase list a text prompt (lesson 4 of this module)
Typical mistake nonsense words, numbers as words fluent wrong words, invented text in silence
Licence Apache-2.0 MIT

On Larkfield’s synthetic calls, Whisper wins on error by a wide margin, at about the same speed. So the rest of this module compares Whisper sizes. The older pipeline still has uses. It runs on very small devices. It can listen for a short, fixed list of phrases, such as “yes”, “no” or a menu choice, with very little compute. And its closed vocabulary means that it cannot invent a word that is not in its list.

Common mistake

Reading a fluent transcript as a right transcript

Symptom: Grace reads twenty Whisper transcripts and finds them excellent: clear sentences, good punctuation. Then a caller complains that nobody sent the “Fern wall planter” replacement. The transcript says “my phone will come to arrive damaged”.

Cause: an end-to-end decoder writes what is likely. Its mistakes are good English, so a reader does not notice them. Vosk’s mistakes (“a neat how quickly the florida”) look wrong at once.

Fix: never judge a recogniser by reading transcripts. Measure it against references, and count the words that matter apart, as in The errors that matter. Whisper small missed 41 of the 60 product names on the test calls, although its WER is only 8.6%.

Consequence in AI work: a system that reads the transcript later, such as a call-reason classifier or an LLM summary, gets fluent wrong text. It cannot see the mistake either.

Check your understanding

Vosk small never wrote "Larkfield" on the test calls, but Whisper small wrote it 55 times. Explain the difference with the parts of each recogniser.

Show the answer

Vosk has a closed vocabulary: its pronunciation list holds every word it can write, and “Larkfield” is not in it. So the search picks listed words that sound alike, such as “luck field”. Whisper’s decoder writes tokens, small pieces of words, so it can spell any word. It writes “Larkfield” when that is the most likely text for the sound and the words before.

Check your understanding

A turn lasts 4 seconds. How many log-mel frames does Whisper make for it, and how many numbers does the picture have? Why is the top of the picture empty for Larkfield's phone files?

Show the answer

One frame every 10 ms gives 4 / 0.01 = 400 frames, each with 80 mel bands: 400 × 80 = 32,000 numbers. The top bands hold pitches above 4,000 Hz. A phone file has 8,000 samples a second, so it can hold pitches only up to 4,000 Hz, and the line removes most sound above 3,400 Hz. Resampling to 16,000 a second does not bring them back.

Check your understanding

Tomás says: "Whisper's transcripts read so well that we can skip the measurement." Give one number from this lesson that shows why that is wrong.

Show the answer

Whisper small missed 41 of the 60 product names on the test calls, with an overall WER of 8.6%. Its mistakes are fluent (“phone will come to arrive” for “Fern wall planter arrived”), so a reader does not see them. Only a measurement against references, with the important words counted apart, shows them.

Recap

  • A recogniser first turns the sound into a picture of pitch over time. Whisper reads a log-mel spectrogram: 80 bands every 10 ms. On phone files, its top bands are empty.
  • The older pipeline (Vosk) has separate parts: an acoustic model (which sound), a pronunciation list (which words) and a language model (which order). Its vocabulary is closed: “Larkfield” became “luck field”.
  • The end-to-end model (Whisper) has an encoder that reads the whole picture and a decoder that writes tokens, one at a time. It can write any word, and its mistakes are fluent.
  • On Larkfield’s synthetic phone test calls, Vosk small has a WER of 31.2% and Whisper small 8.6%, at about the same speed.
  • Never judge a transcript by how well it reads. Measure it.

You now know why Whisper makes fewer, but more fluent, mistakes. But Whisper comes in many sizes, and Tomás wants the biggest one. Next, you run Whisper sizes on your laptop and compare their errors with their speed, memory and cost.

Next: Whisper sizes on a laptop