Skip to content
Français

Choose a recogniser for two languages

  • Case study 1
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we compared recognisers on Larkfield's synthetic calls. In this case study, we use real English and French speech from an open dataset, compare Vosk and Whisper sizes by speaker group, and recommend one.

Case study 1 of 4 · Speech AI: Transcribe, Understand and SpeakChoose a recogniser for two languages

What you will be able to do

Choose a speech recogniser for a real product in two languages, with evidence that you can show to others. You will test five recognisers on real English and French speech. You will check the groups of speakers that an average hides. Then you will recommend one recogniser per language, with its error, speed and cost.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Normalisation guide
  • WER and CER
  • Real-time factor and memory
  • Slices by language and gender
  • Paired comparison
  • Cost per audio hour

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. It scores the recorded transcripts of all five recognisers, so it does not need hours of compute. At the end, it runs the chosen recogniser on 20 new recordings on your computer. Its setup cells download about 30 MB of files and the Whisper model (about 480 MB). It runs in about 3 minutes on a laptop.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole recognition part of the course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • Python 3.12 in a new virtual environment, and the notebook, downloaded from the link above, in VS Code or JupyterLab. The notebook’s first cell installs its packages. Your larkspeech project works too; this case study does not change its files.
  • About 1 GB of free disk for the files and the Whisper model. The setup cell puts the files in a new folder fleurs-recognisers. To start again, delete that folder.

Tested on macOS (Apple M4, 16 GB) with Python 3.12 and faster-whisper 1.2.1, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.

Two Rivers Home Repairs fixes heating, plumbing and electrics in homes on both sides of the Ottawa River, in Ontario and Quebec. Its 150 technicians work in English or French, as each customer prefers. After each visit, a technician writes a short note: what they found, what they did, and what comes next.

Today they type the notes on a phone, in a van, between two visits. The notes are short, late and sometimes missing. Leila, the operations manager, wants dictation: the technician speaks for about 40 seconds, the app shows the text, and the technician checks it and saves it.

Leila has three conditions:

  • Both languages must work. A French note must be as usable as an English one.
  • The app must serve every technician equally. Women and men work in the field, and the app must not work worse for one group.
  • The audio stays with the company. The notes name customers and their addresses. So the recogniser runs on the company’s own CPU server, not on a hosted service. Hugo, the developer, owns that server.

Inès, the company’s data analyst, must recommend a recogniser. She has no recordings of the technicians yet. Recording them needs their consent and a few weeks. So she starts with an open dataset of real speech, to choose the candidates and to build the test that she will later run on the technicians’ own notes.

You will build a fair comparison of five recognisers: the same real recordings in English and French, one normalisation guide, one table. You will look at the results by language and by speaker gender, with the worst recordings. Then you will recommend a recogniser for each language. You will give its cost per audio hour on a CPU server, and say what a person must still check. Last, you will run the recommendation on new recordings, end to end.

A flow chart from left to right. Real speech: the FLEURS test split, 262 English and 230 French recordings. Then a box with the same audio for every recogniser: the baseline Vosk small, and Whisper tiny, base and small, with medium on a sample. Then one guide: the course's normalisation guide plus one rule for big numbers. Then scores by group: language times gender, with WER, CER, real-time factor and memory. Then the decision: one recogniser per language and its cost per audio hour. Last, new input: 20 recordings that nobody looked at.
The case study in six steps: the same audio and the same guide for every recogniser, scores by group, a decision, and a check on new recordings.
Text description of the diagram

The case study has six steps. It starts with real speech: 262 English and 230 French recordings from FLEURS. Every recogniser gets the same audio: Vosk small as the baseline, and Whisper tiny, base and small, with Whisper medium on a sample. One normalisation guide prepares every transcript. The scores are split by language and by gender. The decision names one recogniser per language and its cost. The last step runs the decision on 20 new recordings that nobody looked at before.

FLEURS is an open dataset from Google. Volunteers read sentences aloud in 102 languages. The sentences come from Wikipedia articles, and translators put them into every language. So English and French have the same sentences. Each recording has its text and the speaker’s gender as FLEURS records it.

Inès also looked at Mozilla Common Voice: many speakers, CC0 data, but its downloads now need an account on Mozilla’s own platform. It is a good option for a later test, not for a notebook that every learner must run.

One row is one recording: one volunteer reading one sentence, recorded at 16,000 samples a second. These are the columns the case study uses:

Column What it means
language en (English, United States) or fr (French, France)
sentence_id The sentence. The same number means the same sentence in both languages.
gender female or male, as FLEURS records it
seconds The length of the recording
text What the speaker read: the reference

First checks: the same sentences for both groups

Section titled “First checks: the same sentences for both groups”

In the test split, some sentences were read by a woman and a man, and others by two women or two men. If women and men read different sentences, a gap in their scores could come from the sentences, not from the voices. So Inès keeps only the sentences that both a woman and a man read, with one recording of each:

Counter({('en', 'female'): 131, ('en', 'male'): 131, ('fr', 'female'): 115, ('fr', 'male'): 115})
en 262 recordings, 131 sentences, every sentence read twice: True, 44.2 minutes
fr 230 recordings, 115 sentences, every sentence read twice: True, 38.6 minutes

What this tells us. The sample is 262 English and 230 French recordings, about 10 seconds each. Each group reads the same sentences. With about 5,810 English and 6,052 French reference words, one word error moves the WER of a language by about 0.017 and 0.017 points. Inside one gender, it moves it about twice as much.

Listen to one sentence, read by a woman and by a man in English, and by a man in French:

FLEURS, English test split, a female volunteer (4.3 s). Real speech, CC BY 4.0.
FLEURS, English test split, a male volunteer (6.1 s). Real speech, CC BY 4.0.

Transcript (both English clips): “The scenes are displayed on the pyramids and the different pyramids are lit up.”

FLEURS, French test split, a male volunteer (5 s). Real speech, CC BY 4.0.

Transcript (French clip): “Les scènes sont affichées sur les pyramides et les différentes pyramides sont éclairées.” (In English: the same sentence as above.)

These are real people, not synthetic voices. You hear breath, a little room sound, and each speaker’s own pace.

Before she runs anything, Inès writes down what she measures and what “good enough” means. If she chose the targets after the results, she could pick the targets that make her favourite win.

  • The unit: one recording, one speaker, one sentence.
  • The main measure: WER after one normalisation guide, with the errors summed over all recordings and then divided. CER as a second measure, because French has many short words that a wrong letter changes.
  • Speed and memory: the real-time factor (RTF: compute time ÷ audio time) on 4 CPU threads, and the peak memory of the run.
  • The groups: language × gender. FLEURS gives no accent and no age, so we cannot check those groups here.
  • What a person checks: the technician reads every note before saving it.

Leila and Inès agree on these targets (a team decision, written before the results):

Target Value Why
WER per language at most 10% about one word in ten to fix in a note
Gender gap no clear gap larger than 2 points Leila’s second condition
Real-time factor at most 0.25 a 40-second note is back in 10 seconds
Peak memory at most 2 GB the server also runs the app

What FLEURS cannot tell her. The volunteers read clean sentences in a quiet room. Technicians speak freely, in a van or a basement, with trade words such as “backflow valve” or “chauffe-eau”. So FLEURS chooses the candidates and builds the method. It does not prove the final numbers. Inès will run the same test on 100 consented notes before the app goes live.

The course’s normalisation guide was written for Larkfield’s calls. There, order numbers are said digit by digit, so the guide joins digits (“7 4 2” → “742”). FLEURS sentences have big numbers and years. Vosk writes numbers as words; Whisper mostly writes digits. Look at one recording:

said: Goats seem to have been first domesticated roughly 10,000 years ago in the Zagros Mountains of Iran.
Vosk wrote: good seem to have been first domesticated roughly ten thousand years ago and as are girls mountains of iran
course's guide: ... roughly 10000 years ago ... | ... roughly 10 thousand years ago ...
+ big_numbers: ... roughly 10000 years ago ... | ... roughly 10000 years ago ...

What this tells us. With the course’s guide alone, “ten thousand” becomes “10 thousand”, and Vosk loses two words although it heard the number right. A guide that does not fit the data punishes one recogniser more than another. So Inès adds one rule for this data, big_numbers: a number word with “hundred”, “thousand” or “million” (French “cent”, “mille”, “million”) becomes digits. She applies it to every recogniser and to the references, before any score.

WER after… Vosk small, English Vosk small, French Whisper small, English Whisper small, French
no guide (split on spaces) 41.9% 35.9% 12.9% 21.2%
the course’s guide 25.2% 19.6% 6.8% 13.6%
the guide + big_numbers 24.6% 18.8% 6.7% 13.6%

The new rule moves Vosk by under 1 point, and Whisper by almost nothing. That is small, and it is still worth doing: the gaps between recognisers must come from what they heard, not from how they write numbers. The guide matters much more than the rule: without any guide, every WER is about twice as high.

Two other spelling questions remain, and Inès leaves them as errors. The English references use British spelling (“travellers”) and Whisper writes American spelling (“travelers”). And the guide removes “mm” as a filler, so “35 mm” (millimetres) loses its unit in both texts. Both are rare here. Write such rules down when you find them, and apply them to every recogniser or to none.

The baseline is the simple method that every other method must beat. Here it is Vosk small: the older pipeline from From sound to text, about 40 MB per language, one model for each language.

Vosk small WER CER RTF Peak memory
English 24.6% 12.8% 0.074 252 MB
French 18.8% 8.0% 0.13 298 MB

What this tells us. Vosk small makes about one error in four words in English, and about one in five in French. It is light: under 300 MB of memory. On Larkfield’s synthetic phone calls, Vosk had 24.4% in English and 39.4% in French. Here French is the easier language for Vosk. Keep that in mind: on real speech, the ranking of the languages can change.

The alternatives are Whisper tiny, base and small through faster-whisper, with 8-bit weights, as in Whisper sizes on a laptop. Each one is told the language. Every recogniser gets the same 492 recordings, on the same laptop (Apple M4, 4 CPU threads), and the same guide. Running all of them took about 25 minutes. The notebook does not run them again: it scores the recorded transcripts, so its numbers are exactly the ones on this page.

Here is the table. Every row is the same recordings and the same guide.

Recogniser English WER French WER English CER French CER RTF (en / fr) Peak memory (en / fr)
Vosk small (baseline) 24.6% 18.8% 12.8% 8.0% 0.074 / 0.13 252 / 298 MB
Whisper tiny 15.4% 44.4% 6.7% 19.8% 0.019 / 0.028 545 / 438 MB
Whisper base 11.1% 28.4% 4.8% 11.8% 0.043 / 0.046 626 / 537 MB
Whisper small 6.7% 13.6% 2.7% 5.0% 0.13 / 0.13 1,066 / 1,149 MB
Two scatter charts, English and French. Across: the real-time factor on a log scale; left is faster. Up: WER. A shaded corner marks WER at most 10% and RTF at most 0.25. English: Vosk 24.6% at RTF 0.074, tiny 15.4% at 0.0189, base 11.1% at 0.0425, small 6.7% at 0.1343, medium on 25 sentences 4.7% at 0.2932. French: Vosk 18.8% at 0.1294, tiny 44.4% at 0.0284, base 28.4% at 0.0462, small 13.6% at 0.13, medium on all French recordings 7.4% at 0.3395.
Error against speed on real read speech, English and French (FLEURS, one laptop, 4 CPU threads). Square: Vosk small; circles: Whisper. Whisper medium: an open circle in English (the first 25 sentences only), a filled circle in French (all recordings; see the next section). The shaded corner meets both targets: WER at most 10% and RTF at most 0.25.
Text description of the diagram

Each point is one recogniser. Lower is more accurate, and further left is faster. In English, Whisper small is under 10% WER, at 6.7%; base is just over, at 11.1%. In French, Vosk and the three smaller Whisper sizes are all above 10%. Whisper small is best of them at 13.6%, then Vosk at 18.8%, and Whisper base (28.4%) and tiny (44.4%) are worse than Vosk. Whisper medium is the most accurate in both languages, and the slowest: the next section looks at it.

What this tells us. Three things.

  1. In English, Whisper small meets the error target, and base almost does. Small makes about one error in fifteen words.
  2. In French, the baseline beats two Whisper sizes. Vosk small (18.8%) is better than Whisper base (28.4%) and much better than tiny (44.4%). Whisper’s small sizes saw much more English than French when they learned. A bigger, newer model is not always better in every language: measure it.
  3. Whisper is slower here than on Larkfield’s calls. Small needs an RTF of about 0.13, against 0.075 on the phone calls. One reason is the length: Whisper always reads a 30-second window, so a 10-second recording pays for 30 seconds of work. The check below measures it.

Inès runs one quick check. She takes the first 20 English recordings and runs Whisper on them one by one. Then she joins the same 20 into one long file and runs it once.

Whisper size One by one (about 9.1 s each) Joined into one file
tiny RTF 0.017 RTF 0.011
small RTF 0.11 RTF 0.061
medium RTF 0.336 RTF 0.165

What this tells us. The same audio costs about half the compute when it comes in one long piece. Technicians’ notes are about 40 seconds, longer than these sentences, so the real RTF will be lower than in the table. The table’s speeds are also from a laptop that was running other work at the same time. Measure speed on the server you will use, with audio of the real length.

Two runners race on different days: one in the rain, one in the sun. Who is faster? You cannot tell, because the weather changed too. Make them run the same track on the same day, and the comparison is fair.

Scores by group have the same problem. If women read harder sentences than men, a gap in WER could come from the sentences. Our sample fixes that: every sentence was read once by a woman and once by a man. So we can compare inside each sentence, and count who got more errors on the same words. This is a paired comparison.

A flow chart. One sentence from the test split. A woman reads it: recording 1. A man reads it: recording 2. Both go to the same recogniser and the same guide. Errors in recording 1 and errors in recording 2 go to a comparison: more, fewer or the same? The counts are added over all sentences.
The paired comparison: the same sentence, read by a woman and by a man, through the same recogniser and the same guide. Count which recording got more errors, over all sentences.
Text description of the diagram

One sentence is read twice: once by a woman, once by a man. Both recordings go through the same recogniser and the same guide. We compare their word errors: the woman’s recording had more, fewer or the same. We count this over all sentences. Because the words are the same, a difference comes from the voices and the recordings, not from the sentences.

A tiny example, with constructed numbers for three sentences:

Sentence Errors, woman Errors, man Who had more?
A 4 2 the woman
B 1 1 a tie
C 0 3 the man

Over three sentences, that is one each and one tie: no sign of a gap. Over 115 sentences, a real gap shows as a clear imbalance. To put a number on “clear”, we resample the sentences. We draw 115 sentences at random, with repeats allowed, and compute the gap. We do this 2,000 times. The middle 95% of those gaps is the 95% interval. If it includes 0, we cannot say which group the recogniser serves worse.

Two bar charts, English and French, with one pair of bars per recogniser: female and male WER on the same sentences. English: Vosk 24.9 and 24.4, tiny 14.5 and 16.2, base 11.5 and 10.8, small 6.6 and 6.7. French: Vosk 20.3 and 17.3, tiny 43.7 and 45.2, base 30.1 and 26.7, small 14.4 and 12.9, medium 8.2 and 6.6. A dashed line marks the 10% target.
WER of female and male speakers on the same sentences, for each recogniser (FLEURS test sample). The dashed line is the 10% target.
Text description of the diagram

Each pair of bars is one recogniser: the female speakers’ WER and the male speakers’ WER, on the same sentences. In English, the pairs are close for every recogniser. In French, the female bar is higher for Vosk (20.3% against 17.3%), base (30.1% against 26.7%) small (14.4% against 12.9%) and medium (8.2% against 6.6%), and lower for tiny.

Recogniser Language Gap (female − male) 95% interval Sentences: woman worse / man worse / tie
Vosk small English 0.5 -2.7 to 3.5 50 / 63 / 18
Vosk small French 2.9 0.5 to 5.2 62 / 36 / 17
Whisper base French 3.5 0.1 to 7.1 58 / 47 / 10
Whisper small English -0.1 -1.5 to 1.3 38 / 43 / 50
Whisper small French 1.5 -0.7 to 3.8 54 / 43 / 18
Whisper medium French 1.7 0.1 to 3.3 49 / 31 / 35

What this tells us. In English, Whisper small serves women and men equally: the gap is -0.1 points, and in 50 of 131 sentences both recordings had the same number of errors. In French, Vosk small has a clear gap: women’s recordings got more errors in 62 sentences and men’s in 36, and the interval does not include 0. Whisper small in French leans the same way (1.5 points), but its interval includes 0, so this sample cannot tell. That is a reason to watch, not a proof of fairness.

Be careful with what a gap means. It says this recogniser made more errors on these volunteers’ recordings. It does not say why: the voices, the microphones and the rooms all differ between speakers. FLEURS has no speaker IDs, so one person may have read several sentences. And FLEURS records gender in two values only. Use the label to compare groups, never to guess anyone’s gender from their voice.

The worst recordings and the words that matter

Section titled “The worst recordings and the words that matter”

An average hides the recordings that fail. Here are the worst French recordings of Whisper small (at least 8 words):

Said Whisper small wrote WER
Tout dans l’Univers est fait de matière. Toute la matière est faite de minuscules particules appelées atomes. Tout est dans l'univers et fait de matière. Toute la matière est fait de minus que le particule a plié à tourne. 61.1%
Sundarbans est une réserve naturelle depuis 1966, vous pourrez y trouver environ 400 tigres du Bengale royal et 30 000 cerfs tachetés. Sune d'Arban est une réserve naturelle depuis 1966. Vous pourrez trouver en faisant 4 centigres du Mengal Royal et 30 000 serres à râtocheter. 57.1%

What this tells us. The worst errors are names (“Sundarbans”, “Bengale”) and rare words (“minuscules particules appelées atomes”). Small French words such as “de”, “l’” and “est” also go wrong. A dictation note has names too: customers, streets, parts. Inès counts two kinds of important words, as in The errors that matter. Numbers are anything written with digits. Names are words with a capital letter that do not start a sentence.

Missed Vosk small Whisper tiny Whisper base Whisper small
Numbers, English (of 56) 26 13 8 6
Numbers, French (of 62) 26 43 21 17
Names, English (of 434) 152 158 126 99
Names, French (of 240) 79 150 125 85

Whisper small misses few English numbers, but more than one French number in four. It misses about one English name in four, and one French name in three. Vosk misses almost half of the numbers in both languages. So the person who reads the note must check names and numbers first. That is where a wrong word costs the most: a wrong street, a wrong part number.

Some technicians switch language from one customer to the next. Inès also ran Whisper small without telling it the language. It chose the right language for all 492 recordings. The WER did not change (6.7% English, 13.6% French). Its lowest confidence was 0.828 on an English recording.

So detection works on these clean sentences. The app still keeps each technician’s language setting and sends it with the note, as Accents, noise and two languages recommends. A note that starts with a customer’s name, or with an English part number in a French note, could fool detection, and the setting costs nothing.

None of the four recognisers met the 10% target in French. On a first sample of 25 sentences per language (50 recordings each), Inès had also tried Whisper medium, a larger size (about 769 million parameters, against 244 million for small). On the same 50 French recordings, medium had 7.4% and small 14.1%. That gap is too large to ignore, and 50 recordings are too few to decide on. So she ran medium on all 230 French recordings:

French, all 230 recordings WER CER Female / male WER Gap, 95% interval RTF Peak memory
Whisper small 13.6% 5.0% 14.4% / 12.9% -0.7 to 3.8 0.13 1,149 MB
Whisper medium 7.4% 2.6% 8.2% / 6.6% 0.1 to 3.3 0.34 2,670 MB

What this tells us. Medium halves the French error: 7.4% against 13.6%, and it misses fewer French numbers (15 of 62, against 17 for small) and names (66 of 240, against 85). It is the only recogniser that meets the 10% target in French.

It also misses two targets. Its real-time factor on these 10-second clips is 0.34, above 0.25. Over the whole run, its peak memory grew to 2,670 MB, above 2 GB. On the first 50 recordings, it was 1,864 MB. Memory can grow during a long run, so measure it over a long run.

Women’s recordings still get a little more errors: the gap is 1.7 points, with an interval of 0.1 to 3.3. It is probably real, and it is under the 2-point target. Inès writes it down to check again on the technicians’ notes.

Inès writes the decision for Leila and Hugo in one table. The cost is the real-time factor × the server’s price per hour. The price is US$0.17 an hour for a CPU server with 4 vCPUs and 16 GB. It is an assumption from the course’s cost file, not a quote. The volume is the story’s assumption: 150 technicians × 5 notes × 40 seconds × 22 working days. That is about 183 audio hours a month, 60% English and 40% French.

English French
Recogniser Whisper small (int8, told the language) Whisper medium (int8, told the language)
WER / CER 6.7% / 2.7% 7.4% / 2.6%
Women / men (same sentences) 6.6% / 6.7% 8.2% / 6.6%
Real-time factor (10-second clips; shared laptop) 0.13 0.34
Peak memory 1,066 MB 2,670 MB
Cost per audio hour (assumption: US$0.17 per server hour; an upper bound, longer notes need less) US$0.023 US$0.058
Cost per month (110 and 73 audio hours) US$2.51 US$4.23
What a person still checks names and numbers in every note names and numbers in every note; small French words

Why not the others?

  • Vosk small is the lightest, but it misses about one word in four in English, and in French it serves women clearly worse.
  • Whisper base in English (11.1%) is just over the target, at a third of the compute of small. If the server ever becomes the limit, base is the first fallback for English.
  • Whisper small in French (13.6%) misses the target. It stays the French fallback if medium is too slow on the server.

Speed and memory. Medium misses two targets as measured, so Inès does not hide that. The speed check above shows the way out for speed: on 20 recordings joined into one file, medium had an RTF of 0.165, against 0.336 one by one. A 40-second note is closer to the joined case, so it should come back in under 10 seconds. Hugo will confirm this on the server. Memory is a real cost: about 2.7 GB for French and 1.1 GB for English, out of 16 GB. Leila agrees to change the memory target to 3 GB. She changes it openly, in writing, because accuracy in French matters more to her than 700 MB. If the server cannot spare it, Whisper small is the French fallback, with more review.

Compute is not the expensive part. A whole month costs a few dollars. One wrong address that sends a technician back costs more than a year of compute. So the money question is not “which model is cheapest?” but “how many words must a person fix?”.

A decision is only useful if it works on recordings that nobody used to make it. Inès kept 20 recordings aside from the FLEURS validation split. They have other sentences, with 5 women and 5 men per language. She did not look at them until the decision was written. The decision in code is one small function:

def transcribe_note(path, language):
"""The decision in code: one recording in, its text out, with the compute time."""
x, sr = sf.read(path, dtype="float32")
start = time.perf_counter()
segments, _ = model.transcribe(x, language=language, beam_size=1, condition_on_previous_text=False)
return " ".join(s.text.strip() for s in segments), time.perf_counter() - start

model is Whisper small for English and Whisper medium for French, loaded once at the start. The result on the 20 new recordings:

New recordings Recogniser WER Names missed
10 English Whisper small 4.8% 3 of 11
10 French Whisper medium 9.3% 3 of 9
10 French (for comparison) Whisper small 10.7% 5 of 9

One new French recording, as the technician would see it:

Text
Said Le tronçon entre Point Marion et Fairmont présente les conditions de conduite les plus difficiles sur l’autoroute Buffalo-Pittsburgh, passant fréquemment par des terrains isolés de l’arrière-pays.
Whisper medium Le tronçon entre Pont-Marion et Fermont présente les conditions de conduite les plus difficiles sur l'autoroute Buffalo-Pittsburgh passant fréquemment par des terrains isolés de l'arrière pays.
Whisper small Le transeau entre ponts, marion et ferme on présente les conditions de conduite les plus difficiles sur l'autoroute Pufalo-Pittsburg passant fréquemment par des terrains isolés de l'arrière-pays

What this tells us. On new recordings, the decision holds. English with Whisper small: 4.8%, close to the sample’s 6.7%. French with Whisper medium: 9.3%, close to the sample’s 7.4%, and better than small on the same recordings. With only 10 recordings per language, a few points either way are normal. The example shows where a person still looks: medium writes “Pont-Marion” and “Fermont” for “Point Marion” and “Fairmont”. The rest of the long sentence is right. In a technician’s note, those would be a street and a town: exactly what the technician must check before saving.

Real speech is harder than Larkfield’s synthetic calls

Section titled “Real speech is harder than Larkfield’s synthetic calls”

The course measured the same recognisers on Larkfield’s synthetic calls. The fair comparison is their wideband version, at 16,000 samples a second like FLEURS:

Two bar charts, English and French, with two bars per recogniser: WER on Larkfield's synthetic calls (wideband) and on FLEURS real read speech. English: Vosk 21.6 and 24.6, tiny 5.5 and 15.4, base 5.4 and 11.1, small 3.7 and 6.7. French: Vosk 22.4 and 18.8, tiny 31.9 and 44.4, base 20.7 and 28.4, small 12.4 and 13.6.
WER on Larkfield's synthetic calls (wideband audio, the course's test calls) and on FLEURS real read speech, by language. Different sentences and speakers: compare the pattern, not single points.
Text description of the diagram

Grey bars are Larkfield’s synthetic calls; blue bars are FLEURS. In English, every Whisper size makes about two to three times as many errors on real speech (small: 1.8 times): small 3.7% against 6.7%, tiny 5.5% against 15.4%. In French, Whisper is a little worse on real speech, and Vosk is better: 22.4% on the synthetic calls against 18.8% on FLEURS.

What this tells us. On Larkfield’s synthetic calls, English looked almost solved: 3.7% for Whisper small. Real English speakers, even reading clean sentences, raise the error by about two to three times (1.8 times for small). In French, the course already warned that the three synthetic French voices made French harder; with real French speakers, Vosk does much better than on those voices. The general lesson: a test set must sound like your users. Synthetic speech is good for building and checking a system. It is not good for choosing a model or for promising a number.

  • Read speech, not dictation. Technicians speak freely, with pauses, trade words and noise. Expect higher WERs on their notes.
  • No accents and no ages. FLEURS gives US English and France French. Canadian speakers are not tested here.
  • Small groups and no speaker IDs. Gaps under about 2 points are hard to see, and one person may have read several sentences, so the intervals are a little too narrow.
  • One shared machine. The order of the recognisers is reliable; the exact real-time factors are not.
  • The speakers are real people. FLEURS volunteers recorded their voices for research. Keep the attribution, do not try to identify anyone, and never use these voices to train a voice clone.
  • The gender label is for checking fairness only. Never build a tool that guesses a person’s gender from their voice.
  • The technicians’ recordings. The company asks for consent before any test recording, deletes the audio once the note is saved, and lets a technician type instead. The audio never goes to a hosted service.
  • A machine writes a draft; a person saves it. The app shows the note as a draft until the technician has checked it.
  • Customers’ personal data. Notes name customers and addresses. The rules of Personal data in voice apply to the saved text.
  • Watch the groups after launch. Inès checks the gender gap again on the technicians’ own notes, and every few months after that.

A case study like this shows an employer that you can choose a model with evidence, not with a feeling. Make it yours with one change. Try another language pair from FLEURS, another recogniser, or the phone-line version of the same recordings. Then make it easy to read in five minutes.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • results.csv: one row per recogniser, language and gender, with WER, CER, RTF and memory.
  • GUIDE.md: the normalisation guide you used, with the rule you added and why.
  • The data source and licence: FLEURS, CC BY 4.0, with the attribution text from the box above. Link the dataset; do not re-host its audio.

A README template:

# Choosing a speech recogniser for English and French dictation
## Question
Which local recogniser should a bilingual field-service app use for 40-second notes?
Targets: WER <= 10% per language, no clear gender gap > 2 points, RTF <= 0.25, memory <= 2 GB.
## Data and licence
FLEURS (Google, Conneau et al. 2022), CC BY 4.0, en_us and fr_fr test splits:
every sentence read by both a woman and a man (262 + 230 recordings).
## Approach
Vosk small (baseline) and Whisper tiny, base, small (medium on a sample), int8, 4 CPU threads,
told the language. One normalisation guide (+ a rule for big numbers). Scores by language
and gender; paired comparison and resampled 95% intervals; important words (numbers, names).
## Results
(your table: WER, CER, RTF, memory, gender gap per language)
## Decision and cost
(one recogniser per language, cost per audio hour on a CPU server, what a person checks)
## Limits
Read speech, not dictation; no accents or ages; no speaker IDs; one shared laptop.

How to describe it in an interview, in two sentences: “I compared five open speech recognisers on real English and French speech, by language and by speaker gender, with one normalisation guide; the small baseline beat two Whisper sizes in French. I recommended one recogniser per language with its cost per audio hour, and a plan to confirm it on the company’s own recordings.”

Go further:

  • Send the same recordings through the course’s phone-line code (to_phone in the project) and score again. Predict first which recogniser loses most.
  • Add whisper.cpp from Whisper sizes on a laptop as another row.
  • Give Whisper a vocabulary prompt of names, as in Help the model with your words, and count the names missed again.

Check your understanding

In French, Vosk small (18.8%) made fewer errors than Whisper base (28.4%) and Whisper tiny (44.4%). A colleague says: "Whisper is newer, so it is always better." What do you answer, and how should a team choose?

Show the answer

A newer model family is not better in every language and every size. Whisper’s small sizes learned from much more English than French. On these French recordings, the small Vosk model beat two of them. In English, every Whisper size beat Vosk. So the team measures every candidate on the same recordings, in each language that the product needs. It uses one guide and targets written down before the results. Then it chooses per language, and may choose different recognisers for different languages.

Check your understanding

Vosk small in French: women 20.3%, men 17.3%, 95% interval of the gap 0.5 to 5.2. Whisper small in French: 14.4% and 12.9%, interval -0.7 to 3.8. What can you conclude for each, and why did the sample keep only sentences that both a woman and a man read?

Show the answer

For Vosk, the interval does not include 0. On these recordings, it serves the women worse, by about 2.9 points. For Whisper small, the interval includes 0. This sample cannot tell whether there is a gap, so the team keeps watching it on its own recordings. Neither result says why: voices, microphones and rooms all differ. The sample keeps sentences read by both groups, so the comparison is paired. Both groups read the same words, so a gap cannot come from easier or harder sentences.

Check your understanding

Before any score, the analyst added one rule to the course's normalisation guide: number words with hundred or thousand become digits. Why was it needed, why must it apply to every recogniser, and when would you add such a rule?

Show the answer

The course’s guide was written for order numbers said digit by digit. FLEURS has big numbers. Vosk writes them as words (“ten thousand”), and Whisper writes digits. Without the rule, Vosk lost words that it heard correctly. The comparison was unfair to one recogniser. A rule must apply to the references and to every recogniser, or it becomes a way to favour one. Add a rule when a check on new data finds a difference in spelling, not in hearing. Write it down, and report the scores with and without it.

Recap

  • You checked the normalisation guide on new data first, and added one rule for big numbers that you applied to every recogniser.
  • In English, Whisper small met the targets: 6.7% WER, no gender gap. In French, the baseline Vosk beat Whisper tiny and base, and only Whisper medium came under 10% (7.4%).
  • The paired comparison used the same sentences for women and men. Vosk in French served women clearly worse. For Whisper small, the sample could not show a gap; Whisper medium leans the same way, under the 2-point target.
  • Short clips cost Whisper more compute per second, because it always reads a 30-second window. Compute is still only a few dollars a month; a person checking names and numbers is the real cost.
  • Real read speech had about two to three times the English error of Larkfield’s synthetic calls (small: 1.8 times). A test set must sound like your users.

Next, a different job on real speech: meeting recordings with several speakers, where you split the speakers before you transcribe.

Next: Meeting transcripts with speakers