Choose a recogniser for two languages
- Case study 1
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we compared recognisers on Larkfield's synthetic calls. In this case study, we use real English and French speech from an open dataset, compare Vosk and Whisper sizes by speaker group, and recommend one.
Case study 1 of 4 · Speech AI: Transcribe, Understand and SpeakChoose a recogniser for two languages
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Choose a speech recogniser for a real product in two languages, with evidence that you can show to others. You will test five recognisers on real English and French speech. You will check the groups of speakers that an average hides. Then you will recommend one recogniser per language, with its error, speed and cost.
The notebook runs every step of this page, with the same code and the same output, plus two tasks with checks. It scores the recorded transcripts of all five recognisers, so it does not need hours of compute. At the end, it runs the chosen recogniser on 20 new recordings on your computer. Its setup cells download about 30 MB of files and the Whisper model (about 480 MB). It runs in about 3 minutes on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses the whole recognition part of the course. These lessons matter most:
- WER, CER and the alignment, from Word error rate, step by step.
- The normalisation guide, from Normalise before you compare.
- Important words and slices, from The errors that matter.
- Vosk and Whisper, from From sound to text, and the sizes, speed and memory, from Whisper sizes on a laptop.
- Confounded slices and language detection, from Accents, noise and two languages.
What you need on your computer (only for the local route; the notebook needs none of it)
- Python 3.12 in a new virtual environment, and the notebook, downloaded from the link above, in VS Code or JupyterLab. The notebook’s first cell installs its packages. Your
larkspeechproject works too; this case study does not change its files. - About 1 GB of free disk for the files and the Whisper model. The setup cell puts the files in a new folder
fleurs-recognisers. To start again, delete that folder.
Tested on macOS (Apple M4, 16 GB) with Python 3.12 and faster-whisper 1.2.1, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.
The situation
Section titled “The situation”Two Rivers Home Repairs fixes heating, plumbing and electrics in homes on both sides of the Ottawa River, in Ontario and Quebec. Its 150 technicians work in English or French, as each customer prefers. After each visit, a technician writes a short note: what they found, what they did, and what comes next.
Today they type the notes on a phone, in a van, between two visits. The notes are short, late and sometimes missing. Leila, the operations manager, wants dictation: the technician speaks for about 40 seconds, the app shows the text, and the technician checks it and saves it.
Leila has three conditions:
- Both languages must work. A French note must be as usable as an English one.
- The app must serve every technician equally. Women and men work in the field, and the app must not work worse for one group.
- The audio stays with the company. The notes name customers and their addresses. So the recogniser runs on the company’s own CPU server, not on a hosted service. Hugo, the developer, owns that server.
Inès, the company’s data analyst, must recommend a recogniser. She has no recordings of the technicians yet. Recording them needs their consent and a few weeks. So she starts with an open dataset of real speech, to choose the candidates and to build the test that she will later run on the technicians’ own notes.
What you will build
Section titled “What you will build”You will build a fair comparison of five recognisers: the same real recordings in English and French, one normalisation guide, one table. You will look at the results by language and by speaker gender, with the worst recordings. Then you will recommend a recogniser for each language. You will give its cost per audio hour on a CPU server, and say what a person must still check. Last, you will run the recommendation on new recordings, end to end.
Text description of the diagram
The case study has six steps. It starts with real speech: 262 English and 230 French recordings from FLEURS. Every recogniser gets the same audio: Vosk small as the baseline, and Whisper tiny, base and small, with Whisper medium on a sample. One normalisation guide prepares every transcript. The scores are split by language and by gender. The decision names one recogniser per language and its cost. The last step runs the decision on 20 new recordings that nobody looked at before.
The data
Section titled “The data”FLEURS is an open dataset from Google. Volunteers read sentences aloud in 102 languages. The sentences come from Wikipedia articles, and translators put them into every language. So English and French have the same sentences. Each recording has its text and the speaker’s gender as FLEURS records it.
Inès also looked at Mozilla Common Voice: many speakers, CC0 data, but its downloads now need an account on Mozilla’s own platform. It is a good option for a later test, not for a notebook that every learner must run.
What one row means
Section titled “What one row means”One row is one recording: one volunteer reading one sentence, recorded at 16,000 samples a second. These are the columns the case study uses:
| Column | What it means |
|---|---|
language |
en (English, United States) or fr (French, France) |
sentence_id |
The sentence. The same number means the same sentence in both languages. |
gender |
female or male, as FLEURS records it |
seconds |
The length of the recording |
text |
What the speaker read: the reference |
First checks: the same sentences for both groups
Section titled “First checks: the same sentences for both groups”In the test split, some sentences were read by a woman and a man, and others by two women or two men. If women and men read different sentences, a gap in their scores could come from the sentences, not from the voices. So Inès keeps only the sentences that both a woman and a man read, with one recording of each:
Counter({('en', 'female'): 131, ('en', 'male'): 131, ('fr', 'female'): 115, ('fr', 'male'): 115})en 262 recordings, 131 sentences, every sentence read twice: True, 44.2 minutesfr 230 recordings, 115 sentences, every sentence read twice: True, 38.6 minutesWhat this tells us. The sample is 262 English and 230 French recordings, about 10 seconds each. Each group reads the same sentences. With about 5,810 English and 6,052 French reference words, one word error moves the WER of a language by about 0.017 and 0.017 points. Inside one gender, it moves it about twice as much.
Listen to one sentence, read by a woman and by a man in English, and by a man in French:
Transcript (both English clips): “The scenes are displayed on the pyramids and the different pyramids are lit up.”
Transcript (French clip): “Les scènes sont affichées sur les pyramides et les différentes pyramides sont éclairées.” (In English: the same sentence as above.)
These are real people, not synthetic voices. You hear breath, a little room sound, and each speaker’s own pace.
Frame the problem
Section titled “Frame the problem”Before she runs anything, Inès writes down what she measures and what “good enough” means. If she chose the targets after the results, she could pick the targets that make her favourite win.
- The unit: one recording, one speaker, one sentence.
- The main measure: WER after one normalisation guide, with the errors summed over all recordings and then divided. CER as a second measure, because French has many short words that a wrong letter changes.
- Speed and memory: the real-time factor (RTF: compute time ÷ audio time) on 4 CPU threads, and the peak memory of the run.
- The groups: language × gender. FLEURS gives no accent and no age, so we cannot check those groups here.
- What a person checks: the technician reads every note before saving it.
Leila and Inès agree on these targets (a team decision, written before the results):
| Target | Value | Why |
|---|---|---|
| WER per language | at most 10% | about one word in ten to fix in a note |
| Gender gap | no clear gap larger than 2 points | Leila’s second condition |
| Real-time factor | at most 0.25 | a 40-second note is back in 10 seconds |
| Peak memory | at most 2 GB | the server also runs the app |
What FLEURS cannot tell her. The volunteers read clean sentences in a quiet room. Technicians speak freely, in a van or a basement, with trade words such as “backflow valve” or “chauffe-eau”. So FLEURS chooses the candidates and builds the method. It does not prove the final numbers. Inès will run the same test on 100 consented notes before the app goes live.
Check the guide before you compare
Section titled “Check the guide before you compare”The course’s normalisation guide was written for Larkfield’s calls. There, order numbers are said digit by digit, so the guide joins digits (“7 4 2” → “742”). FLEURS sentences have big numbers and years. Vosk writes numbers as words; Whisper mostly writes digits. Look at one recording:
said: Goats seem to have been first domesticated roughly 10,000 years ago in the Zagros Mountains of Iran.Vosk wrote: good seem to have been first domesticated roughly ten thousand years ago and as are girls mountains of irancourse's guide: ... roughly 10000 years ago ... | ... roughly 10 thousand years ago ...+ big_numbers: ... roughly 10000 years ago ... | ... roughly 10000 years ago ...What this tells us. With the course’s guide alone, “ten thousand” becomes “10 thousand”, and Vosk loses two words although it heard the number right. A guide that does not fit the data punishes one recogniser more than another. So Inès adds one rule for this data, big_numbers: a number word with “hundred”, “thousand” or “million” (French “cent”, “mille”, “million”) becomes digits. She applies it to every recogniser and to the references, before any score.
| WER after… | Vosk small, English | Vosk small, French | Whisper small, English | Whisper small, French |
|---|---|---|---|---|
| no guide (split on spaces) | 41.9% | 35.9% | 12.9% | 21.2% |
| the course’s guide | 25.2% | 19.6% | 6.8% | 13.6% |
the guide + big_numbers |
24.6% | 18.8% | 6.7% | 13.6% |
The new rule moves Vosk by under 1 point, and Whisper by almost nothing. That is small, and it is still worth doing: the gaps between recognisers must come from what they heard, not from how they write numbers. The guide matters much more than the rule: without any guide, every WER is about twice as high.
Two other spelling questions remain, and Inès leaves them as errors. The English references use British spelling (“travellers”) and Whisper writes American spelling (“travelers”). And the guide removes “mm” as a filler, so “35 mm” (millimetres) loses its unit in both texts. Both are rare here. Write such rules down when you find them, and apply them to every recogniser or to none.
A baseline: Vosk small
Section titled “A baseline: Vosk small”The baseline is the simple method that every other method must beat. Here it is Vosk small: the older pipeline from From sound to text, about 40 MB per language, one model for each language.
| Vosk small | WER | CER | RTF | Peak memory |
|---|---|---|---|---|
| English | 24.6% | 12.8% | 0.074 | 252 MB |
| French | 18.8% | 8.0% | 0.13 | 298 MB |
What this tells us. Vosk small makes about one error in four words in English, and about one in five in French. It is light: under 300 MB of memory. On Larkfield’s synthetic phone calls, Vosk had 24.4% in English and 39.4% in French. Here French is the easier language for Vosk. Keep that in mind: on real speech, the ranking of the languages can change.
The alternatives: three Whisper sizes
Section titled “The alternatives: three Whisper sizes”The alternatives are Whisper tiny, base and small through faster-whisper, with 8-bit weights, as in Whisper sizes on a laptop. Each one is told the language. Every recogniser gets the same 492 recordings, on the same laptop (Apple M4, 4 CPU threads), and the same guide. Running all of them took about 25 minutes. The notebook does not run them again: it scores the recorded transcripts, so its numbers are exactly the ones on this page.
A fair comparison under one measure
Section titled “A fair comparison under one measure”Here is the table. Every row is the same recordings and the same guide.
| Recogniser | English WER | French WER | English CER | French CER | RTF (en / fr) | Peak memory (en / fr) |
|---|---|---|---|---|---|---|
| Vosk small (baseline) | 24.6% | 18.8% | 12.8% | 8.0% | 0.074 / 0.13 | 252 / 298 MB |
| Whisper tiny | 15.4% | 44.4% | 6.7% | 19.8% | 0.019 / 0.028 | 545 / 438 MB |
| Whisper base | 11.1% | 28.4% | 4.8% | 11.8% | 0.043 / 0.046 | 626 / 537 MB |
| Whisper small | 6.7% | 13.6% | 2.7% | 5.0% | 0.13 / 0.13 | 1,066 / 1,149 MB |
Text description of the diagram
Each point is one recogniser. Lower is more accurate, and further left is faster. In English, Whisper small is under 10% WER, at 6.7%; base is just over, at 11.1%. In French, Vosk and the three smaller Whisper sizes are all above 10%. Whisper small is best of them at 13.6%, then Vosk at 18.8%, and Whisper base (28.4%) and tiny (44.4%) are worse than Vosk. Whisper medium is the most accurate in both languages, and the slowest: the next section looks at it.
What this tells us. Three things.
- In English, Whisper small meets the error target, and base almost does. Small makes about one error in fifteen words.
- In French, the baseline beats two Whisper sizes. Vosk small (18.8%) is better than Whisper base (28.4%) and much better than tiny (44.4%). Whisper’s small sizes saw much more English than French when they learned. A bigger, newer model is not always better in every language: measure it.
- Whisper is slower here than on Larkfield’s calls. Small needs an RTF of about 0.13, against 0.075 on the phone calls. One reason is the length: Whisper always reads a 30-second window, so a 10-second recording pays for 30 seconds of work. The check below measures it.
Is short audio slower?
Section titled “Is short audio slower?”Inès runs one quick check. She takes the first 20 English recordings and runs Whisper on them one by one. Then she joins the same 20 into one long file and runs it once.
| Whisper size | One by one (about 9.1 s each) | Joined into one file |
|---|---|---|
| tiny | RTF 0.017 | RTF 0.011 |
| small | RTF 0.11 | RTF 0.061 |
| medium | RTF 0.336 | RTF 0.165 |
What this tells us. The same audio costs about half the compute when it comes in one long piece. Technicians’ notes are about 40 seconds, longer than these sentences, so the real RTF will be lower than in the table. The table’s speeds are also from a laptop that was running other work at the same time. Measure speed on the server you will use, with audio of the real length.
The groups that an average hides
Section titled “The groups that an average hides”A new tool: the paired comparison
Section titled “A new tool: the paired comparison”Two runners race on different days: one in the rain, one in the sun. Who is faster? You cannot tell, because the weather changed too. Make them run the same track on the same day, and the comparison is fair.
Scores by group have the same problem. If women read harder sentences than men, a gap in WER could come from the sentences. Our sample fixes that: every sentence was read once by a woman and once by a man. So we can compare inside each sentence, and count who got more errors on the same words. This is a paired comparison.
Text description of the diagram
One sentence is read twice: once by a woman, once by a man. Both recordings go through the same recogniser and the same guide. We compare their word errors: the woman’s recording had more, fewer or the same. We count this over all sentences. Because the words are the same, a difference comes from the voices and the recordings, not from the sentences.
A tiny example, with constructed numbers for three sentences:
| Sentence | Errors, woman | Errors, man | Who had more? |
|---|---|---|---|
| A | 4 | 2 | the woman |
| B | 1 | 1 | a tie |
| C | 0 | 3 | the man |
Over three sentences, that is one each and one tie: no sign of a gap. Over 115 sentences, a real gap shows as a clear imbalance. To put a number on “clear”, we resample the sentences. We draw 115 sentences at random, with repeats allowed, and compute the gap. We do this 2,000 times. The middle 95% of those gaps is the 95% interval. If it includes 0, we cannot say which group the recogniser serves worse.
The gender gap for each recogniser
Section titled “The gender gap for each recogniser”Text description of the diagram
Each pair of bars is one recogniser: the female speakers’ WER and the male speakers’ WER, on the same sentences. In English, the pairs are close for every recogniser. In French, the female bar is higher for Vosk (20.3% against 17.3%), base (30.1% against 26.7%) small (14.4% against 12.9%) and medium (8.2% against 6.6%), and lower for tiny.
| Recogniser | Language | Gap (female − male) | 95% interval | Sentences: woman worse / man worse / tie |
|---|---|---|---|---|
| Vosk small | English | 0.5 | -2.7 to 3.5 | 50 / 63 / 18 |
| Vosk small | French | 2.9 | 0.5 to 5.2 | 62 / 36 / 17 |
| Whisper base | French | 3.5 | 0.1 to 7.1 | 58 / 47 / 10 |
| Whisper small | English | -0.1 | -1.5 to 1.3 | 38 / 43 / 50 |
| Whisper small | French | 1.5 | -0.7 to 3.8 | 54 / 43 / 18 |
| Whisper medium | French | 1.7 | 0.1 to 3.3 | 49 / 31 / 35 |
What this tells us. In English, Whisper small serves women and men equally: the gap is -0.1 points, and in 50 of 131 sentences both recordings had the same number of errors. In French, Vosk small has a clear gap: women’s recordings got more errors in 62 sentences and men’s in 36, and the interval does not include 0. Whisper small in French leans the same way (1.5 points), but its interval includes 0, so this sample cannot tell. That is a reason to watch, not a proof of fairness.
Be careful with what a gap means. It says this recogniser made more errors on these volunteers’ recordings. It does not say why: the voices, the microphones and the rooms all differ between speakers. FLEURS has no speaker IDs, so one person may have read several sentences. And FLEURS records gender in two values only. Use the label to compare groups, never to guess anyone’s gender from their voice.
The worst recordings and the words that matter
Section titled “The worst recordings and the words that matter”An average hides the recordings that fail. Here are the worst French recordings of Whisper small (at least 8 words):
| Said | Whisper small wrote | WER |
|---|---|---|
| Tout dans l’Univers est fait de matière. Toute la matière est faite de minuscules particules appelées atomes. | Tout est dans l'univers et fait de matière. Toute la matière est fait de minus que le particule a plié à tourne. | 61.1% |
| Sundarbans est une réserve naturelle depuis 1966, vous pourrez y trouver environ 400 tigres du Bengale royal et 30 000 cerfs tachetés. | Sune d'Arban est une réserve naturelle depuis 1966. Vous pourrez trouver en faisant 4 centigres du Mengal Royal et 30 000 serres à râtocheter. | 57.1% |
What this tells us. The worst errors are names (“Sundarbans”, “Bengale”) and rare words (“minuscules particules appelées atomes”). Small French words such as “de”, “l’” and “est” also go wrong. A dictation note has names too: customers, streets, parts. Inès counts two kinds of important words, as in The errors that matter. Numbers are anything written with digits. Names are words with a capital letter that do not start a sentence.
| Missed | Vosk small | Whisper tiny | Whisper base | Whisper small |
|---|---|---|---|---|
| Numbers, English (of 56) | 26 | 13 | 8 | 6 |
| Numbers, French (of 62) | 26 | 43 | 21 | 17 |
| Names, English (of 434) | 152 | 158 | 126 | 99 |
| Names, French (of 240) | 79 | 150 | 125 | 85 |
Whisper small misses few English numbers, but more than one French number in four. It misses about one English name in four, and one French name in three. Vosk misses almost half of the numbers in both languages. So the person who reads the note must check names and numbers first. That is where a wrong word costs the most: a wrong street, a wrong part number.
Can Whisper find the language by itself?
Section titled “Can Whisper find the language by itself?”Some technicians switch language from one customer to the next. Inès also ran Whisper small without telling it the language. It chose the right language for all 492 recordings. The WER did not change (6.7% English, 13.6% French). Its lowest confidence was 0.828 on an English recording.
So detection works on these clean sentences. The app still keeps each technician’s language setting and sends it with the note, as Accents, noise and two languages recommends. A note that starts with a customer’s name, or with an English part number in a French note, could fool detection, and the setting costs nothing.
A larger size for French
Section titled “A larger size for French”None of the four recognisers met the 10% target in French. On a first sample of 25 sentences per language (50 recordings each), Inès had also tried Whisper medium, a larger size (about 769 million parameters, against 244 million for small). On the same 50 French recordings, medium had 7.4% and small 14.1%. That gap is too large to ignore, and 50 recordings are too few to decide on. So she ran medium on all 230 French recordings:
| French, all 230 recordings | WER | CER | Female / male WER | Gap, 95% interval | RTF | Peak memory |
|---|---|---|---|---|---|---|
| Whisper small | 13.6% | 5.0% | 14.4% / 12.9% | -0.7 to 3.8 | 0.13 | 1,149 MB |
| Whisper medium | 7.4% | 2.6% | 8.2% / 6.6% | 0.1 to 3.3 | 0.34 | 2,670 MB |
What this tells us. Medium halves the French error: 7.4% against 13.6%, and it misses fewer French numbers (15 of 62, against 17 for small) and names (66 of 240, against 85). It is the only recogniser that meets the 10% target in French.
It also misses two targets. Its real-time factor on these 10-second clips is 0.34, above 0.25. Over the whole run, its peak memory grew to 2,670 MB, above 2 GB. On the first 50 recordings, it was 1,864 MB. Memory can grow during a long run, so measure it over a long run.
Women’s recordings still get a little more errors: the gap is 1.7 points, with an interval of 0.1 to 3.3. It is probably real, and it is under the 2-point target. Inès writes it down to check again on the technicians’ notes.
The recommendation
Section titled “The recommendation”Inès writes the decision for Leila and Hugo in one table. The cost is the real-time factor × the server’s price per hour. The price is US$0.17 an hour for a CPU server with 4 vCPUs and 16 GB. It is an assumption from the course’s cost file, not a quote. The volume is the story’s assumption: 150 technicians × 5 notes × 40 seconds × 22 working days. That is about 183 audio hours a month, 60% English and 40% French.
| English | French | |
|---|---|---|
| Recogniser | Whisper small (int8, told the language) | Whisper medium (int8, told the language) |
| WER / CER | 6.7% / 2.7% | 7.4% / 2.6% |
| Women / men (same sentences) | 6.6% / 6.7% | 8.2% / 6.6% |
| Real-time factor (10-second clips; shared laptop) | 0.13 | 0.34 |
| Peak memory | 1,066 MB | 2,670 MB |
| Cost per audio hour (assumption: US$0.17 per server hour; an upper bound, longer notes need less) | US$0.023 | US$0.058 |
| Cost per month (110 and 73 audio hours) | US$2.51 | US$4.23 |
| What a person still checks | names and numbers in every note | names and numbers in every note; small French words |
Why not the others?
- Vosk small is the lightest, but it misses about one word in four in English, and in French it serves women clearly worse.
- Whisper base in English (11.1%) is just over the target, at a third of the compute of small. If the server ever becomes the limit, base is the first fallback for English.
- Whisper small in French (13.6%) misses the target. It stays the French fallback if medium is too slow on the server.
Speed and memory. Medium misses two targets as measured, so Inès does not hide that. The speed check above shows the way out for speed: on 20 recordings joined into one file, medium had an RTF of 0.165, against 0.336 one by one. A 40-second note is closer to the joined case, so it should come back in under 10 seconds. Hugo will confirm this on the server. Memory is a real cost: about 2.7 GB for French and 1.1 GB for English, out of 16 GB. Leila agrees to change the memory target to 3 GB. She changes it openly, in writing, because accuracy in French matters more to her than 700 MB. If the server cannot spare it, Whisper small is the French fallback, with more review.
Compute is not the expensive part. A whole month costs a few dollars. One wrong address that sends a technician back costs more than a year of compute. So the money question is not “which model is cheapest?” but “how many words must a person fix?”.
The decision on new input, end to end
Section titled “The decision on new input, end to end”A decision is only useful if it works on recordings that nobody used to make it. Inès kept 20 recordings aside from the FLEURS validation split. They have other sentences, with 5 women and 5 men per language. She did not look at them until the decision was written. The decision in code is one small function:
def transcribe_note(path, language): """The decision in code: one recording in, its text out, with the compute time.""" x, sr = sf.read(path, dtype="float32") start = time.perf_counter() segments, _ = model.transcribe(x, language=language, beam_size=1, condition_on_previous_text=False) return " ".join(s.text.strip() for s in segments), time.perf_counter() - startmodel is Whisper small for English and Whisper medium for French, loaded once at the start. The result on the 20 new recordings:
| New recordings | Recogniser | WER | Names missed |
|---|---|---|---|
| 10 English | Whisper small | 4.8% | 3 of 11 |
| 10 French | Whisper medium | 9.3% | 3 of 9 |
| 10 French (for comparison) | Whisper small | 10.7% | 5 of 9 |
One new French recording, as the technician would see it:
| Text | |
|---|---|
| Said | Le tronçon entre Point Marion et Fairmont présente les conditions de conduite les plus difficiles sur l’autoroute Buffalo-Pittsburgh, passant fréquemment par des terrains isolés de l’arrière-pays. |
| Whisper medium | Le tronçon entre Pont-Marion et Fermont présente les conditions de conduite les plus difficiles sur l'autoroute Buffalo-Pittsburgh passant fréquemment par des terrains isolés de l'arrière pays. |
| Whisper small | Le transeau entre ponts, marion et ferme on présente les conditions de conduite les plus difficiles sur l'autoroute Pufalo-Pittsburg passant fréquemment par des terrains isolés de l'arrière-pays |
What this tells us. On new recordings, the decision holds. English with Whisper small: 4.8%, close to the sample’s 6.7%. French with Whisper medium: 9.3%, close to the sample’s 7.4%, and better than small on the same recordings. With only 10 recordings per language, a few points either way are normal. The example shows where a person still looks: medium writes “Pont-Marion” and “Fermont” for “Point Marion” and “Fairmont”. The rest of the long sentence is right. In a technician’s note, those would be a street and a town: exactly what the technician must check before saving.
Insights and limits
Section titled “Insights and limits”Real speech is harder than Larkfield’s synthetic calls
Section titled “Real speech is harder than Larkfield’s synthetic calls”The course measured the same recognisers on Larkfield’s synthetic calls. The fair comparison is their wideband version, at 16,000 samples a second like FLEURS:
Text description of the diagram
Grey bars are Larkfield’s synthetic calls; blue bars are FLEURS. In English, every Whisper size makes about two to three times as many errors on real speech (small: 1.8 times): small 3.7% against 6.7%, tiny 5.5% against 15.4%. In French, Whisper is a little worse on real speech, and Vosk is better: 22.4% on the synthetic calls against 18.8% on FLEURS.
What this tells us. On Larkfield’s synthetic calls, English looked almost solved: 3.7% for Whisper small. Real English speakers, even reading clean sentences, raise the error by about two to three times (1.8 times for small). In French, the course already warned that the three synthetic French voices made French harder; with real French speakers, Vosk does much better than on those voices. The general lesson: a test set must sound like your users. Synthetic speech is good for building and checking a system. It is not good for choosing a model or for promising a number.
Limits of this study
Section titled “Limits of this study”- Read speech, not dictation. Technicians speak freely, with pauses, trade words and noise. Expect higher WERs on their notes.
- No accents and no ages. FLEURS gives US English and France French. Canadian speakers are not tested here.
- Small groups and no speaker IDs. Gaps under about 2 points are hard to see, and one person may have read several sentences, so the intervals are a little too narrow.
- One shared machine. The order of the recognisers is reliable; the exact real-time factors are not.
Responsible use
Section titled “Responsible use”- The speakers are real people. FLEURS volunteers recorded their voices for research. Keep the attribution, do not try to identify anyone, and never use these voices to train a voice clone.
- The gender label is for checking fairness only. Never build a tool that guesses a person’s gender from their voice.
- The technicians’ recordings. The company asks for consent before any test recording, deletes the audio once the note is saved, and lets a technician type instead. The audio never goes to a hosted service.
- A machine writes a draft; a person saves it. The app shows the note as a draft until the technician has checked it.
- Customers’ personal data. Notes name customers and addresses. The rules of Personal data in voice apply to the saved text.
- Watch the groups after launch. Inès checks the gender gap again on the technicians’ own notes, and every few months after that.
Make it a portfolio piece
Section titled “Make it a portfolio piece”A case study like this shows an employer that you can choose a model with evidence, not with a feeling. Make it yours with one change. Try another language pair from FLEURS, another recogniser, or the phone-line version of the same recordings. Then make it easy to read in five minutes.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
results.csv: one row per recogniser, language and gender, with WER, CER, RTF and memory.GUIDE.md: the normalisation guide you used, with the rule you added and why.- The data source and licence: FLEURS, CC BY 4.0, with the attribution text from the box above. Link the dataset; do not re-host its audio.
A README template:
# Choosing a speech recogniser for English and French dictation
## QuestionWhich local recogniser should a bilingual field-service app use for 40-second notes?Targets: WER <= 10% per language, no clear gender gap > 2 points, RTF <= 0.25, memory <= 2 GB.
## Data and licenceFLEURS (Google, Conneau et al. 2022), CC BY 4.0, en_us and fr_fr test splits:every sentence read by both a woman and a man (262 + 230 recordings).
## ApproachVosk small (baseline) and Whisper tiny, base, small (medium on a sample), int8, 4 CPU threads,told the language. One normalisation guide (+ a rule for big numbers). Scores by languageand gender; paired comparison and resampled 95% intervals; important words (numbers, names).
## Results(your table: WER, CER, RTF, memory, gender gap per language)
## Decision and cost(one recogniser per language, cost per audio hour on a CPU server, what a person checks)
## LimitsRead speech, not dictation; no accents or ages; no speaker IDs; one shared laptop.How to describe it in an interview, in two sentences: “I compared five open speech recognisers on real English and French speech, by language and by speaker gender, with one normalisation guide; the small baseline beat two Whisper sizes in French. I recommended one recogniser per language with its cost per audio hour, and a plan to confirm it on the company’s own recordings.”
Go further:
- Send the same recordings through the course’s phone-line code (
to_phonein the project) and score again. Predict first which recogniser loses most. - Add whisper.cpp from Whisper sizes on a laptop as another row.
- Give Whisper a vocabulary prompt of names, as in Help the model with your words, and count the names missed again.
Knowledge check
Section titled “Knowledge check”Check your understanding
In French, Vosk small (18.8%) made fewer errors than Whisper base (28.4%) and Whisper tiny (44.4%). A colleague says: "Whisper is newer, so it is always better." What do you answer, and how should a team choose?
Show the answer
A newer model family is not better in every language and every size. Whisper’s small sizes learned from much more English than French. On these French recordings, the small Vosk model beat two of them. In English, every Whisper size beat Vosk. So the team measures every candidate on the same recordings, in each language that the product needs. It uses one guide and targets written down before the results. Then it chooses per language, and may choose different recognisers for different languages.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Vosk small in French: women 20.3%, men 17.3%, 95% interval of the gap 0.5 to 5.2. Whisper small in French: 14.4% and 12.9%, interval -0.7 to 3.8. What can you conclude for each, and why did the sample keep only sentences that both a woman and a man read?
Show the answer
For Vosk, the interval does not include 0. On these recordings, it serves the women worse, by about 2.9 points. For Whisper small, the interval includes 0. This sample cannot tell whether there is a gap, so the team keeps watching it on its own recordings. Neither result says why: voices, microphones and rooms all differ. The sample keeps sentences read by both groups, so the comparison is paired. Both groups read the same words, so a gap cannot come from easier or harder sentences.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Before any score, the analyst added one rule to the course's normalisation guide: number words with hundred or thousand become digits. Why was it needed, why must it apply to every recogniser, and when would you add such a rule?
Show the answer
The course’s guide was written for order numbers said digit by digit. FLEURS has big numbers. Vosk writes them as words (“ten thousand”), and Whisper writes digits. Without the rule, Vosk lost words that it heard correctly. The comparison was unfair to one recogniser. A rule must apply to the references and to every recogniser, or it becomes a way to favour one. Add a rule when a check on new data finds a difference in spelling, not in hearing. Write it down, and report the scores with and without it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You checked the normalisation guide on new data first, and added one rule for big numbers that you applied to every recogniser.
- In English, Whisper small met the targets: 6.7% WER, no gender gap. In French, the baseline Vosk beat Whisper tiny and base, and only Whisper medium came under 10% (7.4%).
- The paired comparison used the same sentences for women and men. Vosk in French served women clearly worse. For Whisper small, the sample could not show a gap; Whisper medium leans the same way, under the 2-point target.
- Short clips cost Whisper more compute per second, because it always reads a 30-second window. Compute is still only a few dollars a month; a person checking names and numbers is the real cost.
- Real read speech had about two to three times the English error of Larkfield’s synthetic calls (small: 1.8 times). A test set must sound like your users.
Next, a different job on real speech: meeting recordings with several speakers, where you split the speakers before you transcribe.