Accents, noise and two languages
- Lesson 14
- 35 min
- Module 4: Speech recognition: compare the families
Where we are
The vocabulary helped on English calls. Larkfield also has French callers, accents and noisy lines. In this lesson, we see what each recogniser does with two languages, a switch of language in one sentence, and noise. First, think of a school report with one average mark.
Lesson 14 of 36 · Speech AI: Transcribe, Understand and SpeakAccents, noise and two languages
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Find out which callers a recogniser fails, not only how often it fails on average. Then you can see when a slice’s number is really about another slice, and decide when to detect the caller’s language before you recognise.
NotebookDo this lesson in a notebook
The notebook scores four recognisers by language, line, noise and accent. It splits the line and noise slices inside each language, checks Whisper’s language detection, and plays the French caller who switches to English. It downloads only small files and runs no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- WER by slice and the test set by slice, from The errors that matter. The machine learning course explains slices in general, in Slice analysis.
- The vocabulary prompt, from Help the model with your words.
What you need
- Your
larkspeechproject at the end of Module 4 (the same as the labs snapshotend-of-m04), with the Whisper small model downloaded. - Headphones or speakers for one clip, with its transcript on the page.
One average mark on a school report
Section titled “One average mark on a school report”A pupil’s report says: average mark 70%. It sounds fine. Then the parents look at the subjects: 90% in five subjects, and 20% in mathematics. The average was true, and it hid the one subject that needs help.
A WER is an average too. On Larkfield’s 60 test calls, Whisper small has a WER of 8.6%. That number mixes English and French callers, three phone lines, three kinds of background, and several voices. Each of those groups is a slice. A good average can hide a slice that fails.
The problem: which callers does the model fail?
Section titled “The problem: which callers does the model fail?”Larkfield’s callers are not all alike:
- Two languages. 34 of the test calls are in English and 26 in French.
- Accents. The English callers’ voices come from two groups: American English voices and British English voices.
- Three lines. Landline (G.711), mobile (Opus at 8 kbit/s) and internet phone (VoIP, G.726 at 16 kbit/s).
- Background. Quiet, other people talking (background talk), or street noise.
- Two languages in one call. French callers often say the English product names, and some say a whole sentence in English.
Remember that every call is synthetic: open text-to-speech voices read our scripts. The English calls use many voices. The French calls use only three voices. So the French numbers tell less about real French callers, and part of the French error comes from the voices themselves.
Slice by language
Section titled “Slice by language”The language is the largest difference on these calls, for every recogniser:
Text description of the diagram
Left: every recogniser makes more errors on French calls than on English calls. The gap is largest for Whisper tiny: 7.0% in English and 37.0% in French. Whisper small has 3.1% and 15.1%. Right: Whisper small by line. With all calls together, the VoIP line looks worst (12.2%) and the mobile line best (5.9%). Inside English only, the three lines are close: 3.1%, 3.0% and 3.5%. Inside French only, they are 14.3%, 17.1% and 18.4%, from 20, 2 and 4 calls. Each bar shows its number of calls in white.
With the 10% target of Whisper sizes on a laptop, every Whisper size passes in English, and no size passes in French. Whisper medium, on the first 20 test calls only, had 10.2% on its 11 French calls: close to the target, but still over it, and slow.
Is it the language, or the line?
Section titled “Is it the language, or the line?”Look again at the right chart. With all calls, the VoIP line seems to double the error: 12.2% against 5.9% on the mobile line. That looks like a fact about codecs. But look at who is on each line in the test calls:
| Line | English calls | French calls |
|---|---|---|
| Landline | 23 | 20 |
| Mobile | 8 | 2 |
| VoIP (G.726) | 3 | 4 |
The mobile calls are mostly English, and the VoIP calls are mostly French. French calls have more errors. So the “line” slice was mostly measuring the language. Inside English, the lines differ by less than 1 point, which is within noise. When two things change together, you cannot blame one of them until you split the slice inside the other. This mix of two causes is called confounding.
Noise: the same trap
Section titled “Noise: the same trap”The noise slices tell the same story. With all calls, quiet calls look harder than street calls:
| Background | All calls | English only | Calls (English / French) |
|---|---|---|---|
| Quiet | 9.6% | 2.3% | 15 / 15 |
| Background talk | 7.7% | 3.3% | 10 / 7 |
| Street | 7.4% | 4.3% | 9 / 4 |
Half of the quiet calls are French, but only 4 of the 13 street calls are. Inside English, noise does what you expect: quiet calls have the fewest errors. Even so, the effect is small next to the language. On these synthetic calls, noise is not the hard part. French is. Whisper tiny also has the fewest errors on quiet English calls: 5.4% quiet, 6.9% street, 9.4% with background talk.
Accents
Section titled “Accents”The English voices come from two groups. Whisper small has 2.6% on the British English voices (15 calls) and 3.5% on the American English voices (19 calls). One word error moves the English WER by about 0.025 points, so a gap of about 1 point is small. On this set, we cannot claim that one accent is harder. With only a few synthetic voices in each group, these slices also say little about real accents. The case study An IVR for spoken intents tests recognition on real speakers of several English varieties.
Why French is harder here
Section titled “Why French is harder here”Three reasons work together.
- The model. Whisper’s authors report that it had much more English than French in its training data. Smaller sizes suffer most: tiny goes from 7.0% to 37.0%.
- The phone line. French calls lose more on the phone line than English ones. On wideband audio, Whisper small has 12.4% on French calls, against 15.1% on the phone. For Vosk, the change is very large: 22.4% wideband against 39.4% on the phone.
- The voices. Only three synthetic voices read the French calls, and they do not always say every sound clearly. Piper’s French voices warned about missing nasal vowel sounds when the calls were made. “Un” (one) is a nasal vowel. In calls C-0009 and C-0074, Whisper small wrote “en” where the caller said “un”, at the start of an order number. That cost the first digit. We cannot tell how much of the French error comes from the voices: that is a limit of synthetic calls.
Two languages in one call
Section titled “Two languages in one call”Larkfield’s French callers say the English product names: “le Cedar raised bed”, “mon Grow Hub”. Some say a whole sentence in English. Switching language inside a call or a sentence is called code-switching. Listen to the start of call C-0009:
Transcript. “Oui, bonjour. Franchement, le Cedar raised bed ne vaut pas ce que j’espérais. Je veux le retourner et récupérer mon argent.” Then, in English: “Sorry, in English: the app says the device is offline.” (In English, the first part is: “Yes, hello. Honestly, the Cedar raised bed is not worth what I hoped. I want to return it and get my money back.”)
Whisper small was told that the call is French. This is what it wrote, with and without Larkfield’s vocabulary prompt:
| The caller said | Whisper small | Whisper small with the prompt | |
|---|---|---|---|
| Product name | le Cedar raised bed | le SEDAR, le BED | le Cedar raised bed |
| English sentence | Sorry, in English: the app says the device is offline. | Soil, en anglais. The appseid de D.V. is offline. | Soil, in English. The appsaid of the village is offline. |
Told “French”, the model writes French-looking words for English speech: “en anglais” for “in English”. The prompt fixed the product name, because it lists it. It did not fix the English sentence.
Across all French test calls, Whisper small missed 25 of the 26 product names without the prompt, and 7 with it. For a bilingual shop, the vocabulary helps most where the languages mix. On the 11 test calls that have a planted English sentence, the WER of Whisper small is 15.3%, against 14.9% on the other French calls. One English sentence is a small part of a call, so the average hides it again: look at the sentence, not at the call’s WER.
When to detect the language first
Section titled “When to detect the language first”Whisper can detect the language from the first 30 seconds of a call. As the last lesson showed, it found the right language on all 60 test calls. Vosk cannot detect a language: each Vosk model knows one language, so you must choose the model first.
Text description of the diagram
A call starts. If the language is known, because the caller chose it or the phone number is for one language, use it. If not, let Whisper detect it on the first 30 seconds. If the detection is sure enough, recognise with that language. If not, ask the caller: “English or Français?”, with a short prompt that expects those two words. Every path ends in the same step: recognise with that language and its vocabulary.
The rules:
- Recorded calls (batch): detection works well when each call stays in one language, as Larkfield’s calls do. Detection reads only the start of the call: a call that starts with a French greeting and continues in English is read as French.
- A phone line that answers callers: the first answer is short (“Français”), too short to detect. So the IVR asks, and keeps the choice for the rest of the call. Module 7 builds this.
- A sentence in the other language: neither detection nor a fixed language handles it well. The vocabulary prompt helps with names. For whole sentences, you need a recogniser trained on mixed speech, or you accept the errors and measure them.
Guided practice: slice the scores
Section titled “Guided practice: slice the scores”-
In your
larkspeechfolder, with the environment active, score Whisper small by line and by noise:Terminal window speech score --method whisper-small --by linespeech score --method whisper-small --by noisewhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wronglandline 43 5298 8.6% 4.0% 7 of 38 31 of 43mobile 10 1207 5.9% 3.1% 0 of 8 5 of 10voip_g726 7 829 12.2% 7.1% 1 of 6 5 of 7whisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongbabble 17 2151 7.7% 3.6% 0 of 16 15 of 17quiet 30 3593 9.6% 4.8% 8 of 25 18 of 30street 13 1590 7.4% 3.6% 0 of 11 8 of 13The
scorecommand slices one column at a time. To split a slice inside a language, use the notebook, section 3. -
Score by accent:
Terminal window speech score --method whisper-small --by accentwhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongen-GB 15 1760 2.6% 1.3% 0 of 11 8 of 15en-US 19 2228 3.5% 2.1% 2 of 17 8 of 19fr-FR 26 3346 15.1% 7.0% 6 of 24 25 of 26 -
Now run Whisper small on the first 5 French test calls, without and with the vocabulary prompt. This takes about a minute on a laptop.
Terminal window speech transcribe --model small --language fr --limit 5speech transcribe --model small --language fr --limit 5 --promptwhisper-small on 5 test calls (phone): WER 16.6%, CER 9.0%, order numbers wrong 1 of 4, real-time factor 0.067Wrote outputs/transcripts-whisper-small.jsonlwhisper-small-prompt on 5 test calls (phone): WER 11.6%, CER 5.1%, order numbers wrong 1 of 4, real-time factor 0.068Wrote outputs/transcripts-whisper-small-prompt.jsonlYour real-time factors will differ, and a word or two may change.
Your turn: Vosk on the wideband files
Section titled “Your turn: Vosk on the wideband files”Predict: Vosk small has 24.3% in English and 39.4% in French on the phone files. On the wideband files, which language gains more? Then run speech score --method vosk-small --by language --kind wideband.
Check your answer
vosk-small, test calls, wideband audio, normalisation fullslice calls words WER CER order nos wrong products wrongen 34 3988 21.6% 13.1% 10 of 28 22 of 34fr 26 3346 22.4% 11.4% 16 of 24 26 of 26French gains much more: from 39.4% to 22.4%, while English goes from 24.3% to 21.6%. Vosk’s small French model is hurt badly by the phone line. If Larkfield used Vosk, a wideband line would matter most for its French callers.
Common mistake
Blaming the line for the language
Symptom: Malik, the telephony engineer, reads the line slices: VoIP 12.2%, mobile 5.9%. He proposes to move all VoIP customers to another phone provider, at a cost.
Cause: the VoIP test calls are mostly French (4 of 7), and the mobile calls mostly English (8 of 10). French calls have more errors on every line. Inside English, the three lines are within 1 point of each other: 3.1%, 3% and 3.5%.
Fix: before you blame one slice, split it inside the other big slices, here the language. Check the number of calls in each cell: with 3 or 4 calls, a difference of a few points is noise. If a slice matters, collect more calls for it.
Consequence in AI work: a decision built on a confounded slice spends money on the wrong fix, and leaves the real problem (French recognition) as it was.
Check your understanding
All calls together: quiet 9.6%, street 7.4%. English only: quiet 2.3%, street 4.3%. Explain how both can be true.
Show the answer
The quiet slice holds 15 English and 15 French calls; the street slice holds 9 English and 4 French calls. French calls have many more errors. So the quiet slice has a larger share of French, and its overall WER is higher. Inside one language, street noise does add errors. The language confounds the noise slice: split by language first.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A French caller says "le Cedar raised bed" and then one sentence in English. Whisper small was told "French". What do you expect in the transcript, and what helps?
Show the answer
The product name may be written as French-looking words (“le SEDAR, le BED”), and the English sentence as French-looking text (“en anglais” for “in English”). The vocabulary prompt helps with the product names: Whisper small missed 25 of 26 French product names without it and 7 with it. It does not fix a whole English sentence; measure those errors, or use a recogniser trained on mixed speech.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Should Larkfield's batch job detect each call's language, or should the IVR ask the caller? Explain each choice.
Show the answer
Batch: detect. Each recorded call stays in one language. Whisper small detected the right language on all 60 test calls, with almost the same WER (8.7% against 8.6% when told). IVR: ask. The caller’s first answer is a word or two, too short to detect reliably. So the IVR asks “English or Français?” with a short prompt that expects those words. It keeps the choice for the rest of the call.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- An average WER hides slices: Whisper small has 3.1% on English calls and 15.1% on French calls.
- Two slices that change together confound each other. The VoIP line and quiet calls looked hard because they held more French calls. Inside English, line and noise moved the WER by about 2 points or less.
- French is harder here for three reasons: the model, the phone line (Vosk French: 39.4% phone, 22.4% wideband) and the three synthetic voices.
- Code-switching: told “French”, Whisper writes English speech as French-looking words. The vocabulary prompt fixes the product names (25 of 26 missed → 7), not whole sentences.
- Detect the language for recorded calls that stay in one language. Ask the caller on a phone line.
In this module, you compared recognisers, chose a size, helped it with Larkfield’s words and found which callers it fails. But a transcript still does not say who said each word: the agent or the caller. Next, you split each call by speaker, starting with the simplest way: the phone system’s two channels.
Next: Split by channel first