Help the model with your words
- Lesson 13
- 35 min
- Module 4: Speech recognition: compare the families
Where we are
Our recognisers make the same mistakes again and again: product names and order numbers. They do not know Larkfield's words. In this lesson, we give the model our vocabulary and fix spoken order numbers with rules. First, think of how you help a new colleague on their first day.
Lesson 13 of 36 · Speech AI: Transcribe, Understand and SpeakHelp the model with your words
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Make the recogniser write Larkfield’s own words: its product names and its order numbers. A wrong product name breaks a search, and a wrong order number gives a caller someone else’s order.
NotebookDo this lesson in a notebook
The notebook measures the vocabulary prompt on the recorded transcripts of Whisper base and small, and finds the calls where it hurts. Then you write the check-digit rule and count which check catches each wrong order number. It downloads only small text files and runs no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- How Whisper’s decoder writes what is likely, from From sound to text.
- The important words (order numbers, product names, dates, negations), from Count the important words apart in The errors that matter.
What you need
- Your
larkspeechfolder with thetranscribecommand from Whisper sizes on a laptop, and the labs folder next to it. - The Whisper base and small models, which that lesson downloaded.
- Tested on macOS (Apple M4) with Python 3.12. The Windows commands are given but were not run.
Help a new colleague on the first day
Section titled “Help a new colleague on the first day”A new agent starts at Larkfield’s help desk. Before her first call, her team lead gives her a card with Larkfield’s words: “We sell the Grow Hub, the Solara lantern, the Cedar raised bed … Order numbers have six digits.” She has not learned anything new about English. But now, when a caller says “my Fern wall planter”, she knows that name, and she writes it right.
The team lead also tells her: “Read the order number back to the caller, and type it into the order system. If the system does not know it, ask again.” That second rule does not make her hearing better. It catches her mistakes before they cost anything.
This lesson does the same two things for a recogniser: give it a card of Larkfield’s words before it listens, and check its numbers after it listens.
The problem: the same words, wrong every time
Section titled “The problem: the same words, wrong every time”In the last lessons, the same mistakes came back on many calls. On the 60 test calls, Whisper small missed 41 of the 60 product names and 8 of the 52 order numbers. Its overall WER is only 8.6%. These few words matter more than all the others: a product name tells Larkfield what the call is about, and an order number is the key to the order.
Look at one caller. In call C-0077, the caller asks about an Aqua Mist sprinkler:
Transcript. Caller: “Hello there. I’m excited to get started with my Aqua Mist sprinkler, but it hasn’t arrived yet. Could you check where it’s up to?”
Whisper base wrote “my aqua-miss sprinkler”. The sound is clear. The model simply does not know the name, so it wrote a likely spelling.
Idea 1: tell the model the language
Section titled “Idea 1: tell the model the language”The simplest help is the language setting. Whisper can detect the language from the first 30 seconds of a call, or you can tell it. On the test calls, Whisper small found the right language on all 60 calls, and its WER was 8.7% against 8.6% when told. So on Larkfield’s calls, which stay in one language, the setting changes little.
It still matters for two reasons. First, the IVR in Module 7 hears only a few words at a time, such as “yes”. A few words are not enough to detect a language. Second, a wrong language gives a transcript in the wrong language, and all its words count as errors. When you know the language, from the caller’s choice or the phone number they called, tell the model.
Idea 2: a prompt with Larkfield’s words
Section titled “Idea 2: a prompt with Larkfield’s words”Whisper’s decoder writes each token after looking at the tokens before it. Normally, the “tokens before” are only the call’s own words. An initial prompt is text that you put in front, as if the model had just written it. The decoder then expects words like those in the prompt.
The course project has one prompt for each language, in larkspeech/asr.py:
VOCABULARY_PROMPT = { "en": "Larkfield help desk. Grow Hub, Solara lantern, Cedar raised bed, Terracotta Trio, Mistral fan, " "Willow hammock, Ember fire pit, Aqua Mist sprinkler, Fern wall planter, Harbor bench, Pebble path lights, " "Juniper compost bin. Order 742158.", ...}It names the shop, lists the 12 products, and shows an order number written as digits. The --prompt option of speech transcribe passes it to Whisper.
Other tools have other forms of the same idea. Hosted services often take a phrase list (a list of words to prefer). Vosk can take a list of the only phrases it may write. faster-whisper also has a hotwords option, which this course did not measure.
What the prompt did on the test calls
Section titled “What the prompt did on the test calls”Text description of the diagram
Left: the product names missed, of 60. Without the prompt, base missed 43 and small missed 41. With the prompt, base missed 18 and small 10. Right: the overall WER. For small, it went down, from 8.6% to 6.8%. For base, it went up, from 13.7% to 15.2%. The prompt fixed names for both sizes, but it hurt the smaller model overall.
For Whisper small, the prompt is a clear win: product names missed fall from 41 to 10, and the WER falls from 8.6% to 6.8%. For Whisper base, the product names improve too (43 to 18), but the WER rises. On English calls, base went from 5.2% to 8.5%. A prompt can hurt a small model.
Why the prompt hurt Whisper base
Section titled “Why the prompt hurt Whisper base”A WER mixes three kinds of error. Count them apart on the 34 English calls:
| Whisper base, English calls | Substitutions | Deletions | Insertions |
|---|---|---|---|
| Without the prompt | 132 | 17 | 58 |
| With the prompt | 90 | 197 | 52 |
The substitutions went down: more names are right. But the deletions went up more than ten times. With the prompt, base skipped whole stretches of some calls. Call C-0077 shows it:
Text description of the diagram
The call has nine turns: the agent’s turns have blue outlines, the caller’s orange. Without the prompt, Whisper base wrote words for every turn. With the prompt, it dropped 26 of the 27 words of turn 2: the caller’s question about the Aqua Mist sprinkler. It also dropped all of turn 3 (the agent asks for the order number) and all of turn 4 (the caller says “613828”). Then it dropped 6 of the 19 words of turn 5. The transcript jumps from “How can I help you today?” to “It’s shipped on the 6th of June.”
The prompt fixed the product name and lost the whole question, and the order number with it. A small model can treat the prompt as text it has already written, and skip ahead. Always measure a prompt on the size you will use, and count the deletions apart.
Idea 3: rules for order numbers after recognition
Section titled “Idea 3: rules for order numbers after recognition”A prompt helps the model hear better. It cannot promise a right number. The second idea comes after recognition: write the number as digits, then check it before Larkfield acts on it.
From spoken words to digits
Section titled “From spoken words to digits”The normalisation guide of Module 2 already turns spoken numbers into digits: “thirty-eight, forty-five, twenty-nine” becomes 384529, and “oh” between digits becomes 0. The new file larkspeech/numbers.py uses it to find the order number in a caller’s words:
def find_order_number(text, language="en", length=6): """(number, problem): the 6-digit order number in a caller's words, or a reason why there is none.""" runs = digits_in(text, language) if not runs: return None, "no digits" best = max(runs, key=len) if len(best) < length: return best, "too short" if len(best) > length: return best, "too long" if not luhn_ok(best): return best, "check digit wrong" return best, NoneIt returns the number and a problem, or None when there is no problem.
A check digit: a tiny worked example
Section titled “A check digit: a tiny worked example”Larkfield’s order numbers have a check digit: the last digit is computed from the other five, with the Luhn rule that bank cards also use. So most typing or hearing mistakes give a number whose last digit does not fit.
The rule: start from the right. Keep the 1st digit, double the 2nd, keep the 3rd, double the 4th, and so on. If a doubled digit is more than 9, subtract 9. Add everything. The number is valid if the total ends in 0. Here is order 742155:
| Digit (left to right) | 7 | 4 | 2 | 1 | 5 | 5 |
|---|---|---|---|---|---|---|
| Doubled? (every second from the right) | yes | no | yes | no | yes | no |
| Value | 7 × 2 = 14 → 14 − 9 = 5 | 4 | 2 × 2 = 4 | 1 | 5 × 2 = 10 → 10 − 9 = 1 | 5 |
Total: 5 + 4 + 4 + 1 + 1 + 5 = 20. It ends in 0, so 742155 is valid. If a caller’s 1 is heard as 7 (742755), the total changes by 6 and the check fails. If two neighbour digits swap (472155), the total changes too.
Three checks before Larkfield acts
Section titled “Three checks before Larkfield acts”Text description of the diagram
The caller’s words “thirty-eight, forty-five, twenty-nine” are normalised to 384529. The number then meets three questions in a row. Does it have 6 digits? Is its check digit right? Is it in the order system? If all three answers are yes, the IVR reads it back: “Order 3 8 4 5 2 9, is that right?” If any answer is no, it asks again in another way, for example “type it on your keypad”.
How well does this work on real transcripts? For every test call whose order number a recogniser missed, we looked at the digit runs it wrote instead, and at the check that stops each one:
| Recogniser | Order numbers missed (of 52) | Stopped by the length | Stopped by the check digit | No digits at all | Wrong number that passes all three checks |
|---|---|---|---|---|---|
| Whisper tiny | 20 | 18 | 1 | 1 | 0 |
| Whisper base | 13 | 10 | 2 | 1 | 0 |
| Whisper small | 8 | 7 | 0 | 1 | 0 |
| Vosk small | 35 | 22 | 2 | 11 | 0 |
On these calls, no wrong order number passed all three checks. Most mistakes are one digit short: Whisper small wrote “26026” for 260026, where the caller said “two six zero zero two six”. The length check stops those. The checks do not fix a number. They tell the IVR to ask again, which costs a few seconds instead of a wrong order.
Larkfield’s IVR: a different prompt for each question
Section titled “Larkfield’s IVR: a different prompt for each question”The phone line of Module 7 hears short answers: a language, a menu choice, an order number, “yes” or “no”. The first version used the vocabulary prompt for every question, and it failed the language question. Callers who said “Français” got words like “Forfeit” and “Enbush”.
The fix was a short prompt for each state of the dialogue, a little grammar of the answers it expects: “English. Français.” for the language question, the menu phrases for the menu, a sample order number for the order question, and “Yes. No.” for a confirmation. The IVR then used these prompts and two French menu words. On its 60 recorded test calls, the share of calls that it finished without a person rose from 28% to 35%. Module 7 builds it.
Guided practice: the prompt and the rules on your computer
Section titled “Guided practice: the prompt and the rules on your computer”-
In your folder
larkspeech, with the environment active, copy the order-number rules and their tests from the snapshotend-of-m04.macOS and Linux:
Terminal window cp ../nextia-learning-labs/C41/snapshots/end-of-m04/larkspeech/numbers.py larkspeech/cp ../nextia-learning-labs/C41/snapshots/end-of-m04/tests/test_numbers.py tests/Windows PowerShell:
Terminal window Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\larkspeech\numbers.py larkspeech\Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\tests\test_numbers.py tests\Your project is now the same as the snapshot
end-of-m04, except its README. -
Run the tests:
Terminal window python -m pytest -q..................... [100%]21 passed in 0.61s -
Try the rules on three things a caller might say, and on Vosk’s words for call C-0288 from the first lesson of this module:
Terminal window python -c "from larkspeech.numbers import find_order_number as f; print(f(\"Yes, it's thirty-eight, forty-five, twenty-nine.\")); print(f('Sure. The order number is 26026.')); print(f('it is 384592')); print(f('yes has to six there sarah to sick'))"('384529', None)('26026', 'too short')('384592', 'check digit wrong')('6', 'too short')In Vosk’s words, only “six” is a digit, so the length check stops them.
In PowerShell, the quotes inside the Python code need care: if the command fails, put the four lines in a file
try_numbers.pyand runpython try_numbers.py. -
Run Whisper base on the first 10 English test calls, without and with the prompt:
Terminal window speech transcribe --model base --language en --limit 10speech transcribe --model base --language en --limit 10 --promptwhisper-base on 10 test calls (phone): WER 6.1%, CER 3.4%, order numbers wrong 1 of 10, real-time factor 0.029Wrote outputs/transcripts-whisper-base.jsonlwhisper-base-prompt on 10 test calls (phone): WER 9.0%, CER 7.3%, order numbers wrong 1 of 10, real-time factor 0.025Wrote outputs/transcripts-whisper-base-prompt.jsonlYour real-time factors will differ, and a word or two may change.
Your turn: the prompt with Whisper small
Section titled “Your turn: the prompt with Whisper small”Predict: on the same 10 English calls, does the prompt help Whisper small? Then run the two commands with --model small.
Check your answer
whisper-small on 10 test calls (phone): WER 3.7%, CER 1.9%, order numbers wrong 0 of 10, real-time factor 0.068Wrote outputs/transcripts-whisper-small.jsonlwhisper-small-prompt on 10 test calls (phone): WER 2.6%, CER 1.4%, order numbers wrong 0 of 10, real-time factor 0.074Wrote outputs/transcripts-whisper-small-prompt.jsonlOn these 10 calls, the prompt helps Whisper small a little: the WER goes from 3.7% to 2.6%, and no order number is lost. On all 34 English test calls, the recorded runs give 3.1% and 2.6%, but the order numbers missed rose from 2 to 4 of 28. A prompt can move the important words both ways: count each kind apart.
Common mistake
A prompt that helps the words you looked at and hurts the rest
Symptom: Amira adds the vocabulary prompt to Whisper base, because base is fast. She checks ten product names: almost all are right now. She ships it. A week later, Grace finds call records with no order number and no reason for the call.
Cause: the prompt made base skip whole stretches of some calls. On the English test calls, its deletions went from 17 to 197, and order numbers missed from 1 to 2. Amira measured only the words that the prompt was meant to fix.
Fix: measure every change on the whole test set, with the overall WER, the deletions and every kind of important word. Keep the prompt only where it wins, here with Whisper small.
Consequence in AI work: any fix aimed at one kind of error can create another kind. A test set with all the errors that matter counted apart catches it.
Check your understanding
Is 384529 a valid Larkfield order number? Show the Luhn steps.
Show the answer
From the right: 9 (keep), 2 → 4, 5 (keep), 4 → 8, 8 (keep), 3 → 6. Total: 9 + 4 + 5 + 8 + 8 + 6 = 40. It ends in 0, so 384529 is valid. With the last two digits swapped (384592), the total is 2 + 18 − 9 + 5 + 8 + 8 + 6 = 38, and the check fails.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The vocabulary prompt cut Whisper small's missed product names from 41 to 10, but raised Whisper base's WER from 13.7% to 15.2%. What happened to base, and how did we find it?
Show the answer
With the prompt, base skipped whole stretches of some calls: on the English calls, its deletions went from 17 to 197, while its substitutions fell. We found it by counting substitutions, deletions and insertions apart, and by looking at one call (C-0077), where the caller’s question and the order number were gone. Measure a prompt on the model size you will use.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Whisper wrote "26026" for order 260026. Which check stops it, and what should the IVR do next?
Show the answer
The length check: it has 5 digits, not 6. The check does not fix the number. The IVR should ask again in another way, for example “please type your order number on the keypad”. Or it reads back what it heard and asks the caller to correct it. It must not look up or act on a number that failed a check.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Tell the model the language when you know it. On Larkfield’s calls, detection was right on all 60 test calls, but short answers are too short to detect.
- An initial prompt with Larkfield’s words cut Whisper small’s missed product names from 41 to 10 and its WER from 8.6% to 6.8%.
- The same prompt hurt Whisper base: its WER rose from 13.7% to 15.2%, because it skipped whole stretches of calls. Measure a prompt on your model size, and count deletions apart.
- After recognition, turn spoken numbers into digits, then check the length, the check digit (Luhn) and the order system. On the test calls, no wrong order number passed all three.
- The IVR uses a short prompt for each question, such as “English. Français.”, not one big vocabulary.
You can now help the model with Larkfield’s words and catch wrong numbers. But Larkfield’s callers also speak French, switch languages, and call from noisy streets. Next, you see what each recogniser does with two languages, a language switch and noise.