Skip to content
Français

Help the model with your words

  • Lesson 13
  • 35 min
  • Module 4: Speech recognition: compare the families

Where we are

Our recognisers make the same mistakes again and again: product names and order numbers. They do not know Larkfield's words. In this lesson, we give the model our vocabulary and fix spoken order numbers with rules. First, think of how you help a new colleague on their first day.

Lesson 13 of 36 · Speech AI: Transcribe, Understand and SpeakHelp the model with your words

What you will be able to do

Make the recogniser write Larkfield’s own words: its product names and its order numbers. A wrong product name breaks a search, and a wrong order number gives a caller someone else’s order.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook measures the vocabulary prompt on the recorded transcripts of Whisper base and small, and finds the calls where it hurts. Then you write the check-digit rule and count which check catches each wrong order number. It downloads only small text files and runs no model. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

  • Your larkspeech folder with the transcribe command from Whisper sizes on a laptop, and the labs folder next to it.
  • The Whisper base and small models, which that lesson downloaded.
  • Tested on macOS (Apple M4) with Python 3.12. The Windows commands are given but were not run.

A new agent starts at Larkfield’s help desk. Before her first call, her team lead gives her a card with Larkfield’s words: “We sell the Grow Hub, the Solara lantern, the Cedar raised bed … Order numbers have six digits.” She has not learned anything new about English. But now, when a caller says “my Fern wall planter”, she knows that name, and she writes it right.

The team lead also tells her: “Read the order number back to the caller, and type it into the order system. If the system does not know it, ask again.” That second rule does not make her hearing better. It catches her mistakes before they cost anything.

This lesson does the same two things for a recogniser: give it a card of Larkfield’s words before it listens, and check its numbers after it listens.

The problem: the same words, wrong every time

Section titled “The problem: the same words, wrong every time”

In the last lessons, the same mistakes came back on many calls. On the 60 test calls, Whisper small missed 41 of the 60 product names and 8 of the 52 order numbers. Its overall WER is only 8.6%. These few words matter more than all the others: a product name tells Larkfield what the call is about, and an order number is the key to the order.

Look at one caller. In call C-0077, the caller asks about an Aqua Mist sprinkler:

Call C-0077, turn 2, on the phone line (a mono call, so you also hear the line as the agent has it). The caller’s voice is synthetic: Kokoro (voice bf_alice).

Transcript. Caller: “Hello there. I’m excited to get started with my Aqua Mist sprinkler, but it hasn’t arrived yet. Could you check where it’s up to?”

Whisper base wrote “my aqua-miss sprinkler”. The sound is clear. The model simply does not know the name, so it wrote a likely spelling.

The simplest help is the language setting. Whisper can detect the language from the first 30 seconds of a call, or you can tell it. On the test calls, Whisper small found the right language on all 60 calls, and its WER was 8.7% against 8.6% when told. So on Larkfield’s calls, which stay in one language, the setting changes little.

It still matters for two reasons. First, the IVR in Module 7 hears only a few words at a time, such as “yes”. A few words are not enough to detect a language. Second, a wrong language gives a transcript in the wrong language, and all its words count as errors. When you know the language, from the caller’s choice or the phone number they called, tell the model.

Whisper’s decoder writes each token after looking at the tokens before it. Normally, the “tokens before” are only the call’s own words. An initial prompt is text that you put in front, as if the model had just written it. The decoder then expects words like those in the prompt.

The course project has one prompt for each language, in larkspeech/asr.py:

VOCABULARY_PROMPT = {
"en": "Larkfield help desk. Grow Hub, Solara lantern, Cedar raised bed, Terracotta Trio, Mistral fan, "
"Willow hammock, Ember fire pit, Aqua Mist sprinkler, Fern wall planter, Harbor bench, Pebble path lights, "
"Juniper compost bin. Order 742158.",
...
}

It names the shop, lists the 12 products, and shows an order number written as digits. The --prompt option of speech transcribe passes it to Whisper.

Other tools have other forms of the same idea. Hosted services often take a phrase list (a list of words to prefer). Vosk can take a list of the only phrases it may write. faster-whisper also has a hotwords option, which this course did not measure.

Two bar charts. Left: product names missed, of 60. Whisper base: 43 without the prompt, 18 with it. Whisper small: 41 without, 10 with. Right: WER on all test calls in percent. Whisper base: 13.7 without, 15.2 with. Whisper small: 8.6 without, 6.8 with.
Whisper base and small on the 60 phone test calls, without and with Larkfield's vocabulary prompt (recorded transcripts).
Text description of the diagram

Left: the product names missed, of 60. Without the prompt, base missed 43 and small missed 41. With the prompt, base missed 18 and small 10. Right: the overall WER. For small, it went down, from 8.6% to 6.8%. For base, it went up, from 13.7% to 15.2%. The prompt fixed names for both sizes, but it hurt the smaller model overall.

For Whisper small, the prompt is a clear win: product names missed fall from 41 to 10, and the WER falls from 8.6% to 6.8%. For Whisper base, the product names improve too (43 to 18), but the WER rises. On English calls, base went from 5.2% to 8.5%. A prompt can hurt a small model.

A WER mixes three kinds of error. Count them apart on the 34 English calls:

Whisper base, English calls Substitutions Deletions Insertions
Without the prompt 132 17 58
With the prompt 90 197 52

The substitutions went down: more names are right. But the deletions went up more than ten times. With the prompt, base skipped whole stretches of some calls. Call C-0077 shows it:

A timeline of call C-0077 from 0 to 50 seconds with nine turns as boxes, blue outlines for the agent and orange for the caller. Two rows. Whisper base without the prompt: no red in any turn. Whisper base with the prompt: turn 2 (the caller's question) almost all red, turns 3 and 4 all red, turn 5 about one third red, the other turns without red.
Call C-0077, turn by turn. Red shows the share of each turn's words that Whisper base did not write. Recorded transcripts, without and with the vocabulary prompt.
Text description of the diagram

The call has nine turns: the agent’s turns have blue outlines, the caller’s orange. Without the prompt, Whisper base wrote words for every turn. With the prompt, it dropped 26 of the 27 words of turn 2: the caller’s question about the Aqua Mist sprinkler. It also dropped all of turn 3 (the agent asks for the order number) and all of turn 4 (the caller says “613828”). Then it dropped 6 of the 19 words of turn 5. The transcript jumps from “How can I help you today?” to “It’s shipped on the 6th of June.”

The prompt fixed the product name and lost the whole question, and the order number with it. A small model can treat the prompt as text it has already written, and skip ahead. Always measure a prompt on the size you will use, and count the deletions apart.

Idea 3: rules for order numbers after recognition

Section titled “Idea 3: rules for order numbers after recognition”

A prompt helps the model hear better. It cannot promise a right number. The second idea comes after recognition: write the number as digits, then check it before Larkfield acts on it.

The normalisation guide of Module 2 already turns spoken numbers into digits: “thirty-eight, forty-five, twenty-nine” becomes 384529, and “oh” between digits becomes 0. The new file larkspeech/numbers.py uses it to find the order number in a caller’s words:

def find_order_number(text, language="en", length=6):
"""(number, problem): the 6-digit order number in a caller's words, or a reason why there is none."""
runs = digits_in(text, language)
if not runs:
return None, "no digits"
best = max(runs, key=len)
if len(best) < length:
return best, "too short"
if len(best) > length:
return best, "too long"
if not luhn_ok(best):
return best, "check digit wrong"
return best, None

It returns the number and a problem, or None when there is no problem.

Larkfield’s order numbers have a check digit: the last digit is computed from the other five, with the Luhn rule that bank cards also use. So most typing or hearing mistakes give a number whose last digit does not fit.

The rule: start from the right. Keep the 1st digit, double the 2nd, keep the 3rd, double the 4th, and so on. If a doubled digit is more than 9, subtract 9. Add everything. The number is valid if the total ends in 0. Here is order 742155:

Digit (left to right) 7 4 2 1 5 5
Doubled? (every second from the right) yes no yes no yes no
Value 7 × 2 = 14 → 14 − 9 = 5 4 2 × 2 = 4 1 5 × 2 = 10 → 10 − 9 = 1 5

Total: 5 + 4 + 4 + 1 + 1 + 5 = 20. It ends in 0, so 742155 is valid. If a caller’s 1 is heard as 7 (742755), the total changes by 6 and the check fails. If two neighbour digits swap (472155), the total changes too.

A flow chart. The caller's words, thirty-eight, forty-five, twenty-nine, are normalised to 384529. Then three questions in a row: 6 digits? Check digit right? In the order system? If all are yes: read it back, order 3 8 4 5 2 9, is that right? If any answer is no: ask again in another way, type it on the keypad.
After recognition, an order number passes three checks before the IVR uses it: the length, the check digit, and the order system. Any failure asks the caller again.
Text description of the diagram

The caller’s words “thirty-eight, forty-five, twenty-nine” are normalised to 384529. The number then meets three questions in a row. Does it have 6 digits? Is its check digit right? Is it in the order system? If all three answers are yes, the IVR reads it back: “Order 3 8 4 5 2 9, is that right?” If any answer is no, it asks again in another way, for example “type it on your keypad”.

How well does this work on real transcripts? For every test call whose order number a recogniser missed, we looked at the digit runs it wrote instead, and at the check that stops each one:

Recogniser Order numbers missed (of 52) Stopped by the length Stopped by the check digit No digits at all Wrong number that passes all three checks
Whisper tiny 20 18 1 1 0
Whisper base 13 10 2 1 0
Whisper small 8 7 0 1 0
Vosk small 35 22 2 11 0

On these calls, no wrong order number passed all three checks. Most mistakes are one digit short: Whisper small wrote “26026” for 260026, where the caller said “two six zero zero two six”. The length check stops those. The checks do not fix a number. They tell the IVR to ask again, which costs a few seconds instead of a wrong order.

Larkfield’s IVR: a different prompt for each question

Section titled “Larkfield’s IVR: a different prompt for each question”

The phone line of Module 7 hears short answers: a language, a menu choice, an order number, “yes” or “no”. The first version used the vocabulary prompt for every question, and it failed the language question. Callers who said “Français” got words like “Forfeit” and “Enbush”.

The fix was a short prompt for each state of the dialogue, a little grammar of the answers it expects: “English. Français.” for the language question, the menu phrases for the menu, a sample order number for the order question, and “Yes. No.” for a confirmation. The IVR then used these prompts and two French menu words. On its 60 recorded test calls, the share of calls that it finished without a person rose from 28% to 35%. Module 7 builds it.

Guided practice: the prompt and the rules on your computer

Section titled “Guided practice: the prompt and the rules on your computer”
  1. In your folder larkspeech, with the environment active, copy the order-number rules and their tests from the snapshot end-of-m04.

    macOS and Linux:

    Terminal window
    cp ../nextia-learning-labs/C41/snapshots/end-of-m04/larkspeech/numbers.py larkspeech/
    cp ../nextia-learning-labs/C41/snapshots/end-of-m04/tests/test_numbers.py tests/

    Windows PowerShell:

    Terminal window
    Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\larkspeech\numbers.py larkspeech\
    Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\tests\test_numbers.py tests\

    Your project is now the same as the snapshot end-of-m04, except its README.

  2. Run the tests:

    Terminal window
    python -m pytest -q
    ..................... [100%]
    21 passed in 0.61s
  3. Try the rules on three things a caller might say, and on Vosk’s words for call C-0288 from the first lesson of this module:

    Terminal window
    python -c "from larkspeech.numbers import find_order_number as f; print(f(\"Yes, it's thirty-eight, forty-five, twenty-nine.\")); print(f('Sure. The order number is 26026.')); print(f('it is 384592')); print(f('yes has to six there sarah to sick'))"
    ('384529', None)
    ('26026', 'too short')
    ('384592', 'check digit wrong')
    ('6', 'too short')

    In Vosk’s words, only “six” is a digit, so the length check stops them.

    In PowerShell, the quotes inside the Python code need care: if the command fails, put the four lines in a file try_numbers.py and run python try_numbers.py.

  4. Run Whisper base on the first 10 English test calls, without and with the prompt:

    Terminal window
    speech transcribe --model base --language en --limit 10
    speech transcribe --model base --language en --limit 10 --prompt
    whisper-base on 10 test calls (phone): WER 6.1%, CER 3.4%, order numbers wrong 1 of 10, real-time factor 0.029
    Wrote outputs/transcripts-whisper-base.jsonl
    whisper-base-prompt on 10 test calls (phone): WER 9.0%, CER 7.3%, order numbers wrong 1 of 10, real-time factor 0.025
    Wrote outputs/transcripts-whisper-base-prompt.jsonl

    Your real-time factors will differ, and a word or two may change.

Predict: on the same 10 English calls, does the prompt help Whisper small? Then run the two commands with --model small.

Check your answer
whisper-small on 10 test calls (phone): WER 3.7%, CER 1.9%, order numbers wrong 0 of 10, real-time factor 0.068
Wrote outputs/transcripts-whisper-small.jsonl
whisper-small-prompt on 10 test calls (phone): WER 2.6%, CER 1.4%, order numbers wrong 0 of 10, real-time factor 0.074
Wrote outputs/transcripts-whisper-small-prompt.jsonl

On these 10 calls, the prompt helps Whisper small a little: the WER goes from 3.7% to 2.6%, and no order number is lost. On all 34 English test calls, the recorded runs give 3.1% and 2.6%, but the order numbers missed rose from 2 to 4 of 28. A prompt can move the important words both ways: count each kind apart.

Common mistake

A prompt that helps the words you looked at and hurts the rest

Symptom: Amira adds the vocabulary prompt to Whisper base, because base is fast. She checks ten product names: almost all are right now. She ships it. A week later, Grace finds call records with no order number and no reason for the call.

Cause: the prompt made base skip whole stretches of some calls. On the English test calls, its deletions went from 17 to 197, and order numbers missed from 1 to 2. Amira measured only the words that the prompt was meant to fix.

Fix: measure every change on the whole test set, with the overall WER, the deletions and every kind of important word. Keep the prompt only where it wins, here with Whisper small.

Consequence in AI work: any fix aimed at one kind of error can create another kind. A test set with all the errors that matter counted apart catches it.

Check your understanding

Is 384529 a valid Larkfield order number? Show the Luhn steps.

Show the answer

From the right: 9 (keep), 2 → 4, 5 (keep), 4 → 8, 8 (keep), 3 → 6. Total: 9 + 4 + 5 + 8 + 8 + 6 = 40. It ends in 0, so 384529 is valid. With the last two digits swapped (384592), the total is 2 + 18 − 9 + 5 + 8 + 8 + 6 = 38, and the check fails.

Check your understanding

The vocabulary prompt cut Whisper small's missed product names from 41 to 10, but raised Whisper base's WER from 13.7% to 15.2%. What happened to base, and how did we find it?

Show the answer

With the prompt, base skipped whole stretches of some calls: on the English calls, its deletions went from 17 to 197, while its substitutions fell. We found it by counting substitutions, deletions and insertions apart, and by looking at one call (C-0077), where the caller’s question and the order number were gone. Measure a prompt on the model size you will use.

Check your understanding

Whisper wrote "26026" for order 260026. Which check stops it, and what should the IVR do next?

Show the answer

The length check: it has 5 digits, not 6. The check does not fix the number. The IVR should ask again in another way, for example “please type your order number on the keypad”. Or it reads back what it heard and asks the caller to correct it. It must not look up or act on a number that failed a check.

Recap

  • Tell the model the language when you know it. On Larkfield’s calls, detection was right on all 60 test calls, but short answers are too short to detect.
  • An initial prompt with Larkfield’s words cut Whisper small’s missed product names from 41 to 10 and its WER from 8.6% to 6.8%.
  • The same prompt hurt Whisper base: its WER rose from 13.7% to 15.2%, because it skipped whole stretches of calls. Measure a prompt on your model size, and count deletions apart.
  • After recognition, turn spoken numbers into digits, then check the length, the check digit (Luhn) and the order system. On the test calls, no wrong order number passed all three.
  • The IVR uses a short prompt for each question, such as “English. Français.”, not one big vocabulary.

You can now help the model with Larkfield’s words and catch wrong numbers. But Larkfield’s callers also speak French, switch languages, and call from noisy streets. Next, you see what each recogniser does with two languages, a language switch and noise.

Next: Accents, noise and two languages