Skip to content
Français

Normalise before you compare

  • Lesson 5
  • 35 min
  • Module 2: Measure before you choose

Where we are

In the last lesson, we computed the word error rate by counting substitutions, deletions and insertions. But in the explorer, one transcript got 266.7%, although the model wrote every word that the caller said. In this lesson, we see why the way we write a text changes its score, and we write one guide that both texts follow, the way two agents agree how to write their call notes.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 5 of 36 · Speech AI: Transcribe, Understand and SpeakNormalise before you compare

What you will be able to do

Make a word error rate mean something. Write one normalisation guide and apply it to the reference and to the model text. Never compare two scores that used different guides.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It scores one pair at three levels and runs the guide rule by rule. It scores two recognisers on the 60 test calls at each level and shows the one-side mistake. Then you add a rule for French. It needs no model download and no account. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the local route; the notebook needs none of it)

  • Your folder larkspeech with the files of the snapshot end-of-m02, from Word error rate, step by step. speech wer --help shows the option --level.

Tested on macOS with Python 3.12. The commands are the same on Windows; they were not tested there.

Think of two agents at Larkfield’s help desk. They listen to the same recorded call and take notes. Freya writes: “Order 742155. Customer doesn’t want a refund.” Ben writes: “order seven four two one five five, customer does not want a refund”.

Their notes say the same thing. But a strict comparison, word by word, finds many differences. “742155” is one word for Freya and six words for Ben. “Order” has a capital letter in one note and not in the other. “doesn’t” is one word; “does not” is two. A supervisor who marks the notes this way would say that Ben made many mistakes. He made none. The two agents just write in different ways.

A good supervisor first agrees with both agents on how to write: numbers in digits, no capital letters, contractions written out. Then the comparison measures what matters: what they heard, not how they write it.

The problem: the score changes, the transcript does not

Section titled “The problem: the score changes, the transcript does not”

A recogniser and a reference transcript are the two agents. The reference of Larkfield’s calls writes order numbers in digits, with capital letters and punctuation. Some recognisers write numbers in words, with no punctuation at all. Mark them word by word, and you measure the writing style.

Here is a constructed pair. The caller said “Yes, it’s 742155.”, the order number digit by digit, and the model wrote “yes it is seven four two one five five”. Every word is right. speech wer has an option --level that sets how much it cleans the two texts before it compares them. Try the three levels on the same pair:

Terminal window
speech wer "Yes, it's 742155." "yes it is seven four two one five five" --level none
speech wer "Yes, it's 742155." "yes it is seven four two one five five" --level basic
speech wer "Yes, it's 742155." "yes it is seven four two one five five"
REF: *** ** ** ***** **** *** Yes, it's 742155.
HYP: yes it is seven four two one five five
I I I I I I S S S
3 substitutions, 0 deletions, 6 insertions, 3 reference words: WER 300.0%, CER 188.2% (normalisation: none)
REF: yes ** ** ***** **** *** *** it's 742155
HYP: yes it is seven four two one five five
I I I I I I S S
2 substitutions, 0 deletions, 6 insertions, 3 reference words: WER 266.7%, CER 200.0% (normalisation: basic)
REF: yes it is 742155
HYP: yes it is 742155
0 substitutions, 0 deletions, 0 insertions, 4 reference words: WER 0.0%, CER 0.0% (normalisation: full)

The same two texts get 300%, 266.7% and 0%. The model did not change. Only the way we compared the texts did. A WER without its normalisation is not a measurement: you cannot tell what it counts.

The idea: one written guide for both texts

Section titled “The idea: one written guide for both texts”

Normalisation means rewriting both texts in one standard form before you compare them. The rules of that form are the normalisation guide. It is the agreement of the two agents: numbers in digits, no capital letters, contractions written out.

The project has three levels:

Level What it does Use it for
none splits the text on spaces, nothing else showing the problem
basic lower case, punctuation removed a quick look
full basic, plus the rules of Larkfield’s guide every score that you report

Larkfield’s guide is written down. The course’s helper prints it, and the labs folder has it as data/normalisation-guide.md. It has nine rules: lower case, punctuation, fillers, English contractions, French elisions, number words, ordinals, joined digits, and spaces.

Take one caller sentence, “Um, it’s seventy-four, twenty-one, fifty-five.”, and follow it through the rules. Each row shows the text after one more rule:

Rule The text after the rule
The caller’s words Um, it’s seventy-four, twenty-one, fifty-five.
1. Lower case um, it’s seventy-four, twenty-one, fifty-five.
2. Punctuation removed; a hyphen between words becomes a space um it’s seventy four twenty one fifty five
3. Fillers removed (“um”, “uh”, “euh”) it’s seventy four twenty one fifty five
4. Contractions written out it is seventy four twenty one fifty five
6. Number words become digits it is 74 21 55
8. Digits that follow each other are joined it is 742155

Rules 5, 7 and 9 change nothing in this sentence, so they have no row. The reference “It’s 742155.” goes through the same rules and also becomes “it is 742155”. Now the two texts match, and the WER counts only real differences.

Rule 8 needs a reason. Callers say an order number in pieces: digit by digit, in pairs, or with a pause. The model may write “742 155”, “74 21 55” or “742155”. The order system needs only the digits, so the guide must not punish the spaces between them.

The notebook runs six short texts through the guide. Here is what the basic and the full levels make of them:

Text basic full
Yes, it’s 38.45.29. yes it’s 38 45 29 yes it is 384529
It shipped on the fourteenth of March. it shipped on the fourteenth of march it shipped on the 14 of march
Seven oh two, and I don’t want it. seven oh two and i don’t want it 702 and i do not want it
Euh, c’est le sept quatre deux un cinq cinq. euh c’est le sept quatre deux un cinq cinq c est le 742155
Elle est partie le premier juillet. elle est partie le premier juillet elle est partie le 1 juillet

“oh” becomes 0 only next to a digit, so “oh, I see” stays as it is. “fourteenth” becomes 14, but it is never joined to a digit next to it. In “the first 4111”, “first” stays a word of its own, not part of a card number.

Larkfield’s French calls need three more decisions. First, French joins some short words with an apostrophe: “c’est”, “l’article”, “qu’elle”. These are elisions. The guide splits them (“c est”, “l article”), so that “c’est” and “c’ est” do not count as different words. Second, the guide keeps accents: “livré” and “livre” are different words, so “é” is not “e”. Third, the guide knows French number words (“quatre-vingt-dix-sept” is 97) and turns “le premier” into 1. French callers sometimes say an English number, so the French rules also read English number words.

The explorer of the last lesson has a second control now: the normalisation level. The two texts stay the same; only the guide changes.

Explore

One pair, three levels of normalisation

Reference
Yes, it's 742155.
Model transcript
yes it is seven four two one five five

Normalisation: None: split on spaces

A constructed pair: we wrote both texts to show one idea.

  1. insertion: reference nothing, model yes
  2. insertion: reference nothing, model it
  3. insertion: reference nothing, model is
  4. insertion: reference nothing, model seven
  5. insertion: reference nothing, model four
  6. insertion: reference nothing, model two
  7. substitution: reference Yes,, model one
  8. substitution: reference it's, model five
  9. substitution: reference 742155., model five
  • S: substitution (a word became another word)
  • D: deletion (a reference word is missing)
  • I: insertion (the model added a word)
Errors: S + D + I
3 + 0 + 6 = 9
Reference words N
3
WER = errors / N
9 / 3 = 300.0%

3 substitutions, 0 deletions, 6 insertions: 9 errors for 3 reference words, so the WER is 300.0%. The WER is over 100% because the insertions have no limit: the model wrote more words than the reference has. The same pair with the other levels: basic 266.7%, full 0.0%.

Constructed pairs and caller sentences from Larkfield's synthetic test calls. Only the normalisation level changes the alignment and the WER.
Text description of the explorer

The explorer shows the reference and the model transcript. Then it shows the alignment at the chosen level of normalisation: one column per pair of words, with S, D or I under each error. The readout shows S + D + I, the reference words N and the WER. “Only the normalisation differs” gives 300.0% with no normalisation, 266.7% with the basic level and 0.0% with the full guide. “Order number in pairs” (“It’s 742155.” against “It’s seventy-four, twenty-one, fifty-five.”) gives 150.0%, 300.0% and 0.0%. “French” (“C’est le 742155.” against “c’est le sept quatre deux un cinq cinq”) gives 233.3%, 200.0% and 0.0%. Whisper base on call C-0056 wrote “Yes, it’s 38.4529.”. It gets 33.3% with no normalisation and 0.0% with the full guide. With the basic level, “38 4529” becomes two words, and the WER is 66.7%. Vosk small on call C-0011 wrote “sure the order number is three five three one nine three”: 116.7%, 100.0% and 0.0%. The full guide joins the six digits into “353193”. Whisper small on call C-0056 wrote “I do not want to refund, I want a new one.”: 9.1% at every level, because “to” for “a” is a real recognition error.

Three things to notice in the explorer:

  • Yes, from over 100% to 0%. Four of the examples do it, Vosk on call C-0011 among them. In each one, the model heard every word right.
  • More cleaning is not always a lower score. Whisper base wrote “38.4529”. The basic level removes the dot and makes two words, so the WER rises from 33.3% to 66.7%. Only the full guide joins the digits again. A guide is a set of decisions, not a button that makes the score look good.
  • The guide does not hide real errors. Whisper small wrote “I do not want to refund” for “I do not want a refund”. No rule turns “to” into “a”, so the error stays at every level, and the score says so.

Larkfield: the whole test set at three levels

Section titled “Larkfield: the whole test set at three levels”

Now score the recorded transcripts of all 60 test calls at each level. The recognisers do not change; only --level does.

  1. Score Whisper small with no normalisation, then with the basic level:

    Terminal window
    speech score --method whisper-small --level none
    speech score --method whisper-small --level basic
    whisper-small, test calls, phone audio, normalisation none
    slice calls words WER CER order nos wrong products wrong
    all 60 6964 18.1% 7.2% 8 of 52 41 of 60
    whisper-small, test calls, phone audio, normalisation basic
    slice calls words WER CER order nos wrong products wrong
    all 60 7067 11.6% 5.5% 8 of 52 41 of 60

    With the full guide, the last lesson gave 8.6%.

  2. Do the same for Vosk small:

    Terminal window
    speech score --method vosk-small --level none
    speech score --method vosk-small --level basic
    vosk-small, test calls, phone audio, normalisation none
    slice calls words WER CER order nos wrong products wrong
    all 60 6964 60.0% 29.6% 35 of 52 51 of 60
    vosk-small, test calls, phone audio, normalisation basic
    slice calls words WER CER order nos wrong products wrong
    all 60 7067 35.2% 24.1% 35 of 52 51 of 60
  3. Look at the words column. The reference has 6,964 words with no normalisation, 7,067 with the basic level and 7,334 with the full guide. “it’s” is one word before the guide and two words after it, so N changes too.

The columns order nos wrong and products wrong do not change with --level. Those checks always use the full guide, because an order number must be compared as digits.

  • Whisper small
  • Vosk small
The same recorded transcripts of Larkfield's 60 synthetic test calls, scored with no normalisation, the basic level and the full guide. The full guide lowers Whisper small's WER from 18.1% to 8.6%, and Vosk's from 60.0% to 31.2%.
Text description of the chart

A column chart with six columns, on an axis from 0 to 70%. The first three columns are Whisper small: 18.1% with no normalisation, 11.6% with the basic level and 8.6% with the full guide. The last three are Vosk small: 60.0%, 35.2% and 31.2%. Both recognisers fall at each step, and Vosk falls the most, by almost 29 points.

Vosk changes the most because it writes every number in words and has no punctuation or capital letters. With no normalisation, it looks more than three times as bad as Whisper small (60.0% against 18.1%). With the full guide, it is still much worse (31.2% against 8.6%), but now the gap measures what the two recognisers heard.

So a normalisation guide can change the size of a difference between two recognisers, and on other data it can change their order. That is why the course scores every recogniser with the same guide. Each row of the course’s comparison table says which guide it used.

Always normalise before you report a WER, and apply the same guide to the reference and to every model text. Write the guide down, give it a version, and keep it with the scores. A WER in a report without its guide is a number that nobody can check.

Never compare two WERs that used different guides. A model card or a benchmark often uses its own normaliser. Whisper’s authors, for example, published an English text normaliser that also changes British spellings to American ones. Its scores and Larkfield’s are not on the same scale. To compare, score both models on your own calls, with your own guide.

Do not normalise away an error that matters. A rule that turns “firm” into “fern” would make the score look better and hide a real mistake on a product name. A rule must only remove differences in writing, never differences in meaning. The negation “not” must stay, and so must every digit.

The guide costs a little work, and it has one hidden cost. When you change it, every score changes, so you must score every recogniser again. Larkfield keeps one guide for the whole course for this reason.

Guided practice: the guide in the notebook

Section titled “Guided practice: the guide in the notebook”
  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The setup cell ends with a Ready: line.
  3. In section 1, predict the three scores. Then run the cell. You see 300.0%, 266.7% and 0.0%.
  4. In section 2, print the guide, then run the six examples. Compare each line with the table on this page.
  5. In section 3, load the calls. You see 60 test calls.
  6. Run the scores cell. You see 18.1%, 11.6% and 8.6% for Whisper small, and 60.0%, 35.2% and 31.2% for Vosk small.

Whisper small sometimes writes a French ordinal as “1er”, as in “le 1er juillet”. The guide turns “premier” into 1, but it has no rule for “1er”. So “1” and “1er” count as a substitution. In the notebook, section 5, write guide_plus(text, language). It applies the full guide, then one more rule. The rule turns a number followed by “er”, “re”, “e” or “ème” into the number alone. Then compute Whisper small’s WER on the 26 French test calls with the guide and with your rule. Normalise both texts with the same function. Before you run it, predict: will the French WER go down by more or less than one point?

Show the answer
import re
def guide_plus(text, language):
"""The full guide, then one more rule: a French ordinal "1er", "2e" or "3ème" becomes its number."""
out = cp.normalise(text, "full", language)
return re.sub(r"\b(\d+)(er|re|e|ème)\b", r"\1", out)
french = [c for c in test if c["language"] == "fr"]
def french_wer(normalise):
errors = words = 0
for c in french:
n = cp.counts(normalise(reference(c), "fr"), normalise(small[c["call_id"]], "fr"))
errors += n["sub"] + n["del"] + n["ins"]
words += n["ref_words"]
return errors / words
before = french_wer(lambda text, language: cp.normalise(text, "full", language))
after = french_wer(guide_plus)
print(f"French test calls, Whisper small: {before:.2%} with the guide, {after:.2%} with the new rule")
French test calls, Whisper small: 15.06% with the guide, 15.00% with the new rule

The rule is right, and it changes very little: 0.06 points. Most of Whisper small’s French errors are real recognition errors, not differences in writing. That is a useful result too: it tells you to look for the French problem in the audio and the model, not in the guide. If you keep the rule, it is a new version of your guide, and every recogniser must be scored again with it.

Common mistake

The guide on one side only

Symptom: Tomás normalises the model transcripts with the full guide, but forgets the references. Whisper small’s WER on the 60 test calls rises from 8.6% to 53.3%:

errors = words = 0
for c in test:
r = cp.normalise(reference(c), "none", c["language"]) # the reference as it was written
h = cp.normalise(small[c["call_id"]], "full", c["language"]) # only the model text normalised
n = cp.counts(r, h)
errors += n["sub"] + n["del"] + n["ins"]
words += n["ref_words"]
print(f"Whisper small, only the model text normalised: WER {errors / words:.1%}")
Whisper small, only the model text normalised: WER 53.3%

Cause: the reference still has capital letters, punctuation and contractions (“Thank”, “you.”, “it’s”), and the model text has none. Many words that the model heard right now count as substitutions.

Fix: normalise both texts with the same function and the same level. The project’s speech score and cp.corpus_scores do it for you; use them instead of your own loop.

Consequence in AI work: a score that is far too high can make a team drop a good recogniser. Or the team buys a bigger one that it does not need. A score that is too low is worse. It comes from a rule that hides real errors, and nobody looks for the problem.

Check your understanding

The reference is “Yes, it's 384529.” and the model wrote “Yes, it's 38.45.29.” Predict the WER with the basic level and with the full guide, and explain the difference.

Show the answer

With the basic level, the dots become spaces. So the model text has three words, “38”, “45” and “29”, for the one word “384529”. That is 1 substitution and 2 insertions for 3 reference words (“yes”, “it’s”, “384529”), a WER of 100%. With the full guide, “it’s” becomes “it is” on both sides, and rule 8 joins the digits into “384529”. The texts match, and the WER is 0%. The model heard every digit; only the full guide shows it.

Check your understanding

A vendor's page says that its recogniser has a WER of 5% on phone calls. Larkfield's guide gives Whisper small 8.6% on its test calls. Tomás says the vendor is clearly better. What do you answer?

Show the answer

The two numbers are not comparable. They come from different calls, and probably from different normalisation guides. The same transcripts of Whisper small give 18.1%, 11.6% or 8.6%, and only the guide changes. To compare, run both recognisers on Larkfield’s own test calls and score both with Larkfield’s guide. Also, Larkfield’s calls are synthetic, so even that comparison says how they do on these calls, not on real ones.

Check your understanding

Malik suggests a new rule: write every word that sounds like a product name as the product name, for example “firm wall planter” as “fern wall planter”. The WER would drop. Should Larkfield add it to the guide?

Show the answer

No. The guide may only remove differences in writing, such as digits or capital letters. “firm” and “fern” are different words: the model heard the product name wrong. A rule that fixes it inside the score hides a real error. The score would no longer describe what the transcripts can do. To get product names right, change the recogniser or its settings. Then measure the product-name errors on their own, as the next lesson does.

Recap

  • The same transcript can score 300%, 266.7% or 0% when only the normalisation changes. A WER without its guide means nothing.
  • Normalisation rewrites both texts in one standard form. Larkfield’s guide has nine written rules: lower case, punctuation, fillers, contractions, French elisions, number words, ordinals, joined digits and spaces.
  • On the 60 synthetic test calls, Whisper small scores 18.1%, 11.6% and 8.6%: no normalisation, the basic level and the full guide. Vosk small scores 60.0%, 35.2% and 31.2%.
  • Apply the guide to both texts. On one side only, Whisper small’s WER rises to 53.3%.
  • A rule may remove differences in writing, never in meaning. Compare WERs only when they used the same guide on the same calls.

You can now compare transcripts fairly, with one number for each recogniser. But one number still counts a wrong “the” like a lost “not”. Next, you count the errors on the words that matter to Larkfield, and you score the recognisers on each kind of call.

Next: The errors that matter