Word error rate, step by step
- Lesson 4
- 35 min
- Module 2: Measure before you choose
Where we are
In Module 1, the setup check printed a word error rate of 5.5% for one call. We used the number, but we did not see what it counts. In this lesson, we count the mistakes in a transcript by hand, the way a teacher marks a dictation, and then with the project and with jiwer.
Lesson 4 of 36 · Speech AI: Transcribe, Understand and SpeakWord error rate, step by step
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Turn “this transcript is close” into a number that you can compare. Count the wrong, missing and extra words against the reference, and divide by the length of the reference.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It counts the errors of one sentence by hand and with code, and builds the alignment table. Then it checks the result with jiwer and scores two whole calls. It needs no model download and no account. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- What a reference transcript is: the exact words of a call, written by a person or, for Larkfield’s synthetic calls, by the script. See Play one call and read its reference.
- How to run a command in a terminal, with the project’s virtual environment active.
What you need on your computer (only for the local route; the notebook needs none of it)
- Your project folder
larkspeechfrom Set up the speech project, with the course data inlarkspeech/data.speech infoworks in it. - The labs repository
nextia-learning-labsnext tolarkspeech. This lesson copies four files from the snapshotend-of-m02, and installs one new package.
Tested on macOS with Python 3.12 and jiwer 4.0.0. The Windows (PowerShell) commands were not tested.
How a teacher marks a dictation
Section titled “How a teacher marks a dictation”Think of a dictation at school. The teacher reads a short text aloud, and each pupil writes down what they hear. Then the teacher marks each paper against the text in the book.
The teacher finds three kinds of mistake. A pupil wrote a wrong word: “the” where the book says “a”. A pupil left out a word: “not” is missing. A pupil added a word that the teacher never said: “please” at the end. The mark is the number of mistakes for the length of the text. Four mistakes in a text of 40 words is better than four mistakes in a text of 10 words.
A speech recogniser is a pupil that takes a dictation all day. The caller speaks, and the recogniser writes down what it hears. To mark its work, we need the text in the book: the reference transcript, the words that were really said. Then we mark the recogniser in the same way as the teacher.
The problem: “close” is not a number
Section titled “The problem: “close” is not a number”Listen to one sentence from a Larkfield call. The agent has asked for the order number, and the caller answers. This is call C-0056, a synthetic call on a landline with background talk.
The voice reads the order number 384529 in pairs. The picture shows the same sound as a waveform, with each word under the time when the voice said it.
Text description of the diagram
The picture shows 2.6 seconds of sound, from 16.4 to 19.0 seconds into the call. The blue bars show how loud the sound is at each moment. Five bursts stand out from a low, steady noise: the background talk of this call. Under the bursts, a box marks each word: “Yes”, “it’s”, “thirty-eight”, “forty-five” and “twenty-nine”. “it’s” is the shortest word; each number takes about half a second.
The course recorded what three recognisers wrote for this whole call. Whisper tiny and Whisper small are two sizes of the same open model. Vosk small is an older kind of recogniser. From sound to text explains both. Here is the part for this sentence:
| Who | Text |
|---|---|
| Reference | Yes, it’s 384529. |
| Whisper tiny | Yes, it’s 384529. |
| Whisper small | Yes, it’s 384529. |
| Vosk small | yes it’s thirty eight forty five twenty nine |
Grace, the support lead, asks a simple question: “Which recogniser is best for our calls?” For this one sentence, you can read the three texts and judge: all three got the number, and Vosk wrote it as words. But Larkfield’s 60 test calls have about 7,000 words, and the course compares many recognisers and settings. Nobody can read all of that and keep a fair score in their head. Grace needs one number per recogniser, counted the same way every time. Then she can compare recognisers, and see if a change makes things better or worse.
The idea: count three kinds of mistake
Section titled “The idea: count three kinds of mistake”We count exactly the teacher’s three mistakes, word by word:
- A substitution (S): the model wrote a different word in the place of a reference word.
- A deletion (D): a reference word is missing from the model text.
- An insertion (I): the model wrote a word that is not in the reference.
Then we divide by N, the number of words in the reference:
WER = (S + D + I) / N
This is the word error rate, or WER. The name says what it is: the rate of errors, counted in words. A WER of 0 means that every word is right. A WER of 0.25, or 25%, means one error for every four reference words.
A tiny worked example
Section titled “A tiny worked example”Later in the same call, the caller says: “I do not want a refund, I want a new one.” Suppose that a model wrote “I do want a refund I want a new new one”. This pair is constructed: we wrote it to show the three kinds of mistake in one sentence.
We compare the words in lower case and without punctuation, so that “refund,” and “refund” are the same word. The next lesson explains this step in detail. Then we put the two texts one above the other and pair the words:
| Reference | i | do | not | want | a | refund | i | want | a | new | one | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | i | do | want | a | refund | i | want | a | new | new | one | |
| Error | D | I |
“not” is missing, so there is one deletion. The model wrote “new” twice, so there is one insertion. There is no substitution. The reference has 11 words. So the WER is (0 + 1 + 1) / 11 = 2 / 11 = 18.2%.
An 18.2% error rate sounds like “mostly right”. But the deleted word is “not”, and the model text now says the opposite of what the caller wants. Keep this in mind: The errors that matter comes back to it.
Align the words first
Section titled “Align the words first”In the table above, you paired the words by eye. A computer needs a rule for it. The rule is: choose the pairing that needs the fewest changes. This pairing is called the alignment. The fewest number of changes is the edit distance, or Levenshtein distance, after the scientist who described it. So S + D + I is always the edit distance between the two lists of words.
How the alignment is found
Section titled “How the alignment is found”Take a shorter pair. The reference is “i do not want a refund”, and the model wrote “i do want the refund please”. The computer fills a table. Each cell holds the fewest changes that turn the first reference words (the rows) into the first model words (the columns).
| – | i | do | want | the | refund | please | |
|---|---|---|---|---|---|---|---|
| – | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
| i | 1 | 0 | 1 | 2 | 3 | 4 | 5 |
| do | 2 | 1 | 0 | 1 | 2 | 3 | 4 |
| not | 3 | 2 | 1 | 1 | 2 | 3 | 4 |
| want | 4 | 3 | 2 | 1 | 2 | 3 | 4 |
| a | 5 | 4 | 3 | 2 | 2 | 3 | 4 |
| refund | 6 | 5 | 4 | 3 | 3 | 2 | 3 |
Each cell takes the smallest of three moves. A move down from the cell above adds a deletion. A move right from the cell on the left adds an insertion. A move along the diagonal costs nothing for the same word and one change for a different word. The first row and the first column count up from 0, because you must insert or delete every word.
The bottom-right cell holds the answer: 3 changes. The bold cells show the path back to the start. From “do” to “not”, the path goes down: “not” is deleted. At “a” and “the”, the path goes along the diagonal with a cost of 1: a substitution. At the end, it goes right: “please” is inserted. So S = 1, D = 1, I = 1, and the WER is 3 / 6 = 50%.
Two alignments, one WER
Section titled “Two alignments, one WER”Sometimes two pairings need the same number of changes. The project’s own example is “the cat sat on the mat” against “the cat sit on mat today”. One alignment is three substitutions: “sat” with “sit”, “the” with “mat”, and “mat” with “today”. Another is one substitution (“sat” with “sit”), one deletion (“the”) and one insertion (“today”). Both have 3 changes, so the WER is 3 / 6 = 50% either way. The counts of S, D and I can differ between tools, but the WER is the same.
The four steps of a WER, from the two texts to the number:
Text description of the diagram
Two boxes start the diagram: the reference, what the caller said, written by a person; and the model transcript, what the recogniser wrote. Both lead to step 1: normalise both texts with the same written guide. Step 2 aligns the words, with the fewest changes. Step 3 counts the substitutions, deletions and insertions. Step 4 divides their sum by N, the number of words in the reference.
Explore: mark a transcript
Section titled “Explore: mark a transcript”Pick an example, and read the alignment from left to right. In each column, the top row is the reference word and the middle row is the model word. The bottom row is the kind of error. Three stars (***) mean that one side has no word there.
Explore
Mark a transcript word by word
Predict first: In “Only the normalisation differs”, the model wrote every word that the caller said, as words instead of digits. Can its WER be over 100%?
- Reference
- I do not want a refund, I want a new one.
- Model transcript
- I do want a refund I want a new new one
Normalisation: Basic: lower case, no punctuation
A constructed pair: we wrote both texts to show one idea.
- match: reference i, model i
- match: reference do, model do
- deletion: reference not, model nothing
- match: reference want, model want
- match: reference a, model a
- match: reference refund, model refund
- match: reference i, model i
- match: reference want, model want
- match: reference a, model a
- insertion: reference nothing, model new
- match: reference new, model new
- match: reference one, model one
- S: substitution (a word became another word)
- D: deletion (a reference word is missing)
- I: insertion (the model added a word)
- Errors: S + D + I
- 0 + 1 + 1 = 2
- Reference words N
- 11
- WER = errors / N
- 2 / 11 = 18.2%
0 substitutions, 1 deletion, 1 insertion: 2 errors for 11 reference words, so the WER is 18.2%. The same pair with the other levels: none 36.4%, full 18.2%.
Text description of the explorer
The explorer shows two texts, the reference and the model transcript, then the alignment: one column per pair of words. Each column has the reference word on top, the model word under it and an error letter at the bottom. S is a substitution, D a deletion and I an insertion. The readout shows the errors S + D + I, the number of reference words N, and the WER. In this explorer, both texts are in lower case without punctuation. “One of each error” has 1 deletion (“not”) and 1 insertion (“new”) for 11 reference words: 18.2%. In “Only the normalisation differs”, the reference is “yes it’s 742155” and the model wrote “yes it is seven four two one five five”. That is 2 substitutions and 6 insertions for 3 reference words, so 8 / 3 = 266.7%. In call C-0056 with Vosk small, the reference is “yes it’s 384529” and Vosk wrote 8 words: the number became six words. That is 2 matches, 1 substitution and 5 insertions, so 6 / 3 = 200%. In call C-0052 with Whisper tiny, “280917” became “2091 17”: 1 substitution and 1 insertion in 6 words give 33.3%. In call C-0056 with Whisper small, “a refund” became “to refund”: 1 substitution in 11 words, 9.1%.
Why the WER can be over 100%
Section titled “Why the WER can be over 100%”The WER divides by the reference words, but nothing limits the insertions. A model that writes many words where the caller said few gets a WER over 100%. In “Only the normalisation differs”, the model wrote eight words for a reference of three. Vosk small wrote eight words for “Yes, it’s 384529”, because it wrote the number as six words, and got 200%.
Silence can do the same. Suppose that a caller says only “Yes.” and then waits, and the model writes “Yes. Thank you for watching.” This pair is constructed, but some recognisers really write text like this in silence. Four insertions for one reference word give a WER of 400%. A WER over 100% usually means that the model invented text, often in silence or in music on hold. Module 3 shows how to find the parts of a call where nobody speaks.
Character error rate
Section titled “Character error rate”Sometimes a word is almost right. A model writes “teracotta” for “terracotta”: one letter is missing. The WER counts the whole word as a substitution. In “the terracotta trio arrived broken”, that is 1 error in 5 words, a WER of 20%.
The character error rate, or CER, uses the same alignment on letters instead of words. Spaces count as characters too. The same sentence has 34 characters and one missing letter, so the CER is 1 / 34 = 2.9%. CER is kinder to small spelling errors. It is useful for names, for spelling, and for languages that do not put spaces between words. WER is the usual score for transcripts, so this course reports both, and leads with WER.
Larkfield: compute it with the project
Section titled “Larkfield: compute it with the project”The snapshot end-of-m02 adds two commands to the project. speech wer marks one pair of texts and shows the alignment. speech score marks the recorded transcripts of a recogniser on all the test calls.
-
In your folder
larkspeech, with the virtual environment active, copy the files of this module from the snapshotend-of-m02:Terminal window S=../nextia-learning-labs/C41/snapshots/end-of-m02cp "$S/larkspeech/scoring.py" larkspeech/cp "$S/larkspeech/commands/wer.py" larkspeech/commands/cp "$S/tests/test_score.py" tests/cp "$S/requirements.txt" .Terminal window $S = "..\nextia-learning-labs\C41\snapshots\end-of-m02"Copy-Item "$S\larkspeech\scoring.py" larkspeech\Copy-Item "$S\larkspeech\commands\wer.py" larkspeech\commands\Copy-Item "$S\tests\test_score.py" tests\Copy-Item "$S\requirements.txt" .The new
requirements.txtadds one line,jiwer==4.0.0. The project needs no other change:speechfinds a new command by its file inlarkspeech/commands/. -
Install the new package. pip also installs the two packages that jiwer needs, click and rapidfuzz:
Terminal window pip install -r requirements.txt -
Run the tests. Three new tests check the scores and the command:
Terminal window pytest -q............... [100%]15 passed in 12.82sThe time changes from one computer to another.
-
Mark the project’s example pair:
Terminal window speech wer "the cat sat on the mat" "the cat sit on mat today"REF: the cat sat on the matHYP: the cat sit on mat todayS S S3 substitutions, 0 deletions, 0 insertions, 6 reference words: WER 50.0%, CER 36.4% (normalisation: full)REFis the reference andHYPthe model text (the “hypothesis”). The tool chose the alignment with three substitutions. -
Mark the refund pair. Put each text in quotes:
Terminal window speech wer "I do not want a refund, I want a new one." "I do want a refund I want a new new one"REF: i do not want a refund i want a *** new 1HYP: i do *** want a refund i want a new new 1D I0 substitutions, 1 deletions, 1 insertions, 11 reference words: WER 18.2%, CER 21.6% (normalisation: full)This is your result from the table: 2 / 11 = 18.2%. The guide also wrote “one” as the digit 1 on both sides. The next lesson explains why.
-
Score the recorded Whisper tiny transcripts of the 60 test calls.
speech scoreuses Whisper tiny when you give no method:Terminal window speech scorewhisper-tiny, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongall 60 7334 20.7% 10.8% 20 of 52 53 of 60 -
Score two more recognisers. These are recorded transcripts too, so the commands take a few seconds:
Terminal window speech score --method whisper-smallspeech score --method vosk-smallwhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongall 60 7334 8.6% 4.2% 8 of 52 41 of 60vosk-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongall 60 7334 31.2% 20.1% 35 of 52 51 of 60
speech score does not average the WER of each call. It adds the errors of all 60 calls and divides by all 7,334 reference words. A long call then counts more than a short one, as it should: it has more words to get right.
- WER
- CER
Text description of the chart
A column chart with six columns, on an axis from 0 to 40%. The first three columns are the WER: Whisper tiny 20.7%, Whisper small 8.6% and Vosk small 31.2%. The last three are the CER: Whisper tiny 10.8%, Whisper small 4.2% and Vosk small 20.1%. For each recogniser, the CER is about half of the WER, because most wrong words still have many right letters.
These numbers are on Larkfield’s synthetic calls. Open text-to-speech voices read clean scripts, so these calls are easier than real phone calls. The errors that matter compares them with real recorded speech.
jiwer agrees. jiwer is a common Python package for WER and CER. It does not normalise the text for you, so give it texts that already went through the guide. In a Python shell or a notebook:
import jiwerout = jiwer.process_words("i do not want a refund", "i do want the refund please")print(out.substitutions, out.deletions, out.insertions, out.wer)1 1 1 0.5These are the S, D, I and WER of the table above. Use jiwer when you need a quick number. Use the project’s commands when you compare recognisers, because they also apply the guide and the slices.
When to use WER, and what it costs
Section titled “When to use WER, and what it costs”Use WER to compare recognisers, or settings of one recogniser, on the same calls with the same reference transcripts. The same rule on the same calls makes the numbers comparable.
WER has limits that the next two lessons fix. First, it depends on how you write the texts: digits or words, capital letters, punctuation. Two teams can report very different WERs for the same transcripts. Second, every word counts the same. A wrong “the” and a missing “not” each add one error, but only one of them changes what the caller wants.
The cost of a WER is the reference transcripts. Somebody must write down exactly what was said in each call, and check it. Larkfield’s calls are synthetic, so the scripts are the references. For real calls, a person must listen and type, often several times longer than the call itself. That is why a test set is small, and why you keep it for the decisions that need it. The machine learning course explains this for any model.
Guided practice: WER by hand and with code
Section titled “Guided practice: WER by hand and with code”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The setup cell ends with a
Ready:line. - In section 1, run the alignment cell. You see one line per word:
ok,del,subandins. - Run the counting cell. You see
sub1,del1,ins1 andref_words6, and a WER of 0.5. - In section 2, run the table cell. Compare it with the table on this page. The bottom-right number is 3.
- In section 3, run the jiwer cell. jiwer and the helper give the same WER and CER.
- In section 5, predict the WER of the “Yes.” example. Then run the cell. You see a WER of 4.0.
- In section 6, predict which is larger for “teracotta”, the WER or the CER. Then run the cell.
Your turn: one whole call
Section titled “Your turn: one whole call”Call C-0196 is a cancelled order on a VoIP line with street noise. In the notebook, section 8, normalise its reference and the recorded Whisper tiny transcript with the guide. Then count the word steps that are not ok with cp.align, and divide by the reference words. Do the same with the letters for the CER. Before you run it, predict: will this call’s WER be above or below Whisper tiny’s 20.7% on all the test calls?
Show the answer
call = calls["C-0196"]ref = cp.normalise(reference(call), "full", call["language"])hyp = cp.normalise(tiny["C-0196"], "full", call["language"])ops = cp.align(ref.split(), hyp.split())errors = sum(op != "ok" for op, _, _ in ops)task_wer = errors / len(ref.split())char_ops = cp.align(list(ref), list(hyp))task_cer = sum(op != "ok" for op, _, _ in char_ops) / len(ref)print(round(task_wer, 3), round(task_cer, 3))0.096 0.047The WER of this call is 9.6%, and its CER is 4.7%. It is an English call, and Whisper tiny makes far fewer errors in English than in French on these calls. So one call can be much better or much worse than the score of all 60 calls.
Common mistake
The reference and the model text the wrong way round
Symptom: Tomás scores the “Yes.” example and gets 80%. Priya gets 400% for the same two texts. Neither tool shows an error.
ref_short = cp.normalise("Yes.")hyp_long = cp.normalise("Yes. Thank you for watching.")print(cp.wer(ref_short, hyp_long), cp.wer(hyp_long, ref_short))4.0 0.8Cause: the WER divides by the number of reference words. Tomás gave the model text first. So the tool took “yes thank you for watching” as the reference, and the four extra words became four deletions out of five words.
Fix: always pass the reference first, as in jiwer.wer(reference, hypothesis) and speech wer REFERENCE HYPOTHESIS. Name your variables ref and hyp, so that a swap is easy to see.
Consequence in AI work: a swapped WER is too kind to a model that invents text, which is the error that hurts most in silence. You might choose the wrong recogniser, or miss the invented words until a customer reads them.
Knowledge check
Section titled “Knowledge check”Check your understanding
The reference is “my order is late” and the model wrote “my order is very late again”. Count S, D and I, and compute the WER.
Show the answer
Align the words: “my”, “order”, “is” match. The model added “very” before “late”, and “again” after it. So S = 0, D = 0 and I = 2. The reference has 4 words, so the WER is (0 + 0 + 2) / 4 = 0.5, or 50%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Explain to Omar how a WER can be over 100%, and what it often means on a phone call.
Show the answer
The WER divides the errors by the number of reference words. Substitutions and deletions cannot be more than the reference words, but insertions have no limit. If the model writes many words where the caller said few, the errors can be more than the reference words. Then the WER goes over 100%. Vosk wrote 8 words for “Yes, it’s 384529” (the number as six words) and got 200%. On a phone call, a WER over 100% often means that the model invented text, for example in silence or in music on hold.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A transcript of “I do not want a refund, I want a new one” has a WER of 18.2%. Grace says that this is good enough to start a refund. What do you tell her?
Show the answer
The WER counts every word the same. Here the two errors are a missing “not” and an extra “new”, so the text says “I do want a refund”. The meaning is the opposite of what the caller said. A low WER does not tell you that the words of the decision are right. Before Larkfield acts on a transcript, it must check those words apart: negations, order numbers and product names. The next lessons show how.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A recogniser is marked like a dictation, against a reference transcript: substitutions (S), deletions (D) and insertions (I).
- WER = (S + D + I) / N, where N is the number of reference words. The alignment is the pairing with the fewest changes, so S + D + I is the edit distance.
- The WER has no upper limit, because insertions have none: Vosk got 200% on one sentence, and invented text in silence does the same.
- The CER uses the same alignment on letters. It is kinder to small spelling errors.
- On Larkfield’s 60 synthetic test calls: Whisper tiny 20.7%, Whisper small 8.6%, Vosk small 31.2% WER. Always pass the reference first.
You can now mark a transcript and get one number per recogniser. But the same transcript gave 266.7% in the explorer, although the model wrote every word that the caller said. Next, you see how the way we write numbers, capital letters and punctuation changes the score, and you write a guide that both texts follow.