The errors that matter
- Lesson 6
- 40 min
- Module 2: Measure before you choose
Where we are
In the last two lessons, we learned to compute a fair word error rate with one written guide. But one number counts a wrong “the” the same as a lost “not”, and it hides the calls where a recogniser is weak. In this lesson, we count the errors on the words that matter to Grace, and we score the recognisers on each kind of call, the way a waiter checks the one word in an order that the kitchen must not miss.
Lesson 6 of 36 · Speech AI: Transcribe, Understand and SpeakThe errors that matter
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Find out if a transcript is good enough for Larkfield’s decisions. Count the errors on order numbers, product names, dates and negations apart from the WER. Then score each recogniser on each kind of call.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It checks the important words of one call, and counts them on the 60 test calls for three recognisers. It scores Whisper small by slice, and shows a check that passes when it should not. Last, it compares two settings with almost the same WER. It needs no model download and no account. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- WER, CER and the alignment, from Word error rate, step by step.
- The normalisation guide, and why both texts go through it, from Normalise before you compare.
- A slice of a test set: a group of rows that share one property. The machine learning course explains slice analysis for any model.
What you need on your computer (only for the local route; the notebook needs none of it)
- Your folder
larkspeechwith the files of the snapshotend-of-m02.speech score --helpshows the option--by.
Tested on macOS with Python 3.12. The commands are the same on Windows; they were not tested there.
A waiter’s order note
Section titled “A waiter’s order note”Think of a waiter who takes an order: “A burger with fries, no onions, and a glass of water.” He writes “burger fries onions water”. Most of the words are right. But “no” is missing, and the kitchen puts onions on the burger. The customer sends it back.
Now think of a second waiter who writes “burgr, fries, no onions, watter”. He made two spelling mistakes, more than the first waiter. But the kitchen makes exactly the right meal.
A count of wrong words says that the second waiter is worse. The kitchen says the opposite. What matters is not how many words are wrong, but which words. For a meal, “no” matters more than the spelling of “water”. For Larkfield, an order number matters more than “the”.
The problem: one number hides the words that matter
Section titled “The problem: one number hides the words that matter”Listen to a caller in call C-0052, a synthetic call on a mobile line with street noise. The agent has asked for the order number.
Whisper tiny wrote “Sure, the order number is 2091-17.” The caller said 280917. After the guide, the model’s number is 209117: one word in six is wrong, so the WER of this sentence is 16.7%. The WER of the whole call is only 8.7%. Both numbers say “mostly right”. But a phone menu that looks up order 209117 finds no order, or the wrong one.
Now listen again to the caller of call C-0056 from the first lesson of this module. This sentence comes later in the call:
| Who | Text |
|---|---|
| Reference | I do not want a refund, I want a new one. |
| Whisper small | I do not want to refund, I want a new one. |
| Whisper tiny | I do not want to be feeling forward to new work. |
Whisper small wrote “to” for “a”, and the meaning is still right. Whisper tiny kept “not”, but it lost “refund”, and nobody can tell from its text what the caller wants. The WER counts each of these wrong words as one error among thousands, the harmless “to” and the lost “refund” the same. Grace needs to know about the words that matter one by one.
The idea: count the important words apart
Section titled “The idea: count the important words apart”The waiter’s lesson gives the method. First, decide which words the decision needs. Then check those words on their own, next to the WER.
For Larkfield, these important words are:
- Order numbers: the phone menu and the agents look up the order with them.
- Product names: Grace’s team searches the calls for a product, for example to find a pattern of complaints.
- Dates: “it was delivered on the sixth of April” decides what the agent promises.
- Negations: “not”, “no”, “don’t” change the meaning of a whole sentence.
- Card numbers: they must be found so that they can be removed. Module 9 deals with them.
Each call’s labels list its important words in a field important. For call C-0052:
{'product': ['Cedar raised bed'], 'order': ['280917'], 'date': ['the sixth of April'], 'negation': ["No, I don't want it delivered at all."], 'card': []}The check has a simple rule. Normalise the model text with the full guide. Then an item is right only if all its words appear in it, in order. For a negation, only the negation words are checked (“no” and “don’t” here). Any other spelling of a product name is a miss: the reference writes “Grow Hub”, so “GrowHub” counts as a miss. A search for “Grow Hub” would not find it either.
A tiny worked example
Section titled “A tiny worked example”The notebook runs this check on Whisper tiny’s transcript of call C-0052:
hyp = tiny["C-0052"]print("WER of the whole call:", round(cp.wer(cp.normalise(reference(call)), cp.normalise(hyp)), 3))print("whole call: ", cp.important_errors(call["important"], hyp, "en"))caller_said = "Sure, the order number is 2091-17." # Whisper tiny's words for the caller's turnprint("caller's turn:", cp.important_errors({"order": call["important"]["order"]}, caller_said, "en"))WER of the whole call: 0.087whole call: {'product': {'items': 1, 'missed': 1}, 'order': {'items': 1, 'missed': 0}, 'date': {'items': 1, 'missed': 0}, 'negation': {'items': 1, 'missed': 0}, 'card': {'items': 0, 'missed': 0}}caller's turn: {'order': {'items': 1, 'missed': 1}}On the whole call, the product is missed: Whisper tiny wrote “Cedar Rays better” for “Cedar raised bed order”. But the order number is not missed. Why? The agent reads the number back (“That’s order 280917”), and Whisper tiny wrote that part right. The check finds the number there.
On the caller’s turn alone, the order number is missed. This difference matters. A human agent hears the caller and reads the number back from the order system. The phone menu of Module 7 has no agent: the caller’s words are all it gets. So the counts below, on whole calls, are kind to every recogniser. Module 5 splits a call by speaker, so that you can check the caller’s words alone.
What each error costs Larkfield
Section titled “What each error costs Larkfield”The important words are the ones that change a decision. Larkfield’s cost assumptions put a price on each kind of mistake:
Text description of the diagram
The diagram has three rows, each with three boxes from left to right. Row one: an order number is wrong, for example 209117 for 280917. The order system finds no order, or the wrong one. A person takes the call, which costs US$4.00, or the caller gets a wrong status, which costs US$15. Row two: a negation is lost, for example “do want” for “do not want”. The call record says the opposite of what the caller wants. That leads to a wrong action, such as a return started by mistake, at US$15. Row three: a product name is wrong, for example “firm wall planter” for “Fern wall planter”. A search for the product does not find the call, and Grace’s team misses a pattern of complaints.
A wrong “the” costs nothing. That is why the important words are counted apart.
Larkfield: the errors that matter on 60 test calls
Section titled “Larkfield: the errors that matter on 60 test calls”speech score already prints two of these counts: order nos wrong and products wrong. Here are the three recognisers of the first lesson again, with the dates and negations from the notebook:
| Recogniser | WER | Order numbers missed | Product names missed | Dates missed | Negations missed |
|---|---|---|---|---|---|
| Whisper tiny | 20.7% | 20 of 52 | 53 of 60 | 14 of 48 | 0 of 29 |
| Whisper small | 8.6% | 8 of 52 | 41 of 60 | 9 of 48 | 1 of 29 |
| Vosk small | 31.2% | 35 of 52 | 51 of 60 | 24 of 48 | 6 of 29 |
- Order numbers missed
- Product names missed
Text description of the chart
A column chart with six columns, on an axis from 0 to 100%. The first three columns are the order numbers missed. Whisper tiny missed 38% (20 of 52), Whisper small 15% (8 of 52) and Vosk small 67% (35 of 52). The last three are the product names missed. Whisper tiny missed 88% (53 of 60), Whisper small 68% (41 of 60) and Vosk small 85% (51 of 60).
Read the table one column at a time. Whisper small has the lowest WER and also misses the fewest order numbers. But it misses 41 of 60 product names. Larkfield’s names, such as “Fern wall planter” and “Grow Hub”, are not common English. So every recogniser writes words that it knows better: “firm wall planter”. Module 4 gives Whisper a short list of Larkfield’s words, and its product names missed fall from 41 to 10.
Vosk misses 35 of 52 order numbers. Its WER (31.2%) is only about four times Whisper small’s, but it misses more than four times as many order numbers. The WER did not show that.
Build a test set by slice
Section titled “Build a test set by slice”The WER of 60 calls is an average over very different calls. Some callers speak French; some calls come over a mobile line or a VoIP line; some have street noise. A slice is a group of test calls that share one property. Scoring by slice shows where a recogniser is weak, before a customer finds it.
Larkfield’s 60 test calls were built with these slices in mind:
| Slice | Values (test calls) |
|---|---|
| Language | English 34, French 26 |
| Line | landline 43, mobile 10 (Opus at 8 kbit/s), VoIP 7 (G.726 at 16 kbit/s) |
| Noise | quiet 30, background talk 17, street 13 |
| Accent of the caller’s voice | British English 15, American English 19, French 26 |
speech score has an option --by for each slice.
-
Score Whisper small by line:
Terminal window speech score --method whisper-small --by linewhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wronglandline 43 5298 8.6% 4.0% 7 of 38 31 of 43mobile 10 1207 5.9% 3.1% 0 of 8 5 of 10voip_g726 7 829 12.2% 7.1% 1 of 6 5 of 7 -
Score it by language:
Terminal window speech score --method whisper-small --by languagewhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wrongen 34 3988 3.1% 1.7% 2 of 28 16 of 34fr 26 3346 15.1% 7.0% 6 of 24 25 of 26 -
Try
--by noiseand--by accentyourself. Compare your output with the chart below.
- Language
- Line
- Noise
Text description of the chart
A column chart with eight columns, on an axis from 0 to 20%. By language: English (en) 3.1%, French (fr) 15.1%. By line: landline 8.6%, mobile 5.9%, VoIP 12.2%. By noise: quiet 9.6%, background talk 7.7%, street 7.4%.
Three things stand out:
- French is the hard part. Whisper small makes almost five times as many errors in French (15.1%) as in English (3.1%). Part of this comes from the French voices themselves: the course has only three French voices, and they pronounce some sounds badly. Six of the eight order numbers that Whisper small misses are in French calls.
- The VoIP line looks worse than the mobile line: 12.2% against 5.9%. But it is mostly the language again: most VoIP test calls are French, and most mobile calls are English. Inside English, the lines differ by less than 1 point (landline 3.1%, mobile 3.0%, VoIP 3.5%). Module 4 looks at the lines more closely.
- Noise is not the hard part here. The quiet calls (9.6%) score worse than the street calls (7.4%). That surprises most people. The reason is in the test set: half of the quiet calls are French (15 of 30), but only 4 of the 13 street calls. A slice can hide another slice, so look at two properties together before you draw a conclusion.
How big must a slice be?
Section titled “How big must a slice be?”One word error moves the WER of all English test calls by about 0.025 points. For all French calls, it is about 0.03 points. A small slice moves much more. The VoIP slice has only 7 calls and 829 reference words, so one word moves it by 0.12 points. One bad call can move it by several points. Treat a difference between small slices as a question to check, not as a result. If a slice matters to a decision, add more calls to it before you decide.
Our calls are easier than real speech
Section titled “Our calls are easier than real speech”All the numbers in this course so far are on Larkfield’s synthetic calls. Open text-to-speech voices read clean scripts: no one hesitates, coughs or speaks over the agent. So these calls are easier than real phone calls.
The course checked this with real speech. It ran the same Whisper models on 73 real read sentences of LibriSpeech, a public set of audiobook recordings (CC BY 4.0). It ran them as they are, and again through the same phone line as Larkfield’s calls.
- Larkfield's English calls (synthetic, phone)
- LibriSpeech (real read speech)
Text description of the chart
A column chart with three pairs of columns, on an axis from 0 to 14%. Whisper tiny: 7.0% on Larkfield’s English calls and 12.4% on LibriSpeech. Whisper base: 5.2% and 11.0%. Whisper small: 3.1% and 7.8%. Through Larkfield’s phone line, LibriSpeech gives 17.5%, 12.7% and 8.1%.
Whisper small makes 7.8% errors on real read speech, and 8.1% when the same sentences go through Larkfield’s phone line. On Larkfield’s English calls, it makes 3.1%. A recogniser that looks almost perfect on the synthetic calls is not almost perfect on real callers. So when you read a number in this course, read it as “on Larkfield’s synthetic calls”. Use it to compare recognisers and settings, which is fair because they all get the same calls. Do not use it to promise a WER to Grace for real calls. The case studies of this course use real recordings for that.
When to count important words, and what it costs
Section titled “When to count important words, and what it costs”Count the important words whenever a transcript feeds a decision: a phone menu, a call record, a search. The WER tells you about the transcript in general; the important words tell you about the decision.
The method has costs and limits:
- Somebody must list the important words for every test call, and keep the list up to date when the shop adds products. For Larkfield’s synthetic calls, the scripts list them. For real calls, a person must mark them.
- The check is only as good as its rule. “All words in order, anywhere in the call” can be fooled by an agent who repeats the number, as call C-0052 showed. The failure case below shows another trap.
- Small counts move by chance. 52 order numbers is not many. A difference of one or two between two recognisers is noise.
Guided practice: important words and slices in the notebook
Section titled “Guided practice: important words and slices in the notebook”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The setup cell ends with a
Ready:line. - In section 1, run the two cells. You see the important words of C-0052, then
missed0 for the order on the whole call and 1 on the caller’s turn. - In section 2, run the count. You see order numbers missed 20, 8 and 35 of 52.
- In section 3, run the slices. You see 3.1% for English and 15.1% for French.
- In section 4, run the negation cell. You see
missed0 and'refund' kept: False.
Your turn: order numbers by line
Section titled “Your turn: order numbers by line”In the notebook, section 5, count the order numbers that Whisper small missed on each kind of line. Build a dictionary orders_missed_by_line from the line to the number of missed order numbers. Use cp.important_errors with only the order items of each call. Before you run it, predict: the VoIP line has the highest WER. Does it also miss the most order numbers?
Show the answer
orders_missed_by_line = {}for c in test: e = cp.important_errors({"order": c["important"]["order"]}, small[c["call_id"]], c["language"]) orders_missed_by_line[c["line"]] = orders_missed_by_line.get(c["line"], 0) + e["order"]["missed"]print(orders_missed_by_line){'landline': 7, 'voip_g726': 1, 'mobile': 0}No. The landline misses the most order numbers (7), because 43 of the 60 test calls are landline calls. Compare shares, not counts: 7 of 38 order numbers on the landline (18%), 1 of 6 on VoIP (17%) and 0 of 8 on mobile. speech score --method whisper-small --by line prints both numbers. With only 6 order numbers on VoIP, one more miss would change its share by 17 points. The slice is too small to rank the lines on order numbers.
Common mistake
A check that passes when the meaning is gone
Symptom: the negation check says that Whisper tiny missed 0 of 29 negations, the best result of all three recognisers. But look at call C-0056. The caller said “I do not want a refund, I want a new one”. Whisper tiny wrote “I do not want to be feeling forward to new work”.
sentence = "I do not want a refund, I want a new one."print(cp.important_errors({"negation": [sentence]}, "I do not want to be feeling forward to new work.", "en"))print("'refund' kept:", " refund " in " " + cp.normalise("I do not want to be feeling forward to new work.") + " "){'negation': {'items': 1, 'missed': 0}}'refund' kept: FalseCause: the negation check looks only for the negation words: “not”, “no”, “don’t” and “cannot”. “not” is still in the text, so the check passes. It does not check what the “not” belongs to.
Fix: read a sample of transcripts by hand next to the counts, at least a few calls of each slice. When a decision depends on a word, such as “refund” for a refund request, add that word to the important words. Module 6 turns transcripts into call reasons, and measures that decision directly.
Consequence in AI work: a check that looks perfect gives the team false confidence. Here, a recogniser with 20.7% WER looks better on negations than one with 8.6%. A call record built on its text could not say what the caller wanted, and nobody would look, because the check passed.
Knowledge check
Section titled “Knowledge check”Check your understanding
Whisper tiny's transcript of call C-0052 has a WER of 8.7%, but it wrote the caller's order number as 2091-17 instead of 280917. Explain to Omar why the WER did not warn him, and what Larkfield should measure as well.
Show the answer
The WER counts every word the same, so one wrong order number is one error among about 130 words in the call. But the order number decides what happens next. The phone menu finds no order, or the wrong one. Then the call goes to a person (US$4.00 in the course’s assumptions), or the caller gets a wrong status (US$15). Larkfield must count the important words apart: order numbers, product names, dates and negations, next to the WER. It should check the caller’s own words, because on the whole call the agent’s read-back can hide the error.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two settings of Whisper base have almost the same WER on the 60 test calls: A (with a short list of Larkfield's words) 15.2%, B (the same model run by another program, whisper.cpp) 15.3%. A misses 14 order numbers, 18 product names and 3 negations; B misses 18, 45 and 1. Which would you choose for Larkfield's call search, and which for a phone menu that acts on what the caller says? Why?
Show the answer
The WERs differ by 0.1 points, which is noise, so the WER cannot choose. For the call search, A is much better: it misses 18 product names against 45, and the search is by product. Now take a phone menu that acts on what the caller says. A is a little better on order numbers: 14 against 18, a small difference on 52 numbers. But A misses 3 negations against 1, and a lost “not” can start a wrong action at US$15. Before you choose A for the menu, read those three negations. Also add a step where the menu confirms what it heard. The right choice depends on the decision, and the WER cannot show that.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Whisper small has a WER of 8.6% on Larkfield's test calls. Omar plans to open the phone line to French callers next month. What do you tell him, and what are the limits of these numbers?
Show the answer
The 8.6% hides a weak slice: French calls have 15.1% against 3.1% for English. Six of the eight missed order numbers are in French calls. So French callers will meet many more errors. Part of the French error comes from the few French voices in the call set. Also, all these calls are synthetic and easier than real speech: on real read speech (LibriSpeech), Whisper small makes 7.8% errors, not 3.1%. Before the French launch, Larkfield should test on real French calls, by slice, with the important words counted apart.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- One WER counts every word the same. Count the important words apart: order numbers, product names, dates, negations and card numbers. An item is right only if all its words appear in order in the normalised text.
- On Larkfield’s 60 synthetic test calls, Whisper small misses 8 of 52 order numbers and 41 of 60 product names. Vosk small misses 35 order numbers. On whole calls, the agent’s read-back can hide an error in the caller’s words.
- Score by slice: Whisper small makes 3.1% errors in English and 15.1% in French; 5.9% on mobile and 12.2% on VoIP. The line gap is mostly the language (English only: 3.1%, 3.0%, 3.5%). Small slices move by chance, and one slice can hide another.
- The synthetic calls are easier than real speech: Whisper small has 7.8% on real read speech against 3.1% on our English calls. Compare recognisers with these numbers; do not promise them.
- A check is only as good as its rule: the negation check passed on a sentence that lost its meaning.
You can now say whether a recogniser is good enough for each of Larkfield’s decisions, and for which callers. But every recogniser in this module also met parts of the calls where nobody speaks: music on hold, background talk, silence. Next, you find where the speech is, and see why that matters before any recognition starts.
Next: Speech or silence?