Watch the system
- Lesson 36
- 30 min
- Module 9: Run it in production, safely
Where we are
Our system is safe and runs every day. It will not stay good by itself: products, phone providers and callers change. In this lesson, we watch error by slice over time, IVR containment and hang-ups, and review samples.
Lesson 36 of 36 · Speech AI: Transcribe, Understand and SpeakWatch the system
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Keep the speech system good after launch. Watch what comes in and what happens every day, for free. Review a sample of transcripts every week to measure the error rate by slice. Learn how much a small sample moves by chance, so you can tell a real change from noise. Know the changes that will come: a new phone provider, a new region and new product names.
NotebookDo this lesson in a notebook
The notebook scores Whisper small call by call on the 110 validation and test calls. It draws many weekly review samples to show how much a sample of 10, 30 or 100 calls moves by chance. Then it builds eight weeks with a new phone provider and asks you to find the change. It needs no model download and no account. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- WER by slice, from The errors that matter and Accents, noise and two languages.
- Containment and hand-overs, from Test the IVR like a product.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your folder
larkspeechwith the files of Batch and streaming.pytest -qprints41 passed.
Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.
The wallboard and the Friday listen
Section titled “The wallboard and the Friday listen”Walk into any call centre, and you see a big screen on the wall. It shows the calls waiting, the longest wait, and the callers who hung up today. It costs nothing to keep up to date: the phone system counts these things by itself.
On Friday afternoon, the supervisor does something slower. She listens to a few calls from each agent and scores them. That costs her time, so she cannot do it for every call. And once a month, the manager looks at the bill.
Larkfield’s speech system needs the same three rhythms. The daily counts are free. The weekly listening costs review time. The monthly bill closes the loop.
The problem: one number hides the change
Section titled “The problem: one number hides the change”Grace asks for one number: “How good are the transcripts this week?” Priya could give her the WER of a weekly sample. But one number hides two things.
It hides the slices. On Larkfield’s 110 synthetic validation and test calls, Whisper small has a WER of 7.9%. On the English calls it is 4%, and on the French calls 14.1%. If next month brings more French callers, the average rises, although the recogniser did not change at all.
It moves by chance. A person can correct only a few transcripts a week. A small sample gives a different WER every week, even when nothing changes. Then a real change looks like noise, and noise looks like a change.
The idea: watch inputs, outcomes, quality and cost
Section titled “The idea: watch inputs, outcomes, quality and cost”The diagram groups what to watch by how often, and by what it costs.
Text description of the diagram
Four groups, from left to right. Every day, for free, what comes in: the mix of calls by line, language and call length. Also new words, such as product names that the recogniser never saw. Every day, for free, what happens: containment, hand-overs and hang-ups of the IVR. Also the waiting: seconds to the answer, and the time to the first sound back. Every week, at the cost of review time: a person corrects 30 transcripts. From them, you compute the WER by slice (line, language and product). Every month: the cost per audio hour and per IVR call.
Some words:
- Drift means that the calls change after launch. The calls that come in are no longer like the calls you tested on. The MLOps course explains it in Data and concept drift.
- A slice is a group of calls that share one property: a line, a language, a product. You compute the WER of each slice apart.
- A review sample is a small set of calls that a person checks by hand. The corrected transcript becomes the reference, and the WER compares the recogniser’s text with it. The labels arrive late, after a person has worked: see Delayed labels.
- Containment is the share of IVR calls that end without a person. A hand-over sends the caller to a person. A hang-up is a caller who leaves before the end.
Watch the inputs first
Section titled “Watch the inputs first”Inputs are the cheapest signal and the fastest. The phone system knows the line of every call, and the recogniser finds the language. Count them every day. A new phone provider shows up in the line counts on its first morning, long before any review.
A tiny worked example: how much does a sample move?
Section titled “A tiny worked example: how much does a sample move?”Grace has 60 minutes of team-lead listening a day, at US$0.75 a minute (the course’s assumptions). Say that a person needs about 2 minutes to correct the transcript of one call: that is our assumption. One hour a week then buys 30 reviewed calls, for US$45.
How much does the WER of 30 calls move by chance? The course took the per-call results of Whisper small on the 110 calls, and drew 2,000 samples of each size (with replacement). The table shows the range that holds 90% of the samples.
| Calls in the sample | Lowest 5% of samples | Median | Highest 5% of samples |
|---|---|---|---|
| 10 | 4.4% | 7.6% | 12.2% |
| 30 | 5.9% | 7.9% | 10.4% |
| 100 | 6.8% | 7.9% | 9.2% |
Read the middle row. With 30 calls a week, the WER lands anywhere between about 6% and 10%, although the recogniser and the calls are the same every week. So a week at 10% is not news.
- Median sample WER
- 90% of samples fall between the dashed lines
Text description of the chart
A line chart with the sample size from 10 to 100 calls on the horizontal axis. The median stays near 7.6% to 7.9%. The band that holds 90% of the samples is wide for small samples and narrows slowly. It is 4.4% to 12.2% for 10 calls, 5.4% to 10.8% for 20 and 5.9% to 10.4% for 30. It is 6.3% to 9.8% for 50, and 6.8% to 9.2% for 100 calls.
The band narrows slowly. Going from 30 to 100 calls more than triples the review time. The band only shrinks from about 4.5 points to about 2.4 points.
Larkfield: a new phone provider
Section titled “Larkfield: a new phone provider”Now a change. Larkfield moves part of its lines to a new phone provider that uses the G.726 codec. On the 110 calls, Whisper small has a WER of 9.4% on the G.726 line, against 7.9% on the landline and 7.1% on the mobile line. Most of these G.726 calls are French, so this slice mixes the line and the language. Today about 14% of the calls use it. From week 5, 60% of the calls will.
The chart is a constructed example: eight weeks that we built from the real per-call results. Each week, 30 calls are drawn at random. Weeks 1 to 4 have today’s mix of lines; weeks 5 to 8 have the new one. The dashed lines are the 90% band for 30 calls. Because most G.726 calls in our set are French, this drill really adds French calls. Read it as practice in telling a change from noise, not as a fact about the G.726 codec.
- WER of the 30 reviewed calls
- Normal range for 30 calls (90%)
Text description of the chart
A line chart over eight weeks. The weekly WER of 30 reviewed calls is 10.5%, 7.6%, 7.7% and 7.3% in weeks 1 to 4. It is 7.4%, 10.4%, 8.8% and 8.8% in weeks 5 to 8. The normal range for 30 calls runs from 5.9% to 10.4%. Week 1, before any change, is just above the range. The expected WER rises only from 8.0% to 8.8% with the new mix.
What does the chart teach?
- The change is hidden in the noise. With the new mix, the expected WER moves from 8.0% to 8.8%. The weekly samples jump by more than that.
- Week 1 crosses the line with no change at all. A 90% band is crossed in about 1 week in 10 by chance. One point outside the band is not an alarm.
- The inputs show it at once. In weeks 1 to 4, 7, 3, 4 and 5 of the 30 reviewed calls were on the G.726 line. In weeks 5 to 8, 17, 18, 16 and 18 were. The daily line counts, for all calls and for free, would show it on day one.
The right reaction: when an input changes, review more calls from the new slice for two weeks. Measure that slice’s WER on its own, and decide.
The next two changes: a new region and new products
Section titled “The next two changes: a new region and new products”Omar plans a new French-speaking region. On the test calls, Whisper small makes 15.1% errors in French, against 3.1% in English. More French calls raise the average, so watch the WER by language, not only the total.
The region also brings four new products. Whisper small missed 41 of 60 product names on the test calls; with Larkfield’s vocabulary in its prompt, it missed only 10. A product name that is not in the prompt is back to the first case. So a new product is a change to the system: add it to the vocabulary prompt on the day it goes on sale. Then watch how often the call records name an unknown product. The course did not build that count.
IVR outcomes: count them every day
Section titled “IVR outcomes: count them every day”The IVR writes a call record for every call, so its outcomes cost nothing to count. On its 60 recorded test calls, the IVR contained 35% of the calls, sent 63% to a person and chose no wrong intent. Every hand-over costs about US$4 of agent time, and a caller who hangs up about US$8 (assumptions).
Hang-ups cannot be measured in the replay. The recorded callers never hang up, whatever the IVR does. On the real line, count them every day: a rise in hang-ups is often the first sign that a prompt confuses people.
Slices of IVR calls are small. The same IVR contained 25% of the 36 landline test calls, and 50% of the 12 landline calls in the hidden set. The hidden calls were run once, with an earlier version of the IVR, so the IVR changed a little too. But with so few calls, the slices swing a lot.
Captions and accessibility
Section titled “Captions and accessibility”Transcripts help more than Grace’s team. A deaf or hard-of-hearing customer may prefer a written record of the call, and live captions during a video call. Some callers cannot use a voice menu at all. So the IVR keeps the keypad for every choice. A text option, such as a link sent by SMS, is another way in.
If Larkfield offers captions or written records, their quality needs watching too. Put caption calls into the review sample, and measure the WER of that slice. The course did not build captions.
Guided practice: read the slices
Section titled “Guided practice: read the slices”The project already has the commands. speech score scores recorded transcripts by slice, and speech ivr replay counts the IVR outcomes by slice.
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Score Whisper small’s recorded transcripts of the test calls by line:
Terminal window speech score --method whisper-small --by linewhisper-small, test calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wronglandline 43 5298 8.6% 4.0% 7 of 38 31 of 43mobile 10 1207 5.9% 3.1% 0 of 8 5 of 10voip_g726 7 829 12.2% 7.1% 1 of 6 5 of 7 -
Replay the 60 IVR test calls, with the outcomes by line. It takes about 1.5 minutes:
Terminal window speech ivr replay --by lineThe first lines of the output on the course’s computer:
IVR replay: 60 recorded test calls, Whisper base, end of turn 0.8 sline calls passed contained wrong intent to a person s to answerlandline 36 23 25% 0% 72% 51.2mobile 11 7 45% 0% 55% 54.3voip_g726 13 9 54% 0% 46% 50.2A list of the failed calls follows. Your numbers may differ slightly. Whisper on 4 CPU threads does not write exactly the same text every time, so a wrong intent or a second can change. The passed and contained columns stayed the same in our runs.
The G.726 line has the worst WER in step 2 and the best containment in step 3. How can both be true? Look at the kinds of calls on each line. On the landline, 15 of the 36 test calls are meant to end with a person. They are returns, silent callers, callers who press 0 or ask for a person, and unclear callers. On the G.726 line, only 2 of 13 are. So the landline’s containment is low for reasons that have nothing to do with the line. A slice carries a mix of call kinds, and the mix can explain the difference.
Your turn: the same line, other calls
Section titled “Your turn: the same line, other calls”Score the same recorded transcripts on the validation calls, by line. Before you run it, predict: is the G.726 line still the worst?
Show the answer
speech score --method whisper-small --by line --split validationwhisper-small, validation calls, phone audio, normalisation fullslice calls words WER CER order nos wrong products wronglandline 27 3258 6.7% 3.5% 4 of 20 16 of 27mobile 14 1641 8.0% 5.3% 1 of 11 7 of 14voip_g726 9 1086 7.4% 3.7% 0 of 7 5 of 9No. On the validation calls, the G.726 line has 7.4%, better than the mobile line. Same recogniser, same line, other calls: 12.2% on 7 test calls, 7.4% on 9 validation calls. With so few calls, one or two hard calls decide the number, and the mix of languages decides more. Most VoIP test calls are French; inside English, the three lines differ by less than 1 point (the lesson Accents, noise and two languages). Before you act on a slice, count its calls, and join several weeks when the slice is small.
Common mistakes
Section titled “Common mistakes”Common mistake
An alarm on one week's number
Symptom: the WER of the weekly sample jumps from 7.6% to 10.5%. The team rolls back the last change, and nothing improves.
Cause: with 30 calls, the sample WER moves between about 6% and 10% by chance. In the constructed example, such a jump happened in week 1, before anything changed.
Fix: draw the normal range for your sample size. Act on two or three weeks in a row outside it, or on a change in the inputs. Then review more calls from the slice that changed.
Common mistake
Logs that become a privacy leak
Symptom: to debug the IVR, the team logs every recognised turn and every key press. The logs now hold order numbers, names and maybe card digits, kept for years.
Cause: monitoring data is data too. It needs the same rules as the recordings of the last two lessons.
Fix: log counts, call IDs and states; keep text only in the review sample, redacted. The MLOps course explains how in Privacy-aware observability.
When to use each signal
Section titled “When to use each signal”- Every day: the mix of lines, languages and call lengths; containment, hand-overs, hang-ups and waiting times. Free, fast, and the first sign of drift.
- Every week: a reviewed sample of about 30 calls, with the WER by slice over several weeks. More calls from a slice that just changed.
- Before a change: a new provider, region or product. Test on a sample of the new calls first, as in Accents, noise and two languages.
- Every month: the cost per audio hour and per IVR call, as in Batch and streaming. The MLOps course connects quality and cost in Quality and costs.
Knowledge check
Section titled “Knowledge check”Check your understanding
The weekly review sample of 30 calls had a WER of 7.4% last week and 10.4% this week. Grace asks: 'Did the recogniser get worse?' What do you answer, and what do you check first?
Show the answer
Not necessarily. On Larkfield’s synthetic calls, 90% of 30-call samples fall between about 5.9% and 10.4% with no change at all. So 10.4% is right at the top of the normal range. First check the inputs: they are free and counted for every call. Did the mix of lines, languages or products change? Then look at the WER by slice, and over more than one week. If one slice changed, review more calls from it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the test calls, the G.726 line has the worst WER (12.2%) and the IVR's best containment (54%). Tomás concludes that the G.726 line is good for the IVR. What is wrong?
Show the answer
The slices hold different kinds of calls. On the landline, 15 of the 36 IVR test calls are meant to end with a person. On the G.726 line, only 2 of 13 are. So the landline’s containment is low for reasons that have nothing to do with the line. The slices are also small: 13 IVR calls on G.726, and only 7 test calls for the WER. On the 9 validation calls, that WER drops to 7.4%. Compare like with like: the same kind of call on each line, and more calls.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run speech score --method whisper-small --by language. Which slice should Omar's new French-speaking region worry about, and why does the total WER rise even if the recogniser does not change?
Show the answer
The output shows English 3.1% (34 calls) and French 15.1% (26 calls). The French slice makes almost five times more errors. The total WER is a mix of the slices, weighted by their words. If the new region brings more French calls, the French slice weighs more, and the total rises although the recogniser is the same. Watch the WER by language, and the share of French calls every day.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Watch in three rhythms. Every day, for free: the inputs (lines, languages, length) and the IVR outcomes (containment, hand-overs, hang-ups, waiting). Every week: a reviewed sample with the WER by slice. Every month: the cost.
- A sample moves by chance. With 30 reviewed calls of Whisper small on Larkfield’s synthetic calls, 90% of weekly WERs fall between about 5.9% and 10.4%.
- Drift comes from a new phone line, a new region and new product names. In the constructed example, a new provider moved the expected WER from 8.0% to 8.8%. The noise hid it, but the input counts showed it plainly.
- Small slices swing: the G.726 line had 12.2% on 7 test calls and 7.4% on 9 validation calls.
- Captions and written records help callers who cannot use a voice line; put them in the review sample too.
You have built and run the whole system: from audio as data to a phone line that answers by itself, safely. The case studies now test these methods on real recordings. The first one chooses a recogniser for two languages.