Skip to content
Français

Watch the system

  • Lesson 36
  • 30 min
  • Module 9: Run it in production, safely

Where we are

Our system is safe and runs every day. It will not stay good by itself: products, phone providers and callers change. In this lesson, we watch error by slice over time, IVR containment and hang-ups, and review samples.

What you will be able to do

Keep the speech system good after launch. Watch what comes in and what happens every day, for free. Review a sample of transcripts every week to measure the error rate by slice. Learn how much a small sample moves by chance, so you can tell a real change from noise. Know the changes that will come: a new phone provider, a new region and new product names.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook scores Whisper small call by call on the 110 validation and test calls. It draws many weekly review samples to show how much a sample of 10, 30 or 100 calls moves by chance. Then it builds eight weeks with a new phone provider and asks you to find the change. It needs no model download and no account. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the guided practice; the notebook needs none of it)

Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.

Walk into any call centre, and you see a big screen on the wall. It shows the calls waiting, the longest wait, and the callers who hung up today. It costs nothing to keep up to date: the phone system counts these things by itself.

On Friday afternoon, the supervisor does something slower. She listens to a few calls from each agent and scores them. That costs her time, so she cannot do it for every call. And once a month, the manager looks at the bill.

Larkfield’s speech system needs the same three rhythms. The daily counts are free. The weekly listening costs review time. The monthly bill closes the loop.

Grace asks for one number: “How good are the transcripts this week?” Priya could give her the WER of a weekly sample. But one number hides two things.

It hides the slices. On Larkfield’s 110 synthetic validation and test calls, Whisper small has a WER of 7.9%. On the English calls it is 4%, and on the French calls 14.1%. If next month brings more French callers, the average rises, although the recogniser did not change at all.

It moves by chance. A person can correct only a few transcripts a week. A small sample gives a different WER every week, even when nothing changes. Then a real change looks like noise, and noise looks like a change.

The idea: watch inputs, outcomes, quality and cost

Section titled “The idea: watch inputs, outcomes, quality and cost”

The diagram groups what to watch by how often, and by what it costs.

Every day, free, what comes in: the mix of calls (lines, languages, call length) and new words, product names the recogniser never saw. Every day, free, what happens: IVR outcomes (containment, hand-overs, hang-ups) and waiting (seconds to the answer, first sound back). Every week, costs review time: a reviewed sample where a person corrects 30 transcripts, and the WER by slice (line, language, product). Every month: the cost per audio hour and per IVR call.
What to watch on Larkfield's speech system. Inputs and outcomes are counted every day for free; quality needs a person; cost closes the month.
Text description of the diagram

Four groups, from left to right. Every day, for free, what comes in: the mix of calls by line, language and call length. Also new words, such as product names that the recogniser never saw. Every day, for free, what happens: containment, hand-overs and hang-ups of the IVR. Also the waiting: seconds to the answer, and the time to the first sound back. Every week, at the cost of review time: a person corrects 30 transcripts. From them, you compute the WER by slice (line, language and product). Every month: the cost per audio hour and per IVR call.

Some words:

  • Drift means that the calls change after launch. The calls that come in are no longer like the calls you tested on. The MLOps course explains it in Data and concept drift.
  • A slice is a group of calls that share one property: a line, a language, a product. You compute the WER of each slice apart.
  • A review sample is a small set of calls that a person checks by hand. The corrected transcript becomes the reference, and the WER compares the recogniser’s text with it. The labels arrive late, after a person has worked: see Delayed labels.
  • Containment is the share of IVR calls that end without a person. A hand-over sends the caller to a person. A hang-up is a caller who leaves before the end.

Inputs are the cheapest signal and the fastest. The phone system knows the line of every call, and the recogniser finds the language. Count them every day. A new phone provider shows up in the line counts on its first morning, long before any review.

A tiny worked example: how much does a sample move?

Section titled “A tiny worked example: how much does a sample move?”

Grace has 60 minutes of team-lead listening a day, at US$0.75 a minute (the course’s assumptions). Say that a person needs about 2 minutes to correct the transcript of one call: that is our assumption. One hour a week then buys 30 reviewed calls, for US$45.

How much does the WER of 30 calls move by chance? The course took the per-call results of Whisper small on the 110 calls, and drew 2,000 samples of each size (with replacement). The table shows the range that holds 90% of the samples.

Calls in the sample Lowest 5% of samples Median Highest 5% of samples
10 4.4% 7.6% 12.2%
30 5.9% 7.9% 10.4%
100 6.8% 7.9% 9.2%

Read the middle row. With 30 calls a week, the WER lands anywhere between about 6% and 10%, although the recogniser and the calls are the same every week. So a week at 10% is not news.

  • Median sample WER
  • 90% of samples fall between the dashed lines
How much the WER of a review sample moves by chance: Whisper small on Larkfield's 110 synthetic validation and test calls, 2,000 samples of each size (computed from the recorded transcripts).
Text description of the chart

A line chart with the sample size from 10 to 100 calls on the horizontal axis. The median stays near 7.6% to 7.9%. The band that holds 90% of the samples is wide for small samples and narrows slowly. It is 4.4% to 12.2% for 10 calls, 5.4% to 10.8% for 20 and 5.9% to 10.4% for 30. It is 6.3% to 9.8% for 50, and 6.8% to 9.2% for 100 calls.

The band narrows slowly. Going from 30 to 100 calls more than triples the review time. The band only shrinks from about 4.5 points to about 2.4 points.

Now a change. Larkfield moves part of its lines to a new phone provider that uses the G.726 codec. On the 110 calls, Whisper small has a WER of 9.4% on the G.726 line, against 7.9% on the landline and 7.1% on the mobile line. Most of these G.726 calls are French, so this slice mixes the line and the language. Today about 14% of the calls use it. From week 5, 60% of the calls will.

The chart is a constructed example: eight weeks that we built from the real per-call results. Each week, 30 calls are drawn at random. Weeks 1 to 4 have today’s mix of lines; weeks 5 to 8 have the new one. The dashed lines are the 90% band for 30 calls. Because most G.726 calls in our set are French, this drill really adds French calls. Read it as practice in telling a change from noise, not as a fact about the G.726 codec.

  • WER of the 30 reviewed calls
  • Normal range for 30 calls (90%)
A constructed example from real per-call results (Whisper small, Larkfield's synthetic calls): eight weekly samples of 30 calls, with 60% of the calls on the G.726 line from week 5. The change is smaller than the week-to-week noise.
Text description of the chart

A line chart over eight weeks. The weekly WER of 30 reviewed calls is 10.5%, 7.6%, 7.7% and 7.3% in weeks 1 to 4. It is 7.4%, 10.4%, 8.8% and 8.8% in weeks 5 to 8. The normal range for 30 calls runs from 5.9% to 10.4%. Week 1, before any change, is just above the range. The expected WER rises only from 8.0% to 8.8% with the new mix.

What does the chart teach?

  • The change is hidden in the noise. With the new mix, the expected WER moves from 8.0% to 8.8%. The weekly samples jump by more than that.
  • Week 1 crosses the line with no change at all. A 90% band is crossed in about 1 week in 10 by chance. One point outside the band is not an alarm.
  • The inputs show it at once. In weeks 1 to 4, 7, 3, 4 and 5 of the 30 reviewed calls were on the G.726 line. In weeks 5 to 8, 17, 18, 16 and 18 were. The daily line counts, for all calls and for free, would show it on day one.

The right reaction: when an input changes, review more calls from the new slice for two weeks. Measure that slice’s WER on its own, and decide.

The next two changes: a new region and new products

Section titled “The next two changes: a new region and new products”

Omar plans a new French-speaking region. On the test calls, Whisper small makes 15.1% errors in French, against 3.1% in English. More French calls raise the average, so watch the WER by language, not only the total.

The region also brings four new products. Whisper small missed 41 of 60 product names on the test calls; with Larkfield’s vocabulary in its prompt, it missed only 10. A product name that is not in the prompt is back to the first case. So a new product is a change to the system: add it to the vocabulary prompt on the day it goes on sale. Then watch how often the call records name an unknown product. The course did not build that count.

The IVR writes a call record for every call, so its outcomes cost nothing to count. On its 60 recorded test calls, the IVR contained 35% of the calls, sent 63% to a person and chose no wrong intent. Every hand-over costs about US$4 of agent time, and a caller who hangs up about US$8 (assumptions).

Hang-ups cannot be measured in the replay. The recorded callers never hang up, whatever the IVR does. On the real line, count them every day: a rise in hang-ups is often the first sign that a prompt confuses people.

Slices of IVR calls are small. The same IVR contained 25% of the 36 landline test calls, and 50% of the 12 landline calls in the hidden set. The hidden calls were run once, with an earlier version of the IVR, so the IVR changed a little too. But with so few calls, the slices swing a lot.

Transcripts help more than Grace’s team. A deaf or hard-of-hearing customer may prefer a written record of the call, and live captions during a video call. Some callers cannot use a voice menu at all. So the IVR keeps the keypad for every choice. A text option, such as a link sent by SMS, is another way in.

If Larkfield offers captions or written records, their quality needs watching too. Put caption calls into the review sample, and measure the WER of that slice. The course did not build captions.

The project already has the commands. speech score scores recorded transcripts by slice, and speech ivr replay counts the IVR outcomes by slice.

  1. Open a terminal in your project folder larkspeech, and activate .venv.

  2. Score Whisper small’s recorded transcripts of the test calls by line:

    Terminal window
    speech score --method whisper-small --by line
    whisper-small, test calls, phone audio, normalisation full
    slice calls words WER CER order nos wrong products wrong
    landline 43 5298 8.6% 4.0% 7 of 38 31 of 43
    mobile 10 1207 5.9% 3.1% 0 of 8 5 of 10
    voip_g726 7 829 12.2% 7.1% 1 of 6 5 of 7
  3. Replay the 60 IVR test calls, with the outcomes by line. It takes about 1.5 minutes:

    Terminal window
    speech ivr replay --by line

    The first lines of the output on the course’s computer:

    IVR replay: 60 recorded test calls, Whisper base, end of turn 0.8 s
    line calls passed contained wrong intent to a person s to answer
    landline 36 23 25% 0% 72% 51.2
    mobile 11 7 45% 0% 55% 54.3
    voip_g726 13 9 54% 0% 46% 50.2

    A list of the failed calls follows. Your numbers may differ slightly. Whisper on 4 CPU threads does not write exactly the same text every time, so a wrong intent or a second can change. The passed and contained columns stayed the same in our runs.

The G.726 line has the worst WER in step 2 and the best containment in step 3. How can both be true? Look at the kinds of calls on each line. On the landline, 15 of the 36 test calls are meant to end with a person. They are returns, silent callers, callers who press 0 or ask for a person, and unclear callers. On the G.726 line, only 2 of 13 are. So the landline’s containment is low for reasons that have nothing to do with the line. A slice carries a mix of call kinds, and the mix can explain the difference.

Score the same recorded transcripts on the validation calls, by line. Before you run it, predict: is the G.726 line still the worst?

Show the answer
Terminal window
speech score --method whisper-small --by line --split validation
whisper-small, validation calls, phone audio, normalisation full
slice calls words WER CER order nos wrong products wrong
landline 27 3258 6.7% 3.5% 4 of 20 16 of 27
mobile 14 1641 8.0% 5.3% 1 of 11 7 of 14
voip_g726 9 1086 7.4% 3.7% 0 of 7 5 of 9

No. On the validation calls, the G.726 line has 7.4%, better than the mobile line. Same recogniser, same line, other calls: 12.2% on 7 test calls, 7.4% on 9 validation calls. With so few calls, one or two hard calls decide the number, and the mix of languages decides more. Most VoIP test calls are French; inside English, the three lines differ by less than 1 point (the lesson Accents, noise and two languages). Before you act on a slice, count its calls, and join several weeks when the slice is small.

Common mistake

An alarm on one week's number

Symptom: the WER of the weekly sample jumps from 7.6% to 10.5%. The team rolls back the last change, and nothing improves.

Cause: with 30 calls, the sample WER moves between about 6% and 10% by chance. In the constructed example, such a jump happened in week 1, before anything changed.

Fix: draw the normal range for your sample size. Act on two or three weeks in a row outside it, or on a change in the inputs. Then review more calls from the slice that changed.

Common mistake

Logs that become a privacy leak

Symptom: to debug the IVR, the team logs every recognised turn and every key press. The logs now hold order numbers, names and maybe card digits, kept for years.

Cause: monitoring data is data too. It needs the same rules as the recordings of the last two lessons.

Fix: log counts, call IDs and states; keep text only in the review sample, redacted. The MLOps course explains how in Privacy-aware observability.

  • Every day: the mix of lines, languages and call lengths; containment, hand-overs, hang-ups and waiting times. Free, fast, and the first sign of drift.
  • Every week: a reviewed sample of about 30 calls, with the WER by slice over several weeks. More calls from a slice that just changed.
  • Before a change: a new provider, region or product. Test on a sample of the new calls first, as in Accents, noise and two languages.
  • Every month: the cost per audio hour and per IVR call, as in Batch and streaming. The MLOps course connects quality and cost in Quality and costs.

Check your understanding

The weekly review sample of 30 calls had a WER of 7.4% last week and 10.4% this week. Grace asks: 'Did the recogniser get worse?' What do you answer, and what do you check first?

Show the answer

Not necessarily. On Larkfield’s synthetic calls, 90% of 30-call samples fall between about 5.9% and 10.4% with no change at all. So 10.4% is right at the top of the normal range. First check the inputs: they are free and counted for every call. Did the mix of lines, languages or products change? Then look at the WER by slice, and over more than one week. If one slice changed, review more calls from it.

Check your understanding

On the test calls, the G.726 line has the worst WER (12.2%) and the IVR's best containment (54%). Tomás concludes that the G.726 line is good for the IVR. What is wrong?

Show the answer

The slices hold different kinds of calls. On the landline, 15 of the 36 IVR test calls are meant to end with a person. On the G.726 line, only 2 of 13 are. So the landline’s containment is low for reasons that have nothing to do with the line. The slices are also small: 13 IVR calls on G.726, and only 7 test calls for the WER. On the 9 validation calls, that WER drops to 7.4%. Compare like with like: the same kind of call on each line, and more calls.

Check your understanding

Practical check: run speech score --method whisper-small --by language. Which slice should Omar's new French-speaking region worry about, and why does the total WER rise even if the recogniser does not change?

Show the answer

The output shows English 3.1% (34 calls) and French 15.1% (26 calls). The French slice makes almost five times more errors. The total WER is a mix of the slices, weighted by their words. If the new region brings more French calls, the French slice weighs more, and the total rises although the recogniser is the same. Watch the WER by language, and the share of French calls every day.

Recap

  • Watch in three rhythms. Every day, for free: the inputs (lines, languages, length) and the IVR outcomes (containment, hand-overs, hang-ups, waiting). Every week: a reviewed sample with the WER by slice. Every month: the cost.
  • A sample moves by chance. With 30 reviewed calls of Whisper small on Larkfield’s synthetic calls, 90% of weekly WERs fall between about 5.9% and 10.4%.
  • Drift comes from a new phone line, a new region and new product names. In the constructed example, a new provider moved the expected WER from 8.0% to 8.8%. The noise hid it, but the input counts showed it plainly.
  • Small slices swing: the G.726 line had 12.2% on 7 test calls and 7.4% on 9 validation calls.
  • Captions and written records help callers who cannot use a voice line; put them in the review sample too.

You have built and run the whole system: from audio as data to a phone line that answers by itself, safely. The case studies now test these methods on real recordings. The first one chooses a recogniser for two languages.

Next: Choose a recogniser for two languages