Skip to content
Français

Text-to-speech for prompts

  • Lesson 23
  • 35 min
  • Module 7: Build a small IVR

Where we are

An IVR talks to callers, so it needs a voice. In this lesson, we make the Larkfield prompts with two open text-to-speech systems, Piper and Kokoro, compare them by listening, speed, French and licence, and write prompts for the phone.

Lesson 23 of 36 · Speech AI: Transcribe, Understand and SpeakText-to-speech for prompts

What you will be able to do

Give the Larkfield IVR a voice that callers understand on a phone line. Choose it by speed, language and licence, and write prompts that are short enough to wait for.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook plays the Piper and Kokoro prompts, and measures how long each one takes. Then you rewrite the menu so that a caller waits less. It downloads a few short audio files and runs no model. About 20 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need

  • Your larkspeech folder at the end of Module 6, with its environment, and the labs folder next to it.
  • About 130 MB of free disk for the two Piper voices (English and French), which download the first time they are needed.
  • Tested on macOS (Apple M4) with Python 3.12. The Windows commands are given but were not run.

Think of the voice at a train station. “The train to Leeds, on platform 4, is delayed by 10 minutes.” Years ago, a person recorded every word: each city, each platform number, each number of minutes. A computer joined the pieces. When the company added a new city, someone went back to the studio.

Today, many stations use a computer voice that reads any sentence. Nobody records “Leeds” in advance. The sentence is written, and the voice speaks it.

An IVR has the same choice. Some prompts never change (“Welcome to Larkfield”). Others change on every call: “Your Cedar raised bed was delivered on May 11.” Larkfield sells 12 products, has 160 orders in its test system, and has dates all year. Nobody can record every sentence in advance.

The problem: a voice for sentences that change

Section titled “The problem: a voice for sentences that change”

So the IVR needs text-to-speech, often written TTS: a program that turns written text into speech. It must do four things for Larkfield:

  1. Be understood on a phone line. The line removes the low and high pitches and keeps only 8,000 samples a second. A voice that sounds fine on headphones can be hard to follow on a phone.
  2. Be fast. A caller who has just stopped talking waits for the answer. Every half second of silence feels long.
  3. Speak English and French. More than half of Larkfield’s IVR test calls are in French (33 of 60).
  4. Have a licence that Larkfield can use in a product it runs, and maybe shares.

A modern TTS system works in two steps.

First, it turns letters into phonemes, the small sounds of a language. “Order” becomes something like AW R D ER. This step knows that “read” sounds different in “I read yesterday”, and it turns “8 a.m.” into “eight a m”.

Second, a neural network turns the phonemes into audio samples. It learned this from many hours of one person, or several people, reading text aloud. The voice you hear is the voice of those recordings.

Two open systems run well on a laptop CPU:

  • Piper is small and fast. Each voice is a separate file of about 60 MB, learned from one public dataset. The English voice of the Larkfield IVR learned from LJ Speech (one American reader, public domain). The French one learned from SIWIS (one French reader, CC BY 4.0).
  • Kokoro (Kokoro-82M) is one model of 82 million parameters with many voices. It usually sounds more natural, and it is slower.

Here is the same English prompt in both voices. It is the confirmation of an order number. Both clips went through Larkfield’s phone line, so you hear what a caller hears.

The confirmation prompt, 7.71 seconds, on a phone line. A synthetic voice: Piper, trained on the LJ Speech corpus.
The same prompt, 7.62 seconds, on a phone line. A synthetic voice: Kokoro-82M, voice af_heart.

Transcript (both clips): “Order 7 4 2, 1 5 5. Is that right? Say yes or no, or press 1 for yes and 2 for no.”

And a French status answer:

A French status answer, 3.38 seconds, on a phone line. A synthetic voice: Piper, trained on the SIWIS corpus.
The same answer, 3.7 seconds, on a phone line. A synthetic voice: Kokoro-82M, voice ff_siwis (it learned from the same SIWIS reader).

Transcript (both clips): “Votre Grow Hub est parti le 4 mai et il est en route.” (Your Grow Hub shipped on May 4 and is on its way.)

Which one would you rather hear at 10 p.m.? Your answer is a listening test with one listener. It is useful, and it is not enough.

Teams compare voices with a mean opinion score, or MOS. Several listeners hear each prompt and rate it from 1 (bad) to 5 (excellent). The MOS is the mean of all the ratings. Good tests play the voices in a random order, so the listeners do not know which voice they hear. They also play the audio as the caller hears it, through the phone line.

The course team did not run a listening test with people. Instead, we measured something easier: how many words a recogniser gets wrong on each prompt. We call it intelligibility here. It is a rough signal, not a listening test.

Measure speed, first audio and intelligibility

Section titled “Measure speed, first audio and intelligibility”

Three numbers decide most of the choice:

  • The real-time factor: compute time ÷ audio time, as in Set up the speech project. Below 1 means faster than speech.
  • The time to the first audio: how long the caller waits before the first sound. A voice can speak the first sentence while it makes the next ones.
  • The intelligibility: the WER of Whisper small on the prompts after the phone line.
Three bar charts for four voices: Piper English, Kokoro English, Piper French, Kokoro French. Real-time factor: 0.021, 0.098, 0.020, 0.097. First audio in seconds: 0.06, 0.62, 0.05, 0.40. WER of Whisper small on the prompts through the phone line in percent: 10.6, 6.2, 10.4, 8.3.
Piper and Kokoro on the Larkfield prompts, 4 CPU threads on the Apple M4. Intelligibility is a recogniser's WER on the prompts through the phone line, not a listening test with people.
Text description of the diagram

Left: Piper’s real-time factor is about 0.02, Kokoro’s about 0.1, so Piper is about five times faster. Middle: Piper’s first audio comes after 0.06 seconds in English and 0.05 in French; Kokoro’s after 0.62 and 0.40 seconds. Right: Whisper small got fewer words wrong on Kokoro’s prompts (6.2% English, 8.3% French) than on Piper’s (10.6% and 10.4%).

Read the chart as a trade. Piper is about five times faster, and its first sound comes after about 0.05 seconds. Kokoro is easier for the recogniser to follow on the phone line, but the caller waits about half a second more before each answer.

For an IVR, Piper is the better first choice. Its prompts are short and fixed, it runs on any CPU, and every voice is a separate small file. Kokoro is worth a listening test when the voice matters more than the speed.

A WER of about 10% on Piper’s prompts does not mean that callers miss one word in ten. People know the context, and they hear “Order 7 4 2” as a number. A recogniser’s errors show where a voice is less clear; they do not show whether people like it. That is why we call it a rough signal.

Cloud providers sell TTS too. Their voices often sound the most natural, and they speak many languages. But each prompt then goes over the internet, you pay per character, and the first audio waits for the network. The course did not measure a hosted voice, and it needs no account. You can add one with your own key: send the prompt text to the provider’s TTS endpoint, cache the audio it returns, and measure the first-audio time the same way. Check the provider’s current price page first: prices change.

A voice has three parts, and each has its own licence. They are the program that runs the voice, the voice file (the learned weights), and the recordings it learned from.

Program Voice files Recordings it learned from
Piper piper-tts: GPL-3.0-or-later MIT LJ Speech: public domain; SIWIS: CC BY 4.0
Kokoro kokoro: Apache-2.0; its French path uses phonemizer-fork: GPL-3.0 Apache-2.0 public-domain and permissive audio, and, says its model card, synthetic audio from closed TTS services

Two words in the table need an explanation:

  • MIT and Apache-2.0 are permissive licences. You may use the code in your own product, even a closed one, if you keep the copyright notice.
  • GPL is a copyleft licence. If you share a program that includes GPL code, you must share your whole program under the GPL too, with its source code. The licence text says, in section 5(c): “You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy” (GNU GPL version 3).

The course project, larkspeech, is MIT. If it imported Piper’s Python code, a learner who shares the IVR would share a program that includes GPL code. So the project does not import Piper. It runs the piper command as a separate program. Section 5 of the GPL calls “separate and independent works” that are only stored or shipped together an “aggregate”. The GPL then does not cover the other works. Where one program ends and another begins is a legal question. The Free Software Foundation discusses it in its GPL FAQ. The two programs only exchange text and files. The project writes a prompt as one line of text to piper. piper writes a WAV file, and the project reads it.

A separate program has a cost: each start loads the voice model. We measured it on the course laptop. A new piper program for each sentence took about 0.8 s before the first sound, against the 0.05 s in the chart above. So tts.py starts piper once for each language and keeps it running. After that start, the first sentence of a prompt was ready after about 0.05 s (French) and 0.04 s (English). That is close to the chart. Only the first prompt after the IVR starts waits the 0.8 s, once for each language.

Switching to Kokoro does not remove the question, because Kokoro’s French path uses a GPL library too.

These are the licence facts and the choice the course made. They are not legal advice. Before you ship a product, ask whoever checks licences in your company.

A good voice cannot save a bad prompt. The caller hears each word once, cannot scroll back, and must remember the options. So prompts for the phone follow rules that written text does not need.

Here is how long each English prompt of the IVR takes to say, made with Piper:

A bar chart of seconds per prompt: welcome 11.0, menu 10.8, menu retry 9.3, order number 2.8, confirm 8.0, opening hours 10.2, handover 3.9, and a shorter menu written for this lesson 6.6.
How long each English prompt of the IVR takes to say (Piper, LJ Speech voice). The green bar is a shorter menu that we wrote for this lesson. Piper adds a little randomness: the same prompt can differ by a few tenths of a second.
Text description of the diagram

The welcome takes 11 seconds and the menu 10.8 seconds. The menu retry takes 9.3, the opening hours 10.2, the confirmation 8. The handover takes 3.9 and the order-number question only 2.8. Our shorter menu takes 6.6 seconds.

A caller who wants the opening hours hears the welcome and the menu first: more than 20 seconds before they can say a word. The menu says each option twice, once for speech and once for the keypad:

How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person. (32 words)

Here is a shorter version that keeps every option:

Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0. (16 words)

It takes about 6.6 seconds instead of 10.8. Callers who know what they want can also speak or press a key before the prompt ends, if the IVR listens during the prompt. Module 8 deals with these interruptions.

“Press 1 for orders” makes the caller remember a number before they know what it is for. “For orders, press 1” gives the reason first, so the caller waits for the key only when they need it. Our menu breaks this rule. It is a good change to try.

People remember numbers in groups of two or three digits. The IVR reads an order number back as “7 4 2, 1 5 5”: the comma gives a short pause in the middle. A six-digit number read without a pause is easy to mishear, and hard to check.

Our welcome has two languages: “… For English, say English or press 1. Pour le français, dites français ou appuyez sur 2.” The IVR does not know the caller’s language yet, so it says the whole welcome with its English voice. A French caller hears French with an American accent, in the first seconds of the call. You can hear it at the start of the call in the first lesson of this module. A better welcome makes each sentence with the voice of its language, and joins the two pieces of audio.

The way out, in every prompt that can fail

Section titled “The way out, in every prompt that can fail”

Each retry says what to do next (“Please use your keypad”), and the menu says how to reach a person (0). A caller who does not know the way out hangs up.

  1. From the folder that holds larkspeech and nextia-learning-labs, copy the requirements of the end-of-Module-7 snapshot, and the three files of the IVR’s voice.

    macOS and Linux:

    Terminal window
    cp nextia-learning-labs/C41/snapshots/end-of-m07/requirements.txt larkspeech/
    mkdir -p larkspeech/larkspeech/ivr
    cp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/__init__.py larkspeech/larkspeech/ivr/
    cp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/prompts.py larkspeech/larkspeech/ivr/
    cp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/tts.py larkspeech/larkspeech/ivr/

    Windows PowerShell:

    Terminal window
    Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\requirements.txt larkspeech\
    New-Item -ItemType Directory -Force larkspeech\larkspeech\ivr
    Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\__init__.py larkspeech\larkspeech\ivr\
    Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\prompts.py larkspeech\larkspeech\ivr\
    Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\tts.py larkspeech\larkspeech\ivr\
  2. Go into larkspeech, activate the environment, and install the new requirements: piper-tts and the web server of the test phone.

    Terminal window
    pip install -r requirements.txt
  3. Check that the piper program is there, and that the project finds it:

    Terminal window
    piper --help
    python -c "from larkspeech.ivr import tts; print(tts.piper_command())"
    usage: piper [-h] -m MODEL [-c CONFIG] [-i INPUT_FILE] [-f OUTPUT_FILE]
    ...
    /…/larkspeech/.venv/bin/piper

    The path is the piper program inside your environment. On Windows, it ends in Scripts\piper.exe.

  4. Make the English menu prompt:

    Terminal window
    python -c "from larkspeech.ivr import tts, prompts; x = tts.say(prompts.P['en']['menu']); print(f'{len(x) / 8000:.1f} s of audio')"
    Downloading piper/en_US-ljspeech-medium.onnx.json ...
    Downloading piper/en_US-ljspeech-medium.onnx ...
    11.1 s of audio

    The first run downloads the voice (about 60 MB) and its settings. Your length may differ by a few tenths of a second.

  5. Run the same command again. It answers at once: the prompt is now cached. Find the file and play it:

    Terminal window
    ls outputs/prompts
    b70e79392f321e57.wav

    The file name comes from the language and the text, so a changed prompt gets a new file.

Predict: how many seconds does this menu take? “Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0.” Then make it with tts.say(...) as in step 4, with your sentence in place of prompts.P['en']['menu'].

Check your answer
Terminal window
python -c "from larkspeech.ivr import tts; x = tts.say('Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0.'); print(f'{len(x) / 8000:.1f} s of audio')"
6.3 s of audio

About 6.5 seconds instead of about 11: the caller saves more than 4 seconds on every call. Piper speaks at about 2.84 words a second on our prompts, so each word you remove saves about a third of a second. Now listen to the file. Is “1, 2 or 3” clear enough without the words? That is the question a listening test answers.

Common mistake

A prompt that sounds fine on headphones

Symptom: Amira writes a new welcome and listens to it on her headphones: clear and friendly. The first test calls on the phone line hang up more often in the welcome.

Cause: she listened to the 22,050-samples-a-second audio from Piper, not to the phone audio. The phone line removes the high pitches, where the sounds of s, f and th live. So “six” and “fix” sound alike, and fast words run together.

Fix: always listen to and measure prompts after the phone line, at 8,000 samples a second, as the IVR does (tts.say resamples to 8 kHz). Read numbers in groups, and slow down where the meaning is dense.

Consequence in AI work: test any output in the conditions where people use it: a voice on a phone line, a chart on a phone screen, a model on real inputs.

Check your understanding

Piper's first audio comes after about 0.06 s, Kokoro's after about 0.62 s, but Kokoro's prompts had a lower WER. Which would you choose for Larkfield's IVR, and what would change your mind?

Show the answer

Piper. The IVR’s prompts are short and fixed, Piper is about five times faster, and every caller waits less after each turn. Two things could justify the slower voice: a listening test with people (a MOS on phone audio) where callers clearly prefer Kokoro, or many complaints about the voice. The recogniser’s WER alone is not enough: it is a rough signal, not what people hear.

Check your understanding

Why does the course project run Piper as a separate program instead of importing it?

Show the answer

piper-tts is GPL-3.0-or-later. A shared program that includes GPL code must be shared under the GPL. The course project is MIT. By running the piper command and only exchanging text and files with it, the project does not include Piper’s code. Switching to Kokoro would not remove the question, because its French path uses a GPL library.

Check your understanding

Rewrite "Press 2 for opening hours or 3 for returns" for the phone, and say which two rules you used.

Show the answer

For example: “For opening hours, press 2. For returns, press 3.” Rule one: the option first, then the key, so the caller knows what a key is for before they hear it. Rule two: one idea per sentence, so each sentence is short and easy to remember.

Recap

  • Text-to-speech turns letters into phonemes, then phonemes into audio, with a voice learned from recordings.
  • Piper is about five times faster than Kokoro on the Larkfield prompts (real-time factor 0.02 against 0.1), with its first audio after about 0.05 s. Kokoro was easier for a recogniser to follow on the phone line. Neither number is a listening test.
  • A MOS (mean opinion score) asks several listeners to rate each prompt from 1 to 5, on phone audio, in a random order.
  • piper-tts is GPL, so the MIT course project runs piper as a separate program and never imports it.
  • Prompts for the ear: short, one idea per sentence, the option before the key, numbers in groups, and always the way out.

You can now give the IVR a voice and write prompts a caller can follow. But the IVR still cannot hear anything. Next, you build the simplest version that works: a keypad menu. It reads the tones of the phone’s keys, handles silence and wrong keys, and always has a way to a person.

Next: A keypad menu first