Text-to-speech for prompts
- Lesson 23
- 35 min
- Module 7: Build a small IVR
Where we are
An IVR talks to callers, so it needs a voice. In this lesson, we make the Larkfield prompts with two open text-to-speech systems, Piper and Kokoro, compare them by listening, speed, French and licence, and write prompts for the phone.
Lesson 23 of 36 · Speech AI: Transcribe, Understand and SpeakText-to-speech for prompts
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Give the Larkfield IVR a voice that callers understand on a phone line. Choose it by speed, language and licence, and write prompts that are short enough to wait for.
NotebookDo this lesson in a notebook
The notebook plays the Piper and Kokoro prompts, and measures how long each one takes. Then you rewrite the menu so that a caller waits less. It downloads a few short audio files and runs no model. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The IVR’s states and prompts, from What an IVR is, and why companies still use one.
- What a phone line does to audio (8,000 samples a second, a narrow band of pitches), from Audio as data.
- The real-time factor and the WER, from Whisper sizes on a laptop.
What you need
- Your
larkspeechfolder at the end of Module 6, with its environment, and the labs folder next to it. - About 130 MB of free disk for the two Piper voices (English and French), which download the first time they are needed.
- Tested on macOS (Apple M4) with Python 3.12. The Windows commands are given but were not run.
Announcements at a train station
Section titled “Announcements at a train station”Think of the voice at a train station. “The train to Leeds, on platform 4, is delayed by 10 minutes.” Years ago, a person recorded every word: each city, each platform number, each number of minutes. A computer joined the pieces. When the company added a new city, someone went back to the studio.
Today, many stations use a computer voice that reads any sentence. Nobody records “Leeds” in advance. The sentence is written, and the voice speaks it.
An IVR has the same choice. Some prompts never change (“Welcome to Larkfield”). Others change on every call: “Your Cedar raised bed was delivered on May 11.” Larkfield sells 12 products, has 160 orders in its test system, and has dates all year. Nobody can record every sentence in advance.
The problem: a voice for sentences that change
Section titled “The problem: a voice for sentences that change”So the IVR needs text-to-speech, often written TTS: a program that turns written text into speech. It must do four things for Larkfield:
- Be understood on a phone line. The line removes the low and high pitches and keeps only 8,000 samples a second. A voice that sounds fine on headphones can be hard to follow on a phone.
- Be fast. A caller who has just stopped talking waits for the answer. Every half second of silence feels long.
- Speak English and French. More than half of Larkfield’s IVR test calls are in French (33 of 60).
- Have a licence that Larkfield can use in a product it runs, and maybe shares.
The idea: from letters to sounds to audio
Section titled “The idea: from letters to sounds to audio”A modern TTS system works in two steps.
First, it turns letters into phonemes, the small sounds of a language. “Order” becomes something like AW R D ER. This step knows that “read” sounds different in “I read yesterday”, and it turns “8 a.m.” into “eight a m”.
Second, a neural network turns the phonemes into audio samples. It learned this from many hours of one person, or several people, reading text aloud. The voice you hear is the voice of those recordings.
Two open systems run well on a laptop CPU:
- Piper is small and fast. Each voice is a separate file of about 60 MB, learned from one public dataset. The English voice of the Larkfield IVR learned from LJ Speech (one American reader, public domain). The French one learned from SIWIS (one French reader, CC BY 4.0).
- Kokoro (Kokoro-82M) is one model of 82 million parameters with many voices. It usually sounds more natural, and it is slower.
Listen first
Section titled “Listen first”Here is the same English prompt in both voices. It is the confirmation of an order number. Both clips went through Larkfield’s phone line, so you hear what a caller hears.
Transcript (both clips): “Order 7 4 2, 1 5 5. Is that right? Say yes or no, or press 1 for yes and 2 for no.”
And a French status answer:
Transcript (both clips): “Votre Grow Hub est parti le 4 mai et il est en route.” (Your Grow Hub shipped on May 4 and is on its way.)
Which one would you rather hear at 10 p.m.? Your answer is a listening test with one listener. It is useful, and it is not enough.
How a real listening test works
Section titled “How a real listening test works”Teams compare voices with a mean opinion score, or MOS. Several listeners hear each prompt and rate it from 1 (bad) to 5 (excellent). The MOS is the mean of all the ratings. Good tests play the voices in a random order, so the listeners do not know which voice they hear. They also play the audio as the caller hears it, through the phone line.
The course team did not run a listening test with people. Instead, we measured something easier: how many words a recogniser gets wrong on each prompt. We call it intelligibility here. It is a rough signal, not a listening test.
Measure speed, first audio and intelligibility
Section titled “Measure speed, first audio and intelligibility”Three numbers decide most of the choice:
- The real-time factor: compute time ÷ audio time, as in Set up the speech project. Below 1 means faster than speech.
- The time to the first audio: how long the caller waits before the first sound. A voice can speak the first sentence while it makes the next ones.
- The intelligibility: the WER of Whisper small on the prompts after the phone line.
Text description of the diagram
Left: Piper’s real-time factor is about 0.02, Kokoro’s about 0.1, so Piper is about five times faster. Middle: Piper’s first audio comes after 0.06 seconds in English and 0.05 in French; Kokoro’s after 0.62 and 0.40 seconds. Right: Whisper small got fewer words wrong on Kokoro’s prompts (6.2% English, 8.3% French) than on Piper’s (10.6% and 10.4%).
Read the chart as a trade. Piper is about five times faster, and its first sound comes after about 0.05 seconds. Kokoro is easier for the recogniser to follow on the phone line, but the caller waits about half a second more before each answer.
For an IVR, Piper is the better first choice. Its prompts are short and fixed, it runs on any CPU, and every voice is a separate small file. Kokoro is worth a listening test when the voice matters more than the speed.
A WER of about 10% on Piper’s prompts does not mean that callers miss one word in ten. People know the context, and they hear “Order 7 4 2” as a number. A recogniser’s errors show where a voice is less clear; they do not show whether people like it. That is why we call it a rough signal.
A hosted voice
Section titled “A hosted voice”Cloud providers sell TTS too. Their voices often sound the most natural, and they speak many languages. But each prompt then goes over the internet, you pay per character, and the first audio waits for the network. The course did not measure a hosted voice, and it needs no account. You can add one with your own key: send the prompt text to the provider’s TTS endpoint, cache the audio it returns, and measure the first-audio time the same way. Check the provider’s current price page first: prices change.
Licences: what you may do with each voice
Section titled “Licences: what you may do with each voice”A voice has three parts, and each has its own licence. They are the program that runs the voice, the voice file (the learned weights), and the recordings it learned from.
| Program | Voice files | Recordings it learned from | |
|---|---|---|---|
| Piper | piper-tts: GPL-3.0-or-later |
MIT | LJ Speech: public domain; SIWIS: CC BY 4.0 |
| Kokoro | kokoro: Apache-2.0; its French path uses phonemizer-fork: GPL-3.0 |
Apache-2.0 | public-domain and permissive audio, and, says its model card, synthetic audio from closed TTS services |
Two words in the table need an explanation:
- MIT and Apache-2.0 are permissive licences. You may use the code in your own product, even a closed one, if you keep the copyright notice.
- GPL is a copyleft licence. If you share a program that includes GPL code, you must share your whole program under the GPL too, with its source code. The licence text says, in section 5(c): “You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy” (GNU GPL version 3).
The course project, larkspeech, is MIT. If it imported Piper’s Python code, a learner who shares the IVR would share a program that includes GPL code. So the project does not import Piper. It runs the piper command as a separate program. Section 5 of the GPL calls “separate and independent works” that are only stored or shipped together an “aggregate”. The GPL then does not cover the other works. Where one program ends and another begins is a legal question. The Free Software Foundation discusses it in its GPL FAQ. The two programs only exchange text and files. The project writes a prompt as one line of text to piper. piper writes a WAV file, and the project reads it.
A separate program has a cost: each start loads the voice model. We measured it on the course laptop. A new piper program for each sentence took about 0.8 s before the first sound, against the 0.05 s in the chart above. So tts.py starts piper once for each language and keeps it running. After that start, the first sentence of a prompt was ready after about 0.05 s (French) and 0.04 s (English). That is close to the chart. Only the first prompt after the IVR starts waits the 0.8 s, once for each language.
Switching to Kokoro does not remove the question, because Kokoro’s French path uses a GPL library too.
These are the licence facts and the choice the course made. They are not legal advice. Before you ship a product, ask whoever checks licences in your company.
Write prompts for the ear
Section titled “Write prompts for the ear”A good voice cannot save a bad prompt. The caller hears each word once, cannot scroll back, and must remember the options. So prompts for the phone follow rules that written text does not need.
Short, and one idea per sentence
Section titled “Short, and one idea per sentence”Here is how long each English prompt of the IVR takes to say, made with Piper:
Text description of the diagram
The welcome takes 11 seconds and the menu 10.8 seconds. The menu retry takes 9.3, the opening hours 10.2, the confirmation 8. The handover takes 3.9 and the order-number question only 2.8. Our shorter menu takes 6.6 seconds.
A caller who wants the opening hours hears the welcome and the menu first: more than 20 seconds before they can say a word. The menu says each option twice, once for speech and once for the keypad:
How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person. (32 words)
Here is a shorter version that keeps every option:
Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0. (16 words)
It takes about 6.6 seconds instead of 10.8. Callers who know what they want can also speak or press a key before the prompt ends, if the IVR listens during the prompt. Module 8 deals with these interruptions.
The option first, then the key
Section titled “The option first, then the key”“Press 1 for orders” makes the caller remember a number before they know what it is for. “For orders, press 1” gives the reason first, so the caller waits for the key only when they need it. Our menu breaks this rule. It is a good change to try.
Numbers in groups
Section titled “Numbers in groups”People remember numbers in groups of two or three digits. The IVR reads an order number back as “7 4 2, 1 5 5”: the comma gives a short pause in the middle. A six-digit number read without a pause is easy to mishear, and hard to check.
One voice for each language
Section titled “One voice for each language”Our welcome has two languages: “… For English, say English or press 1. Pour le français, dites français ou appuyez sur 2.” The IVR does not know the caller’s language yet, so it says the whole welcome with its English voice. A French caller hears French with an American accent, in the first seconds of the call. You can hear it at the start of the call in the first lesson of this module. A better welcome makes each sentence with the voice of its language, and joins the two pieces of audio.
The way out, in every prompt that can fail
Section titled “The way out, in every prompt that can fail”Each retry says what to do next (“Please use your keypad”), and the menu says how to reach a person (0). A caller who does not know the way out hangs up.
Guided practice: give the IVR its voice
Section titled “Guided practice: give the IVR its voice”-
From the folder that holds
larkspeechandnextia-learning-labs, copy the requirements of the end-of-Module-7 snapshot, and the three files of the IVR’s voice.macOS and Linux:
Terminal window cp nextia-learning-labs/C41/snapshots/end-of-m07/requirements.txt larkspeech/mkdir -p larkspeech/larkspeech/ivrcp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/__init__.py larkspeech/larkspeech/ivr/cp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/prompts.py larkspeech/larkspeech/ivr/cp nextia-learning-labs/C41/snapshots/end-of-m07/larkspeech/ivr/tts.py larkspeech/larkspeech/ivr/Windows PowerShell:
Terminal window Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\requirements.txt larkspeech\New-Item -ItemType Directory -Force larkspeech\larkspeech\ivrCopy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\__init__.py larkspeech\larkspeech\ivr\Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\prompts.py larkspeech\larkspeech\ivr\Copy-Item nextia-learning-labs\C41\snapshots\end-of-m07\larkspeech\ivr\tts.py larkspeech\larkspeech\ivr\ -
Go into
larkspeech, activate the environment, and install the new requirements:piper-ttsand the web server of the test phone.Terminal window pip install -r requirements.txt -
Check that the
piperprogram is there, and that the project finds it:Terminal window piper --helppython -c "from larkspeech.ivr import tts; print(tts.piper_command())"usage: piper [-h] -m MODEL [-c CONFIG] [-i INPUT_FILE] [-f OUTPUT_FILE].../…/larkspeech/.venv/bin/piperThe path is the
piperprogram inside your environment. On Windows, it ends inScripts\piper.exe. -
Make the English menu prompt:
Terminal window python -c "from larkspeech.ivr import tts, prompts; x = tts.say(prompts.P['en']['menu']); print(f'{len(x) / 8000:.1f} s of audio')"Downloading piper/en_US-ljspeech-medium.onnx.json ...Downloading piper/en_US-ljspeech-medium.onnx ...11.1 s of audioThe first run downloads the voice (about 60 MB) and its settings. Your length may differ by a few tenths of a second.
-
Run the same command again. It answers at once: the prompt is now cached. Find the file and play it:
Terminal window ls outputs/promptsb70e79392f321e57.wavThe file name comes from the language and the text, so a changed prompt gets a new file.
Your turn: a shorter menu
Section titled “Your turn: a shorter menu”Predict: how many seconds does this menu take? “Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0.” Then make it with tts.say(...) as in step 4, with your sentence in place of prompts.P['en']['menu'].
Check your answer
python -c "from larkspeech.ivr import tts; x = tts.say('Say orders, hours or returns. Or press 1, 2 or 3. For a person, press 0.'); print(f'{len(x) / 8000:.1f} s of audio')"6.3 s of audioAbout 6.5 seconds instead of about 11: the caller saves more than 4 seconds on every call. Piper speaks at about 2.84 words a second on our prompts, so each word you remove saves about a third of a second. Now listen to the file. Is “1, 2 or 3” clear enough without the words? That is the question a listening test answers.
Common mistake
A prompt that sounds fine on headphones
Symptom: Amira writes a new welcome and listens to it on her headphones: clear and friendly. The first test calls on the phone line hang up more often in the welcome.
Cause: she listened to the 22,050-samples-a-second audio from Piper, not to the phone audio. The phone line removes the high pitches, where the sounds of s, f and th live. So “six” and “fix” sound alike, and fast words run together.
Fix: always listen to and measure prompts after the phone line, at 8,000 samples a second, as the IVR does (tts.say resamples to 8 kHz). Read numbers in groups, and slow down where the meaning is dense.
Consequence in AI work: test any output in the conditions where people use it: a voice on a phone line, a chart on a phone screen, a model on real inputs.
Check your understanding
Piper's first audio comes after about 0.06 s, Kokoro's after about 0.62 s, but Kokoro's prompts had a lower WER. Which would you choose for Larkfield's IVR, and what would change your mind?
Show the answer
Piper. The IVR’s prompts are short and fixed, Piper is about five times faster, and every caller waits less after each turn. Two things could justify the slower voice: a listening test with people (a MOS on phone audio) where callers clearly prefer Kokoro, or many complaints about the voice. The recogniser’s WER alone is not enough: it is a rough signal, not what people hear.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the course project run Piper as a separate program instead of importing it?
Show the answer
piper-tts is GPL-3.0-or-later. A shared program that includes GPL code must be shared under the GPL. The course project is MIT. By running the piper command and only exchanging text and files with it, the project does not include Piper’s code. Switching to Kokoro would not remove the question, because its French path uses a GPL library.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Rewrite "Press 2 for opening hours or 3 for returns" for the phone, and say which two rules you used.
Show the answer
For example: “For opening hours, press 2. For returns, press 3.” Rule one: the option first, then the key, so the caller knows what a key is for before they hear it. Rule two: one idea per sentence, so each sentence is short and easy to remember.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Text-to-speech turns letters into phonemes, then phonemes into audio, with a voice learned from recordings.
- Piper is about five times faster than Kokoro on the Larkfield prompts (real-time factor 0.02 against 0.1), with its first audio after about 0.05 s. Kokoro was easier for a recogniser to follow on the phone line. Neither number is a listening test.
- A MOS (mean opinion score) asks several listeners to rate each prompt from 1 to 5, on phone audio, in a random order.
piper-ttsis GPL, so the MIT course project runspiperas a separate program and never imports it.- Prompts for the ear: short, one idea per sentence, the option before the key, numbers in groups, and always the way out.
You can now give the IVR a voice and write prompts a caller can follow. But the IVR still cannot hear anything. Next, you build the simplest version that works: a keypad menu. It reads the tones of the phone’s keys, handles silence and wrong keys, and always has a way to a person.
Next: A keypad menu first