Skip to content
Français

Set up the speech project

  • Lesson 3
  • 35 min
  • Module 1: The speech problem and its cost

Where we are

We can now read a waveform and a spectrogram. Next, we open the real Larkfield calls. In this lesson, we install the tools, play and transcribe one call, compare it with its reference, and learn to reset the data. First, think of a new agent's first day with a headset.

What you will be able to do

Get the Larkfield calls and the course project on your computer, and prove that they work. You transcribe one call, compare it with its reference, and learn how to go back to a clean start. Without a known-good first run, you cannot tell later whether a bad result comes from a model or from your setup.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook opens call C-0225 as numbers and plays it with its reference transcript. It draws the call’s spectrograms, transcribes it with Whisper tiny and shows the wrong-sample-rate mistake. Its setup cells install the packages and download only the files they need, so it needs no local setup. About 30 minutes. The course project on your own computer comes after it on this page: you need it from Module 2 on.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • Sample rate, channels, mu-law and spectrograms, from Audio as data.
  • How to open a terminal, move between folders with cd, and make a Python virtual environment. If any of this is new, see Isolate dependencies in Python for Practical AI Engineering first.

What you need on your computer

  • Python 3.12. The project was tested with Python 3.12 only. Check it: python3.12 --version on macOS or Linux, py -3.12 --version on Windows.
  • Git, to get the course files (or a browser, to download them as a ZIP file).
  • About 700 MB of free disk space: the calls (190 MB), the Python packages (about 340 MB) and the smallest Whisper model (about 75 MB).
  • An internet connection for the first run. No account, no key and no graphics card.

Tested on macOS (Apple M4) with Python 3.12.13. The Windows (PowerShell) and Linux commands below were not run by the course team.

Think of a new agent on her first day at Larkfield’s help desk. She gets a headset, a computer and a login. Before she takes a real call, her team lead makes a test call: “Can you hear me? Read me back this order number.” If the test call works, any later problem comes from the call, not from her headset. If it does not work, she knows where to look. And she has a way back: log out, plug the headset in again, and try once more.

Your setup needs the same three things: a known-good first run, a result you can see, and a way back to a clean start.

Speech code fails quietly. The notebook of this lesson shows a common mistake with Whisper, the open speech recogniser of this course. It gives Whisper the 8,000-samples-a-second phone audio, but Whisper expects 16,000. For Whisper, the call then plays twice as fast and an octave higher. There is no error message. The model writes confident nonsense: “This is going to be a good one for quality injury. Let me help you today. How many times have you found a new thing in the program?” Its word error rate (WER), the share of words that it got wrong, is 0.802 instead of 0.055.

So the project has a test call of its own. speech check transcribes one known call and compares the result with what was really said. If the result is close to the expected one, the setup works.

The course project is a Python package called larkspeech. Its command is speech. You build it in steps through the course: each module adds files and commands. The course gives you the project at the end of each module as a snapshot: a complete copy of the project at that point. This module gives you the first one, start.

The snapshots and the calls live in the course’s labs repository, nextia-learning-labs, on GitHub. You copy the snapshot and the calls once into your own folder, larkspeech, and work there. You never change the labs folder.

Two folders side by side inside projects, your work folder. The first is nextia-learning-labs/C41, the course's files, which you do not change; it holds data (the calls, labels, recordings and checksums) and snapshots/start (the project at the end of this module). The second is larkspeech, your copy; it holds larkspeech and tests (the code and its tests), data (a copy of the calls), .venv (Python and packages) and outputs (what commands write; speech reset removes it). Two arrows, each labelled copy once: from snapshots/start to the code, and from data to the copy of the calls.
Your folders after the setup. The labs folder stays as it came; you work in larkspeech.
Text description of the diagram

Your work folder, projects, holds two folders. The first, nextia-learning-labs/C41, is the course’s files: the data folder with the calls, labels, recordings and checksums, and the snapshot snapshots/start. You do not change it. The second, larkspeech, is your copy. You copy the snapshot into it once, and the data folder once. It also gets a .venv folder with Python and the packages, and an outputs folder for what the commands write. speech reset removes outputs and checks data.

  1. Open a terminal. Go to the folder where you keep your projects, for example ~/projects. Make it first if it does not exist (mkdir ~/projects).

    Terminal window
    cd ~/projects
  2. Get the course files. If you have Git, clone the labs repository once. If you have the clone already, run git -C nextia-learning-labs pull instead.

    Terminal window
    git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

    Without Git, download the ZIP file of the repository, and unzip it in ~/projects. Its folder is then nextia-learning-labs-main. Use that name in the next steps.

  3. Copy the snapshot start to a new folder larkspeech. Then copy the calls into it, and go into it.

    Terminal window
    cp -R nextia-learning-labs/C41/snapshots/start larkspeech
    cp -R nextia-learning-labs/C41/data larkspeech/data
    cd larkspeech
  4. Make a virtual environment with Python 3.12, and turn it on. Your prompt then starts with (.venv).

    Terminal window
    python3.12 -m venv .venv
    source .venv/bin/activate
  5. Install the packages and the project. This takes a minute or two.

    Terminal window
    python -m pip install -r requirements.txt -e .

    The last line starts with Successfully installed. The -e . part installs the project itself, so the speech command now works. The packages are NumPy, SciPy and soundfile (audio as numbers), faster-whisper (the recogniser) and pytest (the tests), each at a fixed version.

  6. Run the tests. They use a tiny fake data folder, so they need no download.

    Terminal window
    python -m pytest -q
    ............ [100%]
    12 passed in 0.54s

    Your time will differ. The first run is slower.

  1. Ask the project what is in the data.

    Terminal window
    speech info
    Calls: 150 train (transcripts only), 50 validation, 60 test; audio for 110 calls (1.39 hours, phone and wideband)
    Languages: 68 English, 42 French; lines: 70 landline, 24 mobile, 16 voip_g726
    Stereo (agent and caller apart): 66; mono: 44
    IVR test calls: 60; orders in the fake order system: 160
    Every call is synthetic: open text-to-speech voices read invented scripts. The hidden test calls are not here.

    Only the validation and test calls have audio: 110 calls, 1.39 hours. The other lines count those 110 calls. The 150 training calls have only their transcripts and labels.

  2. Look at five calls as audio files. speech inspect shows the first five test calls by default.

    Terminal window
    speech inspect
    call rate ch seconds loudness silence format
    C-0009 8000 2 52.6 -32.0 dB 14% WAV ULAW
    C-0011 8000 2 36.4 -28.4 dB 45% WAV ULAW
    C-0018 8000 2 49.5 -31.6 dB 14% WAV ULAW
    C-0020 8000 2 45.0 -30.3 dB 45% WAV ULAW
    C-0026 8000 2 56.6 -28.9 dB 12% WAV ULAW

    You know every column from the last lesson. rate is the sample rate, and ch is the number of channels. loudness is the level in dBFS after the command mixes both channels. silence is the share of 30 ms frames under −45 dBFS. WAV ULAW means a WAV file with mu-law samples.

  3. Look at the wideband version of one call.

    Terminal window
    speech inspect C-0225 --kind wideband
    call rate ch seconds loudness silence format
    C-0225 16000 1 23.3 -21.3 dB 22% FLAC PCM_16

    The same call: 16,000 samples a second, one channel, 16-bit samples packed in FLAC.

The project’s first test call is C-0225. A caller asks how to put together a Cedar raised bed. Listen to it before you transcribe it.

Call C-0225 on the phone line (8,000 samples a second, both channels added). Both voices are synthetic. The agent, Chloe, is Piper trained on the LibriTTS-R corpus (speaker 5401). The caller is Piper trained on the VCTK corpus (speaker p288).

The reference transcript is what was really said. For a real call, people would write it by listening. For Larkfield’s calls, it comes from the script that the voices read, so it is exact. It is in data/calls/calls.jsonl, one line per call, in the field turns:

Time Who Reference
0.6–5.8 s Agent Thank you for calling Larkfield, this is Chloe. This call is recorded for quality and training. How can I help you today?
6.1–9.6 s Caller Hi. Please explain how to assemble the Cedar raised bed before I fill it with soil.
9.9–14.3 s Agent Good question. For the Cedar raised bed, the first step is to charge it fully before you set it up.
14.7–16.9 s Agent I will email you the setup guide with pictures today.
17.1–18.6 s Agent Is there anything else I can help you with?
18.7–20.1 s Caller That’s everything, thank you.
20.4–22.5 s Agent Thank you for calling Larkfield. Have a good day.

Did you notice the third line? A raised bed has no battery to charge. This is a mistake in our own scripts: the product-question answer says “charge it fully” for every product. We kept it, because a new script would mean new audio and new measurements for the whole course. It does not change anything in this lesson: the recogniser must write what was said, right or wrong.

speech check is the project’s test call. The diagram shows what it does.

A flow from left to right (top to bottom on a phone). C-0225.wav, the phone file with 8,000 samples a second and 2 channels, goes to: mix two channels into one; then resample from 8,000 to 16,000 samples a second; then Whisper tiny, on the CPU, timed. Whisper's transcript goes to: normalise both, lower case, no commas, numbers as digits. The reference, the script's words from calls.jsonl, also goes to normalise both. The last box is the result: WER, the share of words wrong, and the real-time factor.
What speech check does with one call: the right sample rate for Whisper, the same normalisation for both texts, then two numbers.
Text description of the diagram

The check reads the phone file of call C-0225: 8,000 samples a second, two channels. It mixes the two channels into one, and changes the sample rate to 16,000, the rate that Whisper expects. Then Whisper tiny, the smallest size, transcribes the call on the computer’s processor, and the check times it. Next, the check normalises the model’s transcript and the reference in the same way: lower case, no punctuation, numbers as digits. Last, it compares the two texts word by word. It prints the word error rate and the real-time factor.

  1. Run the check. The first time, it downloads Whisper tiny (about 75 MB) from Hugging Face, at a fixed version.

    Terminal window
    speech check
    Call C-0225 (en, landline line, phone audio, 23.3 s, synthetic)
    Model: Thank you for calling Larkfield. This is Chloe. This call is recorded for quality and training. How can I help you today? Hi. Please explain how to assemble the ...
    Reference: Thank you for calling Larkfield, this is Chloe. This call is recorded for quality and training. How can I help you today? Hi. Please explain how to assemble the ...
    WER 5.5% after the normalisation guide; real-time factor 0.019 (0.4 s of compute for 23.3 s of audio)
    Setup check passed.

    During the first run, you also see a line Fetching 6 files, and a warning about unauthenticated requests to the Hugging Face Hub. You can ignore the warning: the model is public, and you need no account.

  2. Check the WER. It should be 5.5%, or very close. The model and its settings are pinned, but on another computer a word can change now and then. If you see a WER far from 5.5%, see the common mistake at the end of this page.

  3. Look at the real-time factor. Yours will differ: it depends on your computer.

The Model line is what Whisper tiny wrote. The Reference line is what was said. Both lines are cut after 160 characters. They already differ: the model wrote “Larkfield. This is Chloe.” with a full stop, and the reference has “Larkfield, this is Chloe.” with a comma. Is that an error?

For Larkfield, it is not. Nobody’s decision changes because of a comma. So before it counts errors, the project changes both texts by the same written rules. This is the normalisation guide: lower case, no punctuation, numbers as digits, and a few more rules. Module 2 builds that guide. Without it, the same transcript of this call has a WER of 12.2% instead of 5.5%, only because of capital letters and commas.

After the guide, 5.5% of the reference words are still wrong. The notebook prints the whole transcript. The errors are in the product name and one verb. The model wrote “the C-Draise Bed” for “the Cedar raised bed”, twice. It also wrote “before I fed it with soil” for “before I fill it with soil”. Names are hard for a recogniser that has never heard them. Module 2 counts errors on names and order numbers apart.

The real-time factor (RTF) is the compute time divided by the length of the audio. Here, 0.4 seconds of compute for 23.3 seconds of audio gives 0.019. An RTF of 0.1 means one hour of audio in six minutes. An RTF under 1 means faster than the audio plays. On the course laptop, the same check gave between 0.015 and 0.027 on different runs. Your computer may be slower or faster.

speech reset is your way back to a clean start. It removes the outputs folder, where later commands write their results. Then it compares every data file with its checksum: a short code computed from the file’s bytes. If a single byte changes, the checksum changes. The list of checksums is in data/SHA256SUMS.

Terminal window
speech reset
Reset: removed nothing; data: all 507 files are right.

“Removed nothing” means that there was no outputs folder yet. The data folder has 508 files: the 507 that the checksums cover, and the checksum list itself.

The data folder has a data card: data/dataset-card.md. A data card is a short document that says what a dataset is, how it was made and where it is weak. Open it in your editor and read it once. It takes five minutes. Three parts matter for the whole course:

  • What is here: every file and folder, and every field of calls.jsonl.
  • How the calls were made: the scripts, the voices and their licences, and the phone lines. It also lists the practice problems that we planted on purpose, such as long silences.
  • Limits: synthetic voices are cleaner than people, and part of the French error comes from the voices. The calls are short and follow a few patterns. Read this part before you trust any number.

The notebook works with the same call, step by step, with the code visible. Open it from the box Do this lesson in a notebook at the top of this page.

  1. Run the setup cells. The data cell ends with Ready: 8 files in larkfield-calls.
  2. In section 1, run the first cell. The phone file has 8000 samples a second and 2 channels; the wideband file has 16000 and 1. The loudness is -30.0 dBFS.
  3. Run the silence cell. You see 0.42: that share of the call’s frames are silent.
  4. In section 2, press play on the player. Read the transcript that the cell prints.
  5. In section 3, predict which spectrogram has bright parts above 3,400 Hz. Then run the cell.
  6. In section 4, load Whisper tiny, then transcribe the call. You see WER after the guide: 0.055.
  7. In section 5, predict whether the WER goes up or down without the guide. Then run the cell. You see 0.122.
  8. In section 6, run the wrong-sample-rate cell. You see a wrong transcript and a WER of about 0.80.

In the notebook, section 7: build a table of five calls (C-0225, C-0083, C-0089, C-0219 and C-0102). For each call, give its sample rate, channels, length, loudness and share of silence. Run the three checks. Before you look, predict: which call has the most silence?

Show the answer
table = []
for call_id in ["C-0225", "C-0083", "C-0089", "C-0219", "C-0102"]:
path = f"data/calls/phone/{call_id}.wav"
info = sf.info(path)
x, sr = read_audio(path)
table.append({"call_id": call_id, "rate": info.samplerate, "channels": info.channels,
"seconds": round(info.duration, 1), "loudness": round(dbfs(x), 1),
"silence": round(silence_share(x, sr), 2)})
for row in table:
print(row)
{'call_id': 'C-0225', 'rate': 8000, 'channels': 2, 'seconds': 23.3, 'loudness': -30.0, 'silence': 0.42}
{'call_id': 'C-0083', 'rate': 8000, 'channels': 1, 'seconds': 32.3, 'loudness': -22.2, 'silence': 0.14}
{'call_id': 'C-0089', 'rate': 8000, 'channels': 1, 'seconds': 56.9, 'loudness': -23.2, 'silence': 0.1}
{'call_id': 'C-0219', 'rate': 8000, 'channels': 2, 'seconds': 64.9, 'loudness': -30.6, 'silence': 0.57}
{'call_id': 'C-0102', 'rate': 8000, 'channels': 1, 'seconds': 32.5, 'loudness': -23.5, 'silence': 0.32}

C-0219 has the most silence (0.57), and C-0225 comes next (0.42). Both are stereo calls. C-0219 also has a long pause of about 12 seconds in the middle of the call. The code mixes the two channels by taking their average, so each turn is about half as loud. So more quiet frames fall under −45 dBFS. The mono calls are about 7 dB louder. So the silence share depends on how the call was recorded, not only on how much people talk. Keep this in mind for Module 3, which finds the speech.

On your computer, change one thing in the setup check: the call. Run it on a French call, C-0065. Before you run it, predict: is the WER higher or lower than 5.5%?

Terminal window
speech check --call C-0065
Show the answer
Call C-0065 (fr, landline line, phone audio, 35.6 s, synthetic)
Model: Merci d'avoir la pluie, ici Juliette. C'est à peine, c'est enregistré pour la qualité et la formation. Comment vous avez aujourd'hui? Bonjour, Madame. Comment s ...
Reference: Merci d'avoir appelé Larkfield, ici Julien. Cet appel est enregistré pour la qualité et la formation. Comment puis-je vous aider aujourd'hui? Bonjour madame. Co ...
WER 37.0% after the normalisation guide; real-time factor 0.023 (0.8 s of compute for 35.6 s of audio)
Setup check passed.

The WER is much higher: 37.0%. Whisper tiny wrote “Merci d’avoir la pluie” (“thank you for the rain”) for “Merci d’avoir appelé Larkfield”, and “Juliette” for “Julien”. Three things add up here: the smallest model, French, and the French synthetic voices, which are harder to understand than the English ones. The wideband file gives the same 37.0% (speech check --call C-0065 --kind wideband), so the phone line is not the problem in this call.

Notice also the last line. The check says “passed” because its only job is to prove that your setup runs: it passes for any WER under 50%. A setup check is not a quality test. Module 4 compares the sizes of Whisper properly, on all the test calls.

Common mistake

No data folder, or a changed data file

Symptom 1: you run a command and see this:

Error: no data folder found (set LARKSPEECH_DATA, or copy the course's data folder here)

Cause: you are not in the larkspeech folder, or the data folder is missing. The project looks for data in the current folder, and then one and two levels up.

Fix: go to the project folder (cd ~/projects/larkspeech), and check that data/calls/calls.jsonl exists. If not, copy the data again, as in step 3 of the setup.

Symptom 2: speech check gives another WER than 5.5%, or a later command gives strange results. Run speech reset. If a file changed or is missing, it says so:

Reset: removed outputs; data: 2 problems:
changed: calls/phone/C-0225.wav
missing: calls/wideband/C-0127.flac
Download the data folder again, or copy the broken files from the course's labs repository.

Fix: copy the named files again from nextia-learning-labs/C41/data, for example cp ../nextia-learning-labs/C41/data/calls/phone/C-0225.wav data/calls/phone/. Then run speech reset again, until it says all 507 files are right.

Consequence in AI work: a changed audio file gives a different WER with no error message, like the wrong sample rate. Without a checksum, you would compare models on different data and never know.

Check your understanding

Why does the project have a setup check with one known call and one expected result? What does the check prove, and what does it not prove?

Show the answer

Speech code fails quietly. A wrong sample rate or a changed file gives a bad transcript with no error message: the wrong rate gave a WER of 0.802 instead of 0.055. A known call with a known result shows that the data, the packages and the model work together. Whisper tiny must give 5.5% on C-0225. The check does not prove that the model is good. It passes for any WER under 50%, and the French call C-0065 gives 37.0% and still “passes”. Model quality is measured later, on all the test calls.

Check your understanding

The model wrote "Larkfield. This is Chloe." and the reference is "Larkfield, this is Chloe.". Should this count as an error? What does the project do about it, and how much does it change the WER of this call?

Show the answer

It should not count as an error for Larkfield: no decision depends on a comma or a capital letter. The project changes both texts by the same written rules, the normalisation guide (lower case, no punctuation, numbers as digits), before it compares them. For this call, the WER is 5.5% after the guide and 12.2% without it. The errors left after the guide are real: the product name (“C-Draise Bed”) and one verb (“fed” for “fill”).

Check your understanding

Your colleague's speech check takes 4.1 seconds of compute for the 23.3-second call. What is the real-time factor? How long would 1,000 hours of calls take on that computer?

Show the answer

RTF = compute time ÷ audio time = 4.1 ÷ 23.3 ≈ 0.18. (4.1 ÷ 23.3 = 0.176.) So 1,000 hours of audio take about 0.176 × 1,000 = 176 hours of compute on that computer, with this model. On the course laptop (RTF about 0.02), the same work takes about 20 hours. The RTF depends on the computer, the model size and its settings, so measure it on the computer that will do the work.

Recap

  • You copied the snapshot start and the calls into larkspeech, made a virtual environment, and installed the project. The tests say 12 passed.
  • speech info and speech inspect show the calls as data: 110 calls with audio, phone (8,000 samples a second, mu-law) and wideband (16,000, FLAC).
  • The reference transcript is what was said. For Larkfield’s synthetic calls, it comes from the script, so it is exact.
  • speech check is the test call. Whisper tiny on C-0225 gives a WER of 5.5% after the normalisation guide. Its real-time factor is near 0.02 on the course laptop.
  • speech reset removes outputs and checks all 507 data files against their checksums. The data card says how the calls were made and where they are weak.

You can now transcribe a call and get one number for its quality. But what exactly is a word error, and why did a comma change the WER from 5.5% to 12.2%? Next, you compute the word error rate by hand, step by step.

Next: Word error rate, step by step