Skip to content
Français

Audio as data

  • Lesson 2
  • 35 min
  • Module 1: The speech problem and its cost

Where we are

We know what Larkfield wants from its calls. We have not yet looked at what a computer gets when it opens a recording. In this lesson, we see sound as numbers, and why a phone line loses sounds that a good microphone keeps. First, think of how a film shows movement.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Read a recording the way a computer does: as numbers. Then you can say what a phone line removes from a call before any model hears it. You can also prepare audio for a model without losing more.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

  • What Larkfield wants from its calls, and why every call in this course is synthetic, from What Larkfield wants from its calls.
  • No mathematics beyond multiplying and dividing.

What you need

  • Headphones or speakers, to hear the clips. The lesson has a transcript of every clip, and a text description of every picture.
  • Nothing to install. You open the same calls on your computer in the next lesson.

A film looks like smooth movement. It is not. A camera takes 24 still pictures every second, and the projector shows them one after the other. Your eye cannot see the gaps, so it sees movement. If the camera took only 2 pictures a second, a running dog would jump from place to place. A fast movement between two pictures would be lost.

A sound recording works the same way. Sound is air that moves back and forth very quickly. A microphone turns that movement into an electric signal. A recorder then measures the signal many times every second and keeps each measurement as a number. Play the numbers back fast enough, and you hear the sound again.

Each measurement is called a sample. In a computer, a sample is usually a number between −1 and 1. The value 0 is the air at rest, and −1 and 1 are the loudest the recording can hold. The number of samples in one second is the sample rate. A phone line takes 8,000 samples a second.

Here is one sentence from a Larkfield call. The caller says: “Hi. Please explain how to assemble the Cedar raised bed before I fill it with soil.” On the phone line, that is 3.75 seconds of sound, so the file holds 29,984 numbers for it.

Two charts. The top chart is the waveform of one caller sentence, 3.75 seconds long, values between minus 1 and 1. Short bursts of sound with quiet gaps between the words. A dotted line marks a point near 0.3 seconds, in the word Hi. The bottom chart zooms into 5 milliseconds at that point: 40 blue dots joined by a thin line, rising and falling between about minus 0.4 and plus 0.55.
One caller sentence of call C-0225 on the phone line, drawn from the course file. Top: all of its numbers as a waveform. Bottom: 5 ms of the word "Hi"; each dot is one sample, 40 of them at 8,000 a second.
Text description of the diagram

The top chart is a waveform: each sample is a point, from left to right in time, and the height is its value. The sentence lasts 3.75 seconds. You can see a burst of sound for each group of words, and quiet gaps between them. The bottom chart zooms into 5 milliseconds of the first word, “Hi”. At 8,000 samples a second, 5 milliseconds hold 40 samples, and each one is a blue dot. The dots go up and down: that is the air moving back and forth.

These are the first eight numbers of the zoom above, read from the file. They cover one millisecond (1/1,000 of a second).

Sample 1 2 3 4 5 6 7 8
Value -0.1405 -0.1073 -0.016 0.076 0.0995 0.0721 0.0321 -0.0199

That is all a recording is. The picture of the numbers over time is called a waveform. A waveform shows when the sound is loud and when it is quiet. It does not show which sound it is: an “s” and an “f” look much the same in it. For that, you need another picture, which comes next.

A sound moves the air back and forth many times a second. The number of times is its frequency, measured in hertz (Hz). 1 Hz is once a second, and 1 kHz is 1,000 times a second. You hear frequency as pitch. A low voice moves the air about 100 times a second. The hiss of an “s” moves it several thousand times a second.

Go back to the film. To show a movement, the camera needs at least two pictures for each back-and-forth. Sound has the same rule: a sample rate can keep sounds only up to half of its rate. This limit is often called the Nyquist frequency, after the engineer who described it.

Sample rate Highest pitch it can keep Where you meet it
8,000 a second (8 kHz) 4,000 Hz the phone line, and Larkfield’s phone files
16,000 a second (16 kHz) 8,000 Hz “HD voice” lines, and what Whisper expects
48,000 a second (48 kHz) 24,000 Hz a good microphone, video, music

People hear sounds up to about 20,000 Hz, so a microphone at 48 kHz keeps everything you can hear. A phone line keeps much less. And the line itself removes more: a classic phone line passes only the sounds between about 300 and 3,400 Hz. Larkfield’s phone files do the same.

Have you ever spelled your name on the phone, and said “S as in Sam, F as in Frank”? There is a reason. The sounds “s” and “f” differ mostly in their high pitches, above about 3,400 Hz. A phone line removes those pitches, so on the phone the two letters sound almost the same. People invented spelling words to get around it.

A speech recogniser has the same problem, and it cannot ask “S as in Sam?”. To see what the line removes, we need a picture of pitch.

A spectrogram shows sound in three dimensions. Time goes from left to right, as in a waveform. Pitch goes from the bottom (low) to the top (high). The colour shows how loud each pitch is at each moment: bright is loud, dark is quiet. A program makes it by cutting the sound into short pieces (here 32 milliseconds) and measuring each pitch in each piece.

Here is the same sentence twice. The first file is what the phone line stored. The second is the same moment of the same call, as a wideband line stores it, at 16,000 samples a second. Listen to both: first the phone, then the wideband version.

Phone line: 8,000 samples a second, mu-law (the caller’s channel of call C-0225).
Wideband: 16,000 samples a second, after the G.722 codec (the same seconds of the same call).

Transcript (caller, 6.0 to 9.7 s): “Hi. Please explain how to assemble the Cedar raised bed before I fill it with soil.” The voice is synthetic: Piper, trained on the VCTK corpus (speaker p288).

Two spectrograms of the same 3.75-second sentence, side by side (one above the other on a phone). Both have time from 0 to 3.7 seconds across and pitch from 0 to 8 kHz up. The left one, the phone line at 8,000 samples a second, has colour only up to 4 kHz; above 4 kHz the area is empty and grey, with the words: nothing above 4 kHz, the file cannot hold it. A white dashed line at 3.4 kHz. The right one, wideband at 16,000 samples a second, has colour up to 8 kHz: bright vertical stripes reach far above the dashed line at several moments.
The same caller sentence on the phone line (left) and on a wideband line (right). The white dashed line is at 3,400 Hz, the top of a classic phone line. Drawn from the course files.
Text description of the diagram

Both pictures show the same sentence. Time goes across, and pitch goes up to 8 kHz. Bright colour means a loud pitch. In both, the bright horizontal bands at the bottom are the voice: its basic pitch and the pitches above it. In the wideband picture on the right, thin bright stripes rise to the top of the picture at several moments. They reach far above the dashed line at 3,400 Hz. These are the hissing sounds, such as the “s” sounds in “Please”, “assemble” and “soil”. In the phone picture on the left, the colour stops at 4 kHz, because a file with 8,000 samples a second cannot hold more. Above 3,400 Hz, the phone line has already made the sound much weaker.

You can hear the difference too. The wideband voice sounds clearer and closer. On the phone, the “s” sounds are dull. A person still understands the sentence, because the words around it help. A recogniser must do the same with less information, and it makes more mistakes on names and numbers.

Now look at a whole call. This is call C-0127 from the last lesson: the caller whose Pebble path lights had not arrived. The top picture is its waveform. The bottom one is its spectrogram. Both come from the wideband file, so the spectrogram reaches 8,000 Hz.

Two charts that share a time axis from 0 to 29 seconds. Top: the waveform of the whole call, with numbered circles 1 to 9 above the nine turns; circles of the agent's turns are blue and the caller's are orange. Bursts of sound for each turn, with short gaps between turns. Bottom: the spectrogram from 0 to 8 kHz. Each turn shows as a block of bright horizontal bands near the bottom, with thin stripes reaching higher. The gaps between turns are dark.
Call C-0127 (wideband file, 16,000 samples a second): the waveform shows when someone speaks; the spectrogram shows which pitches they use. The numbers are the nine turns of the call.
Text description of the diagram

The call lasts 29 seconds and has nine turns. The agent speaks in turns 1, 3, 5, 6, 7 and 9, and the caller in turns 2, 4 and 8. In the waveform, each turn is a burst of sound, and the short gaps between them are silence. In the spectrogram, each turn shows as bright horizontal bands near the bottom: the voice and the pitches above it. Thin stripes reach higher at the hissing sounds. The gaps are dark.

Turn Who What they said
1 Agent Thank you for calling Larkfield, this is Chloe. This call is recorded for quality and training. How can I help you today?
2 Caller Hi, um, good morning. My Pebble path lights haven’t arrived. Please tell me where they are.
3 Agent I can help with that. Can I have your order number, please?
4 Caller Yes, it’s 114942.
5 Agent Thank you. That’s order 114 942. It shipped on the twelfth of March, and it is on its way.
6 Agent I will ask the courier to call you before tomorrow’s delivery.
7 Agent Is there anything else I can help you with?
8 Caller No, that’s it, thanks a lot.
9 Agent Thank you for calling Larkfield. Have a good day.

Speech recognisers do not read the waveform. Most of them start from a picture very close to this spectrogram. You see how in Module 4.

How exact is each number: bit depth and mu-law

Section titled “How exact is each number: bit depth and mu-law”

A sample rate says how often the recorder measures. The bit depth says how exactly. A computer stores each sample as a whole number with a fixed number of bits. With 16 bits, there are 65,536 possible values between −1 and 1. With 8 bits, there are only 256. A value between two steps is rounded to the nearest one.

Phone systems use 8 bits a sample, to keep the data small. But speech is often quiet: many samples are close to 0. With 256 equal steps from −1 to 1, each step is 1/128 (about 0.0078). A quiet sample of 0.001 is rounded to 0, and it is gone.

The phone network’s answer is called mu-law (written μ-law). It still uses 256 steps, but they are not equal: they are very small near 0 and large near the loud end. Quiet sounds keep their detail, and loud sounds lose a little, where you cannot hear it. This table rounds four values both ways, with the course project’s own mu-law code:

Sample value 8 bits, equal steps Error 8 bits, mu-law Error
0.001 (very quiet) 0 100.0% 0.0011 6.0%
0.01 (quiet) 0.0078 21.9% 0.0102 2.3%
0.1 (normal) 0.1016 1.6% 0.1007 0.7%
0.5 (loud) 0.5 0.0% 0.4967 0.7%

The quiet values are where mu-law wins. That is why every Larkfield phone file is 8 bits of mu-law at 8,000 samples a second. Its data rate is 8,000 × 8 = 64,000 bits a second (64 kbit/s). That is about 28.8 MB for one hour of one channel.

Mu-law is one codec: a rule to code sound into fewer bits and to decode it again (the name comes from coder-decoder). Its standard name is G.711. Europe uses a close variant, A-law. Other lines use codecs that pack the sound much smaller, and they lose more. Malik, Larkfield’s telephony engineer, knows all of them. These are the lines in Larkfield’s call set:

Line Codec Data rate Public calls
Landline G.711 mu-law 64 kbit/s 160
Mobile Opus 8 kbit/s 59
Internet phone (VoIP) G.726 16 kbit/s 41
Wideband version of every call G.722 (“HD voice”) 64 kbit/s, 16,000 samples a second all

These codecs lose information: the decoded sound is not the original. In the call set, each line’s sound goes through its codec first. Then our code stores it as 8-bit mu-law at 8 kHz, as a phone system does. A lower data rate does not always mean more recognition errors. On Larkfield’s synthetic calls, Whisper small gets 8.6% of the words wrong on the landline. It gets 5.9% wrong on the mobile line and 12.2% on the VoIP line. But most VoIP calls are French, and French is harder for the recogniser. Inside English, the three lines differ by less than 1 point. Modules 2 and 4 show how to measure this, and how to split a slice.

A recording can have more than one list of numbers. Each list is a channel. Music is often stereo: two channels, for your left and right ear. Many phone systems record a call in stereo too, but in a useful way: the agent on one channel and the caller on the other. Other systems mix both people into one channel: mono.

Three waveforms of call C-0127 that share a time axis from 0 to 29 seconds. Channel 1, the agent, in blue: sound in turns 1, 3, 5, 6, 7 and 9, flat elsewhere. Channel 2, the caller, in orange: sound in turns 2, 4 and 8, flat elsewhere. Mono, both mixed into one channel, in grey: sound in all nine turns, and every turn is smaller than on its own channel.
Call C-0127 as the phone system stored it: the agent on channel 1, the caller on channel 2. The third row mixes both into one channel, as a mono recording does.
Text description of the diagram

On channel 1, only the agent’s turns have sound: 1, 3, 5, 6, 7 and 9. On channel 2, only the caller’s turns have sound: 2, 4 and 8. Each channel is almost flat while the other person speaks. In the mono row, both channels are mixed into one, so all nine turns are there, and you can no longer see who spoke. Every turn is also smaller in the mono row, because the mix is the average of the two channels.

Stereo is a gift. With stereo, you know who spoke each word without any model: the channel tells you. Module 5 uses this. With mono, a model must guess who spoke from the voices. In Larkfield’s 260 public calls, 150 are stereo and 110 are mono.

Mixing has a cost, too. To mix two channels, a program takes the average of the two numbers. When only one person speaks, the other channel is near 0, so that turn is about half as tall in the mix (about 6 dB quieter). Over a whole call the drop looks smaller, because the loudness number averages the turns of both people. You see this in call C-0127. The agent’s channel is -24.9 dBFS and the caller’s channel is -28.1 dBFS. The mono mix is -29.2 dBFS. The next section explains the unit.

Loudness is measured in decibels (dB). A decibel compares two levels: every 6 dB down is half the level, and every 20 dB down is one tenth. For recordings, the reference is the loudest value a file can hold, so the unit is dBFS, decibels below full scale. The loudest possible sound is 0 dBFS. Everything else is a negative number. Speech on a phone line is often between −30 and −15 dBFS.

To find the silence in a call, a program cuts the sound into short frames, here 30 milliseconds each. It measures the loudness of each frame, and calls a frame silent when it is quieter than a threshold. The course project uses −45 dBFS.

A line chart of the caller channel of call C-0127: loudness in dBFS from minus 90 to 0 over 29 seconds, one value per 30 millisecond frame. A dashed line at minus 45 dBFS; the area below it is shaded and labelled: below minus 45 dBFS, silence. The line stays near minus 57 most of the time. It jumps up to between minus 40 and minus 12 in three groups, at about 6 to 9 seconds, 12 to 13.6 seconds and 25 to 25.7 seconds.
The caller channel of call C-0127, one value for each 30 ms frame. Frames under the dashed line at −45 dBFS count as silence.
Text description of the diagram

The caller’s channel has 965 frames of 30 milliseconds. The line stays near -56.7 dBFS most of the time: that is the quiet noise of the phone line, not true silence. It rises above −45 dBFS only in three groups: the caller’s three turns. The loudest frame reaches -11.6 dBFS.

On the caller’s channel, 814 of the 965 frames are silent: 84%. That makes sense: the caller speaks for only about 5.7 seconds of the 29. In the mono mix, 40% of the frames are silent. Notice also that the “silence” is never 0: the line adds a quiet noise at about -56.7 dBFS.

Why does silence matter? A recogniser spends computer time on every second, silent or not. And some recognisers invent words in long silences. Module 3 measures both problems, and finds the speech first.

Every Larkfield call with audio comes in two versions:

  • calls/phone/C-xxxx.wav: what the phone system stored. 8,000 samples a second, 8-bit mu-law, stereo (agent and caller apart) or mono.
  • calls/wideband/C-xxxx.flac: the same call at 16,000 samples a second, mono, after the G.722 codec. FLAC packs audio without losing anything, like a ZIP file for sound.

The calls also have background sound, on purpose. Of the 260 public calls, 136 are quiet, 68 have other people talking in the background, and 56 have street noise. Some calls have music on hold.

Larkfield’s French calls are part of the data, not a translation. But open French voices with clear licences are rare. The call set uses only three: Piper’s fr_FR-gilles-low voice, Piper’s fr_FR-siwis-medium voice, and Kokoro’s ff_siwis voice. The last two learned from the same person, so they are never in the same call. Listen to the start of a French call:

The start of call C-0065 on the phone line (8,000 samples a second). Both voices are synthetic. The agent, Julien, is Piper trained on a public-domain French set with one speaker (CC0). The caller is Piper trained on the SIWIS corpus.

Transcript. Agent (0.6 s): “Merci d’avoir appelé Larkfield, ici Julien. Cet appel est enregistré pour la qualité et la formation. Comment puis-je vous aider aujourd’hui ?” Caller (8.8 s): “Bonjour madame. Comment assembler le Cedar raised bed et fixer les coins ? J’ai besoin des étapes de base pour le monter correctement.” (In English: “Thank you for calling Larkfield, this is Julien. This call is recorded for quality and training. How can I help you today?” “Hello. How do I assemble the Cedar raised bed and fix the corners? I need the basic steps to put it together correctly.”)

Three voices are not many. A recogniser that does well on them may do worse on other French speakers. And part of the French error comes from the voices themselves. Piper’s French voices sometimes drop the nasal vowels: the program that made the calls warned about it. So when a French number is bad in this course, part of the blame is ours, not the recogniser’s. The case studies use recordings of many real French speakers.

  • Keep the original recording as it came from the phone system. Every change can lose information, and no change can bring it back.
  • Change the sample rate only for the model, just before the model reads the audio. Whisper expects 16,000 samples a second, so the course project converts each call to that rate in memory.
  • Record in stereo if your phone system can. It gives you “who spoke” for free.
  • Count the storage. One hour of one phone channel is about 28.8 MB. At 16,000 samples a second with 16 bits, it is 115.2 MB before compression. Recordings of voices are personal data, so every copy must also be protected and deleted on time (Module 9).

Common mistake

A higher sample rate does not bring the lost sounds back

Symptom: Tomás converts every phone call from 8,000 to 16,000 samples a second before the recogniser. “Now it is wideband audio,” he says. The files are twice as big, and the transcripts are no better.

Cause: the sounds above 4,000 Hz were never in the phone file, and the line had already weakened everything above 3,400 Hz. Changing the rate adds new samples between the old ones, but it cannot invent pitches that are not there. On the sentence of this lesson, the converted phone file has 0.002% of its sound energy above 4,000 Hz. The true wideband file has 0.4% there. That is small, but it is where “s” and “f” differ.

Fix: convert to 16,000 samples a second only because the model expects that rate, and keep the stored files as they are. If you need wideband sound, you need a wideband line, such as G.722, from the phone system.

Consequence in AI work: a team that believes its audio is wideband trusts results that it has not measured. Always check the real sample rate of the source, not only of the file.

Check your understanding

A recording has 16,000 samples a second. What is the highest pitch it can keep? How many samples does a 3-second sentence have? And on a phone line at 8,000 samples a second?

Show the answer

At 16,000 samples a second, the highest pitch is half the rate: 8,000 Hz. A 3-second sentence has 3 × 16,000 = 48,000 samples. On the phone line, the highest pitch is 4,000 Hz, and the same sentence has 3 × 8,000 = 24,000 samples. The line also passes only about 300 to 3,400 Hz, so the real limit on the phone is lower than 4,000 Hz.

Check your understanding

Why do phone systems use mu-law instead of 256 equal steps? Use one value from the table in your answer.

Show the answer

Both use 8 bits a sample (256 steps), but speech is often quiet, and equal steps are too coarse near 0. With equal steps, the quiet value 0.01 becomes 0.0078, an error of 21.9%; the very quiet value 0.001 becomes 0. Mu-law has small steps near 0 and large steps for loud sounds, so 0.01 becomes 0.0102 (2.3% error). The loud value 0.5 loses a little (0.7%), where a listener cannot hear it.

Check your understanding

Larkfield's new phone system can record each call in stereo (agent and caller apart) or in mono. Which do you choose, and what does each choice cost?

Show the answer

Stereo. With the agent and the caller on separate channels, you know who said each word without a model. Module 5 uses this. Each voice also keeps its full level. Mono mixes both voices, so a model must guess who spoke from the voices. The mix also makes each turn about half as tall, about 6 dB quieter (over the whole of call C-0127: agent −24.9 dBFS, caller −28.1, mix −29.2). Stereo costs twice the storage: two channels of about 28.8 MB an hour each on a G.711 line.

Recap

  • A recording is a list of samples: numbers between −1 and 1. The sample rate is the number of samples in a second. It is 8,000 on a phone line, 16,000 for Whisper and 48,000 for a good microphone.
  • A sample rate keeps pitches only up to half its rate. The phone line also removes most sound above 3,400 Hz, where “s” and “f” differ. A spectrogram shows it: time across, pitch up, loudness as colour.
  • Bit depth says how exactly each sample is stored. Phone systems use 8 bits of mu-law (G.711), which keeps quiet sounds better than equal steps: 64 kbit/s.
  • Codecs pack sound into fewer bits and lose some of it: G.711, Opus, G.726 and G.722 in Larkfield’s calls.
  • Stereo calls keep the agent and the caller apart; mono mixes them. Loudness is in dBFS (0 is the loudest), and silence is the frames under a threshold, here −45 dBFS.
  • The French calls use only three synthetic voices, so French results tell less about real French callers.

You can now read a call as numbers and see what the phone line removed. But you have not opened a call on your own computer yet. Next, you set up the course project, play and transcribe a call, and check your setup.

Next: Set up the speech project