Split by channel first
- Lesson 15
- 30 min
- Module 5: Who spoke when: diarization and timing
Where we are
We now have good transcripts, but they do not say who spoke. Grace needs to know what the agent promised. In this lesson, we split speakers by channel, the simplest and best method when the phone system records each side apart.
Lesson 15 of 36 · Speech AI: Transcribe, Understand and SpeakSplit by channel first
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Find who spoke when in a phone call. When the phone system records the agent and the caller on separate channels, each channel is one person. Then the job is free, and it is the best method you can get.
NotebookDo this lesson in a notebook
The notebook has the code, the tasks and the checks of this lesson. It computes the diarization error rate of a tiny example by hand, and splits a stereo call by channel. It also shows what background talk and a mono recording do. It needs no model download and no account. About 25 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- What voice activity detection (VAD) does: it marks the parts of a recording where someone speaks. See Speech or silence? and WebRTC VAD and Silero VAD.
- What a stereo phone recording is: two channels, one for each side. See Channels: stereo and mono in Audio as data.
What you need on your computer (only for the local route; the notebook needs none of it)
- Your project folder
larkspeechat the end of Module 4, with the course data inlarkspeech/data.speech transcribe --helpworks in it. - The labs repository
nextia-learning-labsnext tolarkspeech. This lesson copies five files from the snapshotend-of-m05and installs two new packages.
Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.
Two ears, two people
Section titled “Two ears, two people”Think of a team lead at a call centre. She checks a recorded call with headphones. The phone system put the agent in her left ear and the caller in her right ear. She never asks “who said that?”. The side of her head tells her.
Now she gets the same call as one track, both voices in both ears. She can still follow it, but now she must listen to the voices. When the two voices sound alike, she must think hard.
A computer has the same two cases. Some phone systems record each side of the call on its own channel. Others mix both people into one channel. This module is about the job of the team lead: finding who spoke when.
The problem: a transcript without names
Section titled “The problem: a transcript without names”Module 4 gave us good transcripts. But a transcript is one long text. Here is the end of call C-0146, as a recogniser writes it:
… it shipped on the ninth of July and it is on its way I will ask the courier to call you before tomorrow’s delivery …
… is there anything else I can help you with that’s everything thank you …
Grace, the support lead, must check what the agent promised. Here the agent promised a call from the courier. If the call record says that the caller said it, nobody follows up, and the promise is broken. Grace’s cost table gives a missed follow-up a cost of US$25 (an assumption). A transcript without speaker names cannot tell her.
The idea: diarization
Section titled “The idea: diarization”Finding who spoke when is called diarization. The word comes from “diary”: a record of what happened, and when. A diarization system takes a recording and returns turns: a start time, an end time and a speaker for each part of speech. For example: from 0.9 s to 7.9 s, the agent; from 8.5 s to 13.7 s, the caller.
Malik, the telephony engineer, knows a shortcut. Larkfield’s phone system can record each call in stereo: the agent on channel 1 and the caller on channel 2. Then each channel holds one person. Diarization becomes VAD on each channel: the speech on channel 1 is the agent, and the speech on channel 2 is the caller. No speaker model is needed.
Of Larkfield’s 110 validation and test calls, 66 are stereo and 44 are mono. This lesson uses the stereo calls. The next lesson handles the mono calls.
How to measure it: the diarization error rate
Section titled “How to measure it: the diarization error rate”We need one number for “how good are these turns?”, as WER was for transcripts. It is the diarization error rate, or DER. It counts three kinds of error, in seconds:
- Missed speech: someone spoke, but the system found no speaker.
- False speech: the system found a speaker, but nobody spoke.
- Speaker confusion: the system gave the speech to the wrong person.
DER = (missed + false + confusion) / true speech time
The system’s speakers may have other names, such as “A” and “B”. So DER first matches them to the true names in the way that gives the fewest errors. And it does not score a short collar of 0.25 s around each true change of speaker. Nobody can say to the hundredth of a second where a turn starts.
A tiny worked example
Section titled “A tiny worked example”The true turns: the agent speaks from 0 to 2 s, and the caller from 2 to 4 s. The system found speaker “A” from 0 to 3 s and speaker “B” from 3 to 3.5 s. We use no collar here.
Text description of the diagram
The top row is the truth: the agent from 0 to 2 s, then the caller from 2 to 4 s. The middle row is what the system found: A from 0 to 3 s, then B from 3 to 3.5 s. A is matched to the agent, and B to the caller. The bottom row marks the errors. From 0 to 2 s, A is the agent: correct. From 2 to 3 s, A still speaks, but the true speaker is the caller: 1 s of confusion. From 3 to 3.5 s, B is the caller: correct. From 3.5 to 4 s, nobody was found: 0.5 s of missed speech. There is no false speech.
| Part | Seconds | Share of the 4 s of speech |
|---|---|---|
| Missed | 0.5 | 12.5% |
| False | 0 | 0% |
| Confusion | 1.0 | 25.0% |
| DER | 1.5 | 37.5% |
Always look at the three parts, not only the total. They have different causes and different fixes. Missed and false speech come from the VAD: it decides when someone speaks. Confusion comes from the method that decides who speaks.
Split a stereo call
Section titled “Split a stereo call”The method has four steps:
Text description of the diagram
The diagram starts with a stereo call: the agent on channel 1 and the caller on channel 2. Step 1 runs a VAD on each channel. The project uses Silero VAD, and keeps its result as speech or silence for each frame of 10 ms. Step 2 handles the frames where both channels seem to have speech: the louder channel keeps the frame. Step 3 turns the speech of channel 1 into agent turns, and the speech of channel 2 into caller turns. Step 4 measures the DER against the true turns.
Step 2 needs an explanation. In a quiet call, the agent and the caller rarely speak at the same time. So if both channels have speech, something is wrong. Usually, one side has noise that the VAD takes for speech.
When the caller’s side is not quiet
Section titled “When the caller’s side is not quiet”Listen to the caller’s channel of call C-0056 during the first seconds of the call. The caller is in a room where other people talk. At this time, the agent is speaking on the other channel.
Background talk is real speech, so the VAD on the caller’s channel calls it speech. Then the “caller” seems to talk while the agent talks. The picture shows the first 24 seconds of this call.
Text description of the diagram
The top two rows are the sound of the two channels. Channel 1 has the agent’s turns: the greeting from 0.9 to 7.8 s, then turns from 12.8 to 15.9 s and from 19.6 s. Channel 2 has the caller’s turns from 8.5 to 12.1 s and from 16.6 to 18.8 s. It also has a low, steady band of background talk all the time. The three rows below are turns. Truth: agent, caller, agent, caller, agent. Each side alone: the VAD on channel 2 finds speech from 0 s to 38.4 s without a break, so the caller row covers the whole picture, and the agent’s turns are found on top of it. Louder wins: in the frames where both channels have speech, the louder side keeps the frame. The agent keeps most of its turns, in short pieces, and the caller keeps the gaps between the agent’s words.
The fix is the “louder wins” rule of step 2. In a 10 ms frame where both channels have speech, the louder channel keeps the frame. On C-0056, it cuts the confusion from 77.2% to 15.6% of the speech time, and the DER falls from 84.2% to 42.2%. It does not fix everything: 19.6% of the speech is now missed, and the agent’s turns are cut into short pieces. These numbers are on Larkfield’s synthetic calls, as all the numbers of this lesson.
Larkfield: diarize with the project
Section titled “Larkfield: diarize with the project”The snapshot end-of-m05 adds the command speech diarize. It uses Silero VAD from Module 3 and the DER of this lesson. It also brings the speaker model for the next lesson.
-
In your folder
larkspeech, with the virtual environment active, copy the files of this module from the snapshotend-of-m05:Terminal window S=../nextia-learning-labs/C41/snapshots/end-of-m05cp "$S/larkspeech/models.py" larkspeech/cp "$S/larkspeech/diarize.py" larkspeech/cp "$S/larkspeech/commands/diarize.py" larkspeech/commands/cp "$S/tests/test_diarize.py" tests/cp "$S/requirements.txt" .Terminal window $S = "..\nextia-learning-labs\C41\snapshots\end-of-m05"Copy-Item "$S\larkspeech\models.py" larkspeech\Copy-Item "$S\larkspeech\diarize.py" larkspeech\Copy-Item "$S\larkspeech\commands\diarize.py" larkspeech\commands\Copy-Item "$S\tests\test_diarize.py" tests\Copy-Item "$S\requirements.txt" .The new
requirements.txtadds two lines:sherpa-onnx==2.0.1, which runs the speaker model of the next lesson, andscikit-learn==1.9.1, for the clustering. -
Install the new packages:
Terminal window pip install -r requirements.txt -
Run the tests. Three new tests check the DER and the louder-wins rule:
Terminal window pytest -q........................ [100%]24 passed in 12.10sThe time changes from one computer to another.
-
Diarize the 40 stereo test calls by channel. It takes a few seconds:
Terminal window speech diarize --method channel --split testchannel on 40 test calls: mean DER 31.5%missed 26.4%, false 1.2%, confusion 3.9%worst: C-0096 61%, C-0167 56%, C-0248 55%The first line is the mean DER of the 40 calls. The second line gives its three parts. The last line names the three calls with the highest DER.
-
Run it again without the louder-wins rule:
Terminal window speech diarize --method channel --split test --each-side-alonechannel on 40 test calls: mean DER 36.4%missed 23.9%, false 1.2%, confusion 11.3%worst: C-0056 84%, C-0248 76%, C-0167 76%The confusion is almost three times higher. The worst calls are calls with background talk.
-
Look at one quiet call, C-0052, turn by turn:
Terminal window speech diarize --method channel --call C-0052C-0052 (stereo), channel: DER 4.8% (missed 4.8%, false 0.0%, confusion 0.0%)0.89 3.23 agent3.45 5.89 agent6.17 7.52 agent8.53 12.70 caller13.21 16.03 agent…Each line is one turn: start, end, speaker. The VAD splits the agent’s greeting at its pauses, but every turn has the right name.
Where the error comes from
Section titled “Where the error comes from”The chart shows the three parts of the DER on all 66 stereo calls of the validation and test sets.
Text description of the diagram
Two horizontal bars, each split into three parts. Louder wins: missed speech 23.8%, false speech 1.2% and confusion 3.8%, a DER of 28.7%. Each side alone: missed speech 21.2%, false speech 1.2% and confusion 11.1%, a DER of 33.5%. Louder wins drops a little real speech, so missed speech rises by 2.6 points. But it removes 7.3 points of confusion.
Most of the DER is missed speech, about 24%. The channels are not the cause: the VAD is. At its default threshold of 0.5, Silero VAD misses about 19% of the speech time on our phone audio. WebRTC VAD and Silero VAD measured it. Confusion is only 3.8%. So the channels answer “who” very well. To improve the DER further, you would tune the VAD, which answers “when”.
Missed speech matters less for Grace than confusion. A missed half-second at the edge of a turn rarely changes the call record. A sentence given to the wrong person can change it completely.
When to use it, and what it costs
Section titled “When to use it, and what it costs”Use it whenever the phone system can record each side apart. It costs nothing: no model, no download, and a few seconds for 40 calls on a laptop. Its confusion is the lowest of every method in this module. If your calls are mono today, ask the phone provider for stereo recording. It is often a setting. One setting can be worth more than any model.
Do not trust it blindly in these cases:
- One channel holds two people. The caller passes the phone to a partner, or the agent transfers the call to a colleague. Each channel still has one “side”, but not one person.
- Echo. On some lines, the agent’s voice leaks into the caller’s channel. Louder wins helps, because the leak is quieter than the original.
- The channels are in another order. Not every phone system puts the agent on channel 1. Check it once with a known call (see the failure case below).
The cost of this method is a decision at the phone system, not compute. Stereo files are twice as large as mono files. For G.711 phone audio, that is 16 KB a second instead of 8 KB.
Guided practice: split a stereo call
Section titled “Guided practice: split a stereo call”- Open the notebook from the box Do this lesson in a notebook at the top of this page.
- Run the setup cells and the helper cell. The setup cell ends with a
Ready:line. - In section 2, run the cell with
der. Predict the DER of the tiny example, then run the next cell. You see a DER of 0.375: missed 0.125, confusion 0.25. - In section 3, run the split of call C-0069. The notebook uses the energy VAD of Module 3. You see a DER of 0.118, all of it missed speech, and no confusion.
- Run the drawing cell. Compare the true turns with the found turns.
- In section 4, predict what background talk does to call C-0056 at -40 dBFS. Then run the cell. You see a DER of 0.871, and most of it is confusion (0.764). At -30 dBFS, the DER is 0.226.
Your turn: what a mono recording loses
Section titled “Your turn: what a mono recording loses”In section 6 of the notebook, mix the two channels of C-0069 into one, with stereo.mean(axis=1). Run the energy VAD on the mix at -40 dBFS. Give every speech region the name "agent", because one channel cannot tell you who speaks. Then compute the DER against the true turns. Before you run it, predict: which part of the DER will grow most?
Show the answer
mono = stereo.mean(axis=1)mono_turns = [(s, e, "agent") for s, e in energy_vad(mono, sr, -40)]mono_der = der(mono_turns, truth, call["duration"])["der"]print(round(mono_der, 3))0.435The DER rises from 0.118 to 0.435. The new error is confusion. The caller speaks about 43% of the speech time in this call, and all of it now goes to the agent. One channel keeps the words, but it loses the “who”. The next lesson tells the voices apart.
Common mistake
The channels the wrong way round
Symptom: Amira sets up a second phone system for the French line. The DER of its calls looks normal. But in the call records, the caller now promises refunds, and the agent asks for them.
swapped = by_channel(stereo, sr, speakers=("caller", "agent"))print("DER with the names swapped:", round(der(swapped, truth, call["duration"])["der"], 3))first = swapped[0]print("the first turn is labelled", first[2], "- the truth says", truth[0][2])DER with the names swapped: 0.118the first turn is labelled caller - the truth says agentCause: the new system puts the caller on channel 1. The DER does not see this, because it matches the names in the best way before it counts errors. A complete swap costs nothing in DER.
Fix: check the channel order once for each phone system, with a known call. Larkfield’s agents speak first and say “Thank you for calling Larkfield”. Then test the names of the turns, not only the DER: for example, the first turn of every call must be the agent.
Consequence in AI work: a summary model reads the names as facts. With swapped channels, every promise goes to the caller, every follow-up is lost, and the DER report still says that all is well.
Knowledge check
Section titled “Knowledge check”Check your understanding
In a 10-second stretch of a call, people speak for 8 seconds. The system misses 1 second of speech, finds speech in 0.4 seconds of silence, and gives 0.6 seconds to the wrong speaker. What is the DER, and which part should you fix first?
Show the answer
DER = (missed + false + confusion) / true speech time = (1 + 0.4 + 0.6) / 8 = 2 / 8 = 25%. Missed speech is the largest part, so tuning the VAD gives the biggest drop in DER. But for decisions, the 0.6 seconds of confusion can matter more: a sentence with the wrong speaker can change the call record. Look at both.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says that the louder-wins rule is a hack, and that a better VAD would remove the need for it. Is he right on Larkfield's calls?
Show the answer
Only partly. The background talk on the caller’s side is real speech, so a good VAD often calls it speech. A better VAD would not remove it. In a phone call, the two sides rarely speak at the same time, so the louder side is a good guess for who really speaks. On the 66 stereo validation and test calls, louder wins cuts the confusion from 11.1% to 3.8% of the speech time. A better VAD would mostly cut the missed speech, which is a different part of the DER.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks why the course does not use a speaker model on every call, since it would also work on stereo calls. What do you tell him?
Show the answer
When the phone system records each side apart, the channel tells you who speaks for free. It needs no model and no download, and a few seconds for 40 calls. And it confuses speakers less than a speaker model. The confusion is 3.8% by channel on the stereo calls, and 15.6% for speaker embeddings on the mono calls (the next lesson). Use a speaker model only where you have no channels. The cheapest improvement is often to ask the phone provider for stereo recording.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Diarization finds who spoke when: turns with a start, an end and a speaker.
- DER = (missed + false + confusion) / true speech time. Names are matched first, and a collar of 0.25 s around each change is not scored. Always report the three parts apart.
- In a stereo call, each channel is one side, so diarization is a VAD on each channel. Louder wins stops background talk from looking like the caller.
- On Larkfield’s 66 stereo validation and test calls: DER 28.7%, of which missed speech 23.8% (the VAD) and confusion only 3.8%.
- A swapped channel order does not change the DER, but it puts every sentence under the wrong name. Check the names.
Channels give the “who” for free, but 44 of Larkfield’s 110 validation and test calls are mono. Next, you tell two voices apart by how they sound, with speaker embeddings and clustering.