Skip to content
Français

Split by channel first

  • Lesson 15
  • 30 min
  • Module 5: Who spoke when: diarization and timing

Where we are

We now have good transcripts, but they do not say who spoke. Grace needs to know what the agent promised. In this lesson, we split speakers by channel, the simplest and best method when the phone system records each side apart.

What you will be able to do

Find who spoke when in a phone call. When the phone system records the agent and the caller on separate channels, each channel is one person. Then the job is free, and it is the best method you can get.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It computes the diarization error rate of a tiny example by hand, and splits a stereo call by channel. It also shows what background talk and a mono recording do. It needs no model download and no account. About 25 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

What you need on your computer (only for the local route; the notebook needs none of it)

  • Your project folder larkspeech at the end of Module 4, with the course data in larkspeech/data. speech transcribe --help works in it.
  • The labs repository nextia-learning-labs next to larkspeech. This lesson copies five files from the snapshot end-of-m05 and installs two new packages.

Tested on macOS with Python 3.12. The Windows (PowerShell) commands were not tested.

Think of a team lead at a call centre. She checks a recorded call with headphones. The phone system put the agent in her left ear and the caller in her right ear. She never asks “who said that?”. The side of her head tells her.

Now she gets the same call as one track, both voices in both ears. She can still follow it, but now she must listen to the voices. When the two voices sound alike, she must think hard.

A computer has the same two cases. Some phone systems record each side of the call on its own channel. Others mix both people into one channel. This module is about the job of the team lead: finding who spoke when.

Module 4 gave us good transcripts. But a transcript is one long text. Here is the end of call C-0146, as a recogniser writes it:

… it shipped on the ninth of July and it is on its way I will ask the courier to call you before tomorrow’s delivery …

… is there anything else I can help you with that’s everything thank you …

Grace, the support lead, must check what the agent promised. Here the agent promised a call from the courier. If the call record says that the caller said it, nobody follows up, and the promise is broken. Grace’s cost table gives a missed follow-up a cost of US$25 (an assumption). A transcript without speaker names cannot tell her.

Finding who spoke when is called diarization. The word comes from “diary”: a record of what happened, and when. A diarization system takes a recording and returns turns: a start time, an end time and a speaker for each part of speech. For example: from 0.9 s to 7.9 s, the agent; from 8.5 s to 13.7 s, the caller.

Malik, the telephony engineer, knows a shortcut. Larkfield’s phone system can record each call in stereo: the agent on channel 1 and the caller on channel 2. Then each channel holds one person. Diarization becomes VAD on each channel: the speech on channel 1 is the agent, and the speech on channel 2 is the caller. No speaker model is needed.

Of Larkfield’s 110 validation and test calls, 66 are stereo and 44 are mono. This lesson uses the stereo calls. The next lesson handles the mono calls.

How to measure it: the diarization error rate

Section titled “How to measure it: the diarization error rate”

We need one number for “how good are these turns?”, as WER was for transcripts. It is the diarization error rate, or DER. It counts three kinds of error, in seconds:

  • Missed speech: someone spoke, but the system found no speaker.
  • False speech: the system found a speaker, but nobody spoke.
  • Speaker confusion: the system gave the speech to the wrong person.

DER = (missed + false + confusion) / true speech time

The system’s speakers may have other names, such as “A” and “B”. So DER first matches them to the true names in the way that gives the fewest errors. And it does not score a short collar of 0.25 s around each true change of speaker. Nobody can say to the hundredth of a second where a turn starts.

The true turns: the agent speaks from 0 to 2 s, and the caller from 2 to 4 s. The system found speaker “A” from 0 to 3 s and speaker “B” from 3 to 3.5 s. We use no collar here.

Three rows on a time axis from 0 to 4 seconds. Truth: agent from 0 to 2, caller from 2 to 4. Found: A from 0 to 3, B from 3 to 3.5. Errors: correct from 0 to 2, confusion from 2 to 3, correct from 3 to 3.5, missed from 3.5 to 4.
The worked example. A is matched to the agent and B to the caller. 1 s of confusion and 0.5 s of missed speech in 4 s of speech: DER 37.5%.
Text description of the diagram

The top row is the truth: the agent from 0 to 2 s, then the caller from 2 to 4 s. The middle row is what the system found: A from 0 to 3 s, then B from 3 to 3.5 s. A is matched to the agent, and B to the caller. The bottom row marks the errors. From 0 to 2 s, A is the agent: correct. From 2 to 3 s, A still speaks, but the true speaker is the caller: 1 s of confusion. From 3 to 3.5 s, B is the caller: correct. From 3.5 to 4 s, nobody was found: 0.5 s of missed speech. There is no false speech.

Part Seconds Share of the 4 s of speech
Missed 0.5 12.5%
False 0 0%
Confusion 1.0 25.0%
DER 1.5 37.5%

Always look at the three parts, not only the total. They have different causes and different fixes. Missed and false speech come from the VAD: it decides when someone speaks. Confusion comes from the method that decides who speaks.

The method has four steps:

A stereo call, with the agent on channel 1 and the caller on channel 2, goes to step 1, VAD on each channel. Step 2, louder wins: when both channels say speech, the louder side keeps it. Step 3, turns: channel 1 speech is the agent, channel 2 speech is the caller. Step 4, score: DER against the true turns.
Diarization by channel: no speaker model, only a VAD on each side of the call.
Text description of the diagram

The diagram starts with a stereo call: the agent on channel 1 and the caller on channel 2. Step 1 runs a VAD on each channel. The project uses Silero VAD, and keeps its result as speech or silence for each frame of 10 ms. Step 2 handles the frames where both channels seem to have speech: the louder channel keeps the frame. Step 3 turns the speech of channel 1 into agent turns, and the speech of channel 2 into caller turns. Step 4 measures the DER against the true turns.

Step 2 needs an explanation. In a quiet call, the agent and the caller rarely speak at the same time. So if both channels have speech, something is wrong. Usually, one side has noise that the VAD takes for speech.

Listen to the caller’s channel of call C-0056 during the first seconds of the call. The caller is in a room where other people talk. At this time, the agent is speaking on the other channel.

Channel 2 (the caller’s side) of call C-0056, from 0.5 to 8.0 s, phone quality (8 kHz). The caller says nothing. You hear background talk that our code made: three synthetic voices (Piper, trained on the LibriTTS-R corpus) reading other lines at once, quietly. Transcript: no words from the caller. At the same time, on channel 1, the agent says: “Thank you for calling Larkfield, this is Freya. This call is recorded for quality and training. How can I help you today?”

Background talk is real speech, so the VAD on the caller’s channel calls it speech. Then the “caller” seems to talk while the agent talks. The picture shows the first 24 seconds of this call.

Two waveforms, channel 1 with the agent's turns and channel 2 with the caller's turns over a low, steady band of background talk. Under them, three rows of turns. Truth: agent, caller, agent, caller, agent. Each side alone: one long caller bar across the whole picture, with the agent's turns on top of it. Louder wins: the agent keeps its turns, cut into short pieces, and the caller keeps the gaps between them.
Call C-0056, the first 24 seconds. Each side alone, the VAD on the caller's side takes the background talk for speech, so the 'caller' talks during every agent turn. Louder wins gives most of those frames back to the agent. Whole call: DER 84.2% each side alone, 42.2% with louder wins.
Text description of the diagram

The top two rows are the sound of the two channels. Channel 1 has the agent’s turns: the greeting from 0.9 to 7.8 s, then turns from 12.8 to 15.9 s and from 19.6 s. Channel 2 has the caller’s turns from 8.5 to 12.1 s and from 16.6 to 18.8 s. It also has a low, steady band of background talk all the time. The three rows below are turns. Truth: agent, caller, agent, caller, agent. Each side alone: the VAD on channel 2 finds speech from 0 s to 38.4 s without a break, so the caller row covers the whole picture, and the agent’s turns are found on top of it. Louder wins: in the frames where both channels have speech, the louder side keeps the frame. The agent keeps most of its turns, in short pieces, and the caller keeps the gaps between the agent’s words.

The fix is the “louder wins” rule of step 2. In a 10 ms frame where both channels have speech, the louder channel keeps the frame. On C-0056, it cuts the confusion from 77.2% to 15.6% of the speech time, and the DER falls from 84.2% to 42.2%. It does not fix everything: 19.6% of the speech is now missed, and the agent’s turns are cut into short pieces. These numbers are on Larkfield’s synthetic calls, as all the numbers of this lesson.

The snapshot end-of-m05 adds the command speech diarize. It uses Silero VAD from Module 3 and the DER of this lesson. It also brings the speaker model for the next lesson.

  1. In your folder larkspeech, with the virtual environment active, copy the files of this module from the snapshot end-of-m05:

    Terminal window
    S=../nextia-learning-labs/C41/snapshots/end-of-m05
    cp "$S/larkspeech/models.py" larkspeech/
    cp "$S/larkspeech/diarize.py" larkspeech/
    cp "$S/larkspeech/commands/diarize.py" larkspeech/commands/
    cp "$S/tests/test_diarize.py" tests/
    cp "$S/requirements.txt" .

    The new requirements.txt adds two lines: sherpa-onnx==2.0.1, which runs the speaker model of the next lesson, and scikit-learn==1.9.1, for the clustering.

  2. Install the new packages:

    Terminal window
    pip install -r requirements.txt
  3. Run the tests. Three new tests check the DER and the louder-wins rule:

    Terminal window
    pytest -q
    ........................ [100%]
    24 passed in 12.10s

    The time changes from one computer to another.

  4. Diarize the 40 stereo test calls by channel. It takes a few seconds:

    Terminal window
    speech diarize --method channel --split test
    channel on 40 test calls: mean DER 31.5%
    missed 26.4%, false 1.2%, confusion 3.9%
    worst: C-0096 61%, C-0167 56%, C-0248 55%

    The first line is the mean DER of the 40 calls. The second line gives its three parts. The last line names the three calls with the highest DER.

  5. Run it again without the louder-wins rule:

    Terminal window
    speech diarize --method channel --split test --each-side-alone
    channel on 40 test calls: mean DER 36.4%
    missed 23.9%, false 1.2%, confusion 11.3%
    worst: C-0056 84%, C-0248 76%, C-0167 76%

    The confusion is almost three times higher. The worst calls are calls with background talk.

  6. Look at one quiet call, C-0052, turn by turn:

    Terminal window
    speech diarize --method channel --call C-0052
    C-0052 (stereo), channel: DER 4.8% (missed 4.8%, false 0.0%, confusion 0.0%)
    0.89 3.23 agent
    3.45 5.89 agent
    6.17 7.52 agent
    8.53 12.70 caller
    13.21 16.03 agent
    …

    Each line is one turn: start, end, speaker. The VAD splits the agent’s greeting at its pauses, but every turn has the right name.

The chart shows the three parts of the DER on all 66 stereo calls of the validation and test sets.

Two horizontal stacked bars. Louder wins: missed 23.8%, false 1.2%, confusion 3.8%, total 28.7%. Each side alone: missed 21.2%, false 1.2%, confusion 11.1%, total 33.5%.
The parts of the DER by channel, on the 66 stereo validation and test calls (Larkfield's synthetic calls). Missed speech is most of the error in both; louder wins cuts the confusion from 11.1% to 3.8%.
Text description of the diagram

Two horizontal bars, each split into three parts. Louder wins: missed speech 23.8%, false speech 1.2% and confusion 3.8%, a DER of 28.7%. Each side alone: missed speech 21.2%, false speech 1.2% and confusion 11.1%, a DER of 33.5%. Louder wins drops a little real speech, so missed speech rises by 2.6 points. But it removes 7.3 points of confusion.

Most of the DER is missed speech, about 24%. The channels are not the cause: the VAD is. At its default threshold of 0.5, Silero VAD misses about 19% of the speech time on our phone audio. WebRTC VAD and Silero VAD measured it. Confusion is only 3.8%. So the channels answer “who” very well. To improve the DER further, you would tune the VAD, which answers “when”.

Missed speech matters less for Grace than confusion. A missed half-second at the edge of a turn rarely changes the call record. A sentence given to the wrong person can change it completely.

Use it whenever the phone system can record each side apart. It costs nothing: no model, no download, and a few seconds for 40 calls on a laptop. Its confusion is the lowest of every method in this module. If your calls are mono today, ask the phone provider for stereo recording. It is often a setting. One setting can be worth more than any model.

Do not trust it blindly in these cases:

  • One channel holds two people. The caller passes the phone to a partner, or the agent transfers the call to a colleague. Each channel still has one “side”, but not one person.
  • Echo. On some lines, the agent’s voice leaks into the caller’s channel. Louder wins helps, because the leak is quieter than the original.
  • The channels are in another order. Not every phone system puts the agent on channel 1. Check it once with a known call (see the failure case below).

The cost of this method is a decision at the phone system, not compute. Stereo files are twice as large as mono files. For G.711 phone audio, that is 16 KB a second instead of 8 KB.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The setup cell ends with a Ready: line.
  3. In section 2, run the cell with der. Predict the DER of the tiny example, then run the next cell. You see a DER of 0.375: missed 0.125, confusion 0.25.
  4. In section 3, run the split of call C-0069. The notebook uses the energy VAD of Module 3. You see a DER of 0.118, all of it missed speech, and no confusion.
  5. Run the drawing cell. Compare the true turns with the found turns.
  6. In section 4, predict what background talk does to call C-0056 at -40 dBFS. Then run the cell. You see a DER of 0.871, and most of it is confusion (0.764). At -30 dBFS, the DER is 0.226.

In section 6 of the notebook, mix the two channels of C-0069 into one, with stereo.mean(axis=1). Run the energy VAD on the mix at -40 dBFS. Give every speech region the name "agent", because one channel cannot tell you who speaks. Then compute the DER against the true turns. Before you run it, predict: which part of the DER will grow most?

Show the answer
mono = stereo.mean(axis=1)
mono_turns = [(s, e, "agent") for s, e in energy_vad(mono, sr, -40)]
mono_der = der(mono_turns, truth, call["duration"])["der"]
print(round(mono_der, 3))
0.435

The DER rises from 0.118 to 0.435. The new error is confusion. The caller speaks about 43% of the speech time in this call, and all of it now goes to the agent. One channel keeps the words, but it loses the “who”. The next lesson tells the voices apart.

Common mistake

The channels the wrong way round

Symptom: Amira sets up a second phone system for the French line. The DER of its calls looks normal. But in the call records, the caller now promises refunds, and the agent asks for them.

swapped = by_channel(stereo, sr, speakers=("caller", "agent"))
print("DER with the names swapped:", round(der(swapped, truth, call["duration"])["der"], 3))
first = swapped[0]
print("the first turn is labelled", first[2], "- the truth says", truth[0][2])
DER with the names swapped: 0.118
the first turn is labelled caller - the truth says agent

Cause: the new system puts the caller on channel 1. The DER does not see this, because it matches the names in the best way before it counts errors. A complete swap costs nothing in DER.

Fix: check the channel order once for each phone system, with a known call. Larkfield’s agents speak first and say “Thank you for calling Larkfield”. Then test the names of the turns, not only the DER: for example, the first turn of every call must be the agent.

Consequence in AI work: a summary model reads the names as facts. With swapped channels, every promise goes to the caller, every follow-up is lost, and the DER report still says that all is well.

Check your understanding

In a 10-second stretch of a call, people speak for 8 seconds. The system misses 1 second of speech, finds speech in 0.4 seconds of silence, and gives 0.6 seconds to the wrong speaker. What is the DER, and which part should you fix first?

Show the answer

DER = (missed + false + confusion) / true speech time = (1 + 0.4 + 0.6) / 8 = 2 / 8 = 25%. Missed speech is the largest part, so tuning the VAD gives the biggest drop in DER. But for decisions, the 0.6 seconds of confusion can matter more: a sentence with the wrong speaker can change the call record. Look at both.

Check your understanding

Tomás says that the louder-wins rule is a hack, and that a better VAD would remove the need for it. Is he right on Larkfield's calls?

Show the answer

Only partly. The background talk on the caller’s side is real speech, so a good VAD often calls it speech. A better VAD would not remove it. In a phone call, the two sides rarely speak at the same time, so the louder side is a good guess for who really speaks. On the 66 stereo validation and test calls, louder wins cuts the confusion from 11.1% to 3.8% of the speech time. A better VAD would mostly cut the missed speech, which is a different part of the DER.

Check your understanding

Omar asks why the course does not use a speaker model on every call, since it would also work on stereo calls. What do you tell him?

Show the answer

When the phone system records each side apart, the channel tells you who speaks for free. It needs no model and no download, and a few seconds for 40 calls. And it confuses speakers less than a speaker model. The confusion is 3.8% by channel on the stereo calls, and 15.6% for speaker embeddings on the mono calls (the next lesson). Use a speaker model only where you have no channels. The cheapest improvement is often to ask the phone provider for stereo recording.

Recap

  • Diarization finds who spoke when: turns with a start, an end and a speaker.
  • DER = (missed + false + confusion) / true speech time. Names are matched first, and a collar of 0.25 s around each change is not scored. Always report the three parts apart.
  • In a stereo call, each channel is one side, so diarization is a VAD on each channel. Louder wins stops background talk from looking like the caller.
  • On Larkfield’s 66 stereo validation and test calls: DER 28.7%, of which missed speech 23.8% (the VAD) and confusion only 3.8%.
  • A swapped channel order does not change the DER, but it puts every sentence under the wrong name. Check the names.

Channels give the “who” for free, but 44 of Larkfield’s 110 validation and test calls are mono. Next, you tell two voices apart by how they sound, with speaker embeddings and clustering.

Next: Speaker embeddings and clustering