Skip to content
Français

Speaker embeddings and clustering

  • Lesson 16
  • 40 min
  • Module 5: Who spoke when: diarization and timing

Where we are

Splitting by channel is free, but some calls have only one channel. In this lesson, we turn short pieces of speech into vectors with an open speaker model, cluster them into speakers, and measure the diarization error rate.

Lesson 16 of 36 · Speech AI: Transcribe, Understand and SpeakSpeaker embeddings and clustering

What you will be able to do

Tell two voices apart when they share one channel. Turn short pieces of speech into lists of numbers that describe the voice. Put similar lists in the same group. Then measure how often the groups are wrong.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It computes a cosine similarity by hand and embeds turns of a Larkfield call. It clusters and draws the pieces of a mono call, and shows where two similar voices merge. It downloads a 27 MB speaker model once and needs no account. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

  • Turns and the diarization error rate (DER), from Split by channel first.
  • What an embedding is, in general: a list of numbers that places similar things close together. The RAG course explains embeddings of text; this lesson does the same for voices.

What you need on your computer (only for the local route)

  • Your project folder larkspeech with the files of end-of-m05, from Split by channel first. speech diarize --help works in it.
  • An internet connection the first time: the project downloads the speaker model (27 MB) and checks its SHA-256.

Tested on macOS with Python 3.12 and sherpa-onnx 2.0.1. Windows and Linux were not tested.

A call-centre agent picks up the phone. A regular customer says “Hi, it’s me again”, and the agent knows the voice. The agent did not need a name. The voice has a sound of its own: high or low, fast or slow, its own way of saying “s”.

Now give the agent a recording of a call between two strangers, on one track. After a few sentences, the agent can still tell “voice one” from “voice two”. The agent cannot name them, but can group the sentences by how they sound. This lesson teaches a computer to do the same.

44 of Larkfield’s 110 validation and test calls are mono: the phone system mixed the agent and the caller into one channel. A VAD finds speech in them, but every piece of speech is in the same channel. In Split by channel first, the notebook gave all of the speech of a mixed call to one speaker. The DER rose from 11.8% to 43.5%, and all of the new error was confusion.

Listen to the start of mono call C-0285:

Call C-0285, mono, from 0.4 to 12.0 s, phone quality (8 kHz). Two synthetic voices. The agent is Piper, trained on the VCTK corpus (speaker p240). The caller is Piper, trained on the LibriTTS-R corpus (speaker 925). Transcript. Agent: “Thank you for calling Larkfield, this is Victor. How can I help you today?” Caller: “Yes, hi. I’m annoyed to see two payments for the Juniper compost bin on my card. Please explain why this happened.”

You hear two different voices at once. A computer gets one list of numbers. We need a way to describe “how this piece sounds”, so that pieces from the same voice look alike.

The idea: speaker embeddings, then clustering

Section titled “The idea: speaker embeddings, then clustering”

A speaker embedding is a list of numbers that describes how a voice sounds, not what it says. A speaker model makes it. The model was trained on the voices of thousands of people. For pieces from the same person, it learned to give lists that point in nearly the same direction. For pieces from different people, the lists point in different directions.

The project uses an open speaker model: WeSpeaker ResNet-34, trained on VoxCeleb, a large set of interview clips of public figures. Its licence is Apache-2.0. It turns any piece of speech into a list of 256 numbers, and runs on a laptop CPU through sherpa-onnx.

Then we need clustering: put the lists into groups by how alike they are, without labels. Nobody tells the computer which piece is the agent. It only finds two groups. Clustering is a classic method of unsupervised learning; the machine learning course explains it for any data.

We compare two embeddings with the cosine similarity. It is 1 when two lists point the same way, 0 when they are unrelated, and -1 when they point opposite ways. The project first scales each list so that its squares add up to 1. For such lists, the cosine is the sum of the products of their numbers.

Take three tiny “voices” with two numbers each. Their squares add up to 1, for example 0.6 × 0.6 + 0.8 × 0.8 = 1:

Pair Products Cosine
a = (0.6, 0.8), b = (0.8, 0.6) 0.6 × 0.8 + 0.8 × 0.6 0.96
a = (0.6, 0.8), c = (-0.6, 0.8) 0.6 × (-0.6) + 0.8 × 0.8 0.28

a and b are alike; a and c are not. Real embeddings have 256 numbers instead of 2, but the sum is the same idea.

With the real model, the notebook embeds three turns of call C-0052, mixed into one channel. The two agent turns have a cosine of 0.901. The agent and the caller have 0.787 and 0.737. The same voice is closer, but not 1. The words and the line change the numbers a little. So the gap between “same” and “different” is real, but small.

A mono call, with both voices in one channel, goes to step 1, find speech with Silero VAD. Step 2, cut pieces: 1.5 second windows, a new one every 0.75 seconds. Step 3, embed: the speaker model gives 256 numbers a piece. Step 4, cluster: two groups of similar vectors. Step 5, name: the group that speaks first is the agent.
Diarization of a mono call with speaker embeddings and clustering, as the project's diarize.py does it.
Text description of the diagram

The diagram has a mono call and five steps. Step 1 finds the speech with Silero VAD. Step 2 cuts each region of speech into pieces of 1.5 seconds. It starts a new piece every 0.75 seconds, so the pieces overlap. Step 3 gives each piece to the speaker model, which returns 256 numbers. Step 4 clusters the pieces into two groups of similar vectors. Step 5 names the groups: the group of the first piece is the agent, because Larkfield’s agents answer the call and speak first.

Each step has a reason. A piece of 1.5 seconds holds enough sound to describe a voice, but is short enough to hold one person most of the time. The pieces overlap, so a change of speaker always falls near the middle of some piece. The project’s clustering is agglomerative. It starts with every piece alone. Then it joins the two closest groups again and again, until two groups are left. “Closest” uses the cosine. At the end, each 10 ms frame takes the speaker of the piece whose centre is nearest.

Call C-0285 gives 54 pieces. Each piece has 256 numbers, which nobody can draw. So the picture shows the two directions in which the pieces differ most, found by PCA (principal component analysis). The two directions keep about 38% of the spread; the rest is hidden. Each dot is one piece.

A scatter plot of 54 dots. On the left, a group of orange squares: caller pieces in the caller cluster. On the right, a group of blue circles: agent pieces in the agent cluster. Between them, 11 hollow blue squares: caller pieces that the clustering put with the agent.
The 54 pieces of mono call C-0285, in two dimensions. Colour: the cluster. Shape: the true speaker. 11 caller pieces (hollow squares) went to the agent's cluster. DER 22.7%: missed 8.2%, confusion 14.4%.
Text description of the diagram

The picture is a scatter plot. Each dot is one piece of 1.5 seconds. Its place comes from the two directions in which the embeddings differ most. The colour is the cluster: blue for the cluster named agent, orange for the cluster named caller. The shape is the truth: a circle when the agent spoke in the piece, a square when the caller spoke. A solid dot means the cluster agrees with the truth; a hollow dot means it does not. On the right, 26 blue circles form the agent’s group. On the left, 17 orange squares form the caller’s group. Between the two groups, 11 hollow blue squares are caller pieces that the clustering put with the agent. They are close to the agent’s group in this picture, and closer still in the 256 numbers.

The two voices mostly form two groups, but the border is not clean. Some caller pieces sit nearer the agent’s group. Those pieces become confusion: caller speech under the agent’s name. On this call, confusion is 14.4% of the speech time, close to the mean of all mono calls.

The project’s speech diarize uses this method by default (--method embeddings). The speaker model is downloaded the first time, and checked against its SHA-256.

  1. Diarize one mono call, turn by turn:

    Terminal window
    speech diarize --call C-0285
    Downloading spk/wespeaker_en_voxceleb_resnet34_LM.onnx ...
    C-0285 (mono), embeddings: DER 22.7% (missed 8.2%, false 0.0%, confusion 14.4%)
    0.63 3.84 agent
    4.44 7.07 agent
    7.07 9.32 caller
    9.32 10.07 agent
    10.07 11.27 caller
    …

    The download line shows only the first time. The truth says that the caller speaks from 4.4 to 11.7 s. The method gives the first 2.6 seconds of it to the agent, and later swaps once more. That is the confusion of the picture.

  2. Diarize the 60 test calls. The stereo calls are mixed to mono first, so all 60 calls are used. It takes about 40 seconds on a laptop:

    Terminal window
    speech diarize --method embeddings --split test
    embeddings on 60 test calls: mean DER 37.8%
    missed 20.5%, false 0.3%, confusion 16.9%
    worst: C-0194 73%, C-0018 72%, C-0248 71%

    Missed speech is the same kind of error as with channels: the VAD. The new part is confusion: 16.9% instead of about 4% with channels.

  3. Compare with channels on one stereo call. C-0052 had a DER of 4.8% by channel:

    Terminal window
    speech diarize --call C-0052
    C-0052 (stereo), embeddings: DER 26.0% (missed 3.8%, false 0.0%, confusion 22.2%)
    0.86 3.26 agent
    3.45 5.92 agent
    6.17 7.55 agent
    8.50 12.30 agent
    12.30 12.70 caller
    …

    The caller’s first turn, from 8.5 to 12.6 s, goes almost all to the agent.

The run time is small. The speaker model needs about 0.012 seconds of compute for each second of audio, on a laptop CPU with 4 threads. So a 45-second call takes about half a second.

The timeline shows call C-0052: the truth, the turns by channel, and the turns by embeddings on the call mixed to mono.

A waveform of the 42-second call, then three rows of turns. Truth: agent and caller turns take turns. By channel: the same pattern, with the agent's long turns split at pauses. Embeddings: almost every turn is blue, the agent; only one short orange caller piece is left.
Call C-0052: who spoke when, by the truth, by channel (DER 4.8%, no confusion) and by embeddings on the mix (DER 26.0%, confusion 22.2%). The embedding method gives most of the caller's speech to the agent.
Text description of the diagram

The top row is the sound of the call, both voices mixed. The three rows below are turns, blue for the agent and orange for the caller. The truth has ten turns. After the agent’s greeting, the caller asks from 8.5 to 12.6 s. The caller gives the order number from 17.0 to 19.3 s, and speaks again from 27.0 to 28.2 s. The caller’s thanks run from 37.2 to 38.2 s. All the other turns are the agent’s. By channel, the pattern is the same; the VAD only splits some long agent turns at pauses. By embeddings, nearly every turn is blue. Only one short orange piece remains, from 12.3 to 12.7 seconds.

On this call, the caller’s voice was too close to the agent’s for the clustering. The method found two groups, but the second group is small: one piece at the end of the caller’s first turn.

The chart compares the parts of the DER for every method of this module, on all 110 validation and test calls.

Four horizontal stacked bars. By channel, louder wins: missed 23.8, false 1.2, confusion 3.8, DER 28.7%. By channel, each side alone: missed 21.2, false 1.2, confusion 11.1, DER 33.5%. Embeddings, mono calls: missed 20.9, false 0.5, confusion 15.6, DER 37.0%. Embeddings, stereo mixed: missed 17.5, false 1.1, confusion 16.0, DER 34.6%.
The parts of the DER on Larkfield's synthetic validation and test calls: channels on the 66 stereo calls, embeddings on the 44 mono calls and on the 66 stereo calls mixed to mono. Missed speech is similar everywhere; confusion is what the method changes.
Text description of the diagram

Four bars, each split into missed speech, false speech and confusion, in percent of the true speech time. By channel with louder wins: 23.8, 1.2 and 3.8, a DER of 28.7%. By channel, each side alone: 21.2, 1.2 and 11.1, a DER of 33.5%. Embeddings on the 44 mono calls: 20.9, 0.5 and 15.6, a DER of 37.0%. Embeddings on the 66 stereo calls mixed to mono: 17.5, 1.1 and 16.0, a DER of 34.6%. The missed speech is between 17% and 24% for every method, because they all use the same VAD. The confusion is about 4% with channels and about 16% with embeddings.

The last bar is a fair test. The same 66 calls lose about 4 points of confusion with channels and about 16 points with embeddings. So, on these calls, the channel is about four times better at “who” than the speaker model. These numbers are on Larkfield’s synthetic calls.

Use it when you have only one channel and two speakers who rarely talk at the same time: a recorded phone call, an interview. It needs no labels and no training: an open model and a few lines of clustering. It runs on a CPU at about 1% of real time.

Do not use it, or do not trust it alone, in these cases:

  • You can get stereo. Channels confuse speakers about four times less on our calls, for no compute.
  • The voices sound alike. Two voices of the same accent and pitch can merge into one cluster. The next lesson shows the worst case of our call set.
  • More than two people, or people talking at the same time. The project asks for exactly two clusters, and each frame has one speaker. A meeting or a call with a transfer needs more: the next lesson.
  • Short calls. A call with only a few pieces gives the clustering too little to work with.

What it costs: the model download once (27 MB), about half a second of CPU per call, and its error. The confusion is the expensive part: Grace’s team must not act on a sentence under the wrong name. Word times and alignment shows a cheap check for the sentences that matter.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the two tool cells (the energy VAD and der).
  3. In section 2, run the two cells that download and load the speaker model. You see that each embedding has 256 numbers.
  4. In section 3, predict which pair is more alike, then run the cell. You see 0.96 and 0.28.
  5. In section 4, run the cell on call C-0052. You see 0.901 for the two agent turns, and 0.787 and 0.737 for the agent and the caller.
  6. In sections 5 and 6, cut call C-0285 into pieces, cluster them and score them. The notebook uses the energy VAD at -40 dBFS, so its numbers differ from the project’s. You see 34 pieces and a DER of 0.189, almost all of it missed speech (0.169), with confusion 0.02.
  7. Run the map cell, and find the pieces whose colour and shape disagree.
  8. In section 7, predict what pieces of 0.75 s do to the confusion. Then run the cell. The confusion rises from 0.02 to 0.184: a shorter piece describes the voice less well.

In section 9 of the notebook, run the same pipeline on mono call C-0217. Cut it into pieces with pieces_of, embed them, and cluster them with cluster_turns. Then score the turns against true_turns(task_call). Put the confusion part, rounded to three decimals, in task_confusion. Before you run it, predict: more or less confusion than C-0285 in the notebook?

Show the answer
task_call = calls["C-0217"]
z, zsr = read_audio("data/calls/phone/C-0217.wav", 16000)
zp = pieces_of(z, zsr)
ze = np.stack([embed(z[int(s * zsr):int(e * zsr)], zsr) for s, e in zp])
task_result = der(cluster_turns(zp, ze)[0], true_turns(task_call), task_call["duration"])
task_confusion = round(task_result["confusion"], 3)
print(task_confusion)
0.125

The confusion of C-0217 is 0.125: more than C-0285. One call can be easy and the next one hard, so always measure on many calls before you trust a method.

Common mistake

One odd piece takes a whole cluster

Symptom: Priya’s first notebook gives call C-0285 a DER of 0.563. Its confusion is 0.412, much worse than the project’s. In the map, nearly every dot has the same colour.

import collections
print(collections.Counter(labels.tolist()))
Counter({0: 46, 1: 1})

Cause: the first version kept every leftover piece at the end of a speech region, even pieces of half a second. One short piece had an odd embedding, far from all the others. Agglomerative clustering with two clusters put that one piece alone in the second cluster, and all the other 46 pieces, both voices, in the first.

Fix: drop leftover pieces shorter than 1 second, as the notebook now does, and always count the pieces in each cluster. A cluster of one or two pieces is a sign of an outlier, not of a speaker. With the fix, the two clusters have 20 and 14 pieces, and the confusion falls to 0.02.

Consequence in AI work: the DER report shows only a bad number; it does not say why. Without the count, you might blame the speaker model and replace it, when the problem was one piece of noise.

Check your understanding

Two embeddings are (0.8, 0.6) and (0.6, -0.8). What is their cosine similarity, and what does it tell you?

Show the answer

The squares of each list add up to 1, so the cosine is the sum of the products. That is 0.8 × 0.6 + 0.6 × (-0.8) = 0.48 − 0.48 = 0. The two lists are unrelated: they point at right angles. For speaker embeddings, that means two very different voices. Real pairs of different voices on Larkfield’s calls are not this far apart: their cosines are often between 0.6 and 0.9.

Check your understanding

Why does the project cut the speech into overlapping pieces of 1.5 seconds, and not into whole turns?

Show the answer

In a mono call, we do not know where the turns are: finding them is the job. So we cut all speech into short pieces. A piece of 1.5 seconds holds enough sound to describe a voice, and is short enough to hold one person most of the time. The pieces overlap, so a change of speaker falls near the middle of some piece. The frames near the change take the speaker of the nearest piece. Shorter pieces describe the voice less well. In the notebook, pieces of 0.75 s raised the confusion of C-0285 from 0.02 to 0.184.

Check your understanding

Tomás wants to use speaker embeddings on every call, stereo or mono, to keep one pipeline. What do the numbers say?

Show the answer

On the 66 stereo validation and test calls, splitting by channel gives a confusion of 3.8%. Mixing the same calls to mono and using embeddings gives 16.0%. So one pipeline would add about 12 points of confusion on the stereo calls, for more compute. Keep channels where they exist, and use embeddings only for the mono calls. The code can choose the method from the number of channels in the file.

Recap

  • A speaker embedding describes how a voice sounds as a list of numbers (256 for the project’s open WeSpeaker model). The cosine similarity says how alike two lists are.
  • Diarization of a mono call: find speech and cut it into overlapping pieces of 1.5 seconds. Embed each piece, cluster the pieces into two groups, and call the first group “agent”.
  • On Larkfield’s 44 synthetic mono calls: DER 37.0%, of which confusion 15.6%. The same method on the stereo calls mixed to mono: confusion 16.0%, against 3.8% by channel.
  • The border between two voices is not clean: in call C-0285, 11 of 54 pieces went to the wrong cluster.
  • Count the pieces in each cluster: a cluster of one piece is an outlier, not a speaker.

Two clusters work for two speakers who sound different. Next, you see what a full pretrained pipeline adds for overlap and many speakers. You also see what its terms ask of you, and where two similar voices still merge.

Next: Pretrained diarization pipelines