Pretrained diarization pipelines
- Lesson 17
- 30 min
- Module 5: Who spoke when: diarization and timing
Where we are
Our clustering finds two speakers in most mono calls. Full diarization pipelines go further. In this lesson, we see what a pipeline such as pyannote adds, what its terms ask of you, and where two similar voices still merge.
Lesson 17 of 36 · Speech AI: Transcribe, Understand and SpeakPretrained diarization pipelines
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Know what a full, pretrained diarization pipeline adds to the method of the last lesson. Know what it asks of you before you can use it. And know where no method helps: two voices that sound alike.
Before you startWhat to know and to have ready
What you need to know
- Speaker embeddings, clustering and the DER, from Speaker embeddings and clustering.
What you need on your computer (only for the steps)
- Your project folder
larkspeechwith the files ofend-of-m05. This lesson adds no file.
This lesson has no notebook: the course does not run a pretrained pipeline. The optional path at the end was not tested by the course team.
Taking notes in a meeting
Section titled “Taking notes in a meeting”Think of a person who writes the minutes of a team meeting. Six people sit at the table. Two of them often talk at the same time. One person joins late and says only one sentence. The writer must note who said each thing, including the sentence that two people said at once.
Our method from the last lesson could not do this job. It asks for exactly two speakers. It gives each moment to one speaker only. And it knows voices only by how they sound, so two people with similar voices become one.
A phone call at Larkfield is easier than a meeting: two people, who rarely talk at once. But calls have these problems too. A caller passes the phone to a partner. An agent brings in a colleague. And sometimes, the two voices on the line sound alike.
The problem: two voices that sound alike
Section titled “The problem: two voices that sound alike”Call C-0144 is a practice problem that we made on purpose. The caller’s voice was chosen to sound like the agent’s voice. Listen to the middle of the call:
A person hears two voices, but they are close. For the speaker model, they are closer still. The project’s command shows what happens:
speech diarize --call C-0144C-0144 (mono), embeddings: DER 49.9% (missed 3.9%, false 0.0%, confusion 46.1%) 0.57 4.70 agent 4.70 5.82 caller 6.14 11.78 agent 12.41 13.92 agent 14.42 16.13 agent 16.79 19.15 agent 19.15 19.62 caller…The caller’s long turn from 6.1 to 11.7 s is under the agent’s name. Almost half of the speech time is confusion.
Text description of the diagram
The top row is the sound of the mono call. Below it is the truth. The agent speaks from 0.6 to 5.8 s, and the caller from 6.1 to 11.7 s. After a short agent turn, the caller speaks from 14.4 to 16.4 s. The agent’s promise runs from 16.8 to 19.7 s. The caller thanks the agent from 21.7 to 22.8 s, and the agent says goodbye. The bottom row is the embedding method. Nearly the whole call is blue, the agent. Only three short orange pieces are called the caller. They start at 4.7 s, 19.2 s and 24.8 s. All three are in fact the agent.
The method still found two clusters, because we asked for two. But the second cluster is not the caller. It is three short pieces of the agent’s voice that sound a little different.
Is it only this call?
Section titled “Is it only this call?”Only 7 of Larkfield’s 260 public calls have these planted similar voices. C-0144 is the only one among the mono validation and test calls. One call is an example, not a fact. So we asked a wider question. On each validation and test call, how alike are the two voices? And how much confusion does the method make?
For each call, we averaged the embeddings of the agent’s true turns. We did the same for the caller’s true turns. Then we took the cosine of the two averages.
Text description of the diagram
Each dot is one call: circles for mono calls, triangles for stereo calls mixed to mono. Left to right: how alike the two voices are, as the cosine of their mean embeddings. Bottom to top: the confusion of the embedding method. The median cosine is 0.80. The calls on the left, under about 0.65, have almost no confusion. The calls on the right have much more. In the quarter of calls with the most alike voices, from a cosine of 0.86, the mean confusion is 24.6%. In the other calls, it is 12.8%. C-0144 is marked in red: cosine 0.954, confusion 46.1%. Two other calls have a slightly higher cosine, 0.961 and 0.967, with less confusion.
The picture shows a clear trend: the more alike two voices are, the more the clustering confuses them. It also shows that this is not a rule for each call. Two calls with background talk have voices as alike as C-0144 by this measure, but less confusion. The background talk is probably the reason: it is under both voices, so it makes their averages closer. These numbers are on Larkfield’s synthetic voices. Real voices differ more, but similar voices are real too: a parent and an adult child, two colleagues of the same age and accent.
The idea: a pretrained pipeline
Section titled “The idea: a pretrained pipeline”A diarization pipeline is a chain of models that was trained, as a whole, to answer “who spoke when”. The best-known open one is pyannote (the pyannote.audio library and its pretrained pipelines on Hugging Face). Its pipelines have the shape below.
Text description of the diagram
The diagram has audio and three steps. Step 1 is a segmentation model. It looks at short windows of the call, a few seconds each, and says which speakers are active in each moment, including two at once. Step 2 makes one embedding for each speaker inside each window. Step 3 clusters these embeddings across the whole call. Then “speaker 1 of window 3” and “speaker 2 of window 7” can become the same person. The output has turns, overlapping speech and as many speakers as the clustering finds.
Compare it with our method. We cut the speech into fixed pieces, and each piece has one speaker. A pipeline first finds where each speaker is active, with a model trained for that job. It can mark two speakers at the same moment. This adds three things:
- Overlap. It can give one moment to two people. Our method always gives it to one, so overlap is always half wrong.
- Many speakers. The clustering can choose the number of speakers, or take a number or a range from you.
- Better borders. The segmentation model is trained on real conversations, so it places the start and end of turns more precisely than fixed pieces.
What it does not add: a different way to hear voices. Step 2 is still a speaker embedding. If two voices give nearly the same embedding, the pipeline must still decide by how they sound. So two very similar voices can still merge, in any method that tells speakers apart only by their voice.
What the terms ask of you
Section titled “What the terms ask of you”The pyannote.audio library is open-source code. The pretrained models have their own terms, and you must read them before you use them. As an example, here is what the page of the open pipeline pyannote/speaker-diarization-community-1 on Hugging Face says. Its gate reads: “You need to agree to share your Hugging Face information to access this model.”
| Question | What the model page says |
|---|---|
| Licence of the pipeline | CC BY 4.0 (free to use, with attribution) |
| Can you just download it? | No. The model is gated: you must accept its conditions and share your Hugging Face contact information with its authors. |
| What do you need to run it? | A Hugging Face access token, passed to the code, and pyannote.audio 4.x (it brings PyTorch). |
| Number of speakers | Optional: num_speakers, or min_speakers and max_speakers. |
| Overlap | The output marks overlapping speech; an “exclusive” output gives one speaker per moment, which is easier to join with word times. |
Terms change: check the current model page before you use it. Three questions for Larkfield follow from this table:
- Privacy. The model runs on your own computer after the download, so the audio stays at Larkfield. The token only gives access to the download. The same team also sells hosted models. With those, the audio leaves Larkfield’s servers, so Larkfield’s privacy rules apply (Personal data in voice).
- Accounts and tokens. Every developer and every server needs a token that has accepted the conditions. A token is a secret: keep it out of the code and the logs.
- The course’s choice. This course asks for no account and no token. So the project uses channels and an open model under Apache-2.0, and does not run pyannote. That is a choice of this course, not a judgement of pyannote.
An optional path (not tested by the course)
Section titled “An optional path (not tested by the course)”If you accept the terms, you can measure a pretrained pipeline on Larkfield’s calls with the same DER as the rest of the course. The course team did not run these steps; your results will differ from any number on this page.
-
Read and accept the conditions on the model page, and create a Hugging Face access token. Put it in an environment variable, for example
HF_TOKEN, not in your code. -
In a separate virtual environment (pyannote.audio brings PyTorch, which is large), install the library:
Terminal window pip install "pyannote.audio>=4,<5" -
Run the pipeline on one mono call, and tell it that there are two speakers. This sketch follows the model page:
import osfrom pyannote.audio import Pipelinepipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-community-1",token=os.environ["HF_TOKEN"])output = pipeline("data/calls/phone/C-0144.wav", num_speakers=2)turns = [(t.start, t.end, speaker) for t, speaker in output.exclusive_speaker_diarization] -
Score the turns with the project’s DER, in the same environment as
larkspeech:from larkspeech import data as D, diarize as Zfrom larkspeech.paths import data_dircall = D.call(data_dir(None), "C-0144")print(Z.der(turns, Z.truth_turns(call), call["duration"]))The speaker names are “SPEAKER_00” and “SPEAKER_01”. The DER matches them to “agent” and “caller” first.
-
Do the same for all 44 mono validation and test calls. Put the mean missed, false and confusion next to the chart of Speaker embeddings and clustering. Note the compute time too.
Decide from the three parts, not from the total DER. A pipeline that cuts the missed speech but not the confusion helps Grace less than the total suggests.
Which method for which calls
Section titled “Which method for which calls”Text description of the diagram
The chart starts with one question: is each side on its own channel? If yes, split by channel: it is free, and its confusion is about 4% on our calls. If no, ask: are there two speakers, with little overlap? If yes, use embeddings and clustering: an open model on a CPU, with a confusion of 15.6% here. If no, ask: can you accept the model terms and the token? If yes, use a pretrained pipeline, and measure it on your own calls first. If no, change the recording: ask the phone system for stereo.
For Larkfield, the answer is clear. Stereo calls use channels. Mono calls use embeddings, and the most important sentences get an extra check (Word times and alignment). And Malik asks the phone provider to record every call in stereo, which removes the problem at its source.
When to use a pretrained pipeline, and what it costs
Section titled “When to use a pretrained pipeline, and what it costs”Use it for meetings, interviews with several guests, calls with transfers, and any audio with overlap or more than two speakers. Use it when its terms fit your company, and after you measure it on your own audio.
Do not expect it to separate two voices that sound alike, or to replace stereo recording. And do not expect it to be right on your data because it was good on a benchmark. Benchmarks use other audio: other languages, other microphones, other lines.
What it costs: an account and a token for every place it runs, and PyTorch in your environment. It needs more compute than our method; a GPU helps with long recordings. And the model terms need a legal review. Measure the compute on your own computer: this course has no number for it.
Guided practice: find the merged voices
Section titled “Guided practice: find the merged voices”-
In your project folder, with the virtual environment active, diarize the 50 validation calls with embeddings. It takes about 30 seconds:
Terminal window speech diarize --method embeddings --split validationembeddings on 50 validation calls: mean DER 32.9%missed 16.8%, false 1.6%, confusion 14.5%calls where the agent and the caller sound alike (planted): 1, mean DER 49.9%worst: C-0251 115%, C-0230 78%, C-0296 70%The third line names the planted calls in the split: here, one call, C-0144.
-
Diarize C-0144, and compare the turns with the true turns on this page. Which true turn is missing from the output?
Terminal window speech diarize --call C-0144 -
Look at the worst call, C-0251: its DER is over 100%. Diarize it, and read its missed, false and confusion. Which part makes the DER larger than 100%?
Show the answers
In step 2, the caller’s turn from 6.1 to 11.7 s (“Yes, hi. I’m a bit lost here…”) is missing: the output gives 6.14 to 11.78 to the agent.
In step 3, the false speech is large. C-0251 is a stereo call with background talk. Mixed to mono, the VAD finds speech in the background talk, where nobody of the call speaks. Like the WER, the DER divides by the true speech, so false speech can push it over 100%.
Your turn: let the clustering choose
Section titled “Your turn: let the clustering choose”The project’s by_embeddings can choose the number of speakers itself: with threshold=, the clustering stops joining groups when they are further apart than this cosine distance. A pretrained pipeline makes the same choice when you do not give it num_speakers. Run this in your project folder:
from larkspeech import data as D, diarize as Zfrom larkspeech.paths import data_dir
d = data_dir(None)for cid in ("C-0285", "C-0144", "C-0052"): c = D.call(d, cid) x, sr = D.load_audio(d, cid) for th in (0.3, 0.5, 0.7): turns = Z.by_embeddings(x, sr, threshold=th) r = Z.der(turns, Z.truth_turns(c), c["duration"]) print(cid, th, len({w for _, _, w in turns}), "speakers", round(r["der"], 3))Before you run it, predict: is there one threshold that finds two speakers in all three calls?
Show the answer
C-0285 0.3 6 speakers 0.436C-0285 0.5 1 speakers 0.519C-0285 0.7 1 speakers 0.519C-0144 0.3 2 speakers 0.499C-0144 0.5 1 speakers 0.443C-0144 0.7 1 speakers 0.443C-0052 0.3 4 speakers 0.178C-0052 0.5 2 speakers 0.26C-0052 0.7 1 speakers 0.262No. At 0.3, C-0285 gets six “speakers” and C-0052 gets four. At 0.5, C-0052 gets two, but C-0285 gets one. On C-0144, one speaker even scores better than two: 0.443 against 0.499. When the voices merge anyway, “one speaker” costs less than a wrong second speaker. When you know the number of speakers, as Larkfield does for its calls, give it to the method.
Common mistake
A good benchmark number, the wrong comparison
Symptom: Tomás reads a low DER for a pretrained pipeline on a public meeting benchmark. He wants to mix every Larkfield call to mono and run the pipeline on all of them, “to have one modern pipeline”.
Cause: two mistakes. The benchmark is other audio: meetings with good microphones, not 8 kHz phone lines with codecs. And on stereo calls, the channel gets the “who” from the recording, so a model that listens to the mix starts with less information. The course did not run pyannote, but it measured what mixing costs our own speaker-embedding method on the same 66 stereo calls:
| Method on the 66 stereo validation and test calls | Confusion |
|---|---|
| By channel, louder wins | 3.8% |
| Embeddings on the same calls mixed to mono | 16.0% |
Fix: compare methods on your own calls, with the same DER, the same collar and the same three parts. Keep channels for stereo calls, and test a pipeline only where channels do not exist.
Consequence in AI work: a model chosen from a benchmark can make call records worse, at a higher compute cost. Meanwhile, the slide with the benchmark looks better.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name two things that a pretrained pipeline such as pyannote adds to the project's embedding method, and one thing that it does not fix.
Show the answer
It adds overlap: a segmentation model can mark two speakers at the same moment, while our method gives each moment to one speaker. It adds many speakers: its clustering can choose the number, or take a number or a range. It also places turn borders better. It does not fix two voices that sound alike. Its step 2 is still a speaker embedding, so voices with nearly the same embedding can still merge, as in call C-0144.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Omar asks if Larkfield can use the pyannote pipeline from Hugging Face tomorrow. What must happen first?
Show the answer
Someone must read the model’s terms on its page. The open pipeline is gated: each user must accept its conditions and share contact information with its authors. Each place that runs it needs a Hugging Face access token, kept as a secret. The team must check the licence (CC BY 4.0 for the community pipeline; check the current page) with legal review. The model runs locally, so the audio stays at Larkfield; a hosted version would send audio out. And the team must measure it on Larkfield’s own calls before it replaces anything.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On call C-0144, one cluster scores a lower DER (44.3%) than two clusters (49.9%). Should the project use one cluster when two voices are very alike?
Show the answer
No. One cluster scores better only because the second cluster was wrong too. With one speaker, the error is the caller’s speech time. A bad second cluster adds more error on top. But a transcript with one speaker cannot tell Grace who promised what, which is the whole job. The better answers are stereo recording, a check of the important sentences, and sending such calls to a person.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A pretrained diarization pipeline such as pyannote adds a segmentation model. It finds who is active in short windows, with overlap. Then it clusters any number of speakers.
- It still tells people apart by their voice. Similar voices can merge in any such method: on call C-0144, the planted pair, our method’s confusion is 46.1%.
- On Larkfield’s synthetic calls, the more alike the voices, the more confusion. It is 24.6% in the quarter of calls with the most alike voices, and 12.8% in the rest.
- The pipeline’s terms: the model is gated (accept the conditions, share your contact information), needs an access token, and has its own licence. Check the current page, and measure it on your own calls.
- Choose the method from the recording first: stereo → channels; mono with two speakers → embeddings; overlap or many speakers → a pretrained pipeline, if its terms fit.
You know who spoke when, and you know what was said. Next, you join the two: each word gets a time and a speaker, and you catch a promise that went to the wrong person.
Next: Word times and alignment