Skip to content
Français

Meeting transcripts with speakers

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we split Larkfield calls by speaker and wrote speaker-labelled transcripts. In this case study, we diarize and transcribe real recorded meetings, measure DER and WER, and make action-item notes.

Case study 2 of 4 · Speech AI: Transcribe, Understand and SpeakMeeting transcripts with speakers

What you will be able to do

Make meeting notes that say who said what, from real recordings of real people. You will compare a microphone on the table with one headset per person. You will measure who spoke when and the words, and score whole notes with names in one number. Then you will choose a setup and say what a person must still check.

Case studyPortfolio piece

Licence
CC BY 4.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Overlap-aware DER
  • Channel diarization with crosstalk
  • Speaker embeddings on a far microphone
  • WER and cpWER
  • Per-person checks
  • Recorded LLM action items
  • Cost per audio hour

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs the steps of this page with the same code. It downloads about 27 MB. That is a 5-minute excerpt of one meeting, the word reference of four meetings, and the recorded outputs of the full meetings. It scores the recorded outputs and runs the pipeline live on the excerpt. It needs no account and no key, and runs in a few minutes on a laptop.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses most of the course. These lessons matter most:

What you need on your computer (only for the local route; the notebook needs none of it)

  • A virtual environment with the packages of the project at the end of Module 5 (faster-whisper, sherpa-onnx, scikit-learn), and jiwer. The larkspeech project works; this case study does not change its files.
  • The notebook, downloaded from the link above. Its setup cell makes a new folder ami-meetings. To start again, delete that folder.

Tested on macOS with Python 3.12. Windows, Linux, Colab and Kaggle were not tested.

Omar, Larkfield’s product manager, runs a product meeting every week. Somebody writes the notes by hand, and they are late, short and sometimes wrong about who agreed to do what. Omar asks Priya: “Can the speech tools of the help desk write our meeting notes? I want who said what, and a list of action items with an owner each.”

Priya does not want to record her colleagues for a test: that needs their consent and a rule for how long it is kept. So she tests first on public meetings, recorded with consent, under an open licence.

Two errors matter here, and they are not the same:

  • A wrong name. “Grace will order the samples” when Amira said it. The wrong person gets the task, and the right person forgets it.
  • A lost sentence. A quiet person’s idea or promise is missing from the notes. Nobody sees what is not there.

You will build three setups for meeting notes and compare them on the same meetings and the same measures:

Three rows. Baseline: far microphone, find speech with Silero, everyone is one speaker, Whisper on the far microphone, words with names. Embeddings: far microphone, find speech, speaker embeddings and four clusters, Whisper on the far microphone, words with names. Headsets: one headset per person, find speech on each headset, louder wins, Whisper on each headset, words with names.
The three setups of this case study. Each ends in the same thing: words with names, which become the notes.
Text description of the diagram

The diagram has three rows, one for each setup. Each row goes from a recording to words with names. The baseline takes the far microphone on the table and finds speech with Silero. It gives all of it to one speaker, and Whisper transcribes it. The embeddings setup uses the same microphone. It cuts the speech into pieces, makes a speaker embedding for each piece and clusters them into four speakers, as Module 5 taught. The headsets setup has one headset per person. It finds speech on each headset and gives a moment that several headsets hear to the loudest one. Then Whisper transcribes each headset.

You will measure who spoke when (DER), the words (WER), and the notes with names in one number (cpWER, a new measure). Then you choose a setup, run it on a meeting kept apart, and turn the transcript into action items with a recorded LLM.

The data is the AMI Meeting Corpus: 100 hours of meetings recorded in three rooms. In most meetings, four paid participants play a design team that plans a new TV remote control. The talk is spontaneous; only the topic is set. Each person wore a headset, and microphones on the table heard the whole room.

We use four short kick-off meetings. Three are for the comparison. The fourth, IS1008a, is kept apart for the end-to-end run at the end.

Meeting Room Length The four people (role, sex, first language)
ES2004a Edinburgh 17.5 min interface designer (M, Hindi), project manager (F, English), industrial designer (M, English), marketing expert (F, English)
IS1009a IDIAP 14.0 min project manager (F, English), interface designer (F, Tamil), industrial designer (F, Telugu), marketing expert (F, Swiss German)
IS1001a IDIAP 15.2 min project manager (M, French), interface designer (M, Greek), industrial designer (M, French), marketing expert (F, French)
IS1008a (new input) IDIAP 15.7 min project manager (M, Hindi), interface designer (F, English), industrial designer (F, English), marketing expert (M, English)

For every word, the annotators wrote its speaker and its start and end time. That gives the truth for WER and the true turns for DER: one person’s words, joined while the gap is under half a second.

These two clips are the same 8 seconds of ES2004a. The first is the far microphone on the table. The second is the headset of the industrial designer (C).

Meeting ES2004a, 7:00 to 7:08, the far microphone (Array1-01), made louder. Real people, from the AMI Meeting Corpus (CC BY 4.0). Transcript. Project manager (B): “That’s lovely.” Industrial designer (C): “Yeah, and uh, I’ll have to think on the spot of, uh, things that it is. Um.” Marketing expert (D), at the same time: “Beauti-, that’s…”
The same 8 seconds from the headset of the industrial designer (C), made louder. Real people, from the AMI Meeting Corpus (CC BY 4.0). Same transcript. C’s own voice is close and clear; the other two people are quieter, but you can still hear them.

On the far microphone, everyone sounds far away, with the room’s echo. On the headset, one voice is close, but the neighbours are there too. That is crosstalk.

People talk at the same time. The share of the speech time with two or more voices is 15.8% in ES2004a, 13.6% in IS1009a and 9.4% in IS1001a. Most overlaps are short: a “yeah” or “mm-hmm” while someone else talks. Larkfield’s synthetic calls almost never overlap.

Every headset hears everyone. The timeline shows 30 seconds of ES2004a: the four headsets, the far microphone, and three rows of turns.

Five waveforms, then three rows of turns, for 30 seconds of meeting ES2004a. Each headset shows its wearer loudly and the others more quietly. The far microphone is one quiet mixture. Truth and louder wins have similar coloured turns; the far microphone with embeddings has almost only one colour.
30 seconds of ES2004a (9:35 to 10:05). Each headset hears its wearer loudly and the neighbours more quietly. Louder wins finds nearly the true turns; the far microphone with embeddings gives almost everything to one person.
Text description of the diagram

The top four rows are the sound of the four headsets, each in its wearer’s colour: A blue, B orange, C green, D purple. Each headset has tall shapes when its wearer speaks, and smaller shapes at the same times as the other headsets’ tall ones: that is crosstalk. The fifth row is the far microphone: one quiet mixture of everyone. Below are three rows of turns. In the truth, all four people speak. C opens, and A talks for about five seconds. Then B, D and B speak, and C talks for the last ten seconds. “Louder wins” from the headsets has nearly the same turns, with small gaps. “Far mic, embeddings” is all orange: the clustering gave the whole 30 seconds to the project manager, B.

The DER of Module 5 gave each 10 ms moment to one speaker. In a meeting, one moment can have two true speakers. The case study’s der counts per moment. Missed: fewer speakers found than true. False: more found than true. Confusion: a found speaker with the wrong name. As before, it first matches the found names to the true names in the best way. It does not score 0.25 s around each true change of speaker.

A method that gives each moment to one speaker misses one voice in every overlap. The collar removes many short overlaps from the score, but in the scored time, overlap is still 5.7% to 8.4% of the speech.

The simplest notes put every word under one name. We find speech on the far microphone with Silero, and call all of it “speaker 1”. The far microphone is quiet (34 to 48 dB below full scale). So all far-microphone setups first make it louder, to −25 dBFS. They use the lower Silero threshold of 0.3 from WebRTC VAD and Silero VAD.

On the three meetings, the baseline’s DER is 48.2%: missed 17.2%, false 2.7% and confusion 28.3%. The confusion is the speech of everyone except the person who talks most. Any useful method must beat this number.

Alternatives on the far microphone and the headsets

Section titled “Alternatives on the far microphone and the headsets”

We tried five methods against the baseline, on the same three meetings and with the same der. The chart shows the parts of the DER for each.

Six horizontal stacked bars of missed, false and confusion. Baseline far mic one speaker, far mic embeddings 1.5 s, far mic embeddings 3 s: about half of the speech time in error, mostly confusion. Headset mix embeddings: lower. Headsets each alone: a very long false bar, cut at the edge. Headsets louder wins: the shortest bar, almost all missed.
DER parts on the three AMI meetings (ES2004a, IS1009a, IS1001a), pooled by speech time. The far microphone with embeddings does not beat the baseline; headsets with louder wins do, by a lot.
Text description of the diagram

Six bars, each split into missed (grey), false (gold) and confusion (red), in percent of the true speech time. The baseline, the far microphone as one speaker, has a DER of 48.2%. The far microphone with embeddings has 50.3% with pieces of 1.5 s and 53.3% with pieces of 3 s: no better. The four headsets added together into one track, with embeddings, have 35.9%. The headsets, each alone, have 145.3%: the false part alone is 131.6%, so the bar is cut. The headsets with louder wins have 26.9%, of which 24.4% is missed and only 1.5% confusion.

This is the honest surprise of the case study. The method of Speaker embeddings and clustering, with four clusters, gives a DER of 50.3%, against 48.2% for “one speaker”. Longer pieces (3 s) make it worse: 53.3%. The clustering puts nearly all speech in one or two big groups. In ES2004a, the project manager gets 84% of her speech under her own name, and the other three people get almost none.

Why? We compared pieces of 1.5 s that hold one person only. On the far microphone, pieces of the same person and pieces of different people are almost equally alike.

Grouped bars for three meetings. On the far microphone, the same-person and different-people bars have nearly equal heights, around 0.8. On the headset mix, the same-person bar is clearly higher than the different-people bar.
The mean cosine similarity of speaker embeddings (WeSpeaker ResNet-34) for pieces of the same person and of different people. On the far microphone the two are nearly equal: the model hears the room more than the voice.
Text description of the diagram

For each of the three meetings, four bars. On the far microphone, the same person scores 0.83 and different people 0.83 in ES2004a; 0.77 and 0.76 in IS1009a; 0.84 and 0.84 in IS1001a. On the four headsets added into one track, the gap is clear in ES2004a (0.49 against 0.33), smaller in IS1009a (0.61 against 0.51) and small in IS1001a (0.64 against 0.62).

The speaker model learned from interview clips with close microphones. At two metres, in a room with echo, every voice gets the same “room colour”, and the model describes the room more than the person. No clustering can split what the embeddings do not separate. On the four headsets added into one track, the gap comes back in the Edinburgh meeting (DER 29.6%), less in the IDIAP meetings.

With one headset per person, each headset is a channel, as in Split by channel first. But “each headset alone” gives a DER of 145.3%. The VAD on each headset finds the neighbours’ voices too, so the same sentence appears under three or four names: false speech of 131.6% of the true speech time. It is fine in the Edinburgh room (23.1%) and over 200% in the IDIAP rooms.

Louder wins fixes this. When several headsets call a moment speech, the loudest headset gets it. The wearer is always closest to their own microphone. The DER falls to 26.9%, and confusion to 1.5%.

What is left is mostly missed speech (24.4%). Part of it is overlap: louder wins keeps one person per moment. The rest is short “mm-hmm”s and the pauses inside true turns, which Silero does not call speech.

Measure the words, and the words with names

Section titled “Measure the words, and the words with names”

We ran Whisper tiny, base and small (int8, laptop CPU, as in the project) on every stream. On the far microphone, the reference is all the words in time order. On a headset, it is the wearer’s words, and we keep only the Whisper words inside the wearer’s louder-wins turns. Without that step, each headset transcript also holds the neighbours’ words: WER 152.8% with Whisper small.

Grouped bars of WER for Whisper tiny, base and small. For each size, the headsets bar is lower than the far-microphone bar.
WER on the three AMI meetings by Whisper size and microphone (normalisation guide plus the meeting rules). With Whisper tiny or base, a headset helps more than a bigger model.
Text description of the diagram

Three pairs of bars. Whisper tiny: headsets 30.1%, far microphone 43.6%. Whisper base: headsets 26.9%, far microphone 37.4%. Whisper small: headsets 26.5%, far microphone 30.7%.

Two things stand out. First, the microphone matters more than the model. A headset with Whisper tiny (30.1%) is about as good as the far microphone with Whisper small (30.7%), for about the same compute. With Whisper small, the headsets still win, but by less: 26.5% against 30.7%. Second, these WERs are far above Larkfield’s synthetic calls (Whisper small: 3.1% in English). Real meetings have hesitations, broken words (“Beauti-”), overlap and many accents.

The normalisation guide needs three meeting rules

Section titled “The normalisation guide needs three meeting rules”

AMI’s transcribers write letters said one by one as L_C_D_, and Whisper writes “LCD”. They write “Mm-hmm” and “gonna”; Whisper writes “Mhm” and “going to”. As in Normalise before you compare, we write the rules once and apply them to both texts: letters said one by one become one word, more sounds count as fillers, and short forms become full words.

Normalisation (Whisper small, far microphone, three meetings) WER
none 47.7%
basic (lower case, no punctuation) 35.6%
the course’s full guide 31.6%
the full guide plus the three meeting rules 30.7%

Every number in this case study uses the last line.

Think of a person who checks the minutes of a meeting. A line is right only when the words are right and under the right name. DER checks only names and times; WER checks only words. Neither says how good the notes are.

The cpWER (concatenated minimum-permutation word error rate) checks both. It joins (“concatenates”) each person’s words into one text. It matches each found speaker to a true speaker. It chooses the matching (“permutation”) with the fewest errors. Then it divides all word errors by the number of true words. A word under the wrong name counts twice: missing for one person, extra for another.

Truth: A says we need a budget; B says I will write it. Found: speaker 2 says we need a budget; speaker 1 says I will right it. Best match: 2 to A, 1 to B. One error. cpWER 1 divided by 8 equals 12.5 percent. The wrong match would give 8 errors.
cpWER on a tiny example: the found speakers have numbers, not names, so the best matching is chosen first.
Text description of the diagram

Two true speakers: A says “we need a budget” and B says “I will write it”, 8 words in all. The system found speaker 2 with “we need a budget” and speaker 1 with “I will right it”. The best matching is 2 to A and 1 to B. Then there is one error, “right” for “write”, so the cpWER is 1 divided by 8, or 12.5%. The other matching, 1 to A, would give 8 errors: that is why cpWER chooses the best one.

Meeting-transcription challenges such as CHiME use cpWER. It needs words with names: as in Word times and alignment, each word goes to the speaker whose turn holds its middle.

Now the four setups, with Whisper small, on the same three meetings, in one table. Compute is the real-time factor of the whole setup on the laptop CPU: for the headsets, Whisper runs four times, once for each headset. The cost uses the course’s assumption of a CPU server at US$0.17 an hour.

Setup (Whisper small) DER WER cpWER Compute (× audio time) US$ per audio hour
Baseline: far mic, one speaker 48.2% 30.7% 89.3% 0.068 0.012
Far mic, embeddings 50.3% 30.7% 86.9% 0.083 0.014
Headsets, each alone 145.3% 152.8% 152.8% 0.258 0.044
Headsets, louder wins 26.9% 26.5% 26.5% 0.258 0.044
Headsets, louder wins, Whisper small with its VAD 26.9% 22.9% 22.9% 0.152 0.026
Headsets, louder wins, Whisper base 26.9% 26.9% 26.9% 0.115 0.019
Headsets, louder wins, Whisper tiny 26.9% 30.1% 30.2% 0.062 0.011

What this tells us. On the far microphone, the words are nearly as good as on the headsets. The notes are not: a cpWER of 86.9% means most words are under a wrong name. With headsets and louder wins, the cpWER equals the WER: almost no word has a wrong name. Every setup costs a few cents per hour of meeting, so compute does not decide.

Change one thing: Whisper’s own VAD. faster-whisper can skip the silence with its built-in Silero VAD (vad_filter=True), as in Why VAD makes recognition better. On the headsets it lowers the WER from 26.5% to 22.9% and nearly halves the compute. A headset is mostly silence and quiet crosstalk, where Whisper invents or copies words. On the far microphone it raises the WER, from 30.7% to 33.1%, because it cuts quiet speech. The same setting helps one microphone and hurts the other.

An average hides people. The chart shows each person’s WER on their own headset, without and with Whisper’s VAD.

Two marks per person for twelve people on a WER axis. Filled marks, Whisper small: the worst are IS1001a D at about 89%, IS1009a C at about 61% and ES2004a A at about 50%; the best are the three project managers, around 18 to 22%. Hollow marks, with the VAD: lower for most people, but IS1001a D goes off the chart at 162%.
WER of each person on their own headset (louder wins), on the three AMI meetings: Whisper small (filled) and with its VAD (hollow). The people who speak least are hit hardest, and the VAD that helps ten people makes the worst one worse.
Text description of the diagram

Twelve rows, one per person, sorted from the highest WER to the lowest. The worst three are the marketing expert of IS1001a (a woman whose first language is French), the industrial designer of IS1009a (a woman whose first language is Telugu) and the interface designer of ES2004a (a man whose first language is Hindi). The best three are the three project managers, between 18% and 22%. Hollow marks show Whisper with its VAD. They are lower for ten people and slightly higher for the interface designer of IS1009a. For the marketing expert of IS1001a, the hollow mark is off the chart: 161.7%.

The worst person, the marketing expert of IS1001a, says only 47 words, and 89% of them are wrong or lost. The project managers, who talk most, do best. The quiet people’s errors are mostly deletions. The VAD misses their short turns, or louder wins drops them when they speak over someone. Only 39% of the two worst people’s speech was found under their name, against 73% to 75% for the project managers.

With the VAD, her WER rises to 161.7%: her headset kept extra words from her neighbours. A better average can hide a worse worst case.

By group (without the VAD): women 23.8%, men 30.9%; native English speakers 21.0%, others 33.5%. These are not facts about groups. There are twelve people, and the groups are mixed up with how much each person talks. The lesson is the check: measure every person, and look at those who say little.

As in Summaries and follow-ups, Azure chat-small reads the transcript (text only, never audio). It returns action items in JSON: owner, task, due date and a short quote. Each answer was recorded once, so the notebook needs no key. One answer costs about US$0.0008; all 12 cost US$0.0099.

AMI’s human summaries list seven actions for the three meetings. The author compared each recorded answer with them (a judgement, not a measured score):

Transcript given to the LLM AMI actions found (of 7) Owner named right Items that are not AMI actions
The human reference, with roles 6 6 0
Headsets, louder wins, Whisper small (roles from the seats) 5 5 1 (not an action: “I’ll see what I can do” while practising on the whiteboard)
Far mic, embeddings, Whisper small (“Speaker 1”…“Speaker 4”) 6 6 1 (a real promise by the project manager)

The far microphone does as well here for a reason that will not last: the project managers said the roles aloud (“I will ask the industrial designer to …”). When people say “I’ll do it”, only the speaker name tells who “I” is. The end-to-end run shows this. The headsets missed the action of ES2004a because Whisper lost the sentence “and then we’ll come back and liaise again”. One lost sentence can be one lost action.

For Larkfield’s product meetings, Priya recommends:

  1. A microphone per person. In a room, a headset or a clip-on microphone for each person. In an online meeting, each person’s own audio stream, which most meeting tools can record apart: it is the same setup for free.
  2. Silero VAD at 0.5 on each channel, then louder wins. No smarter margin: the plain rule worked in every room.
  3. Whisper small with its VAD on each channel, keeping only the words inside the wearer’s turns: cpWER 22.9%. It was better for ten of the twelve people; the check below covers the others.
  4. The names come from the seats, not from voices. The setup links a person to a channel when they join, never by a voice print.
  5. Action items by the LLM, as a draft, from the text only, with a quote for each item.

What a person still checks, before the notes go out:

  • Every action item’s owner and date, against its quote. The model invented one action and lost another in our three meetings.
  • The quiet people. Read the lines of the people who said least: their words are the most likely to be missing, or to be a neighbour’s words.
  • Numbers, names and decisions. A WER near 25% means about one word in four is wrong; a price or a date must be read against the recording.

Cost per hour of meeting: compute US$0.026 (CPU server at US$0.17 an hour, an assumption). The LLM costs about US$0.0029 (recorded tokens). The person’s check is 5 minutes at the course’s assumed US$0.75 a minute: US$3.75. The person’s time is the biggest cost. So make the check short: a quote next to every item, and the quiet people’s lines marked. We did not price the headsets.

For a room with only a table microphone, this case study has no good open answer. Use the far microphone for the words without names, or test a pretrained pipeline after you accept its terms (see Responsible use).

IS1008a was never used to choose anything. We ran the recommended setup on it once: louder wins on the four headsets, then Whisper small with its VAD on each.

IS1008a (new input) DER WER cpWER
Headsets, louder wins, Whisper small with its VAD 20.1% 13.0% 13.0%
Far mic, embeddings, Whisper small (for contrast) 37.9% 15.2% 70.3%

The new meeting is easier than the three: less overlap (4.1%) and longer turns. The ranking is the same. Without the VAD, the headsets give 13.4%: here, the difference is within noise.

The notebook runs this setup live on a 5-minute excerpt of IS1008a, with Whisper tiny so that it is fast. The last lines of its notes (real output; the roles come from the seats):

[14:46] PM: And if anything you need any time, so please, either you can call me or just send me email or just comment, knock my door. So I'm available here.
[14:57] UI: Okay.
[14:58] ID: Thank you.
[14:58] PM: Thanks for coming and I will be sure, I will time
[15:01] UI: Thank you.
[15:01] PM: see you later.
[15:33] ME: a look at

From the headsets transcript of the whole meeting (Whisper small), the recorded action items found all four AMI actions, each with the right role. From the far microphone, the model found the same four tasks, but named the owners “Speaker 1”, “Speaker 3”, “Speaker 4” and “unclear”. Here people said “I’ll do it” more than they named roles, so those notes cannot say who.

  • The microphone decides. One microphone per person made names nearly free (confusion 1.5%). On the far microphone, the open speaker model could not tell voices apart.
  • Simple rules travel. Louder wins worked in both rooms; a tuned margin worked in one (the failure case below).
  • Real meetings are harder than Larkfield’s synthetic calls: WER near 25% even on headsets (3.1% on the English calls); far-microphone confusion 30% (16% on the synthetic mono calls).

Limits. Three meetings and twelve people, all kick-off meetings of one role-play, in rooms from 2005. Word-based true turns raise the missed part for every method. Whisper ran with one setting (greedy decoding, int8). The action items are one recorded answer per transcript, judged by the author; another run can differ. We did not run a pretrained pipeline.

Common mistake

A setting tuned in one room

Symptom: on ES2004a, Priya tried a softer louder wins, to keep real overlap. She compares each headset’s loudness with its own speech level, and every headset within 10 dB of the loudest keeps the moment. It lowered the DER of ES2004a from 23.3% to 21.7%. Then she ran it on the other meetings: 129.0% on IS1009a and 174.8% on IS1001a.

Cause: the IDIAP headsets hear much more of the room. There, a neighbour is often within 10 dB of the wearer, so the crosstalk came back as false speech. The setting fitted one room.

Fix: keep the plain louder wins, which works in every room we tried. Choose settings on several meetings and rooms, and keep a meeting apart that you never use to choose.

Consequence in AI work: in meetings, the room is part of the data.

Meeting notes are records of people. Before Larkfield records its own meetings, these rules apply, as in Personal data in voice:

  • Consent and a notice. Everyone knows the meeting is recorded and transcribed, and why. A person who does not agree can ask for notes by hand.
  • Delete the audio, keep the notes. Delete the recording when the notes are checked.
  • Text only to a hosted model. Here only transcripts went to the LLM, and its answers were recorded. The provider’s contract must cover meeting text.
  • Do not measure people with it. The microphone and the recogniser bias talk time. The quiet people, with the most missed speech, would look as if they said even less. The notes are for tasks, not for judging people.
  • No voice identification. Names come from the seats, not from voices; see Voice cloning, speaker checks and misuse.
  • The AMI people are real. We play only short clips with their transcript, use AMI’s letters and roles, and never try to find out who they are.

A pretrained pipeline such as pyannote (Pretrained diarization pipelines) is the usual next step for a far microphone. Its model page reports a DER of 19.9% on AMI’s far microphone for speaker-diarization-community-1 (a vendor claim, scored without a collar). We did not run it: it needs a token and acceptance of its terms. Before you compare, check whether AMI’s training meetings were part of its training.

Put the case study in a public repository:

  • README: the problem, the data with its licence and attribution, the three setups, the results table with the baseline, the recommendation and the limits.
  • The notebook, run from the top, with the package versions.
  • A model card for the setup: what it is for (notes for a team that agreed to it), what it is not for (judging people, identifying voices), and what a person checks.

In an interview, say it in two sentences: “I compared a table microphone with one headset per person for meeting notes on the AMI corpus. The open speaker-embedding method did not beat a one-speaker baseline on the far microphone. Headsets with a loudness rule cut the word-and-name error from 86.9% to 26.5%.”

Go further: run the notebook on more AMI meetings from all three rooms. Score a pretrained pipeline with the same der and cpwer, if you accept its terms. Mark the lines with an action item, a number or a date for the person who checks.

Check your understanding

The far microphone with speaker embeddings had a higher DER than putting all speech under one name. How can a method that finds four speakers be worse than one that finds one?

Show the answer

When the embeddings do not separate the voices, the clusters do not match people, and each wrong cluster adds confusion. “One speaker” is at least right for the person who talks most. On the far microphone, pieces of the same person and of different people were almost equally alike (0.83 against 0.83 in ES2004a). Before you trust a clustering, check that the embeddings separate the speakers on your own audio.

Check your understanding

With one headset per person, why was 'each headset alone' so bad in the IDIAP meetings, and what fixed it?

Show the answer

Each headset also hears the neighbours (crosstalk). So one sentence appears under three or four names: false speech of more than 200% in the IDIAP meetings. Louder wins gives a moment that several headsets hear to the loudest one, because the wearer is closest to their own microphone. The confusion fell to about 1–2%.

Check your understanding

Omar wants the notes to show how much each person talked, to see who is engaged. What do you answer?

Show the answer

No. The setup misses more speech of some people: even with headsets, it found only 39% to 75% of a person’s speech under their name. The recogniser also makes more errors for some voices. Those people would look less engaged because of the tools. The notes are for tasks, and people did not agree to have their talk time measured.

Recap

  • In meetings, people talk at the same time: the DER counts overlap, with missed, false and confusion apart.
  • On a far microphone, the open speaker-embedding method did not beat “everyone is one speaker”: the room, not the clustering, was the problem.
  • With one headset per person, louder wins removes the crosstalk and leaves mostly missed speech.
  • cpWER scores notes with names in one number: words and names must both be right. With headsets and louder wins it fell to 26.5%, against 86.9% on the far microphone with embeddings (Whisper small).
  • Check every person, choose settings on more than one room, and let a person check every action item’s owner.

Next, a different job on a small device: spotting short spoken commands in the audio itself, and counting false alarms per hour.

Next: Spot commands on a small device