Skip to content
Français

Word times and alignment

  • Lesson 18
  • 35 min
  • Module 5: Who spoke when: diarization and timing

Where we are

We know who spoke when, and we know what was said. The two are still separate. In this lesson, we join words to speakers with timestamps, check how accurate the word times are, and write a transcript with names and times.

What you will be able to do

Write a transcript that says who said what, and when. Give each word a time, give each word the speaker whose turn holds it, and check the sentences that matter before anyone acts on them.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the code, the tasks and the checks of this lesson. It joins words to turns by hand, and measures Whisper’s word times against the voices’ own times. It writes a transcript with names, and catches a promise under the wrong name. It downloads Whisper tiny (75 MB) once and needs no account. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the steps)

  • Your project folder larkspeech with the files of end-of-m05. The steps use Whisper tiny, which the project downloads the first time (about 75 MB).

Tested on macOS with Python 3.12 and faster-whisper 1.2.1. Windows and Linux were not tested.

Think of the subtitles of a film for people who cannot hear. Each line appears at the right moment, and when two people talk, the line says who speaks: “ANNA: I told you already.” The subtitle writer joins two things: the words, and the person who says them, at a time.

Larkfield needs the same thing for its calls. Grace reads call records, not audio. Take the line “[15.1 s] agent: I will email you the setup guide today”. It tells her who promised what. It also tells her where to listen if she has a doubt.

After Module 4 and the first lessons of this module, we have two separate outputs for each call:

  • The recogniser gives the words, as one long text.
  • The diarization gives the turns: start, end and speaker, with no words.

Neither is enough. And a mistake in one of them can change a decision. In this lesson’s main example, call C-0108, the agent says: “I will email you the setup guide with pictures today.” The embedding method of Speaker embeddings and clustering gives that whole sentence to the caller. A call record built from it says that the caller promised to send a guide. Nobody at Larkfield sends it, and Grace’s cost table puts a missed follow-up at US$25 (an assumption).

The idea: a time for every word, then a join

Section titled “The idea: a time for every word, then a join”

Whisper can return a start and an end time for each word, not only for each sentence. This is called word timestamps (or word times). With word times, we can align the two outputs: give each word to the turn that holds it in time.

Two boxes, the recogniser with each word's start and end, and the diarization with turns of start, end and speaker, both go to Join: the word's middle falls in which turn? Then Group: words in a row with the same speaker. Then the transcript: time, speaker and words.
Who said what, and when: join each word to a turn by the middle of the word, then group the words of one speaker into lines.
Text description of the diagram

Two boxes start the diagram. The recogniser gives each word with a start and an end. The diarization gives turns with a start, an end and a speaker. Both go to the join step: for each word, find the turn that holds the middle of the word. Then the group step puts the words in a row with the same speaker into one line. The result is a transcript where each line has a time, a speaker and the words.

Why the middle of the word? Word times and turn borders both have small errors. A word’s start can fall a little before its turn starts; its middle is safer. Sometimes the middle falls in a short gap between two turns. Then the project gives the word to the nearest turn, if that turn is at most 0.5 s away. Further away, the word gets “?”: nobody was found speaking there.

The agent’s turn runs from 0.0 to 1.0 s, and the caller’s turn from 1.2 to 3.2 s. These are the words, with their times (a constructed example):

Word Start End Middle Turn that holds the middle
Can 0.10 0.30 0.20 agent
I 0.30 0.38 0.34 agent
have 0.38 0.62 0.50 agent
it? 0.62 0.95 0.785 agent
Yes, 1.30 1.55 1.425 caller
it’s 1.55 1.80 1.675 caller
742155. 1.80 3.10 2.45 caller

Grouped into lines:

[ 0.1 s] agent Can I have it?
[ 1.3 s] caller Yes, it's 742155.

Word times are estimates. Whisper does not measure them; it guesses them from where its model “looks” in the sound while it writes each word. So we must check them. Larkfield’s calls are synthetic, and that helps: the voices reported when they said each word. These are true word times, which real recordings never have.

Listen to a short part of call C-0003, where the agent asks for the order number:

Call C-0003, from 13.7 to 20.9 s, wideband (16 kHz), both voices mixed. Two synthetic voices, both Piper trained on the VCTK corpus: the agent (speaker p240) and the caller (speaker p330). Transcript. Agent: “I can help with that. Can I have your order number, please?” Caller: “Yes, it’s five nine seven … four one nine.” (The caller pauses inside the number.)

The picture puts the voice’s own word times above Whisper base’s word times for the same seconds.

A waveform from 13.7 to 20.9 seconds, then two rows of word boxes. The voice's own times: I, can, help, with, that, Can, I, have, your, order, number, please, Yes, it's, five, nine, seven, and after a pause four, one, nine. Whisper base: the same words up to Yes and it's, each a little earlier; then one box 597 that covers five nine seven, and one box 419 after the pause.
Word times in call C-0003: the voice's own times (top) and Whisper base (bottom). Whisper's boxes start a little early, and it writes the digits as one word, 597, where the voice said three words.
Text description of the diagram

The top row is the sound. The middle row has one box for each word as the voice said it: “I can help with that. Can I have your order number, please?”, then “Yes, it’s five nine seven”, a pause of about 1.6 seconds, and “four one nine”. The bottom row has Whisper base’s words. Up to “it’s”, each Whisper box matches a true word, and most start a few hundredths of a second earlier. Then Whisper writes “597” as one word from 17.3 to 17.8 s, where the voice said three words, “five nine seven”. After the pause, it writes “419”. On a phone screen, the picture shows only the part from 16.4 s.

Two things to see. First, the times are close: a few hundredths of a second, much shorter than a word. Second, a word is not always the same unit on both sides. Whisper writes “597” for the three spoken words “five nine seven”. Its time covers all three. That is fine for a join with turns, but it means that you cannot always pair the words one to one.

The course measured Whisper base’s word starts on 20 English calls (wideband). For each true word, we looked for a Whisper word with the same spelling that starts within 1 second, and took the time difference.

A histogram of start time errors from 0 to 400 milliseconds. The highest bar is the first, 0 to 20 ms, with 465 words. The bars fall quickly; few words are over 200 ms. Dashed lines mark the median at 55 ms and the 90th percentile at 137 ms.
How far Whisper base's word starts are from the true starts: 1,924 matched words of 20 English synthetic calls. Median 55 ms, 90th percentile 137 ms; 80% of the words are within 0.1 s.
Text description of the diagram

The histogram counts words by the error of their start time, in steps of 20 milliseconds. 465 words are within 20 ms, 264 between 20 and 40 ms, 321 between 40 and 60 ms, and 288 between 60 and 80 ms. Then the bars fall: 200, 119, 82, 41, and fewer than 30 for each step after 160 ms. The last bar, at 380 ms and more, holds 18 words. A dashed line marks the median, 55 ms: half of the words are closer than this. Another marks the 90th percentile, 137 ms: nine words in ten are closer than this.

Measure (Whisper base, 20 English calls) Value
True words / matched within 1 s 2,201 / 1,924
Median start error 0.055 s
90th percentile 0.137 s
Words within 0.1 s 80%
Words over 0.25 s 4%
Whisper’s start earlier than the true start 81% of the words (median −0.053 s)

277 words did not match. Some of them are numbers that Whisper wrote as digits, like “597”, and some are words that Whisper got wrong. On Larkfield’s synthetic calls, a word time is good to about a twentieth of a second. That is plenty to give a word to a turn: turns last seconds. It is not enough to cut one word out of the audio exactly. When you beep a card number in Personal data in voice, leave a margin around the words.

The project’s speech diarize --call … --words does the join of this lesson. It transcribes the call with Whisper tiny (fast), joins each word to a turn by its middle, and prints one line per speaker change.

  1. Write call C-0108 with the embedding method. C-0108 is stereo, so the method first mixes it to mono:

    Terminal window
    speech diarize --call C-0108 --words
    C-0108 (stereo), embeddings: DER 42.7% (missed 0.0%, false 0.8%, confusion 41.9%)
    [ 0.5 s] agent Thank you for calling Luxeil's
    [ 2.0 s] caller Discipictor. Discold's
    [ 3.4 s] agent recorded for quality and training.
    [ 5.0 s] caller How can I help you today? Hello there. I'm getting frustrated with the war haven't. You can clean huggutaths with drugs probably. Good question. For the willow hammock, the first step is to charge it to fully before you set it up. I will email you the set of guides and pictures today.
    [ 17.7 s] agent Is there anything else I
    [ 18.4 s] caller could help you with? With everything thank you. Thank you for calling Luxeil to have a good day.

    The long line from 5.0 s holds the agent’s question, the caller’s question and the agent’s answer, all under “caller”. It ends with the agent’s promise, “I will email you the set of guides and pictures today”, under the caller’s name.

    Two other things are not diarization errors. Whisper tiny on a phone line with background talk writes poor text (“Luxeil” for “Larkfield”). And “charge it fully before you set it up”, said about a hammock, is a mistake in our own scripts. The agent says it for every product.

  2. Write the same call by channel:

    Terminal window
    speech diarize --call C-0108 --method channel --words
    C-0108 (stereo), channel: DER 17.1% (missed 12.7%, false 0.0%, confusion 4.4%)
    [ 0.5 s] agent Thank you for calling Luxeil's Discipictor. Discold's recorded for quality and training. How can I
    [ 5.2 s] caller help you
    [ 5.6 s] agent today?
    [ 6.5 s] caller Hello there. I'm getting frustrated with the war haven't. You can clean huggutaths with drugs probably.
    [ 10.7 s] agent Good question. For the willow hammock, the first step
    …
    [ 15.2 s] agent I will email you the set of guides and pictures today. Is there anything else I could help you with?
    [ 19.4 s] caller With everything thank you.
    [ 20.8 s] agent Thank you for calling Luxeil to have a good day.

    The promise is now under the agent. A few short words still go to the caller (“help you”): background talk on the caller’s side, which louder wins does not remove everywhere.

Of the 110 validation and test calls, C-0108 is the only one where the embedding method gives most of the promise to the caller. In 10 calls, it gives more than a tenth of the promise to the caller. A rare error, but an expensive one. Three cheap checks catch most of it:

  1. A rule on the roles. At Larkfield, agents promise; callers ask. A caller line that says “I will”, “I’ll” or “I have” (done something) is suspect. In the notebook, this rule flags one line of C-0108 with the embedding turns, and none with the true turns.
  2. A second method. When the call is stereo, compare the speaker of the promise by channel and by embeddings. If they disagree, trust the channel.
  3. A person for the rest. A flagged call goes to Grace’s team with the time of the sentence. A team lead listens to 5 seconds, not the whole call.

None of these checks fixes the names. They stop a wrong name from becoming a wrong action.

When to use word times, and what they cost

Section titled “When to use word times, and what they cost”

Use them for every transcript that people or programs act on. Examples are call records, searches (“find the calls where the agent said ‘refund’”), compliance checks with a time, and redaction. They come with the recognition: word_timestamps=True in faster-whisper. They can make the transcription a little slower, and they need no other model.

Do not trust them to the hundredth of a second. And do not pair words one to one: a number can be one word on one side and three on the other. On real recordings, you have no true word times to check against. Check a sample by listening, or use a forced aligner, a tool that places a known text in the audio, when you need precise times.

What it costs: almost nothing more than the transcript. The expensive part is a wrong speaker under an important sentence, and the cheap checks above cost a few lines of code.

  1. Open the notebook from the box Do this lesson in a notebook at the top of this page.
  2. Run the setup cells and the helper cell. The setup installs faster-whisper the first time.
  3. In section 1, predict who said “it?”, then run the cell. You see two lines: the agent from 0.1 s, the caller from 1.3 s.
  4. In sections 2 and 3, transcribe call C-0011 with Whisper tiny and measure its word starts. You see 97 of 113 words matched, a median error of 0.074 s and a 90th percentile of 0.140 s. Whisper tiny is a little less precise than Whisper base.
  5. In section 4, write the first lines of C-0011 with names and times.
  6. In section 5, write call C-0108 with the turns of the embedding method. Find the promise, and read its speaker.
  7. In section 6, run the role rule. You see one line flagged with the embedding turns, and none with the true turns.

In section 8 of the notebook, use the start errors of call C-0011 from section 3 (errors). Compute the share of matched words whose start error is 0.1 s or less, round it to three decimals, and put it in task_share. Before you run it, predict: more or less than the 80% of Whisper base on 20 calls?

Show the answer
task_share = round(float(np.mean(errors <= 0.1)), 3)
print(task_share)
0.66

With Whisper tiny, 66% of the matched words of C-0011 start within 0.1 s. That is less than the 80% of Whisper base on 20 calls. A smaller model, one call: both make the number lower. For a join with turns that last seconds, it is still enough.

Common mistake

Words that belong to nobody

Symptom: an early version of the project’s --words gave lines like these for call C-0108, by channel. Sentences break into pieces, and some pieces have no speaker:

[ 0.5 s] agent Thank you for calling Luxeil's Discipictor. Discold's recorded
[ 3.7 s] ? for quality
[ 4.2 s] agent and training.
…
[ 15.2 s] agent I will email you the
[ 15.8 s] ? set of guides
[ 16.3 s] agent and pictures today. Is there anything else I could help you with?

Cause: the turns come from the VAD, which ends a turn at each short pause. A word whose middle falls in the gap belongs to no turn, so the join wrote “?”. Whisper’s word times are often a little early, which can move a word’s middle into a gap.

Fix: give a word in a short gap to the nearest turn, if that turn is at most 0.5 s away. The project’s speaker_at in diarize.py does this now, and the output of step 2 above has no “?”. Words further than 0.5 s from any turn still get “?”: they are a sign that the VAD missed speech.

Consequence in AI work: a summary model reads a line with “?” as a third speaker, or drops it. Here, part of the agent’s promise sat in a “?” line, so a record could lose the words “set of guides”.

Check your understanding

A word runs from 4.80 to 5.30 s. The agent's turn ends at 5.00 s, and the caller's turn starts at 5.20 s. Who said the word, by the project's rule?

Show the answer

The middle of the word is (4.80 + 5.30) / 2 = 5.05 s. It is in the gap between the turns, so the rule takes the nearest turn within 0.5 s. The agent’s turn ends 0.05 s before, and the caller’s turn starts 0.15 s after. So the word goes to the agent.

Check your understanding

Whisper base's median word-start error on Larkfield's calls is 0.055 s. Is that good enough to give words to speakers? Is it good enough to beep out one word of a card number exactly?

Show the answer

It is good enough to give words to speakers: turns last seconds, and nine words in ten are within 0.137 s. It is not good enough to cut one word exactly. A word lasts about 0.2 to 0.5 s. And one word in ten is off by more than 0.137 s. Whisper’s starts are also usually early. So a beep needs a margin before and after the words, and a check that nothing is left.

Check your understanding

Grace asks how she will know that a promise in a call record was made by the agent and not by the caller. Name one check, and what it costs.

Show the answer

Use a rule on the roles. At Larkfield, agents promise and callers ask. So the rule flags a caller line with “I will”, “I’ll” or “I have”. It costs a few lines of code. In the notebook, it flags C-0108’s promise with the embedding turns, and nothing with the true turns. For stereo calls, she can also compare the channel method, which confuses speakers about four times less. A flagged call goes to a team lead with the time of the sentence, so a check takes seconds of listening, not the whole call.

Recap

  • Word timestamps give each word a start and an end. Alignment joins each word to the turn that holds its middle. A word in a short gap goes to the nearest turn within 0.5 s.
  • On 20 English synthetic calls, the median error of Whisper base’s word starts is 0.055 s. The 90th percentile is 0.137 s. Its starts are usually early, and it can write three spoken words as one (“597”).
  • Good enough to give words to speakers; not good enough to cut one word exactly without a margin.
  • A speaker error can put a promise under the wrong name. In C-0108, the embedding method gives the agent’s whole promise to the caller. By channel, it is right.
  • Catch it before anyone acts: a role rule, a second method for stereo calls, and a person for flagged calls.

You now have transcripts that say who said what, and when. Module 6 reads them: why did the caller call, what did the agent promise, and did the agent say the recording notice?

Next: Why did they call?