Skip to content
Français

Compliance phrases and keyword spotting

  • Lesson 21
  • 35 min
  • Module 6: From transcript to decision

Where we are

Summaries tell us what happened in a call. Some phrases must be said in every call, such as the recording notice. In this lesson, we search transcripts exactly and with fuzzy match, spot keywords in the audio, and compare recall and false alarms.

Lesson 21 of 36 · Speech AI: Transcribe, Understand and SpeakCompliance phrases and keyword spotting

What you will be able to do

Check that the agent said the recording notice in every call. Compare three ways. Search the transcript for the exact phrase, search it with a fuzzy match, and listen for the key words in the audio itself. Measure how many notices each one finds, and how often it says “said” when the notice was not said.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds the exact search and the fuzzy match, and computes one similarity by hand. It measures both searches on the test calls and on all 260 public calls. It shows two sentences that fool the fuzzy match, and asks you to choose a threshold. It needs no model download and no account. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the guided practice; the notebook needs none of it)

  • Your folder larkspeech with the files of Why did they call?. pytest -q prints 27 passed.

Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.

Think of a supervisor in a call centre. Once a week, she listens to ten calls from each agent. For each call, she has a checklist: “Said the recording notice? Checked the caller’s name? Offered more help?” She ticks yes or no.

The first item is special. Callers must hear that the call is recorded, before they say anything personal. Larkfield’s agents start every call with the same sentence: “This call is recorded for quality and training.”

Ten calls a week from each agent is a small sample. Larkfield takes about 4,000 calls a month. Grace wants every call checked, not a sample.

Listen to the start of call C-0064.

Call C-0064, from 0.5 to 9.7 seconds, phone audio over a VoIP line (8 kHz). The agent is a synthetic voice: Kokoro, voice bf_emma. Every Larkfield call is synthetic: open text-to-speech voices read scripts that we wrote.

Transcript. Agent: “Thank you for calling Larkfield, this is Freya. Just so you know, calls may be recorded for training and quality purposes. How can I help you today?”

Freya said the notice. She used other words, in another order. A supervisor ticks “yes”. A program that looks for the exact sentence says “no”.

The opposite problem is worse. A call where the agent forgot the notice, but the program says “yes”, is a false alarm. Nobody looks at that call again, and the missing notice stays hidden.

Now listen to a French call, C-0020. Here, the agent says the exact notice, but the recogniser makes a small mistake.

Call C-0020, from 0.5 to 9.4 seconds, phone audio over a VoIP line (8 kHz). The agent is a synthetic voice: Kokoro, voice ff_siwis.

Transcript. Agent: « Merci d’avoir appelé Larkfield, ici Camille. Cet appel est enregistré pour la qualité et la formation. Comment puis-je vous aider aujourd’hui ? » (In English: “Thank you for calling Larkfield, this is Camille. This call is recorded for quality and training. How can I help you today?”)

Whisper small wrote: “Merci d’avoir appelé Lockfield, ici Camille. Ce tapel est enregistré pour la qualité et la formation.” “Cet appel” and “ce tapel” sound the same. The letters moved by one space, and the exact search fails.

The idea: count the notices found, and the false alarms

Section titled “The idea: count the notices found, and the false alarms”

Two numbers score any notice check. Both use the labels of our calls:

  • Recall is the share of the calls with the notice where the check finds it. A check with low recall sends good calls to a person for nothing.
  • False alarms are the calls without the notice that the check marks as compliant. Each one hides a real problem.

There are three ways to find the notice:

  1. Exact search: look for the sentence, letter for letter, in the transcript after the basic normalisation (lower case, no punctuation). It is simple and never fooled by a near miss. But it misses every other wording and every small recognition error.
  2. Fuzzy match: accept text that is close to the notice. “Fuzzy” means “not exact”. The project accepts two kinds of “close”, shown below.
  3. Keyword spotting: do not make a transcript at all. Listen to the audio for a few chosen words only, such as “recorded”, “quality” and “training”.
The agent's words go through three tests in order. 1, exact phrase: does this call is recorded for quality and training appear letter for letter? If yes, notice found. If no, 2, key words: do recorded, quality and training all appear, in any order? If yes, notice found. If no, 3, similar window: slide a window as long as the notice, 8 words, over the text; is the similarity 0.6 or more, with at least one key word? If yes, notice found. If no, the notice is missing and a person listens.
The project's notice check. Each test catches what the one before it misses; a call that fails all three goes to a person.
Text description of the diagram

The agent’s words go through three tests, one after the other. Test 1, exact phrase: does “this call is recorded for quality and training” appear letter for letter? If yes, the notice is found. If no, test 2, key words: do “recorded”, “quality” and “training” all appear, in any order? If yes, the notice is found. If no, test 3, a similar window: slide a window of words as long as the notice, 8 words in English, along the text. Is its similarity to the notice 0.6 or more, and does it contain at least one key word? If yes, the notice is found. If no, the notice is missing, and a person listens to the call.

The key-word test catches Freya’s other wording in C-0064. The window test catches small recognition errors, as in C-0020. In French, the key words are « enregistré », « qualité » and « formation ».

A tiny worked example: how similar is “ce tapel”?

Section titled “A tiny worked example: how similar is “ce tapel”?”

The similarity of two texts, as Python’s difflib computes it, counts the letters that appear in both texts in the same order. Call that number M. Then:

similarity = 2 × M ÷ (length of text A + length of text B)

The result goes from 0 (nothing alike) to 1 (the same text). For C-0020, after the basic normalisation:

Text Letters and spaces
The notice cet appel est enregistré pour la qualité et la formation 56
Whisper heard ce tapel est enregistré pour la qualité et la formation 55
In both, in the same order everything except one t and one space 54

The similarity is 2 × 54 ÷ (55 + 56) = 108 ÷ 111 = 0.973. That is far above the threshold of 0.6. The window also contains key words. So the fuzzy match says “notice found”, and the exact search says “no”.

Now a near miss that we made up on purpose: “This call is not recorded for quality and training.” Its similarity to the notice is 0.958, and all three key words are there. The fuzzy match says “notice found”, although the sentence says the opposite. Close is not the same.

Priya runs the checks on the 60 test calls. 57 of them have the notice (48 exact, 9 other wordings), and 3 do not. She also reads the recording_notice field of the LLM’s call records from the last lesson. The keyword spotter is in the next section.

  • Exact transcript
  • Whisper small's transcript
  • Audio (keyword spotting)
Recall of each notice check on Larkfield's 60 synthetic test calls (measured; the LLM answers are recorded). No check made a false alarm, but only 3 test calls miss the notice.
Text description of the chart

A column chart with seven columns on an axis from 0 to 100%. On the exact transcripts: exact search 84.2% (48 of 57), fuzzy match 100%, LLM 100%. On Whisper small’s transcripts: exact search 71.9% (41 of 57), fuzzy match 100%, LLM 100%. Keyword spotting in the audio, with Vosk and three key words in the first 20 seconds: 56.1% (32 of 57).

  • The exact search misses the 9 other wordings on the exact transcripts (84.2%). On Whisper’s transcripts it also misses 7 French notices, all with “ce tapel” or a similar spelling (71.9%).
  • The fuzzy match finds all 57, on both transcripts. Recognition errors do not move it.
  • The LLM also finds all 57. It costs a model call, but the call record already has the field, so it is free here.
  • No check made a false alarm on the 3 test calls without the notice.

Three calls are not enough for false alarms

Section titled “Three calls are not enough for false alarms”

“No false alarm” on 3 calls says almost nothing. One false alarm in three would already be 33%. To measure false alarms, you need many calls without the notice.

The train calls have no audio, but they have transcripts and labels. Priya runs the two searches on all 260 public calls: 221 with the notice and 39 without.

Check (exact transcripts) Notices found False alarms
Exact search 183 of 221 (82.8%) 0 of 39
Fuzzy match 221 of 221 (100%) 0 of 39

Still no false alarm, now on 39 calls. But be careful again. In our scripts, an agent who forgets the notice says nothing like it. A real agent may say “I’m not recording this” or talk about “training”. The two sentences that we made up show that the fuzzy match passes such near misses. Before you trust it on real calls, collect real near misses, and let a person listen to a sample of the calls that pass.

The third method skips the transcript. A keyword spotter listens to the audio for a short list of words and ignores everything else. The course used Vosk, the older recogniser of Module 4, with a phrase list: Vosk may only choose between “recorded”, “quality”, “training” and “unknown word”. The spotter listens to the first 20 seconds of each call, where the notice should be. It marks the notice as said when it hears at least two of the three words.

Why a phrase list? A general recogniser writes the words that it knows. In Module 4, Vosk never once wrote “Larkfield” in 120 tries: it wrote “luck field” 16 times and “lark feel” 9 times. With a phrase list, the recogniser can only answer with your words, so it cannot drift to similar common words.

On the 60 test calls (measured):

Keyword spotting (Vosk, phrase list)
Notices found 32 of 57 (56.1%)
False alarms 0 of 3
Compute real-time factor 0.009: about 34 seconds for an hour of audio

It is very fast and needs no full transcript. But it found only about half the notices. The small Vosk model on 8 kHz phone audio often hears one key word but not two. Lower the rule to one word, and recall would rise, and false alarms too: “training” alone could be about something else.

You already copied understand.py in Why did they call?. The command speech compliance runs the exact search and the fuzzy match on the agent’s turns of a split.

  1. Open a terminal in your project folder larkspeech, and activate .venv.

  2. Check the notice on the reference transcripts of the test calls:

    Terminal window
    speech compliance

    The output on the course’s computer:

    Recording notice on 60 test calls (57 said it, 3 did not), transcripts: reference
    exact phrase found 48 of 57 notices (recall 84.2%); false alarms 0 (a call without the notice marked as compliant)
    fuzzy match found 57 of 57 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant)
  3. Check it on Whisper small’s recorded transcripts:

    Terminal window
    speech compliance --transcripts whisper-small
    Recording notice on 60 test calls (57 said it, 3 did not), transcripts: whisper-small
    exact phrase found 41 of 57 notices (recall 71.9%); false alarms 0 (a call without the notice marked as compliant)
    fuzzy match found 57 of 57 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant)

    Whisper’s transcripts have both speakers and no labels, so the search reads the whole call.

The test split has only 3 calls without the notice. Which split has the most? Run the check on it, and say what the false-alarm numbers now tell you.

Show the answer

The train split: 30 of its 150 calls have no notice. Its calls have no audio, but the reference transcripts are enough for this check.

Terminal window
speech compliance --split train
Recording notice on 150 train calls (120 said it, 30 did not), transcripts: reference
exact phrase found 100 of 120 notices (recall 83.3%); false alarms 0 (a call without the notice marked as compliant)
fuzzy match found 120 of 120 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant)

No false alarm in 30 calls is better evidence than none in 3. It is still evidence about our scripts only: none of them has a near miss such as “this call is not recorded”.

Common mistake

The exact search reports many missing notices after a script change

Symptom: one Monday, the number of calls “without the notice” jumps. The agents say they did not change anything.

Cause: the team changed one word of the notice, or a new recogniser spells a word differently (“Lockfield”, “ce tapel”). The exact search fails on every call.

Fix: use the fuzzy match, with the key words of the notice. Keep the exact search as a second number: when the two numbers move apart, something changed.

Common mistake

Searching the whole call instead of the agent's turns

Symptom: a call is marked compliant, but the agent never said the notice.

Cause: the caller said the words: “Is this call recorded? I hope it is, for quality.” The search read both speakers. On Whisper’s mono transcripts, there are no speaker labels.

Fix: search the agent’s turns only, from a stereo recording or from diarization. Search the first 30 seconds, where the notice belongs. Our scripts have no such caller sentence, so the course did not measure this case.

  • Exact search: when the wording is fixed and the text is exact, such as a chat or an email. As a second number beside the fuzzy match.
  • Fuzzy match on the transcript: when you already transcribe every call, as Larkfield does. It costs nothing more, and it accepts other wordings and small recognition errors. Test it on near misses.
  • Keyword spotting in the audio: when you do not want a full transcript. Examples are a live check during the call, a small device, or a rule that forbids storing transcripts. It is fast, but here it found only 56% of the notices. The case study Spot commands on a small device measures a spotter on real recordings.
  • The LLM’s field: when you make call records anyway. Check it like any other field.

Check your understanding

Tomás says: 'The fuzzy match found 57 of 57 notices with no false alarm. It is perfect.' What do you tell him?

Show the answer

The recall is measured well: 57 of 57 on the test calls, and 221 of 221 on all public calls. The false alarms are not. Only 3 test calls miss the notice. The 39 public calls without it all come from our scripts, where a forgetful agent says nothing similar. Take a made-up near miss: “This call is not recorded for quality and training.” It has a similarity of 0.958 and all three key words, so the fuzzy match passes it. Test on real near misses, and listen to a sample of the calls that pass.

Check your understanding

Predict: you switch from the exact transcripts to Whisper small's transcripts. What happens to the recall of the exact search, and of the fuzzy match? Why?

Show the answer

The exact search falls, from 84.2% to 71.9%. Whisper wrote seven French notices with a small error, such as “ce tapel” for “cet appel”. One changed letter or space breaks an exact match. The fuzzy match stays at 100%. The key words are still there. The similarity of “ce tapel est enregistré…” to the notice is 0.973, far above the threshold of 0.6.

Check your understanding

Practical check: run speech compliance --split validation. How many calls miss the notice, what does each search find, and why does Grace need both numbers?

Show the answer

The output is: 50 validation calls, 44 said the notice and 6 did not. The exact phrase finds 35 of 44 (recall 79.5%), the fuzzy match 44 of 44 (100.0%), with no false alarm for either. Grace uses the fuzzy match to decide, because it accepts fair other wordings. She keeps the exact number too. The gap between the two shows how many agents use another wording. A sudden change in the gap shows that a script or the recogniser changed.

Recap

  • Score a notice check with recall (notices found) and false alarms (calls without the notice marked as compliant).
  • Exact search found 84.2% of the notices on Larkfield’s 60 synthetic test calls, and 71.9% on Whisper small’s transcripts. The fuzzy match (key words in any order, or a similar window) found 100% on both.
  • Similarity = 2 × letters in common ÷ the two lengths: “ce tapel est enregistré…” is 0.973 similar to the notice.
  • Only 3 test calls miss the notice, so false alarms need the train and validation labels too: 0 of 39 on all public calls. Near misses that we made up still fool the fuzzy match.
  • Keyword spotting with Vosk and a phrase list is very fast (real-time factor 0.009), but found only 56% of the notices.

You can now turn a call into decisions: its reason, a checked record and a compliance check. All of that happens after the call. Next, Larkfield wants a phone line that answers by itself, during the call. We start with the oldest and safest kind: a phone menu.

Next: What an IVR is, and why companies still use one