Compliance phrases and keyword spotting
- Lesson 21
- 35 min
- Module 6: From transcript to decision
Where we are
Summaries tell us what happened in a call. Some phrases must be said in every call, such as the recording notice. In this lesson, we search transcripts exactly and with fuzzy match, spot keywords in the audio, and compare recall and false alarms.
Lesson 21 of 36 · Speech AI: Transcribe, Understand and SpeakCompliance phrases and keyword spotting
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Check that the agent said the recording notice in every call. Compare three ways. Search the transcript for the exact phrase, search it with a fuzzy match, and listen for the key words in the audio itself. Measure how many notices each one finds, and how often it says “said” when the notice was not said.
NotebookDo this lesson in a notebook
The notebook builds the exact search and the fuzzy match, and computes one similarity by hand. It measures both searches on the test calls and on all 260 public calls. It shows two sentences that fool the fuzzy match, and asks you to choose a threshold. It needs no model download and no account. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The call record and its
recording_noticefield, from Summaries and follow-ups. - Recall and false alarms: the machine learning course explains them in The precision–recall trade-off. This lesson explains what you need.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your folder
larkspeechwith the files of Why did they call?.pytest -qprints27 passed.
Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.
The supervisor’s checklist
Section titled “The supervisor’s checklist”Think of a supervisor in a call centre. Once a week, she listens to ten calls from each agent. For each call, she has a checklist: “Said the recording notice? Checked the caller’s name? Offered more help?” She ticks yes or no.
The first item is special. Callers must hear that the call is recorded, before they say anything personal. Larkfield’s agents start every call with the same sentence: “This call is recorded for quality and training.”
Ten calls a week from each agent is a small sample. Larkfield takes about 4,000 calls a month. Grace wants every call checked, not a sample.
The problem: the notice has many forms
Section titled “The problem: the notice has many forms”Listen to the start of call C-0064.
Transcript. Agent: “Thank you for calling Larkfield, this is Freya. Just so you know, calls may be recorded for training and quality purposes. How can I help you today?”
Freya said the notice. She used other words, in another order. A supervisor ticks “yes”. A program that looks for the exact sentence says “no”.
The opposite problem is worse. A call where the agent forgot the notice, but the program says “yes”, is a false alarm. Nobody looks at that call again, and the missing notice stays hidden.
Now listen to a French call, C-0020. Here, the agent says the exact notice, but the recogniser makes a small mistake.
Transcript. Agent: « Merci d’avoir appelé Larkfield, ici Camille. Cet appel est enregistré pour la qualité et la formation. Comment puis-je vous aider aujourd’hui ? » (In English: “Thank you for calling Larkfield, this is Camille. This call is recorded for quality and training. How can I help you today?”)
Whisper small wrote: “Merci d’avoir appelé Lockfield, ici Camille. Ce tapel est enregistré pour la qualité et la formation.” “Cet appel” and “ce tapel” sound the same. The letters moved by one space, and the exact search fails.
The idea: count the notices found, and the false alarms
Section titled “The idea: count the notices found, and the false alarms”Two numbers score any notice check. Both use the labels of our calls:
- Recall is the share of the calls with the notice where the check finds it. A check with low recall sends good calls to a person for nothing.
- False alarms are the calls without the notice that the check marks as compliant. Each one hides a real problem.
There are three ways to find the notice:
- Exact search: look for the sentence, letter for letter, in the transcript after the basic normalisation (lower case, no punctuation). It is simple and never fooled by a near miss. But it misses every other wording and every small recognition error.
- Fuzzy match: accept text that is close to the notice. “Fuzzy” means “not exact”. The project accepts two kinds of “close”, shown below.
- Keyword spotting: do not make a transcript at all. Listen to the audio for a few chosen words only, such as “recorded”, “quality” and “training”.
Fuzzy match in three steps
Section titled “Fuzzy match in three steps”Text description of the diagram
The agent’s words go through three tests, one after the other. Test 1, exact phrase: does “this call is recorded for quality and training” appear letter for letter? If yes, the notice is found. If no, test 2, key words: do “recorded”, “quality” and “training” all appear, in any order? If yes, the notice is found. If no, test 3, a similar window: slide a window of words as long as the notice, 8 words in English, along the text. Is its similarity to the notice 0.6 or more, and does it contain at least one key word? If yes, the notice is found. If no, the notice is missing, and a person listens to the call.
The key-word test catches Freya’s other wording in C-0064. The window test catches small recognition errors, as in C-0020. In French, the key words are « enregistré », « qualité » and « formation ».
A tiny worked example: how similar is “ce tapel”?
Section titled “A tiny worked example: how similar is “ce tapel”?”The similarity of two texts, as Python’s difflib computes it, counts the letters that appear in both texts in the same order. Call that number M. Then:
similarity = 2 × M ÷ (length of text A + length of text B)
The result goes from 0 (nothing alike) to 1 (the same text). For C-0020, after the basic normalisation:
| Text | Letters and spaces | |
|---|---|---|
| The notice | cet appel est enregistré pour la qualité et la formation |
56 |
| Whisper heard | ce tapel est enregistré pour la qualité et la formation |
55 |
| In both, in the same order | everything except one t and one space |
54 |
The similarity is 2 × 54 ÷ (55 + 56) = 108 ÷ 111 = 0.973. That is far above the threshold of 0.6. The window also contains key words. So the fuzzy match says “notice found”, and the exact search says “no”.
Now a near miss that we made up on purpose: “This call is not recorded for quality and training.” Its similarity to the notice is 0.958, and all three key words are there. The fuzzy match says “notice found”, although the sentence says the opposite. Close is not the same.
Larkfield: the notice on 60 test calls
Section titled “Larkfield: the notice on 60 test calls”Priya runs the checks on the 60 test calls. 57 of them have the notice (48 exact, 9 other wordings), and 3 do not. She also reads the recording_notice field of the LLM’s call records from the last lesson. The keyword spotter is in the next section.
- Exact transcript
- Whisper small's transcript
- Audio (keyword spotting)
Text description of the chart
A column chart with seven columns on an axis from 0 to 100%. On the exact transcripts: exact search 84.2% (48 of 57), fuzzy match 100%, LLM 100%. On Whisper small’s transcripts: exact search 71.9% (41 of 57), fuzzy match 100%, LLM 100%. Keyword spotting in the audio, with Vosk and three key words in the first 20 seconds: 56.1% (32 of 57).
- The exact search misses the 9 other wordings on the exact transcripts (84.2%). On Whisper’s transcripts it also misses 7 French notices, all with “ce tapel” or a similar spelling (71.9%).
- The fuzzy match finds all 57, on both transcripts. Recognition errors do not move it.
- The LLM also finds all 57. It costs a model call, but the call record already has the field, so it is free here.
- No check made a false alarm on the 3 test calls without the notice.
Three calls are not enough for false alarms
Section titled “Three calls are not enough for false alarms”“No false alarm” on 3 calls says almost nothing. One false alarm in three would already be 33%. To measure false alarms, you need many calls without the notice.
The train calls have no audio, but they have transcripts and labels. Priya runs the two searches on all 260 public calls: 221 with the notice and 39 without.
| Check (exact transcripts) | Notices found | False alarms |
|---|---|---|
| Exact search | 183 of 221 (82.8%) | 0 of 39 |
| Fuzzy match | 221 of 221 (100%) | 0 of 39 |
Still no false alarm, now on 39 calls. But be careful again. In our scripts, an agent who forgets the notice says nothing like it. A real agent may say “I’m not recording this” or talk about “training”. The two sentences that we made up show that the fuzzy match passes such near misses. Before you trust it on real calls, collect real near misses, and let a person listen to a sample of the calls that pass.
Keyword spotting: listen for the words
Section titled “Keyword spotting: listen for the words”The third method skips the transcript. A keyword spotter listens to the audio for a short list of words and ignores everything else. The course used Vosk, the older recogniser of Module 4, with a phrase list: Vosk may only choose between “recorded”, “quality”, “training” and “unknown word”. The spotter listens to the first 20 seconds of each call, where the notice should be. It marks the notice as said when it hears at least two of the three words.
Why a phrase list? A general recogniser writes the words that it knows. In Module 4, Vosk never once wrote “Larkfield” in 120 tries: it wrote “luck field” 16 times and “lark feel” 9 times. With a phrase list, the recogniser can only answer with your words, so it cannot drift to similar common words.
On the 60 test calls (measured):
| Keyword spotting (Vosk, phrase list) | |
|---|---|
| Notices found | 32 of 57 (56.1%) |
| False alarms | 0 of 3 |
| Compute | real-time factor 0.009: about 34 seconds for an hour of audio |
It is very fast and needs no full transcript. But it found only about half the notices. The small Vosk model on 8 kHz phone audio often hears one key word but not two. Lower the rule to one word, and recall would rise, and false alarms too: “training” alone could be about something else.
Guided practice: run the compliance check
Section titled “Guided practice: run the compliance check”You already copied understand.py in Why did they call?. The command speech compliance runs the exact search and the fuzzy match on the agent’s turns of a split.
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Check the notice on the reference transcripts of the test calls:
Terminal window speech complianceThe output on the course’s computer:
Recording notice on 60 test calls (57 said it, 3 did not), transcripts: referenceexact phrase found 48 of 57 notices (recall 84.2%); false alarms 0 (a call without the notice marked as compliant)fuzzy match found 57 of 57 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant) -
Check it on Whisper small’s recorded transcripts:
Terminal window speech compliance --transcripts whisper-smallRecording notice on 60 test calls (57 said it, 3 did not), transcripts: whisper-smallexact phrase found 41 of 57 notices (recall 71.9%); false alarms 0 (a call without the notice marked as compliant)fuzzy match found 57 of 57 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant)Whisper’s transcripts have both speakers and no labels, so the search reads the whole call.
Your turn: more calls without the notice
Section titled “Your turn: more calls without the notice”The test split has only 3 calls without the notice. Which split has the most? Run the check on it, and say what the false-alarm numbers now tell you.
Show the answer
The train split: 30 of its 150 calls have no notice. Its calls have no audio, but the reference transcripts are enough for this check.
speech compliance --split trainRecording notice on 150 train calls (120 said it, 30 did not), transcripts: reference exact phrase found 100 of 120 notices (recall 83.3%); false alarms 0 (a call without the notice marked as compliant) fuzzy match found 120 of 120 notices (recall 100.0%); false alarms 0 (a call without the notice marked as compliant)No false alarm in 30 calls is better evidence than none in 3. It is still evidence about our scripts only: none of them has a near miss such as “this call is not recorded”.
Common mistakes
Section titled “Common mistakes”Common mistake
The exact search reports many missing notices after a script change
Symptom: one Monday, the number of calls “without the notice” jumps. The agents say they did not change anything.
Cause: the team changed one word of the notice, or a new recogniser spells a word differently (“Lockfield”, “ce tapel”). The exact search fails on every call.
Fix: use the fuzzy match, with the key words of the notice. Keep the exact search as a second number: when the two numbers move apart, something changed.
Common mistake
Searching the whole call instead of the agent's turns
Symptom: a call is marked compliant, but the agent never said the notice.
Cause: the caller said the words: “Is this call recorded? I hope it is, for quality.” The search read both speakers. On Whisper’s mono transcripts, there are no speaker labels.
Fix: search the agent’s turns only, from a stereo recording or from diarization. Search the first 30 seconds, where the notice belongs. Our scripts have no such caller sentence, so the course did not measure this case.
When to use each method
Section titled “When to use each method”- Exact search: when the wording is fixed and the text is exact, such as a chat or an email. As a second number beside the fuzzy match.
- Fuzzy match on the transcript: when you already transcribe every call, as Larkfield does. It costs nothing more, and it accepts other wordings and small recognition errors. Test it on near misses.
- Keyword spotting in the audio: when you do not want a full transcript. Examples are a live check during the call, a small device, or a rule that forbids storing transcripts. It is fast, but here it found only 56% of the notices. The case study Spot commands on a small device measures a spotter on real recordings.
- The LLM’s field: when you make call records anyway. Check it like any other field.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'The fuzzy match found 57 of 57 notices with no false alarm. It is perfect.' What do you tell him?
Show the answer
The recall is measured well: 57 of 57 on the test calls, and 221 of 221 on all public calls. The false alarms are not. Only 3 test calls miss the notice. The 39 public calls without it all come from our scripts, where a forgetful agent says nothing similar. Take a made-up near miss: “This call is not recorded for quality and training.” It has a similarity of 0.958 and all three key words, so the fuzzy match passes it. Test on real near misses, and listen to a sample of the calls that pass.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you switch from the exact transcripts to Whisper small's transcripts. What happens to the recall of the exact search, and of the fuzzy match? Why?
Show the answer
The exact search falls, from 84.2% to 71.9%. Whisper wrote seven French notices with a small error, such as “ce tapel” for “cet appel”. One changed letter or space breaks an exact match. The fuzzy match stays at 100%. The key words are still there. The similarity of “ce tapel est enregistré…” to the notice is 0.973, far above the threshold of 0.6.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run speech compliance --split validation. How many calls miss the notice, what does each search find, and why does Grace need both numbers?
Show the answer
The output is: 50 validation calls, 44 said the notice and 6 did not. The exact phrase finds 35 of 44 (recall 79.5%), the fuzzy match 44 of 44 (100.0%), with no false alarm for either. Grace uses the fuzzy match to decide, because it accepts fair other wordings. She keeps the exact number too. The gap between the two shows how many agents use another wording. A sudden change in the gap shows that a script or the recogniser changed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Score a notice check with recall (notices found) and false alarms (calls without the notice marked as compliant).
- Exact search found 84.2% of the notices on Larkfield’s 60 synthetic test calls, and 71.9% on Whisper small’s transcripts. The fuzzy match (key words in any order, or a similar window) found 100% on both.
- Similarity = 2 × letters in common ÷ the two lengths: “ce tapel est enregistré…” is 0.973 similar to the notice.
- Only 3 test calls miss the notice, so false alarms need the train and validation labels too: 0 of 39 on all public calls. Near misses that we made up still fool the fuzzy match.
- Keyword spotting with Vosk and a phrase list is very fast (real-time factor 0.009), but found only 56% of the notices.
You can now turn a call into decisions: its reason, a checked record and a compliance check. All of that happens after the call. Next, Larkfield wants a phone line that answers by itself, during the call. We start with the oldest and safest kind: a phone menu.