Personal data in voice
- Lesson 34
- 40 min
- Module 9: Run it in production, safely
Where we are
Our pipeline now runs every day on every recorded call. Calls hold names, voices and sometimes card numbers. In this lesson, we redact personal data in the transcript and in the audio, and we write rules for consent, retention and deletion.
Lesson 34 of 36 · Speech AI: Transcribe, Understand and SpeakPersonal data in voice
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Keep personal data out of what Larkfield stores. Never ask for a card number by speech. When a caller reads one aloud anyway, find it in the transcript and replace it with a label. Replace the same seconds of audio with a beep. Then run a check that proves no card number is left. Write down how long Larkfield keeps each kind of data.
NotebookDo this lesson in a notebook
The notebook finds the card number in call C-0128 with the word times, beeps it in the audio, and redacts the transcript. It shows how one misheard digit hides a card number from the strict rule. Then you write the privacy check yourself. It needs no model download and no account. About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Word times, from Word times and alignment.
- The check digit (the Luhn rule), from Help the model with your words.
- The normalisation guide, from Normalise before you compare: it turns “four one one one” into “4111”.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your folder
larkspeechwith the five files of Batch and streaming.pytest -qprints41 passed.
Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.
The black marker
Section titled “The black marker”Think of a call-centre agent who takes notes on paper. A caller suddenly reads out a card number, and the agent writes it down without thinking. At the end of the day, the supervisor reads the notes. She takes a black marker and covers the card number, so nobody can read it again. If the call was also recorded on tape, she must find the same seconds on the tape and cover them too.
Larkfield does the same thing, with code. But a computer only covers what it finds. A card number that the recogniser wrote in a strange way stays in the notes.
The problem: a card number in the recording
Section titled “The problem: a card number in the recording”Listen to call C-0245. The agent asked for the order number. The caller offers more than that.
Transcript. Caller: “I can give you my card number, it’s 4111 1111 1111 1111.” Agent: “Please do not read your card number on the phone…”
The agent did the right thing. But the card number is now in the recording. Whisper base wrote it like this:
It’s for 111 -111111111111111111.
The first “4” became the word “for”. The rest became 21 ones: a group of three, then a run of 18. That is not a valid card number any more. A program that looks only for valid card numbers finds nothing, and the card number stays in the stored text, almost complete.
Why one card number matters
Section titled “Why one card number matters”Card numbers have their own rules. The card industry’s security standard, PCI DSS (PCI Security Standards Council), says how card data must be protected wherever it is stored. One card number in a recording can bring the whole recording system under those rules. Names, voices and order numbers are personal data too: they can point to a person. Laws about personal data differ from country to country. In the European Union, for example, the GDPR defines personal data in Article 4(1) as “any information relating to an identified or identifiable natural person”. This lesson shows the technical side only. It is not legal advice: Larkfield decides its rules with a lawyer.
The idea: prevent, redact, prove, keep less
Section titled “The idea: prevent, redact, prove, keep less”There are four parts. Each one catches what the one before it misses.
- Prevent. The IVR never asks for a card number by speech. If a card number is ever needed, the caller types it on the keypad, and the key presses are never written to logs.
- Redact. “Redact” means remove sensitive parts before a document is kept or shared. When a caller reads a card number aloud anyway, replace it in the transcript with
[CARD NUMBER REMOVED]. Use the word times to find the same seconds in the audio, and replace them with a beep: a 1,000 Hz tone. - Prove. Run a privacy check on every text file before it is stored. It fails if any card number is left.
- Keep less. Store only the redacted copies, and delete them after a fixed time.
The diagram shows where each step sits in Larkfield’s pipeline.
Text description of the diagram
Seven steps in three groups. During the call: step 1, the phone line. The IVR asks for card numbers by keypad only. Step 2, the recorder. Keypad tones are not kept in the recording. After the call, in the course project: step 3, the recogniser writes a transcript with word times. Step 4, redact: the card digits go out of the text, and a beep replaces them in the audio. Step 5, the privacy check: no run of 12 digits may be left in any text. What is kept: step 6, storage. Only redacted copies are stored, and they are deleted on time. Step 7, logs and records. They hold the call ID, but no raw text and no key presses.
A tiny worked example: the check digit
Section titled “A tiny worked example: the check digit”How does a program know that 16 digits are a card number? Card numbers end with a check digit, made with the same Luhn rule as Larkfield’s order numbers. You used it in Help the model with your words. Double every second digit from the right, and subtract 9 from any result above 9. Add everything: a valid number gives a total that ends in 0.
For the test number 4111 1111 1111 1111 (16 digits):
| Digits (from the right) | Doubled? | Value |
|---|---|---|
| eight 1s in positions 1, 3, 5 … 15 | no | 8 × 1 = 8 |
| seven 1s in positions 2, 4 … 14 | yes | 7 × 2 = 14 |
| the 4 in position 16 | yes | 4 × 2 = 8 |
| Total | 8 + 14 + 8 = 30: ends in 0, valid |
Now Whisper’s version: 21 ones. That is more than 19 digits, so it is too long for a card number. The check digit fails too. Ten ones are doubled (10 × 2 = 20) and eleven are not (11 × 1 = 11). The total is 31. It does not end in 0.
The project has two rules, in redact.py:
- The strict rule accepts 13 to 19 digits that pass the Luhn check. It almost never marks an order number or a phone number by mistake. But one wrong digit hides the card.
- The broad rule accepts any run of 12 or more digits, valid or not. Larkfield’s order numbers have 6 digits, so they stay. Whisper’s 21 ones are caught.
Both rules first apply the normalisation guide, so “four one one one” and “4111 1111” both become digits.
Larkfield: 15 calls with a spoken card number
Section titled “Larkfield: 15 calls with a spoken card number”The call set has 15 public calls where a caller reads a card number aloud: a practice problem we planted on purpose. The course team ran both rules on Whisper base’s transcripts, with word times, and beeped the audio. Then it transcribed the beeped audio again, to check that no card number could still be heard.
- Strict rule (13–19 digits, Luhn)
- Broad rule (12 or more digits)
Text description of the chart
Three pairs of columns. Card numbers found: strict 2, broad 13. Calls with a card-like number still in the text: strict 11, broad 0. Calls with a card-like number in the re-transcribed audio after the beep: strict 10, broad 0.
- The strict rule found only 2 of 15. Whisper wrote “for 1111…”, “full 111…”, “4, 1, 1, 1…”, and each version broke the check digit. In 11 calls, a card-like number stayed in the text.
- The broad rule found 13 of 15. No card-like number was left in the text, or in the beeped audio, of any call.
- The 2 calls that no rule found are French. Whisper wrote “quatre nannas, un nannan” (C-0008) and « qu’être en en en, non, non… » (C-0075). There are no digits in that text, so no digit rule can find them. The text is clean. The audio is not: nothing was beeped, and the card number can still be heard.
For privacy, a missed card number is worse than a false alarm. So redact.py uses the broad rule by default.
What the beep looks like
Section titled “What the beep looks like”The picture shows the same 6.6 seconds of C-0245 before and after redaction, with the words of the script under them.
Text description of the diagram
Two waveforms of the same 6.6 seconds of call C-0245, from 29.6 to 36.2 seconds. The first one is the original: speech all the way. In the second one, the part from 31.2 to 35.1 seconds is a flat, even block: the 1,000 Hz tone. The words “I can give you my card number, it’s” stay before it. Under the waveforms are the words of the script with their times. All the digits fall inside the beeped block. Whisper base wrote: “It’s for 111 -111111111111111111.”
Transcript after redaction (from speech redact C-0245, Whisper base): “I can give you my card number. It’s for [CARD NUMBER REMOVED] Please do not read your card number on the phone.” Whisper base gives no speaker labels on this mono call.
Look closely: the word “for” is still there. It is the first digit, “4”, written as a word. One digit is not a card number, but it shows that redaction is never perfect.
The beep uses the word times from the recogniser. The digits were spoken from 31.4 to 34.9 seconds. The beep adds 0.15 seconds on each side. So a word time that is a little early or late still falls inside it. In Word times and alignment, Whisper’s word starts were off by 0.055 seconds at the median. For 9 words in 10, they were off by 0.137 seconds or less.
Keypad digits are personal data too
Section titled “Keypad digits are personal data too”The IVR’s call records keep a timeline of the call. Every key press is in it, with its time and its digit: call I-017 has [50.64, "key", "5"], [50.86, "key", "8"], …, an order number typed on the keypad. For order numbers, that helps when Amira debugs a call. For a card number, it would be a leak. Rule: in any state that collects payment data, the IVR writes “a key was pressed”, never which key.
Guided practice: redact a call and check the outputs
Section titled “Guided practice: redact a call and check the outputs”You copied redact.py and commands/privacy.py in the last lesson. Two commands use them: speech redact (one call, text and audio) and speech privacy-check (every text file in a folder).
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Redact call C-0245:
Terminal window speech redact C-0245The output on the course’s computer:
C-0245: 1 card number(s) heard, 1 beeped 31.4-34.9 sWrote outputs/redacted/C-0245.txt and outputs/redacted/C-0245.wavThe command transcribes the call with Whisper base, so it takes a few seconds.
-
Listen to
outputs/redacted/C-0245.wavat 30 seconds, and readoutputs/redacted/C-0245.txt. Find the label[CARD NUMBER REMOVED]in the text. -
Check every text file in
outputs:Terminal window speech privacy-check outputsPrivacy check: 2 text files in outputs; no card numbers found. Passed.The two files are the batch results from the last lesson and the redacted transcript. Your count is different if you have other files there.
-
Now check the recorded transcripts that came with the course data:
Terminal window speech privacy-check data/recordingsThe first lines of the output:
Privacy check: 20 text files in data/recordings; 48 card numbers found:data/recordings/asr/vosk-small.jsonl: 1111...1111data/recordings/asr/vosk-small.jsonl: 1111...1111data/recordings/asr/vosk-small.jsonl: 4111...1111The check fails, and the command returns an error code. These files are the raw transcripts of Modules 4 and 7. Nobody redacted them, so they still hold the card numbers.
Every failure is in a raw transcript, so these are true card numbers, not false alarms. Look at how the check reads the .jsonl files: field by field. It scans each piece of text on its own. It skips numbers with a decimal point, and the fields that hold times and IDs, such as start, end, call_id and order. Without that rule, the check would read a whole record as one text. Then a list of turn times, or two order numbers side by side, makes 12 digits in a row. In IVR call I-054, the caller said “Six zero three one three seven”, and then typed “6-0-3134”. Read as one text, that record looks like a card number. Such a false alarm costs a person a minute. A check that cries wolf on every file costs more: people stop reading it. So keep the broad rule for the words people said, and keep numbers that are not card numbers in their own fields.
Your turn: the French call
Section titled “Your turn: the French call”Run speech redact C-0075. Before you run it, predict: how many card numbers does it find, and does the privacy check then pass?
Show the answer
C-0075: 0 card number(s) heard, 0 beepedWrote outputs/redacted/C-0075.txt and outputs/redacted/C-0075.wavIt finds none. Then the check:
speech privacy-check outputsPrivacy check: 3 text files in outputs; no card numbers found. Passed.But the caller did read a card number. Open outputs/redacted/C-0075.txt: Whisper wrote « Ma carte, c’est qu’être en en en, non, non, non… » instead of the digits. The text is clean, so the check passes. The audio file in outputs/redacted/ still holds the spoken card number. A passed check proves that the text has no card number. It proves nothing about the audio. So the IVR must not ask for card numbers by speech at all. And Larkfield keeps audio for a shorter time than text.
Consent, retention and deletion
Section titled “Consent, retention and deletion”Redaction is one rule. Larkfield needs a short written list for every kind of data. The AI security course teaches how to build it in Data inventory and Minimize and retain deliberately. For a phone line, the list has these rows:
| Data | Why Larkfield keeps it | Rule to decide |
|---|---|---|
| Consent: the recording notice | proof that the caller was told | Check it in every call, as in Compliance phrases and keyword spotting. No notice: do not keep the recording. |
| Raw recording | training agents, checking quality | The shortest time; delete it after the redacted copy exists |
| Redacted recording and transcript | search, reports, the review sample | A fixed number of days or months, then delete |
| Call record (reason, summary) | reports, follow-ups | As long as the customer record; no card digits, ever |
| Voice samples for a speaker check | the next lesson | Only with the caller’s explicit permission; many laws treat voiceprints as sensitive |
| Logs and key presses | debugging | Call ID and state only |
Deletion must reach every copy: the batch inbox, the results file, the redacted folder, backups and any data sent to a hosted service. Keep a list of where each call’s data goes, so one request (“delete my calls”) can find them all.
Common mistakes
Section titled “Common mistakes”Common mistake
The privacy check passes, but the audio still holds a card number
Symptom: the check says Passed. A person who listens to a stored recording still hears the digits.
Cause: the check reads text only. When the recogniser writes the digits as other words, as in the two French calls, nothing is found and nothing is beeped.
Fix: prevent first: keypad only for card numbers. Delete raw audio soon. Treat “no card found” as “nothing found in the text”, not “no card in the call”. Listen to a sample of calls where a caller talks about a card. The course did not build this sample check.
Common mistake
The strict rule passes every test, then misses most cards
Symptom: your unit tests pass with a clean “4111 1111 1111 1111”. On real transcripts, most spoken card numbers stay.
Cause: the tests use perfect text. A recogniser drops, adds or changes digits, and every change breaks the check digit. On our 15 calls, the strict rule found 2.
Fix: use the broad rule (12 or more digits), test it on recognised transcripts, and accept a few false alarms.
When to use each tool
Section titled “When to use each tool”- Keypad only: always, for card numbers and other payment data. It is the only step that keeps the data out of the audio from the start.
- Redaction in text and audio: on every recorded call, before anything is stored or sent to a hosted service.
- The broad rule: for privacy, where a miss is costly. Use the strict rule only where a false alarm is costly and a miss is not, for example to count card mentions in a report.
- The privacy check: on every folder before storage, and on old data too. The course’s own
data/recordingsfailed it.
Knowledge check
Section titled “Knowledge check”Check your understanding
Whisper wrote a spoken card number as 'for 1111 1111 1111 1111'. Which rule finds it, the strict or the broad one? Explain with the digits.
Show the answer
After the normalisation guide, the text holds 16 ones: the 4 became the word “for”. For the strict rule, 16 digits is a valid length, but the Luhn check fails. Eight ones are doubled (8 × 2 = 16) and eight are not (8 × 1 = 8), so the total is 24, which does not end in 0. The broad rule finds it, because 16 digits is 12 or more. Larkfield’s 6-digit order numbers stay.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Grace says: 'The privacy check passed on all our outputs, so our recordings are clean.' What do you tell her?
Show the answer
The check reads the text, not the audio. On Larkfield’s 15 synthetic calls with a spoken card number, the broad rule found 13. In the 2 French calls, Whisper wrote words instead of digits. No rule found them, and the code beeped nothing. The text passes, and the audio still holds the card number. The real protection has three parts. Never ask for card numbers by speech. Delete raw audio soon. Listen to a sample of calls about cards.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run speech redact C-0245. Between which times is the card number beeped, and why does the beep in the audio file start a little before that time?
Show the answer
The command prints 1 card number(s) heard, 1 beeped 31.4-34.9 s. These are the times of the first and last digit words from Whisper base. The beep adds 0.15 seconds on each side, so it covers about 31.25 to 35.05 seconds. Word times are not exact. Whisper’s word starts were off by 0.055 seconds at the median, and by up to 0.137 seconds for 9 words in 10. The extra time keeps an early or late digit inside the beep.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Prevent, redact, prove, keep less. Card numbers by keypad only. A label in the text and a beep in the audio, from the word times. A privacy check before storage. Short retention.
- The Luhn check finds valid card numbers, but one misheard digit breaks it. On Larkfield’s 15 synthetic calls, the strict rule found 2 card numbers. The broad rule (12 or more digits) found 13, with none left in the text or the beeped audio.
- No digit rule finds a card number that the recogniser wrote as other words. A passed privacy check is about the text only.
- Key presses in logs and old raw transcripts are leaks too: the course’s own recorded transcripts failed the check.
We now protect what callers say. Next, we look at the voice itself: can a voice prove who is calling?