Skip to content
Français

Personal data in voice

  • Lesson 34
  • 40 min
  • Module 9: Run it in production, safely

Where we are

Our pipeline now runs every day on every recorded call. Calls hold names, voices and sometimes card numbers. In this lesson, we redact personal data in the transcript and in the audio, and we write rules for consent, retention and deletion.

What you will be able to do

Keep personal data out of what Larkfield stores. Never ask for a card number by speech. When a caller reads one aloud anyway, find it in the transcript and replace it with a label. Replace the same seconds of audio with a beep. Then run a check that proves no card number is left. Write down how long Larkfield keeps each kind of data.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook finds the card number in call C-0128 with the word times, beeps it in the audio, and redacts the transcript. It shows how one misheard digit hides a card number from the strict rule. Then you write the privacy check yourself. It needs no model download and no account. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the guided practice; the notebook needs none of it)

  • Your folder larkspeech with the five files of Batch and streaming. pytest -q prints 41 passed.

Tested on macOS with Python 3.12.13. The commands are the same on Windows; they were not run there.

Think of a call-centre agent who takes notes on paper. A caller suddenly reads out a card number, and the agent writes it down without thinking. At the end of the day, the supervisor reads the notes. She takes a black marker and covers the card number, so nobody can read it again. If the call was also recorded on tape, she must find the same seconds on the tape and cover them too.

Larkfield does the same thing, with code. But a computer only covers what it finds. A card number that the recogniser wrote in a strange way stays in the notes.

The problem: a card number in the recording

Section titled “The problem: a card number in the recording”

Listen to call C-0245. The agent asked for the order number. The caller offers more than that.

Call C-0245, from 29.6 to 36.2 seconds, phone audio over a landline (8 kHz), with background talk. The caller is a synthetic voice: Piper, trained on the VCTK corpus (speaker p264). The card number is 4111 1111 1111 1111, a well-known test number that no bank issues.

Transcript. Caller: “I can give you my card number, it’s 4111 1111 1111 1111.” Agent: “Please do not read your card number on the phone…”

The agent did the right thing. But the card number is now in the recording. Whisper base wrote it like this:

It’s for 111 -111111111111111111.

The first “4” became the word “for”. The rest became 21 ones: a group of three, then a run of 18. That is not a valid card number any more. A program that looks only for valid card numbers finds nothing, and the card number stays in the stored text, almost complete.

Card numbers have their own rules. The card industry’s security standard, PCI DSS (PCI Security Standards Council), says how card data must be protected wherever it is stored. One card number in a recording can bring the whole recording system under those rules. Names, voices and order numbers are personal data too: they can point to a person. Laws about personal data differ from country to country. In the European Union, for example, the GDPR defines personal data in Article 4(1) as “any information relating to an identified or identifiable natural person”. This lesson shows the technical side only. It is not legal advice: Larkfield decides its rules with a lawyer.

The idea: prevent, redact, prove, keep less

Section titled “The idea: prevent, redact, prove, keep less”

There are four parts. Each one catches what the one before it misses.

  1. Prevent. The IVR never asks for a card number by speech. If a card number is ever needed, the caller types it on the keypad, and the key presses are never written to logs.
  2. Redact. “Redact” means remove sensitive parts before a document is kept or shared. When a caller reads a card number aloud anyway, replace it in the transcript with [CARD NUMBER REMOVED]. Use the word times to find the same seconds in the audio, and replace them with a beep: a 1,000 Hz tone.
  3. Prove. Run a privacy check on every text file before it is stored. It fails if any card number is left.
  4. Keep less. Store only the redacted copies, and delete them after a fixed time.

The diagram shows where each step sits in Larkfield’s pipeline.

During the call: 1, the phone line: the IVR asks for card numbers by keypad only. 2, the recorder: keypad tones are not kept in the recording. After the call, in the project: 3, the recogniser writes a transcript with word times. 4, redact: card digits out of the text, a beep in the audio. 5, privacy check: no run of 12 digits left in any text. What is kept: 6, storage: redacted copies only, deleted on time. 7, logs and records: call ID, no raw text, no key presses.
The production pipeline with its redaction points. Steps 3 to 5 are in the course project; steps 1, 2, 6 and 7 are rules for the phone system and the storage.
Text description of the diagram

Seven steps in three groups. During the call: step 1, the phone line. The IVR asks for card numbers by keypad only. Step 2, the recorder. Keypad tones are not kept in the recording. After the call, in the course project: step 3, the recogniser writes a transcript with word times. Step 4, redact: the card digits go out of the text, and a beep replaces them in the audio. Step 5, the privacy check: no run of 12 digits may be left in any text. What is kept: step 6, storage. Only redacted copies are stored, and they are deleted on time. Step 7, logs and records. They hold the call ID, but no raw text and no key presses.

How does a program know that 16 digits are a card number? Card numbers end with a check digit, made with the same Luhn rule as Larkfield’s order numbers. You used it in Help the model with your words. Double every second digit from the right, and subtract 9 from any result above 9. Add everything: a valid number gives a total that ends in 0.

For the test number 4111 1111 1111 1111 (16 digits):

Digits (from the right) Doubled? Value
eight 1s in positions 1, 3, 5 … 15 no 8 × 1 = 8
seven 1s in positions 2, 4 … 14 yes 7 × 2 = 14
the 4 in position 16 yes 4 × 2 = 8
Total 8 + 14 + 8 = 30: ends in 0, valid

Now Whisper’s version: 21 ones. That is more than 19 digits, so it is too long for a card number. The check digit fails too. Ten ones are doubled (10 × 2 = 20) and eleven are not (11 × 1 = 11). The total is 31. It does not end in 0.

The project has two rules, in redact.py:

  • The strict rule accepts 13 to 19 digits that pass the Luhn check. It almost never marks an order number or a phone number by mistake. But one wrong digit hides the card.
  • The broad rule accepts any run of 12 or more digits, valid or not. Larkfield’s order numbers have 6 digits, so they stay. Whisper’s 21 ones are caught.

Both rules first apply the normalisation guide, so “four one one one” and “4111 1111” both become digits.

Larkfield: 15 calls with a spoken card number

Section titled “Larkfield: 15 calls with a spoken card number”

The call set has 15 public calls where a caller reads a card number aloud: a practice problem we planted on purpose. The course team ran both rules on Whisper base’s transcripts, with word times, and beeped the audio. Then it transcribed the beeped audio again, to check that no card number could still be heard.

  • Strict rule (13–19 digits, Luhn)
  • Broad rule (12 or more digits)
The two rules on the 15 calls with a spoken card number in Larkfield's synthetic call set (measured, Whisper base). Pairs, left to right: card numbers found; calls with a card-like number still in the text; calls with one still in the re-transcribed beeped audio.
Text description of the chart

Three pairs of columns. Card numbers found: strict 2, broad 13. Calls with a card-like number still in the text: strict 11, broad 0. Calls with a card-like number in the re-transcribed audio after the beep: strict 10, broad 0.

  • The strict rule found only 2 of 15. Whisper wrote “for 1111…”, “full 111…”, “4, 1, 1, 1…”, and each version broke the check digit. In 11 calls, a card-like number stayed in the text.
  • The broad rule found 13 of 15. No card-like number was left in the text, or in the beeped audio, of any call.
  • The 2 calls that no rule found are French. Whisper wrote “quatre nannas, un nannan” (C-0008) and « qu’être en en en, non, non… » (C-0075). There are no digits in that text, so no digit rule can find them. The text is clean. The audio is not: nothing was beeped, and the card number can still be heard.

For privacy, a missed card number is worse than a false alarm. So redact.py uses the broad rule by default.

The picture shows the same 6.6 seconds of C-0245 before and after redaction, with the words of the script under them.

Two waveforms of call C-0245 from 29.6 to 36.2 seconds. Before: speech all the way. After: from 31.2 to 35.1 seconds the waveform is a flat, even block, the 1,000 Hz tone; the words I can give you my card number it's stay before it. Under the waveforms, the script's words with their times; the digits four one one one and the rest fall inside the beeped block. Whisper base wrote: It's for 111 -111111111111111111.
C-0245 before and after the broad rule. The tone covers the digits plus 0.15 s on each side; the words before them stay.
Text description of the diagram

Two waveforms of the same 6.6 seconds of call C-0245, from 29.6 to 36.2 seconds. The first one is the original: speech all the way. In the second one, the part from 31.2 to 35.1 seconds is a flat, even block: the 1,000 Hz tone. The words “I can give you my card number, it’s” stay before it. Under the waveforms are the words of the script with their times. All the digits fall inside the beeped block. Whisper base wrote: “It’s for 111 -111111111111111111.”

The same seconds of C-0245 after redaction (labs file data/redacted/C-0245.wav). The caller is a synthetic voice: Piper, trained on the VCTK corpus.

Transcript after redaction (from speech redact C-0245, Whisper base): “I can give you my card number. It’s for [CARD NUMBER REMOVED] Please do not read your card number on the phone.” Whisper base gives no speaker labels on this mono call.

Look closely: the word “for” is still there. It is the first digit, “4”, written as a word. One digit is not a card number, but it shows that redaction is never perfect.

The beep uses the word times from the recogniser. The digits were spoken from 31.4 to 34.9 seconds. The beep adds 0.15 seconds on each side. So a word time that is a little early or late still falls inside it. In Word times and alignment, Whisper’s word starts were off by 0.055 seconds at the median. For 9 words in 10, they were off by 0.137 seconds or less.

The IVR’s call records keep a timeline of the call. Every key press is in it, with its time and its digit: call I-017 has [50.64, "key", "5"], [50.86, "key", "8"], …, an order number typed on the keypad. For order numbers, that helps when Amira debugs a call. For a card number, it would be a leak. Rule: in any state that collects payment data, the IVR writes “a key was pressed”, never which key.

Guided practice: redact a call and check the outputs

Section titled “Guided practice: redact a call and check the outputs”

You copied redact.py and commands/privacy.py in the last lesson. Two commands use them: speech redact (one call, text and audio) and speech privacy-check (every text file in a folder).

  1. Open a terminal in your project folder larkspeech, and activate .venv.

  2. Redact call C-0245:

    Terminal window
    speech redact C-0245

    The output on the course’s computer:

    C-0245: 1 card number(s) heard, 1 beeped 31.4-34.9 s
    Wrote outputs/redacted/C-0245.txt and outputs/redacted/C-0245.wav

    The command transcribes the call with Whisper base, so it takes a few seconds.

  3. Listen to outputs/redacted/C-0245.wav at 30 seconds, and read outputs/redacted/C-0245.txt. Find the label [CARD NUMBER REMOVED] in the text.

  4. Check every text file in outputs:

    Terminal window
    speech privacy-check outputs
    Privacy check: 2 text files in outputs; no card numbers found. Passed.

    The two files are the batch results from the last lesson and the redacted transcript. Your count is different if you have other files there.

  5. Now check the recorded transcripts that came with the course data:

    Terminal window
    speech privacy-check data/recordings

    The first lines of the output:

    Privacy check: 20 text files in data/recordings; 48 card numbers found:
    data/recordings/asr/vosk-small.jsonl: 1111...1111
    data/recordings/asr/vosk-small.jsonl: 1111...1111
    data/recordings/asr/vosk-small.jsonl: 4111...1111

    The check fails, and the command returns an error code. These files are the raw transcripts of Modules 4 and 7. Nobody redacted them, so they still hold the card numbers.

Every failure is in a raw transcript, so these are true card numbers, not false alarms. Look at how the check reads the .jsonl files: field by field. It scans each piece of text on its own. It skips numbers with a decimal point, and the fields that hold times and IDs, such as start, end, call_id and order. Without that rule, the check would read a whole record as one text. Then a list of turn times, or two order numbers side by side, makes 12 digits in a row. In IVR call I-054, the caller said “Six zero three one three seven”, and then typed “6-0-3134”. Read as one text, that record looks like a card number. Such a false alarm costs a person a minute. A check that cries wolf on every file costs more: people stop reading it. So keep the broad rule for the words people said, and keep numbers that are not card numbers in their own fields.

Run speech redact C-0075. Before you run it, predict: how many card numbers does it find, and does the privacy check then pass?

Show the answer
C-0075: 0 card number(s) heard, 0 beeped
Wrote outputs/redacted/C-0075.txt and outputs/redacted/C-0075.wav

It finds none. Then the check:

Terminal window
speech privacy-check outputs
Privacy check: 3 text files in outputs; no card numbers found. Passed.

But the caller did read a card number. Open outputs/redacted/C-0075.txt: Whisper wrote « Ma carte, c’est qu’être en en en, non, non, non… » instead of the digits. The text is clean, so the check passes. The audio file in outputs/redacted/ still holds the spoken card number. A passed check proves that the text has no card number. It proves nothing about the audio. So the IVR must not ask for card numbers by speech at all. And Larkfield keeps audio for a shorter time than text.

Redaction is one rule. Larkfield needs a short written list for every kind of data. The AI security course teaches how to build it in Data inventory and Minimize and retain deliberately. For a phone line, the list has these rows:

Data Why Larkfield keeps it Rule to decide
Consent: the recording notice proof that the caller was told Check it in every call, as in Compliance phrases and keyword spotting. No notice: do not keep the recording.
Raw recording training agents, checking quality The shortest time; delete it after the redacted copy exists
Redacted recording and transcript search, reports, the review sample A fixed number of days or months, then delete
Call record (reason, summary) reports, follow-ups As long as the customer record; no card digits, ever
Voice samples for a speaker check the next lesson Only with the caller’s explicit permission; many laws treat voiceprints as sensitive
Logs and key presses debugging Call ID and state only

Deletion must reach every copy: the batch inbox, the results file, the redacted folder, backups and any data sent to a hosted service. Keep a list of where each call’s data goes, so one request (“delete my calls”) can find them all.

Common mistake

The privacy check passes, but the audio still holds a card number

Symptom: the check says Passed. A person who listens to a stored recording still hears the digits.

Cause: the check reads text only. When the recogniser writes the digits as other words, as in the two French calls, nothing is found and nothing is beeped.

Fix: prevent first: keypad only for card numbers. Delete raw audio soon. Treat “no card found” as “nothing found in the text”, not “no card in the call”. Listen to a sample of calls where a caller talks about a card. The course did not build this sample check.

Common mistake

The strict rule passes every test, then misses most cards

Symptom: your unit tests pass with a clean “4111 1111 1111 1111”. On real transcripts, most spoken card numbers stay.

Cause: the tests use perfect text. A recogniser drops, adds or changes digits, and every change breaks the check digit. On our 15 calls, the strict rule found 2.

Fix: use the broad rule (12 or more digits), test it on recognised transcripts, and accept a few false alarms.

  • Keypad only: always, for card numbers and other payment data. It is the only step that keeps the data out of the audio from the start.
  • Redaction in text and audio: on every recorded call, before anything is stored or sent to a hosted service.
  • The broad rule: for privacy, where a miss is costly. Use the strict rule only where a false alarm is costly and a miss is not, for example to count card mentions in a report.
  • The privacy check: on every folder before storage, and on old data too. The course’s own data/recordings failed it.

Check your understanding

Whisper wrote a spoken card number as 'for 1111 1111 1111 1111'. Which rule finds it, the strict or the broad one? Explain with the digits.

Show the answer

After the normalisation guide, the text holds 16 ones: the 4 became the word “for”. For the strict rule, 16 digits is a valid length, but the Luhn check fails. Eight ones are doubled (8 × 2 = 16) and eight are not (8 × 1 = 8), so the total is 24, which does not end in 0. The broad rule finds it, because 16 digits is 12 or more. Larkfield’s 6-digit order numbers stay.

Check your understanding

Grace says: 'The privacy check passed on all our outputs, so our recordings are clean.' What do you tell her?

Show the answer

The check reads the text, not the audio. On Larkfield’s 15 synthetic calls with a spoken card number, the broad rule found 13. In the 2 French calls, Whisper wrote words instead of digits. No rule found them, and the code beeped nothing. The text passes, and the audio still holds the card number. The real protection has three parts. Never ask for card numbers by speech. Delete raw audio soon. Listen to a sample of calls about cards.

Check your understanding

Practical check: run speech redact C-0245. Between which times is the card number beeped, and why does the beep in the audio file start a little before that time?

Show the answer

The command prints 1 card number(s) heard, 1 beeped 31.4-34.9 s. These are the times of the first and last digit words from Whisper base. The beep adds 0.15 seconds on each side, so it covers about 31.25 to 35.05 seconds. Word times are not exact. Whisper’s word starts were off by 0.055 seconds at the median, and by up to 0.137 seconds for 9 words in 10. The extra time keeps an early or late digit inside the beep.

Recap

  • Prevent, redact, prove, keep less. Card numbers by keypad only. A label in the text and a beep in the audio, from the word times. A privacy check before storage. Short retention.
  • The Luhn check finds valid card numbers, but one misheard digit breaks it. On Larkfield’s 15 synthetic calls, the strict rule found 2 card numbers. The broad rule (12 or more digits) found 13, with none left in the text or the beeped audio.
  • No digit rule finds a card number that the recogniser wrote as other words. A passed privacy check is about the text only.
  • Key presses in logs and old raw transcripts are leaks too: the course’s own recorded transcripts failed the check.

We now protect what callers say. Next, we look at the voice itself: can a voice prove who is calling?

Next: Voice cloning, speaker checks and misuse