Skip to content
Français

Voice cloning, speaker checks and misuse

  • Lesson 35
  • 30 min
  • Module 9: Run it in production, safely

Where we are

We protect what callers say. We must also think about the voice itself. In this lesson, we see why a voice is not a password, how a similar voice can pass a weak speaker check, and why callers must know they talk to a machine.

Lesson 35 of 36 · Speech AI: Transcribe, Understand and SpeakVoice cloning, speaker checks and misuse

What you will be able to do

Decide what a voice may prove on Larkfield’s phone line. Enrol a voice, score new speech against it, and see how many other voices pass at each threshold. Learn why a speaker check is one signal among several, and never the only key to an account. Tell every caller that they are talking to a machine.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook enrols the agent Chloe from three of her turns in one call. Then it scores her other turns, the similar voice and 28 other callers against that enrolment. You choose a threshold and count who gets in. It downloads the 27 MB speaker model once and needs no account. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

This lesson has no new project command. The practice is in the notebook.

Think of a bank clerk who takes phone calls. A regular customer calls every week, and the clerk knows her voice well. “Hello, it’s Maria.” The clerk relaxes. But before she moves any money, she still asks for the account number and sends a code to Maria’s phone.

Why? Because voices can be copied, and people sound alike. A familiar voice is a good hint. It is not proof.

Larkfield’s product team asks: “Could the phone line recognise returning callers by their voice? Then they skip the questions.” Priya tests the idea on the course’s synthetic voices. She uses the voice of the agent Chloe as the “customer”, because Chloe speaks in many calls.

First, listen to two voices.

Chloe, the agent in call C-0225, from 0.45 to 5.9 seconds, agent channel, phone audio over a landline (8 kHz). A synthetic voice: Piper, trained on the LibriTTS-R corpus (speaker 5401).

Transcript. Chloe: “Thank you for calling Larkfield, this is Chloe. This call is recorded for quality and training. How can I help you today?”

The caller in call C-0144, from 6.0 to 11.9 seconds, phone audio over a landline with street noise. A synthetic voice: Piper, trained on the LibriTTS-R corpus (speaker 1914), chosen because it sounds close to Chloe.

Transcript. Caller: “Yes, hi. I’m a bit lost here; I need to change the email on my account, and I think it may be connected to my Grow Hub.”

The second voice is a different speaker of the same voice model. Of all the synthetic voices in the call set, it is the one closest to Chloe. Their embeddings have a cosine similarity of 0.952. The median of the other voices is 0.717.

Voice cloning means making a text-to-speech voice that sounds like one particular person, often from a short recording of them. Modern tools need only seconds of speech. That is the threat here: an attacker records a customer, clones the voice, and calls the bank.

The course clones no voice, and records no real person. The “similar voice” above is not a clone. It is the closest voice that we found among open synthetic voices. A real clone is built on purpose to sound like its target. We did not measure one, so we cannot say how it would score. It is not safe to assume that it would do worse than a voice that is close by chance.

Speaker verification answers one question: “Is this the person they say they are?” It has three steps:

  1. Enrol. Record the person once, here three sentences. Turn each sentence into a speaker embedding, and keep their average: the voiceprint.
  2. Score. When someone calls and says “I am Chloe”, embed their speech and compute the cosine similarity to Chloe’s voiceprint.
  3. Decide. Accept if the score is at or above a threshold.

Two mistakes are possible. A false reject turns away the real person. A false accept lets in someone else. Raising the threshold makes fewer false accepts and more false rejects. There is no setting that removes both.

A tiny worked example: three test sentences

Section titled “A tiny worked example: three test sentences”

The course enrolled Chloe from three sentences (“Thank you for calling Larkfield.”, “I can help with that.”, “Is there anything else I can help with?”). The voices themselves read every sentence, through the phone line. The project’s speaker model (WeSpeaker ResNet-34) embedded each one. Then the course scored three new sentences of Chloe, three of the similar voice, and one sentence each of 30 other voices.

Sentence Chloe Similar voice Accepted at 0.7? (Chloe / similar) At 0.8?
“Please hold while I check your order.” 0.855 0.854 yes / yes yes / yes
“Your order shipped on Monday.” 0.794 0.769 yes / yes no / no
“Have a good day.” 0.734 0.667 yes / no no / no
30 other voices median 0.743, highest 0.902 19 of 30 accepted 7 of 30 accepted

Read the table row by row. On the first sentence, the similar voice and Chloe score almost the same (0.854 and 0.855). Short sentences score lower for everyone. The short sentence “Have a good day” gives Chloe only 0.734, her lowest score. And one of the 30 other voices scores 0.902, higher than any sentence of Chloe herself.

The chart shows the share of speech that the check accepts at each threshold, for the three groups.

  • Chloe herself (should pass)
  • The similar voice (should fail)
  • 30 other voices (should fail)
A speaker check enrolled on Chloe, on Larkfield's synthetic voices through the phone line (measured; WeSpeaker embeddings). No threshold accepts Chloe and refuses the others. Each Chloe and similar-voice point is only 3 sentences.
Text description of the chart

A line chart with the threshold from 0.5 to 0.9 on the horizontal axis and the share accepted on the vertical axis. Chloe: 100% at 0.5, 0.6 and 0.7, then 33% at 0.8 and 0% at 0.9. The similar voice: 100% at 0.5 and 0.6, 67% at 0.7, 33% at 0.8, 0% at 0.9. The 30 other voices: 97% at 0.5, 90% at 0.6, 63% at 0.7, 23% at 0.8, 3% at 0.9. The lines for Chloe and for the similar voice lie almost on top of each other.

  • At 0.7, Chloe passes 3 of 3. So do 2 of 3 sentences of the similar voice and 19 of 30 other voices.
  • At 0.8, Chloe passes only 1 of 3, and 7 of 30 other voices still pass.
  • At 0.9, Chloe never passes, and 1 other voice still does.

On these synthetic voices, through a phone line, the speaker check is weak at every threshold. The phone line removes much of what makes a voice unique: only 300 to 3,400 Hz are left. Short sentences also carry little of the voice. Real voices may separate better or worse. The course did not test real people, on purpose. Numbers this small (3 sentences) are a warning, not a measurement of a product.

The notebook repeats the check on the call audio itself, with Chloe’s turns from several calls. It shows the same picture: other callers pass at thresholds where Chloe passes.

A speaker check can still be useful: as one signal among several. Think of the bank clerk again. The familiar voice made her relax a little, but the account number and the code decided.

A caller asks for something risky, such as changing the email or sending a refund. Three signals go to a decision: the voice score, the cosine to the enrolled voice; something they know, the order number and postcode match; something they have, a one-time code sent to the email on file. Enough signals for this action? Yes: do it, and log which signals passed. No or unsure: hand over to a person, never refuse in silence.
The voice score is one input to the decision, never the key by itself. Risky actions need something the caller knows and something they have.
Text description of the diagram

A caller asks for something risky, such as changing the email on the account or sending a refund. Three signals go to one decision. Signal 1, the voice score: the cosine similarity to the enrolled voice. Signal 2, something they know: the order number and the postcode match. Signal 3, something they have: a one-time code sent to the email that Larkfield already has. The decision asks: are there enough signals for this action? If yes, do it, and log which signals passed. If no or unsure, hand over to a person. Never refuse in silence.

For Larkfield, the rules are simple:

  • Order status and opening hours need no identity check. The IVR gives an order’s status only to someone who has its 6-digit order number, which has a check digit.
  • A change to the account or a refund needs something the caller knows and something they have, such as a one-time code. The voice score may add a hint. A low score can send the call to a person. A high score never skips a step.
  • Voiceprints are sensitive personal data. Many laws treat them as biometric data, with strict rules. For example, the EU’s GDPR lists “biometric data for the purpose of uniquely identifying a natural person” among the special categories of Article 9(1). Such data may be processed only under the conditions of Article 9(2). Enrol a caller only with their explicit permission, store the voiceprint like a password, and delete it when they ask. The last lesson’s retention table has a row for it.

The AI security course teaches a method for this: list what you protect and who might attack it, in Assets and actors, then Prioritize threats. For a voice line, the usual attempts are:

Attempt Example What stops it
Replay play a recording of the real customer a one-time code; questions that change every call
Cloned voice a text-to-speech voice made from a customer’s video never let a voice score unlock an action alone
Social engineering “I’m in a hurry, please just change it” fixed rules: the IVR and the agent cannot skip steps
Fishing for data “Read me the address on order 742155” the IVR never reads addresses or other personal data back

Tell callers they are talking to a machine

Section titled “Tell callers they are talking to a machine”

The misuse can also go the other way: a company’s own machine pretends to be a person. Callers have a right to know who, or what, they talk to. Some laws require it. For example, Article 50(1) of the EU’s AI Act says that people must be “informed that they are interacting with an AI system”, unless this is obvious. Check the rules where you operate.

Larkfield’s IVR says it in its first sentence. From the recorded replay of the IVR test calls:

“Welcome to Larkfield. You are talking to an automated assistant. For English, say English or press 1. Pour le français, dites français ou appuyez sur 2.”

Three more rules follow from it:

  • The voice agent must never claim to be a person, even when a caller asks “Are you a real person?”. It says what it is and offers a person.
  • The prompts use open synthetic voices with licences that allow this use (Module 7). Never use a real person’s voice, such as a staff member or an actor, without their written permission for exactly this use.
  • Pressing 0 or saying “person” always reaches a person, as in Test the IVR like a product.

Omar wants refunds by phone, out of hours. Use the numbers above. Which of these rules would you accept, and why?

  1. Accept if the voice score is 0.8 or more.
  2. Accept if the voice score is 0.7 or more and the order number is right.
  3. Accept if the order number is right and the caller reads back a one-time code sent to the email on the account. Hand over to a person if the voice score is under 0.5.
Show the answer

Rule 3. Rule 1 lets in 7 of 30 other voices on our synthetic voices and turns away the real Chloe 2 times in 3. Rule 2 still lets in anyone who has an order number and a voice that passes at 0.7. 19 of 30 other voices did. Order numbers also appear on delivery notes and emails. Rule 3 uses something the caller knows and something they have. The voice only sends doubtful calls to a person. At 0.5, Chloe never failed, so she is not bothered.

Common mistake

Raising the threshold to stop impostors

Symptom: the check accepted other voices, so the team raised the threshold to 0.8. Now customers complain that the line refuses them.

Cause: on these voices, the scores of the real speaker and of other speakers overlap. At 0.8, Chloe passed 1 of 3 sentences, and 7 of 30 other voices still passed. A higher threshold moves the errors; it does not remove them.

Fix: keep the threshold where real customers pass. Do not let the voice score decide anything important. Add a second factor for risky actions.

Common mistake

Testing on the enrolment sentences

Symptom: the speaker check looks excellent in the test: the real speaker scores above 0.95 every time.

Cause: the test used the same sentences as the enrolment, or the same words (“Thank you for calling Larkfield”). Embeddings of the same words are closer, whoever speaks. An attacker does not say the enrolment sentence for you.

Fix: test with other sentences, other days, other lines and short answers. Report false accepts and false rejects apart, with the number of tests.

Check your understanding

Tomás says: 'We can enrol every caller's voice and skip the order number. Voices are unique.' Use the course's numbers to answer him.

Show the answer

On Larkfield’s synthetic voices through the phone line, a check enrolled on Chloe accepted 19 of 30 other voices. That was at a threshold of 0.7. At 0.8, it still accepted 7 of 30. It accepted Chloe herself only 1 time in 3. One other voice scored 0.902, higher than any sentence of Chloe. Voices may be unique, but a phone line and short sentences hide much of what makes them unique. And a clone is made to sound like its target. The voice can be one signal. The order number, or better a one-time code, must still decide.

Check your understanding

The course's 'cloned voice' scored 0.952 against Chloe. Why does the lesson refuse to call it a clone, and why does that matter for the conclusion?

Show the answer

No voice was cloned. It is the closest voice that the course found among open synthetic voices, close to Chloe by chance. The course clones no voice and records no real person, on purpose. It matters because a real clone is built to sound like its target. We did not measure one, so we must not claim that a clone would fail the check. The safe conclusion is only that even a voice that is close by chance passes a weak check.

Check your understanding

Which three things must Larkfield's IVR and voice agent do so that callers know they talk to a machine and can get a person?

Show the answer

First, say it at the start: “You are talking to an automated assistant.” Second, never claim to be a person, even when asked, and offer a person instead. Third, always let the caller reach a person by pressing 0 or saying “person”. Also, use only voices with permission and a licence for this use, never a real person’s voice without their written permission.

Recap

  • Speaker verification: enrol a voiceprint, score new speech with the cosine similarity, accept at a threshold. False accepts and false rejects move in opposite directions.
  • On Larkfield’s synthetic voices through the phone line, the check was weak at every threshold. At 0.7: Chloe 3 of 3, other voices 19 of 30. At 0.8: Chloe 1 of 3, others 7 of 30.
  • The “cloned voice” is the closest synthetic voice we found (cosine 0.952), not a clone. No voice was cloned.
  • One signal, never the only one: risky actions need something the caller knows and something they have. Voiceprints are sensitive data.
  • Tell callers they talk to a machine, never pretend to be a person, and always offer a person.

The system is now safe by design. But it will not stay good by itself: callers, products and phone lines change. Next, we decide what to watch every day, every week and every month.

Next: Watch the system