Skip to content
Français

Final assignment

  • After the case studies
  • about 3 hr

Where we are

Each module added one piece and tried it alone: the test set, VAD, recognition, diarization, the call record, the IVR and the voice agent. In this assignment, you use them all at once, for Larkfield's new French-language line.

What you will be able to do

Use every module of the course on one new situation, and deliver a speech pack that Omar, Grace and Amira can act on. The calls look like the course’s calls, but a new language mix, a new phone line and new product names change them. So you must measure each choice again, not copy it.

Before you startWhat to know and to have readyAbout 180 minutes
  • You completed Modules 1 to 9. Each part below names the lessons that it uses.
  • You do not need the case studies of Module 10 for this assignment. (The certificate asks for at least one of them.) To see a recogniser chosen for two languages on real speech first, open Choose a recogniser for two languages.
  • Your folder larkspeech works at the end of Module 9: pytest -q prints 41 passed, as in Batch and streaming. The folder nextia-learning-labs is beside it, as in Set up the speech project.
  • About 3 hours of work, plus run time. In the author’s solution, the longest run (Vosk and five Whisper settings on all 40 calls) took about 17 minutes on a laptop CPU.
  • Tested on macOS with Python 3.12.13 and the project’s pinned packages. The Windows commands are written, but they were not run.

The situation: a French line in a new region

Section titled “The situation: a French line in a new region”

Larkfield is growing. Omar, the product manager, opens a new region where callers speak French. Three things change at once:

  • The callers. Every call on the new line is in French. Some callers switch to English for a moment, as in the course’s French calls.
  • The phone line. The region uses a new phone provider. Malik, the telephony engineer, knows its codec: G.726 at 16 kbit/s, the same codec as the course’s VoIP line. The provider also says that it loses up to about 3% of the 20 ms audio packets. A lost packet becomes 20 ms of silence.
  • The products. The region sells four products that the course’s calls never name: Boreal heat lamp, Lumen deck tiles, Saule rain barrel and Nordik snow shovel. Their order numbers start with 8 or 9.

Omar wants the out-of-hours line to answer French callers by itself, as the IVR does for the first region. Grace, the support lead, wants searchable transcripts and a call record for every call, with the recording-notice check. Amira owns the IVR. Tomás, the AI engineer, has an answer ready: “Put the voice agent on the new line. It speaks French, and it does not need a menu.”

Priya asks you to answer all of them with evidence. This is what the new line does to a caller’s voice:

A waveform of 1.2 seconds of a caller's speech on the new line. The speech is in bursts. Four narrow red bands mark 20 ms frames where the sound drops to near silence in the middle of a word.
1.2 seconds of the caller's channel of call N-005 (a synthetic voice: Kokoro, which learned French from the SIWIS corpus, with street noise made by our code). Red bands: 20 ms frames that are near silence inside speech, the packets that the line lost. Drawn by our code from the public file.
Text description of the diagram

A waveform over 1.2 seconds, from left to right. The caller’s speech comes in bursts, one or two syllables each. Four narrow red bands cross the waveform. In each band, the sound drops to almost nothing for 20 ms, in the middle of a burst, and then comes back. This is what a lost packet does: a short hole in a word.

A person hardly hears a 20 ms hole. A recogniser can hear a different word. Our code finds about 2 such holes in every 100 frames of speech in the 40 calls. That is a lower limit: it cannot see a hole at the edge of a word. Whether the holes matter for the transcripts is a question you answer with a measurement.

Every call is synthetic: open text-to-speech voices read invented scripts, and Larkfield is fictional. The French voices are few. Only three voices speak French, and two of them learned from the same speaker, as in Audio as data. Part of every French error comes from the voices themselves. So quote your numbers “on Larkfield’s synthetic calls”, and do not promise them for real callers.

The data is in the labs repository, in C41/final/. The course’s own data, C41/data/, stays as it was.

File or folder Rows What one row is
calls/calls.jsonl, calls/phone/N-*.wav 40 One conversation between a Larkfield agent and a French caller, 32 to 75 seconds, 8 kHz mu-law. Columns: call_id, split (always final), language, line (always new_provider), channels (27 stereo, 13 mono), duration. No transcript and no label.
ivr/calls.jsonl, ivr/NI-*/NN.wav 20 One test call to the IVR: the caller’s turns only, one file per turn (speech, keypad tones or silence), 76 files in all. No expected outcome.
orders.json 60 One order of the new region: the number (6 digits, starting with 8 or 9, with a check digit), the product, the status, the date and whether it can be returned.
products.json 4 The four new product names.

The folder also has costs.json (the course’s cost assumptions), README.md, dataset-card.md and SHA256SUMS, and is about 29 MB. The IDs are new: an ID says nothing about a call. The course team keeps the reference transcripts, the labels and the expected IVR outcomes. Nobody chooses anything with them.

Check the files, and give the folder to the project. Run this in your larkspeech folder, with .venv active:

Terminal window
git -C ../nextia-learning-labs pull
F=../nextia-learning-labs/C41/final
(cd "$F" && shasum -a 256 -c SHA256SUMS | grep -vc ': OK$')
speech --data "$F" info

The number that shasum prints is the number of wrong files. It must be 0. Then speech info prints:

Calls: 0 train (transcripts only), 0 validation, 0 test; audio for 40 calls (0.58 hours, phone and wideband)
Languages: 0 English, 40 French; lines: 40 new_provider
Stereo (agent and caller apart): 27; mono: 13
IVR test calls: 20; orders in the fake order system: 60
Every call is synthetic: open text-to-speech voices read invented scripts. The hidden test calls are not here.

The global option --data gives the folder to any command, for example speech --data "$F" redact N-001 or speech --data "$F" ivr serve. (LARKSPEECH_DATA="$F" does the same.) Commands that score, such as speech transcribe, also need your labels: Part 1 shows how. The first line says “phone and wideband” for any data folder. This one has no wideband copy: the phone audio is all that the new provider gives Larkfield.

Make a folder speech-pack/ in your larkspeech folder. Put every file of the assignment in it. What to submit lists the files.

The pack follows the course, one module after another. Each part is one required item of the course’s completion rules.

A grey box on top: the public data, 40 French calls and 20 IVR calls on the new line, 60 new orders, 4 new products, no labels. Below, six blue boxes: 1 Test set, 2 Pipeline, 3 Call record, 4 Privacy, 5 IVR, 6 Agent or menu, each with what it holds and its file. An orange box at the bottom: the course team checks your labels and scores against the answers it keeps, once; nothing is chosen with that score.
The six parts of the speech pack and the files that they hand in. You make the labels that you need; the course team scores your pack once against the answers that it keeps.
Text description of the diagram

At the top, a grey box: the public data, 40 French calls and 20 IVR calls on the new line, 60 new orders and 4 new products, with no labels. Below it, six boxes. 1 Test set: 12 or more reference transcripts, a normalisation guide, 3 recognisers by slice; files testset/ and GUIDE.md. 2 Pipeline: VAD, recogniser, diarization, word times, with the real-time factor and the cost per audio hour; file pipeline.py. 3 Call record: reason, summary, notice, measured on your labels, with the main error chain; file records.jsonl. 4 Privacy: redaction in text and audio, the privacy check, a retention rule; file PRIVACY.md. 5 IVR: French prompts, the new names, one new task, a way to a person, replayed test calls; files ivr/ and ivr-replay.jsonl. 6 Agent or menu: a latency budget for one flow, IVR, agent or hybrid, who goes to a person; file OMAR.md. At the bottom, an orange box: the course team checks your labels and scores against the answers it keeps, once. Nothing is chosen with that score.

Part 1: a test set, a guide and three recognisers

Section titled “Part 1: a test set, a guide and three recognisers”

Uses Word error rate, step by step, Normalise before you compare, The errors that matter, Whisper sizes on a laptop, Help the model with your words and Accents, noise and two languages.

There are no reference transcripts. A WER needs one. So your first job is the one that a real team does on a new line: listen, and write down what was said.

  1. Choose the test calls before you run any recogniser. Take at least 12 of the 40 calls, with both stereo and mono calls. Choose them from the public columns only (for example, the first 6 stereo and the first 6 mono calls by ID). Write the rule in REPORT.md. Never choose the calls where a recogniser looks good or bad.

  2. Write the reference transcripts by listening. Write every word of both speakers, turn by turn, as it was said. Write numbers as the speaker said them. Write the product names as products.json spells them. If you start from a machine draft, say which recogniser made it, and correct every word by ear. A draft from one recogniser makes that recogniser look better: see Common mistakes.

  3. Mark the important words in each call: the order number, the product name, dates, negations and any card number, as in The errors that matter.

  4. Write your normalisation guide in speech-pack/GUIDE.md. Start from the course’s guide (data/normalisation-guide.md). Add a rule for each new case that you meet, for example how French writes a number of six digits, or the product names inside French sentences. A WER with no written guide is not accepted.

  5. Make your test set a data folder that the project can read. Write speech-pack/testset/calls/calls.jsonl with one row per test call. Copy the audio files to speech-pack/testset/calls/phone/. Each row keeps the public columns and adds turns (each with speaker and text) and important (the words of step 3, in the course’s format). Then the course’s commands score it:

    Terminal window
    speech --data speech-pack/testset transcribe --split final --model small --limit 0
  6. Compare at least three recognisers on the same test calls, with the same guide: for example Whisper tiny or base, Whisper small, Whisper small with a prompt, and Vosk small. Give the WER, the CER, the important-word errors and the real-time factor of each, with the number of calls and words.

  7. Test the prompt with the new names. The course’s vocabulary prompt (--prompt) names the old products. Make a prompt with the four new names, and compare three settings: no prompt, the old prompt, the new prompt. Count the product names that each one gets right.

  8. Slice. On this line, the language and the line do not change. Slice by what does change: stereo or mono, and a slice that you label while you listen (background noise, a caller who switches to English, a long hold). Give the size of every slice. With 12 calls, a slice of 3 calls proves very little: say so.

Uses Why VAD makes recognition better, Split by channel first, Speaker embeddings and clustering, Word times and alignment and Batch and streaming.

Build the pipeline that Grace’s team will run on every call of the new line: VAD, your chosen recogniser, diarization and word times. Save it as speech-pack/pipeline.py. It can call the project’s functions (larkspeech.asr, larkspeech.vad, larkspeech.diarize).

  1. VAD. Use the VAD settings that you tuned in Why VAD makes recognition better, and check them again on your test calls: the lost packets make new short silences inside words. Compare the WER with and without VAD on your test set.
  2. The recogniser of Part 1, with word times (word_timestamps=True).
  3. Diarization. Split the 27 stereo calls by channel. Cluster speaker embeddings for the 13 mono calls. Give each word the speaker of the turn that holds it, as speech diarize --words does.
  4. Measure. On your test calls, give the DER with missed speech, false speech and confusion apart. Give also the share of reference words that get the right speaker. You need speaker turns in your references for this: mark them while you listen.
  5. Speed and cost. Run the whole pipeline on all 40 calls. Give the real-time factor of each step, and the cost per audio hour on the course’s CPU server (US$0.17 an hour, an assumption in costs.json). Then give the cost of one month of the new region’s calls, with the number of calls a month as your assumption.

Uses Why did they call?, Summaries and follow-ups and Compliance phrases and keyword spotting.

  1. Label your test calls while you listen. Write the call reason (one of the course’s seven), the order number and the product. Write the agent’s promise, the next step, and whether the agent said the recording notice.
  2. Make a call record for every call from the pipeline’s transcript: the reason, a short summary, the order number, the product, the promise and the notice check. Use at least two of the course’s methods for the reason, for example the keyword rules and the classifier trained on the course’s train calls. A recorded LLM is optional; if you use your own key, give the cost.
  3. Measure each field on your labels: the share right for the reason, the order number, the product, the next step; the notices found and the false alarms.
  4. Explain the main error chain with one call. Start from the sound and end at the record: for example, a lost packet, then a misheard digit, then an order number that fails the check digit, then a record with no order. Say which step you would fix first, and why.

Uses Personal data in voice and Voice cloning, speaker checks and misuse.

Some callers on the new line read a card number aloud, as some do on the old line. The French spoken card numbers are the hard case: the course showed that Whisper can write them as words that are not digits, so speech privacy-check passes while the audio still holds the number.

  1. Redact every call in text and in audio with the project’s broad rule (speech --data "$F" redact N-001, or the same functions in a loop).
  2. Close the French gap. Add a rule of your own that does not need digits: for example, a caller who says « carte » and then several number words, or a call that your listening marked. Beep that span too, or send the call to a person for checking. Measure your rule on your test calls: how many card numbers it finds, and how many beeps fall where there was no card number.
  3. Run the privacy check on every output of the pack: transcripts, call records, summaries, logs and the IVR’s call records. A card number that stays in a transcript file fails the pack, even if the call record is clean.
  4. Listen to the beeped audio of each call where a caller talks about a card. Write what you heard.
  5. Write the rules in speech-pack/PRIVACY.md. Keypad only for card numbers on the IVR. The recording notice before recording. How long Larkfield keeps raw audio, redacted audio, transcripts and call records. How one deletion request reaches every copy.

Uses What an IVR is, and why companies still use one, Text-to-speech for prompts, A keypad menu first, Let callers say what they need, Numbers, confirmation and recovery and Test the IVR like a product.

The IVR already speaks French. On the new line it must also know the new orders and the new products, and it gets one new task. The new task is “a question about a new product”. The caller says « question sur un produit » or presses 4. The IVR asks which product, with the four names in its prompt. It reads the name back and asks for a yes or a no. Then it gives a short answer (for example how to set the product up) and ends the call. Two misses, or a product that the region does not sell, go to a person.

The project can support this task: it is one new intent, one new state with a closed list of names, and the confirmation and miss rules that the IVR already has.

  1. Point the IVR at the new orders. speech --data "$F" ivr serve loads orders.json of the new region. Call the test phone once, in French, and ask for an order.

  2. Change the state prompts. The number prompt’s sample order starts with 7. Give it a sample order that starts with 8 or 9. Add the product names to the prompt of the new state, as in Help the model with your words.

  3. Add the new task in larkspeech/ivr/dialogue.py and prompts.py: the menu word and key, the product state, the confirmation, the answer, and two misses to a person. Write the French prompts for the ear: short, one idea a sentence, the option before the key. Add unit tests in tests/test_ivr.py, as the course’s tests do with text and keys only.

  4. Fix the replay’s key-press bug from Test the IVR like a product before you report the time to the answer.

  5. Label the 20 recorded test calls. Listen to each call’s turns in order. Write what a good IVR should do: the language, the intent, and the order number if the caller gives one. Write the expected outcome (status_given, hours_given, return_started or handover) and a kind that describes the call. Write the labels in speech-pack/ivr-labels.jsonl.

  6. Make test calls for the new task. No recorded call asks about a product. So make at least four: two that should pass, one with an unknown product, and one where the caller says “non” twice. Use a French voice that is not the IVR’s own voice, for example Piper’s fr_FR-gilles-low. Put each turn through the phone band, and drop 3% of the 20 ms frames, as the new line does.

  7. Replay the 20 calls and your new-task calls through the engine, with your labels. The command speech ivr replay reads the course’s test calls, so call the same function from a short script, speech-pack/replay_final.py:

    import json
    from pathlib import Path
    from larkspeech.ivr import replay as R
    from larkspeech.ivr.engine import Recogniser
    final = Path("../nextia-learning-labs/C41/final")
    calls = [json.loads(x) for x in (final / "ivr/calls.jsonl").read_text(encoding="utf-8").splitlines() if x]
    labels = {r["call_id"]: r for r in map(json.loads, Path("speech-pack/ivr-labels.jsonl").read_text().splitlines())}
    for c in calls: # your labels: kind, noise and expected (language, intent, order, outcome)
    c.update(kind=labels[c["call_id"]]["kind"], noise=labels[c["call_id"]].get("noise", "quiet"),
    expected=labels[c["call_id"]]["expected"])
    rows = R.replay(final, Recogniser("base"), calls)
    for k, v in R.summary(rows, "kind").items():
    print(k, v)
    R.write_records(rows, "speech-pack/ivr-replay.jsonl")

    R.summary counts status_given, hours_given and return_started as contained. Count your new task’s outcome as contained too.

  8. Report the calls passed, the containment, the wrong-intent rate and the share sent to a person, overall and by kind. Report the hang-ups apart: a call that ends with no answer and no person is the worst ending.

Uses Where a menu stops working, The voice-agent pipeline, Turn-taking and interruptions and Menu, agent or both?.

Tomás wants the voice agent on the new line. Answer him with a latency budget and a recommendation.

  1. Choose one flow, for example “where is my order”. Write the turns of a caller who uses it.
  2. Measure the parts on the new line. Recognition: the seconds of each caller turn in your replay (the IVR’s call records keep them in their timeline as asr_seconds). The LLM’s first words: the course’s recorded agent turns (speech agent budget shows them). The voice: run the French voice as the project does, as a separate program that is kept running, on a few agent-style sentences. Time the first sound of each sentence, and the program’s start apart.
  3. Add up the budget for at least two settings (for example end of turn 0.8 s and 0.5 s, Whisper base and tiny), with the median and a high value. Say which part you would cut first, and what it would take to get under one second.
  4. Compare the IVR, the agent and a hybrid for the new line. The recorded agent only answered the course’s calls, so the agent’s containment on the new line is an assumption: say which number you take and why. Count the hang-ups at US$8 each: speech agent compare leaves them out.
  5. Write the recommendation for Omar in speech-pack/OMAR.md: one page, no code. Choose the IVR, the agent or a hybrid. Name the calls that always go to a person: card numbers, a caller who asks, two misses, a product that the region does not sell. Say what you would measure in a pilot.

The folder speech-pack/ in your larkspeech folder:

speech-pack/
GUIDE.md Part 1: your normalisation guide (the course's guide plus your rules)
testset/ Part 1: calls/calls.jsonl with your references and important words, calls/phone/ audio
comparison.json Parts 1 and 2: one row per recogniser and setting, the course's columns
pipeline.py Part 2, with any other script or notebook that you used
records.jsonl Part 3: one call record per call (40), card numbers removed
PRIVACY.md Part 4: the rules, the check output, what you heard in the beeped audio
ivr/ Part 5: your changes to dialogue.py, prompts.py and tests/test_ivr.py (or a diff)
ivr-labels.jsonl Part 5: your labels of the 20 test calls, written before the replay
product-calls/ Part 5: your test calls for the new task, with how you made them
replay_final.py Part 5
ivr-replay.jsonl Part 5: the IVR's call records
budget.json Part 6: the parts of the budget and their sources
OMAR.md Part 6: one page for Omar
REPORT.md Parts 1 to 6, with the commands that make every number
requirements.txt exact versions

Build comparison.json with the course’s Table (larkspeech/compare.py, the columns of Whisper sizes on a laptop). Put the test set in slice, for example new line, 12 test calls or new line, mono.

REPORT.md has these sections:

# Speech pack for the new French line
## Data checks (files, checksums, what you heard on the new line)
## Test set and guide (Part 1: the rule that chose the calls, how you transcribed, the guide's new rules)
## Recognisers by slice (Part 1: the table, the prompt test, the important words, the choice and why)
## Pipeline (Part 2: VAD, diarization, word times, RTF and cost per audio hour)
## Call record (Part 3: each field measured on your labels, the error chain)
## Personal data (Part 4: both rules, the check, what you heard, the retention rules)
## The IVR on the new line (Part 5: the changes, the new task, the labels, the replay, by kind)
## Menu, agent or both (Part 6: the budget, the comparison with hang-ups, the recommendation)
## Scores from the course team (filled in once; no change after it)
## Limits (what the pack cannot see, and what you would measure next)
## How to reproduce (commands, versions, run times, a fresh environment)

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a concrete reason. Without a reviewer, use the table on your own pack, or ask a colleague. The course is complete only when all six items are demonstrated. You can improve an item and send it again.

Required item Demonstrated when Partially demonstrated, for example Not yet demonstrated, for example
1. Test set and recognisers (Modules 2 and 4) At least 12 calls chosen by a written rule before any run, transcribed by ear, with important words. A written guide with the new rules. At least three recognisers on the same calls and the same guide, with WER, CER, important-word errors and RTF, by at least one slice with its size. The prompt with the new names is tested. The slices have no sizes. Or the references are machine drafts that were not corrected by ear. Or only two recognisers. A WER with no written guide. Or the test calls chosen after seeing the scores.
2. Pipeline (Modules 3, 5 and 9) VAD, the chosen recogniser, diarization by channel and by embeddings, and word times run on all 40 calls. DER with its three parts, and the share of words with the right speaker, on the test calls. RTF per step and the cost per audio hour, with the server price as an assumption. No DER parts. Or the cost has no assumption. No diarization. Or no RTF.
3. Call record (Module 6) A record for every call. Reason, order number, product, next step and notice measured on your labels, with the counts. One error chain explained from sound to record. Only the reason is measured. Or the chain is general, with no call. No measurement against labels.
4. Privacy (Module 9) Redaction in text and audio. A rule for the French spoken card numbers, measured. The privacy check passes on every output of the pack. The beeped audio was heard. Retention and deletion rules. The check runs on the call records only. Or the French rule is not measured. A spoken card number left in any output. Or real people’s voices used without consent.
5. The IVR (Module 7) French prompts and voice, the new orders and names, the new task with confirmation and two misses to a person, unit tests. The 20 calls labelled before the replay, at least four new-task calls, the key-press fix. Passed, containment, wrong-intent rate and hand-overs, overall and by kind; hang-ups apart. No test calls for the new task. Or the time to the answer without the fix. The IVR has no way to reach a person. Or labels changed after the replay.
6. Latency and recommendation (Module 8) A budget for one flow with each part measured or recorded, and its source. At least two settings. IVR, agent and hybrid compared with hang-ups counted. A one-page recommendation that names the calls that go to a person. The voice part is the course’s number, not measured with the project’s own setup (the separate piper program). Or hang-ups are not counted. No budget. Or “the agent is better” with no number.

Not accepted, whatever the rest shows (the item is “not yet demonstrated”):

  • A WER without a normalisation guide. (Item 1.)
  • Real people’s voices without their consent, in the test calls, the new-task calls or anywhere else. Use synthetic voices. (Items 1 and 5.)
  • A spoken card number in any output: a transcript, a record, a summary, a log or an audio file that you keep. (Item 4.)
  • An IVR with no way to reach a person. Key 0 and two misses must always lead to a person. (Item 5.)

A good pack is honest before it is impressive. It has three things.

  • Numbers with their labels and sizes. Each WER, DER, share right and containment has the number of calls behind it, and says what it was measured on: your test set, all 40 calls, or the course team’s score.
  • A simple baseline in each comparison. The course’s settings as they are: Whisper without a prompt, the IVR without your changes, the keyword rules. If your change does not beat it, the pack says so.
  • A decision with its cost. “On the new line, the IVR does this, and these calls go to a person. It costs about this much a call, within this range.”

These ranges come from the author’s own solution. The author built it from the public data only, with the course’s code, and wrote no label by hand: the course team’s references stood in for the transcripts and IVR labels that you write by ear. Then the course team scored it against the answers that it keeps. Your methods can differ. Compare the method and the size of the result, not the exact number.

  • The new line is not much harder than the old French calls. Whisper small made about 15% errors on the new line, close to its 15.1% on the course’s French test calls. The lost packets cost little: the French voices and the new names cost more.
  • The prompt with the new names is the biggest single gain. Without a prompt, Whisper small wrote no new product name exactly. The course’s old prompt did not help with the names. A prompt with the four new names brought the WER down to about 11%, and about 6 product names in 10 out right.
  • Size still matters a lot. Whisper base made about 27% errors and tiny about 43%. Vosk small made about 41% and missed every order number. Whisper small missed about 1 order number in 10; base missed more than half.
  • The tuned VAD made the WER worse here, by about 3 points, mostly on the calls with background talk. Test the course’s settings again on each new line.
  • 12 test calls give the right picture. The author’s 12 calls ranked Whisper tiny, base and small, Vosk and the new prompt in the same order as all 40 calls.
  • The guide almost halves the score. Whisper small’s WER was about 26% with no normalisation and about 15% with the full guide.
  • The pipeline is cheap. Whisper small, VAD and diarization ran at a real-time factor of about 0.09: about US$0.015 per audio hour on the course’s CPU server (an assumption). Diarization by channel had almost no confusion; nearly all of its DER (about 28%) was missed speech. Embeddings on the mono calls gave about 42%, with about 14% confusion. About 4 reference words in 5 got the right speaker.
  • The call record is harder than in the course. The keyword rules and the classifier got about 7 reasons in 10 right, against 90% and more on the course’s test calls. The classifier learned from the old products and from mostly English calls. The fuzzy notice check found every notice, with no false alarm. The exact phrase found only about 1 in 3. About 4 order numbers in 10 were lost on the way to the record, mostly as “too short”: the caller’s digits were split.
  • The broad card rule found no French card number at all. Whisper wrote the digits as words or as nonsense. A context rule (« carte » followed by number words) found half to three quarters of them in two runs, with one beep where there was no card. Whisper’s text changes a little from run to run, and so does what the rule finds: listen to every card call. A pack that only ran the project’s rule would keep card numbers in its audio.
  • The privacy check had false alarms too. In the author’s first run, it read the diarization turns as long digit runs, in 38 of the 40 calls. The turns were stored as lists of times (“[5.7, 11.82, …”). The project’s check now reads a JSON file field by field. It skips numbers with a decimal point and the fields for times and IDs. It keeps the broad rule for the text. In a plain text file, the check can still read times as digits, so store them in JSON fields.
  • The IVR contains about a third of the new calls with Whisper base, as on the course’s calls: about 10 of the 20 calls passed. Spoken order numbers almost always failed. Whisper small contained about half and passed more; tiny contained a fifth. Some callers’ « Français » was heard as English words. With the key-press fix and the new prompts, the mean time to an answer fell from about 51 to about 45 seconds.
  • The new task worked on the keypad, and mostly failed by speech. In the author’s first version, the French test voice read the English product names in a French way, and Whisper did not match them. The call that used keys passed. One call ended with no answer and no person: the IVR waited for a turn that never ended. That is a hang-up, and the pack must count it.
  • The voice can be slower than the course’s budget. Started again for each sentence, Piper took about 0.8 s to the first sound of a French sentence. The course measured 0.05 s inside one program. Recognition took a median of about 0.26 s (base, new line), and the recorded LLM about 1.0 s. So the median wait was about 2.9 s at 0.8 s end of turn, and 2.6 s at 0.5 s. The project now keeps the piper program running. After its start, the first sentence was ready after about 0.05 s. That brings those sums down by about 0.75 s. No setting came near one second.

Common mistake

A test set transcribed by the recogniser that it tests

Symptom: Whisper small has a WER of 3% on your test set, much lower than on the course’s French calls.

Cause: you started the references from Whisper small’s drafts and corrected only the words that sounded wrong. Its mistakes that sound right stayed in the references, so they do not count as errors.

Fix: write the references by ear, or correct a draft word by word while you listen, and say which recogniser made the draft. Compare the drafted recogniser and the others on the same references, and be careful with the drafted one.

Consequence in AI work: the team chooses a model because it graded its own test. The first real complaint shows the gap.

Common mistake

The privacy check passes, and the audio keeps the card

Symptom: speech privacy-check says Passed. on every output. A person who listens to a stored call still hears a card number.

Cause: the recogniser wrote the French digits as words or as nonsense, so the text has no digit run to find, and nothing was beeped.

Fix: add a rule that does not need digits, listen to every call where a caller talks about a card, and keep the IVR’s rule: keypad only for card numbers. Delete raw audio soon.

Consequence in AI work: a green check gives false comfort. A stored card number is a breach, whatever the check said.

  • Labels changed after the replay. The IVR’s result is not a label. Write what a good IVR should do first.
  • The old vocabulary prompt kept. It names the old products and an order that starts with 7. It does not help the new names.
  • The course’s VAD settings copied. They were tuned for the old lines. Measure them again.
  • A hang-up counted as nothing. A call with no answer and no person is the worst ending. Count it at US$8, as the course does.
  • The voice’s time copied from the course. The course measured Piper inside one program. The project runs it as a separate program: a program started for each sentence pays about 0.8 s each time. Keep it running, and measure it.
  • One number for the whole line. Give stereo and mono, and the slices that you labelled, with their sizes.
  • Test calls for the new task in the IVR’s own voice. A recogniser hears its own synthetic voice better than callers. Use another voice, the phone band and the lost packets.
Show the public check values

Open this only after you have your own numbers. These values come from the public data, so you can check them. They are not the answers to the assignment.

  • Data: 40 calls, 27 stereo and 13 mono, 32 to 75 seconds each, 34.5 minutes in all. 20 IVR calls with 76 turn files. 60 orders. Every file in SHA256SUMS is OK.
  • Speed on the author’s laptop (Apple M4, 4 threads, all 40 calls): real-time factor about 0.017 for Whisper tiny, 0.026 for base, 0.07 to 0.08 for small, and 0.12 for Vosk small. The whole pipeline with Whisper small, VAD and diarization: about 0.09.
  • Lost packets: our code marks about 2 frames in 100 of the speech as near silence inside speech. It misses the holes at the edges of words.
  • The privacy check on the pipeline’s raw output: the first version of the check reported “card numbers” in almost every call. All of them were diarization times. The project’s check now skips the numbers and time fields of JSON records.
  • The voice: Piper’s French voice takes about 0.8 s to a WAV file for a short sentence when it starts again for each sentence. Kept running, as the project does, it takes about 0.05 s for the first sentence.

Recap

You can now take a speech problem from a new line to a decision. You build a test set by ear, with a written guide, and compare recognisers on the words that matter. You run a pipeline with VAD, diarization and word times, and give its cost per audio hour. You measure call records against your own labels, and you keep card numbers out of every output. You extend the IVR with a new task, test it with labelled calls, and decide between a menu and an agent with a latency budget and a full cost. The case studies of Module 10 use the same steps on real speech: Choose a recogniser for two languages, Meeting transcripts with speakers, Spot commands on a small device and An IVR for spoken intents.