Test the IVR like a product
- Lesson 27
- 40 min
- Module 7: Build a small IVR
Where we are
Our IVR handles three tasks, confirms what it heard and hands over when it must. Is it good? In this lesson, we replay the recorded test calls as automated tests and measure containment, wrong intents and time to the answer by slice.
Lesson 27 of 36 · Speech AI: Transcribe, Understand and SpeakTest the IVR like a product
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Judge the Larkfield IVR the way a product team does. Run every recorded test call as an automated test, read the results by slice, and listen to the failures to find what to fix next.
NotebookDo this lesson in a notebook
The notebook reads the recorded replay of the IVR on the 60 test calls. It computes the measures by kind and by line, makes the test stricter, and asks you to find the calls with a retry right after a key press. It downloads only small text files and runs no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The four measures (containment, time to the answer, hang-ups, wrong intents), from What an IVR is, and why companies still use one.
- How the IVR handles keys, speech and order numbers, from the three lessons before this one.
- Slices: why an average can hide a group that fails, from The errors that matter.
What you need
- Your
larkspeechfolder at the end-of-Module-7 snapshot, with Whisper base. - For the test phone: Chrome or Firefox, a microphone, and headphones.
- Tested on macOS (Apple M4) with Python 3.12. The Windows commands are the same in PowerShell, but were not run. A live call with a real microphone was not tested.
A driving test on a fixed route
Section titled “A driving test on a fixed route”In a driving test, every candidate drives the same route. The examiner has a sheet: a roundabout, a hill start, a parking space, a busy junction. Each part is marked “passed” or “not passed”. At the end, the examiner does not only say “you drove well”. The sheet shows which parts went wrong, so the next lesson can practise them.
Testing an IVR works the same way. The same recorded calls, every time. An expected result for each call. And the results split by kind of call, so you see which part of the menu needs work.
The problem: a few calls at a time is not a test
Section titled “The problem: a few calls at a time is not a test”So far, you replayed four or five calls at a time. That is how you debug: you look closely at one call. It is not how you judge. A change that fixes I-003 can break I-049, and you would not see it. Also, five calls cannot tell you whether 35% or 40% of the callers get an answer.
Product teams solve this with a regression test: a fixed set of cases that runs after every change. “Regression” means going back: the test catches a change that makes something that worked fail again.
The idea: every recorded call is an automated test
Section titled “The idea: every recorded call is an automated test”Each of the 60 public IVR test calls has an expected result: the language, the intent, the outcome and the order number. The command speech ivr replay plays each call through the same engine as the test phone: the same keypad detector, the same VAD and end of turn, the same Whisper and dialogue. Then it compares. The rule is in larkspeech/ivr/replay.py:
ok = r["outcome"] == exp["outcome"] and (exp.get("order") in (None, r["order"]) or r["outcome"] == "handover")In words: a call passes when its outcome is the expected one (status_given, hours_given, return_started or handover). If the IVR kept an order number, it must also be the right one.
What “passed” does not check
Section titled “What “passed” does not check”Read the rule again. It does not check the reason for a handover. In the lesson about speech, test call I-073 asked for a person. Whisper wrote “I want to talk to her.”, no rule matched, and the call reached a person after two misses: no_input, not caller_asked. It passed, but for the wrong reason, after a caller heard “Sorry, I did not get that” once.
It is not the only one. If a handover must also have the expected reason, 33 of the 60 calls pass instead of 39. Six calls reached a person for another reason. Five of them are return calls: the test expects not_returnable, but the IVR never understood the order number, and the call ended with no_input before the IVR could look at the order. A test checks only what you wrote into it. The notebook makes the test stricter.
A tiny worked example: from records to measures
Section titled “A tiny worked example: from records to measures”Here are four call records, shortened, to see how the replay counts:
| Call | Expected | Outcome | Seconds | Passed? | Contained? |
|---|---|---|---|---|---|
| I-009 | status given | status given | 46.0 | yes | yes |
| I-029 | hours given | hours given | 27.1 | yes | yes |
| I-003 | status given | handover | – | no | no |
| I-062 | handover | handover | – | yes | no |
- Passed: 3 of 4. I-062 passes because the caller pressed 0 for a person, and got one.
- Containment: 2 of 4 = 50%. Only the calls that the IVR finished by itself count.
- Time to the answer: (46.0 + 27.1) ÷ 2 = 36.6 seconds, over the contained calls only.
- To a person: 2 of 4 = 50%.
Passed and contained are different questions. “Passed” asks whether the IVR did the right thing. “Contained” asks whether it saved a person’s time. A test set with many calls that should reach a person can have a high pass rate and a low containment, and both are correct.
The results on Larkfield’s 60 test calls
Section titled “The results on Larkfield’s 60 test calls”With Whisper base and an end of turn of 0.8 seconds, on Larkfield’s synthetic calls:
| Measure | Result |
|---|---|
| Tests passed | 39 of 60 |
| Containment | 35% |
| To a person | 63% |
| Wrong intent | 0% (no call) |
| Time to the answer (contained calls) | 50 seconds |
Of the 60 calls, 19 should reach a person. They are returns of orders that cannot be returned, silent callers, callers who ask for a person, and unclear requests. So the best possible containment on this set is 41 of 60, or 68%. The IVR reached 35%.
By kind of call
Section titled “By kind of call”Text description of the diagram
Every keypad kind passes: status 6 of 6, hours 3 of 3. Hours by speech pass 6 of 6. The failures gather where the caller says an order number: status by speech 2 of 12, a number with a pause 1 of 4, a wrong number then fixed 1 of 3, a change of mind 0 of 3. All the kinds that should reach a person pass, except one silent caller (2 of 3).
The chart tells Amira where to work next: spoken order numbers, as the last lesson found. It also shows what already works and must not break: the keypad, the opening hours, and every way to a person.
By language and by line
Section titled “By language and by line”| Slice | Calls | Passed | Containment |
|---|---|---|---|
| English | 27 | 21 | 41% |
| French | 33 | 18 | 30% |
| Landline | 36 | 23 | 25% |
| Mobile (Opus) | 11 | 7 | 45% |
| VoIP (G.726) | 13 | 9 | 54% |
French callers get an answer less often than English ones. Whisper base is weaker in French, and the first question (“Français”) already fails in six French calls.
The lines look like a surprise: the landline, the cleanest line, has the lowest containment. Do not read it as “landlines are worse”. The landline slice holds most of the calls that must not be contained, and most of the French ones (the numbers are in the common mistake below). A slice is only fair when the slices hold the same mix of calls. With 11 or 13 calls, one call moves a slice by about 8 points.
The hidden calls, once
Section titled “The hidden calls, once”The course also has 20 hidden IVR test calls that no lesson used. They were run once, at the end, before the last change to dialogue.py (not re-run): 13 of 20 passed, containment 40%. That is close to the public 35%, so the IVR was not tuned to the public calls by accident.
What the replay cannot tell you
Section titled “What the replay cannot tell you”Three limits matter as much as the numbers:
- The recorded callers cannot react. When the IVR asks again in another way (“please type it on your keypad”), the recorded caller says their next recorded line. A real caller would press the keys. So a miss usually ends with a person in the replay. This lowers containment against what real callers would get.
- “Return started” is tested only by unit tests. None of the six public return calls has an order that can be returned. So all six end with a person, as expected. Only one of them got as far as the order check. The path that starts a return runs only in
tests/test_ivr.py. - The calls are synthetic. Real callers have more accents, more noise, and longer, messier answers. Expect lower numbers on a real line.
Listen to the failures
Section titled “Listen to the failures”Numbers tell you where; listening tells you why. The replay writes every call record, with its timeline, to outputs/ivr-replay.jsonl. When we read the timelines of the keypad calls, we found something odd in a call that passed:
Text description of the diagram
The caller presses 1 for English. The IVR plays the menu, then, about 5 seconds after the menu ends, the menu retry: “Sorry, I did not get that.” Only then does the next key, 1, arrive. The same happens in the number state (number retry before the keys 586032) and in the confirm state (confirm retry before the last key 1). The call passes, after 69.02 seconds.
The IVR did nothing wrong here. Our replay did. When the engine reacts in the middle of a caller’s recording (a key is found after 40 ms), the replay keeps the rest of that recording as the caller’s next turn. That rule is right for speech, when a caller talks over a prompt. For a key, the rest is only the tail of the tone and silence. So after each key, the replay played about half a second of silence as a “turn”, waited 5 seconds, and counted a no-input.
We tested a fix: after a key press, drop the rest of the recording. On the 60 calls:
| Passed | Containment | Time to the answer: status by keypad | hours by speech | |
|---|---|---|---|---|
| The replay in the snapshot | 39 | 35% | 67.8 s | 39.5 s |
| With the fix | 39 | 35% | 38.1 s | 27.2 s |
The fix changes no pass or containment, so every number in this module stays true. But the time to the answer was too long: a keypad caller gets the status about 30 seconds sooner than the snapshot’s replay says. The snapshot keeps the replay as it is, so that your numbers match the next module. The notebook asks you to find every call where it happened. A test harness is code too: it has bugs, and listening to the calls that pass is how you find them.
Guided practice: the whole test set, and the test phone
Section titled “Guided practice: the whole test set, and the test phone”-
In
larkspeech, with the environment active, run the unit tests:Terminal window python -m pytest -q................................. [100%]33 passed in 1.69s -
Replay all 60 public test calls, by kind. It takes about 1 to 2 minutes:
Terminal window speech ivr replay --by kindIVR replay: 60 recorded test calls, Whisper base, end of turn 0.8 skind calls passed contained wrong intent to a person s to answeragent_request 3 3 0% 0% 100% -change_of_mind 3 0 0% 0% 100% -hours_keypad 3 3 100% 0% 0% 30.4hours_speech 6 6 100% 0% 0% 39.5noisy_fallback 4 2 50% 0% 50% 66.1number_pause 4 1 25% 0% 75% 59.7return_speech 6 6 0% 0% 100% -silent 3 2 0% 0% 67% -status_keypad 6 6 100% 0% 0% 67.8status_speech 12 2 17% 0% 83% 45.5unclear 4 4 0% 0% 100% -wrong_then_fix 3 1 33% 0% 67% 62.4zero_again 3 3 0% 0% 100% -Failed: I-001 (status_speech: handover, expected status_given), I-003 (status_speech: handover, expected status_given), …Wrote outputs/ivr-replay.jsonl (one call record per line)The passed counts match the chart. The wrong-intent column can differ by a call from run to run: Whisper on 4 CPU threads does not give exactly the same text every time.
-
Replay by language:
Terminal window speech ivr replay --by languageIVR replay: 60 recorded test calls, Whisper base, end of turn 0.8 slanguage calls passed contained wrong intent to a person s to answeren 27 21 41% 0% 56% 46.3fr 33 18 30% 3% 70% 57.0... -
Start the IVR with its test phone:
Terminal window speech ivr serveOpen http://127.0.0.1:8041/ and press Call. Stop with Ctrl+C.In a second terminal, check that it answers:
Terminal window curl -s http://127.0.0.1:8041/health{"status":"ok","orders":160}On Windows PowerShell, use
Invoke-RestMethod http://127.0.0.1:8041/health. -
Open http://127.0.0.1:8041/ in Chrome or Firefox. Put on headphones, so the IVR does not hear its own voice. Press Call, allow the microphone, and press 1, then 2 on the page’s keypad.

The test phone after a keypad call: key 1 (English), then key 2 (opening hours). The IVR wrote this call to outputs/calls/live-001.json. The IVR writes each call to
outputs/calls/. Then try speech: call again and say “English”, then “What are your opening hours?”. The audio stays on your computer. Stop the server with Ctrl+C.
Your turn: one change, measured on all calls
Section titled “Your turn: one change, measured on all calls”Choose one change from this module, make it in your copy, and replay all 60 calls. Good candidates:
- add “where’s my order” to the menu rules (I-006 failed on it);
- remove the number prompt for English (the last lesson’s measurement);
- the shorter menu prompt from the lesson on text-to-speech (it changes only the time to the answer).
Predict first: which kinds will change, and in which direction? Then compare the table with the one in step 2, kind by kind. Keep a change only if nothing that passed now fails.
What to look for
A good change moves its own kind and leaves the others alone. Suppose a change to the menu rules gives an “unclear request” call a wrong intent. Then a test that passed before fails: that is the regression the test set is for. With a few calls per kind, one call more or less is a real change, but be careful with conclusions: 3 calls of a kind are an example, not a rate. Before you trust a gain, run the hidden calls once more, as the course did once.
Optional: a real phone number, or Asterisk
Section titled “Optional: a real phone number, or Asterisk”The same app can answer a real phone call. Two paths are written in docs/phone-line-options.md in your project. The course team did not test them. They need an account with a phone provider, or Docker and some telephony knowledge, and a real number costs money: check the provider’s current price page first.
- A real phone number. Many providers can send a call’s audio to your app over a WebSocket, as 8 kHz mu-law audio, the same format as Larkfield’s phone calls. You add one route to
larkspeech/ivr/app.pythat speaks the provider’s format, make your laptop reachable with a tunnel, and call the number. - Asterisk and a softphone (a stretch). Asterisk is an open-source phone system that many contact centres run. A softphone on your laptop calls an Asterisk in a container, which sends one extension to the IVR.
Never record real callers without their consent. Module 9 deals with consent, retention and redaction.
Common mistake
A slice that blames the wrong thing
Symptom: Malik sees “landline 25%, mobile 45%, VoIP 54%” and asks the phone provider to check the landline.
Cause: the slices hold different calls. 15 of the 36 landline calls should reach a person, against 2 of 11 mobile calls and 2 of 13 VoIP calls. Most of the French calls are on the landline too. The pass rates (23 of 36, 7 of 11, 9 of 13) are close.
Fix: before you compare slices, compare what is in them: the kinds of call, the languages and the expected outcomes. Compare pass rates, or containment among the calls that should be contained. With slices of 11 or 13 calls, treat a difference of one or two calls as noise.
Consequence in AI work: a slice is a group with its own mix. A difference between slices can come from the mix, not from the thing that names the slice.
Check your understanding
Ten test calls: 6 should be answered by the IVR and 4 should reach a person. The IVR answers 4 of the 6 correctly and sends all 10 others to a person. What are the pass rate and the containment?
Show the answer
Passed: 8 of 10 (the 4 right answers and the 4 calls that should reach a person). Containment: 4 of 10 = 40%. The 2 calls that should have been answered and were sent to a person fail. The best possible containment on this set is 60%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the replay probably give a lower containment than real callers would get, and what is one way to measure the difference?
Show the answer
The recorded callers cannot react. When the IVR asks again in another way (“type it on your keypad”), the recording plays the next recorded line instead, so a miss usually ends with a person. Real callers would follow the new instruction. To measure it: record a few test callers who react to the prompts (people calling the test phone, with consent), or make test calls whose recordings include the fallback answer, and compare.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A change raises containment from 35% to 38%, but one "asks for a person" call now ends with an order status. Do you keep the change?
Show the answer
Not as it is. A caller who asked for a person got something else. That is a wrong intent, and it costs more (US$15 in Larkfield’s assumptions) than a call to a person (US$4). Find why the change broke that call, fix it, and replay all calls again. Keep a change only when it adds wins without new failures in the kinds that worked.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Every recorded call with an expected result is an automated test. Replay them all after every change: it is a regression test.
- Passed (the right thing happened) and contained (no person needed) are different. On Larkfield’s 60 synthetic test calls: 39 passed, containment 35% (best possible 68%), English 41%, French 30%. The hidden calls, once (before the last change to
dialogue.py): 13 of 20, 40%. - A test checks only what you wrote into it: with the handover reason checked too, 33 pass instead of 39.
- Read the results by kind: spoken order numbers fail most; the keypad, the hours and the way to a person work.
- Compare slices only after you check their mix. The landline looked worst because it held the calls that must reach a person.
- Listen to the calls that pass too. Our replay added a retry after every key press. Pass and containment did not change, but the time to the answer was about 30 seconds too long for keypad callers.
- The replay’s callers cannot react, “return started” is only unit-tested, and the calls are synthetic: real callers will differ.
You can now test the IVR like a product and say what it does well and badly. But some callers do not fit any menu: “my parcel came, but the lid was cracked and I need it before Saturday”. Next, you look at where a menu stops working, and what a voice agent could do there.