Numbers, confirmation and recovery
- Lesson 26
- 40 min
- Module 7: Build a small IVR
Where we are
Our IVR now understands what callers want. Order numbers are still hard: callers pause, say "oh" for zero, and the line adds noise. In this lesson, we get the number right, confirm it, ask again after a miss, and hand over after two.
Lesson 26 of 36 · Speech AI: Transcribe, Understand and SpeakNumbers, confirmation and recovery
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Make the Larkfield IVR safe with order numbers. It must never act on a misheard number, it reads the number back before it acts, and it recovers from a miss without trapping the caller.
NotebookDo this lesson in a notebook
The notebook turns the recorded transcripts of the callers’ order numbers into digits. It checks each one (length, check digit, order system), and counts the wrong numbers that pass. Then you write the read-back in groups and a yes-or-no rule. It downloads only small text files and runs no model. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Spoken numbers to digits, the check digit (Luhn) and the three checks of an order number, from Help the model with your words.
- Counting the errors that matter, per speaker, from The errors that matter.
- The IVR’s states, misses and speech path, from Let callers say what they need.
What you need
- Your
larkspeechfolder at the end-of-Module-7 snapshot, with Whisper base. - Tested on macOS (Apple M4) with Python 3.12. The Windows commands are the same in PowerShell, but were not run.
A phone number to a pizza shop
Section titled “A phone number to a pizza shop”You order a pizza by phone. At the end, the person asks for your number. You say “oh-seven-seven, nine-oh-oh, one-two-three-four”. The line is bad and the shop is loud. A good shop does not just write something down. They read it back: “zero seven seven, nine zero zero, one two three four?” You say “yes”. If they got a digit wrong, you hear it, and you correct it.
Three habits make this work. Read it back in groups. Wait for a clear yes. And when something is wrong, ask again in another way (“can you text it to us?”). The IVR needs the same three habits. A wrong order number is worse than no answer, because the caller hears someone else’s order.
The problem: the spoken order number fails most often
Section titled “The problem: the spoken order number fails most often”On the 60 recorded test calls, the IVR passed 2 of the 12 calls that ask for an order status by speech. It passed 1 of 4 calls where the caller pauses inside the number. It passed 1 of 3 where the caller first says a wrong number, then fixes it. Here are the 12 status-by-speech calls, replayed through the end-of-Module-7 snapshot:
| Call | Language | The caller said | The IVR heard | Digits found, and the check that failed | Outcome |
|---|---|---|---|---|---|
| I-001 | en | eleven, forty-nine, forty-two (114942) | “114940.” | 114940: check digit wrong | a person |
| I-002 | fr | un quatre cinq huit huit huit (145888) | “Un quatre cinq, 8, 8, 8.” | 145888: all checks pass | status given |
| I-003 | en | five two two nine one two (522912) | “Two two nine one two.” | 22912: too short | a person |
| I-004 | en | thirty-nine, eleven, sixty-nine (391169) | “Third 9, 11, 69.” | 91169: too short | a person |
| I-005 | fr | un trois six un sept six (136176) | “Fortsis, c’est…” | no digits | a person |
| I-009 | en | four seven nine nine zero seven (479907) | “479907.” | 479907: all checks pass | status given |
| I-010 | en | three oh three three six two (303362) | “Three, both three, three.” | 33: too short | a person |
| I-012 | fr | un sept un quatre un trois (171413) | “Un site un 413.” | 1413: too short | a person |
| I-006, I-007 | en, fr | (never asked: the menu answer was not understood) | a person | ||
| I-008, I-011 | fr | (never asked: “Français” was not understood) | a person |
Read the table twice. First, the bad news: the IVR understood only 2 of the 8 order numbers it asked for. Second, the good news: every wrong number was caught. No caller heard the status of someone else’s order. The checks did their job: the IVR is safe, but it is not yet helpful.
The order numbers are counted per caller turn. The IVR test calls hold only the caller’s side, so nothing else can hide an error. On whole conversations, in Module 2, the agent’s read-back could hide a caller’s wrong digit: count the caller’s turns alone.
Idea 1: from words to digits, then three checks
Section titled “Idea 1: from words to digits, then three checks”The IVR uses the function from Module 4, find_order_number. It turns the caller’s words into digits with the normalisation guide (“thirty-nine” → 39, “oh” between digits → 0). Then it keeps the longest run of digits, and checks it:
python -c "from larkspeech.numbers import find_order_number as f; print(f('Two two nine one two.')); print(f('114940.')); print(f('three oh three three six two')); print(f('Three, both three, three.'))"('22912', 'too short')('114940', 'check digit wrong')('303362', None)('33', 'too short')“three oh three three six two” is a valid order number: “oh” became 0. But Whisper wrote “Three, both three, three.”, so the IVR found only “33”. No rule can bring back digits that are not in the text.
Then the IVR looks the number up in the order system. A six-digit number with a right check digit can still be an order that does not exist. Only a number that passes all three checks goes to the next step.
Idea 2: ask for the rest, then another way
Section titled “Idea 2: ask for the rest, then another way”A failed check is not the end. The IVR has two ways to recover, in this order.
Ask for the rest. When the caller paused in the middle of the number, the IVR heard only the first part. This happens the first time it finds from 2 to 5 digits. It reads back what it heard and asks for the rest: “I heard 7 9 8. Please say the rest of the number.” Then it joins the two parts.
Ask again, in another way. When the number fails a check, the IVR does not ask the same question again. It switches to the keypad: “That number does not look right. Please type it on your keypad, one digit at a time.” A key is never misheard. When the number is not in the order system: “I cannot find that order. Please type the number on your keypad.”
Text description of the diagram
The IVR asks for the number, turns the words into digits and checks them. A number that is too short, the first time, leads to “I heard 7 4 2. Please say the rest.” A number that fails the check digit or the order system, or that the caller rejects at the read-back, leads to “Please type it on your keypad”. A second miss leads to a person, who sees what the IVR heard.
A tiny worked example: a number in two parts
Section titled “A tiny worked example: a number in two parts”Here is the dialogue alone, with no audio, on the words of test call I-049. The caller says the order number in two parts:
python -c "from larkspeech import data as D; from larkspeech.paths import data_dir; from larkspeech.ivr.dialogue import Dialogue; d = Dialogue(D.orders(data_dir(None))); d.start(); [print(a['say']) for e in [('speech', 'English.'), ('speech', 'Where is my order?'), ('speech', 'Seven nine eight.'), ('speech', 'Zero nine zero.'), ('speech', 'Correct.')] for a in d.step(*e) if 'say' in a]"How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person.Please say or type your six-digit order number.I heard 7 9 8. Please say the rest of the number.Order 7 9 8, 0 9 0. Is that right? Say yes or no, or press 1 for yes and 2 for no.Your Mistral fan shipped on October 11 and is on its way. Thank you for calling Larkfield. Goodbye.Step by step: “Seven nine eight.” gives 798, which is too short, so the IVR asks for the rest. “Zero nine zero.” gives 090. Joined, 798090 has 6 digits, a right check digit, and it is in the order system. So the IVR reads it back in groups. “Correct.” is a yes, and the IVR gives the status.
Idea 3: confirm before you act
Section titled “Idea 3: confirm before you act”The read-back is the IVR’s last safety check. It says the number in groups of three, “Order 7 9 8, 0 9 0”, because a caller checks three digits at a time more easily than six. Then it waits for a clear answer.
The yes-or-no rule is a small list of words, in both languages:
YES = {"yes", "yeah", "yep", "correct", "right", "that's right", "oui", "exact", "exactement", "c'est ça", "ouais"}NO = {"no", "nope", "wrong", "not", "incorrect", "non", "pas", "faux"}The rule checks for a no first: “That’s not right” contains “right”, but it is a no. A “no” takes the caller back to the number, and counts as a miss. No answer, or an answer that is neither yes nor no, asks again: “Please say yes or no, or press 1 for yes and 2 for no.”
Is the read-back worth its 8 seconds? Larkfield counts a wrong action at US$15 and a call to a person at US$4 (both assumptions). For opening hours, no read-back is needed: there is nothing to confirm. For a return, a wrong order number would start a return on the wrong order. So the IVR confirms before every action on an order, and never before a simple answer.
Idea 4: two misses, then a person, with what you know
Section titled “Idea 4: two misses, then a person, with what you know”After the second miss in the same state, the IVR stops. It says “I am putting you through to a person now. They will see what you told me.” and transfers the call with a reason: no_match, no_input, caller_asked or not_returnable.
The person who answers must not ask everything again. So the call record goes with the call: the language, the intent, and everything the IVR heard. Here is the record when a caller says a wrong number, then types a wrong number too:
{'language': 'en', 'intent': 'order_status', 'order': None, 'outcome': 'handover', 'handoff_reason': 'no_match', 'heard': ['English.', 'Where is my order?', 'seven nine eight zero nine one', '798091'], 'prompts': ['welcome', 'menu', 'number', 'number_retry', 'handover'], 'misses': 2}The person sees at once: English, an order status, and two tries at a number close to 798090. That person can ask “Is your order 7 9 8, 0 9 0?” instead of “How can I help?”
Larkfield: does the number prompt help?
Section titled “Larkfield: does the number prompt help?”The last lesson gave Whisper a short prompt in each state. In the number state, it is “My order number is 742155. Seven four two, one five five.” It fixed the language question. Does it help with numbers too?
We measured it. We took every caller turn of the 60 public test calls that says an order number: 31 turns, 13 in English and 18 in French. Whisper base wrote each one four ways. It heard either the whole recorded turn, or the turn as the IVR cuts it (with the VAD and the end of turn). And it worked either without or with the number prompt.
Text description of the diagram
English: on the whole turn without a prompt, Whisper base got 11 of 13 numbers right; with the number prompt, 10. On the turn as the IVR cuts it, 10 without the prompt and 7 with it, which is how the IVR runs today. French: 2, 3, 1 and 2 of 18.
Two lessons come out of it:
- In English, the number prompt does not help Whisper base. On the turns as the IVR cuts them, it got 7 of 13 right with the prompt and 10 without it. In I-003, the caller said “five two two nine one two”. Without the prompt, Whisper base wrote “5, 2, 2, 9, 1, 2.”; with it, “Two two nine one two.” The first digit was lost.
- French numbers are hard for Whisper base in every setting: 1 to 3 of 18. The voices’ French and the small model both play a part. Here, the keypad is the real way in.
This is one run on 31 turns, so a difference of one or two turns is noise. A gap of 7 against 10 or 11 in English is worth a test with more calls. The IVR’s settings stay as they are in this module, so that the numbers match from lesson to lesson. Removing the number prompt for English is a good change to try in your own copy.
Guided practice: replay the number calls
Section titled “Guided practice: replay the number calls”-
In
larkspeech, with the environment active, replay five calls with spoken order numbers: I-001, I-003 and I-010 (status by speech), I-049 (a pause) and I-054 (a wrong number, then fixed).Terminal window speech ivr replay --calls I-001,I-003,I-010,I-049,I-054IVR replay: 5 recorded test calls, Whisper base, end of turn 0.8 scalls passed contained wrong intent to a person s to answerall 5 2 40% 0% 60% 61.0Failed: I-001 (status_speech: handover, expected status_given), I-003 (status_speech: handover, expected status_given), I-010 (status_speech: handover, expected status_given)Wrote outputs/ivr-replay.jsonl (one call record per line) -
Read the record of I-003, the second call:
Terminal window python -c "import json; r = [json.loads(x) for x in open('outputs/ivr-replay.jsonl')][1]; print(r['call_id'], r['outcome'], r['handoff_reason']); print(r['heard']); print(r['prompts'])"I-003 handover no_input['English.', 'I want to track my order.', 'Two two nine one two.', 'Yes.']['welcome', 'menu', 'number', 'number_rest', 'number_retry', 'handover']The IVR heard “Two two nine one two.”, found 5 digits, and asked for the rest. The recorded caller then said “Yes.” That is their next recorded line, the answer to a read-back that never came. A real caller would have said the number again. The replay cannot react, and this lowers the IVR’s score against a real caller.
-
Check that the checks are safe: in
outputs/ivr-replay.jsonl, no call has anorderthat differs from the expected one.Terminal window python -c "import json; rows = [json.loads(x) for x in open('outputs/ivr-replay.jsonl')]; print([(r['call_id'], r['order'], r['expected'].get('order')) for r in rows])"[('I-001', None, '114942'), ('I-003', None, '522912'), ('I-010', None, '303362'), ('I-049', '798090', '798090'), ('I-054', '603134', '603134')]Every
orderis either the expected number orNone: the IVR never kept a wrong number.
Your turn: a wrong number, said and typed
Section titled “Your turn: a wrong number, said and typed”Predict: a caller says “seven nine eight zero nine one” (one digit wrong), then types 798091 on the keypad. What does the IVR say after each, and how does the call end? Use the dialogue alone, as in the worked example, with ('keys', '798091') for the keypad.
Check your answer
python -c "from larkspeech import data as D; from larkspeech.paths import data_dir; from larkspeech.ivr.dialogue import Dialogue; d = Dialogue(D.orders(data_dir(None))); d.start(); [print(a.get('say') or a) for e in [('speech', 'English.'), ('speech', 'Where is my order?'), ('speech', 'seven nine eight zero nine one'), ('keys', '798091')] for a in d.step(*e)]; print(d.record())"How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person.Please say or type your six-digit order number.That number does not look right. Please type it on your keypad, one digit at a time.I am putting you through to a person now. They will see what you told me.{'transfer': 'no_match'}{'call_id': None, 'language': 'en', 'intent': 'order_status', 'order': None, 'outcome': 'handover', 'handoff_reason': 'no_match', 'heard': ['English.', 'Where is my order?', 'seven nine eight zero nine one', '798091'], 'prompts': ['welcome', 'menu', 'number', 'number_retry', 'handover'], 'misses': 2}798091 fails the check digit, so the first try is a miss and the IVR switches to the keypad. The typed number fails again: a second miss, and a person takes over with everything the caller said. The wrong order was never read out.
Common mistake
A read-back that says yes for the caller
Symptom: while writing this lesson, we tried the yes-or-no rule on a few answers a caller might give. “That’s not right.” came back as yes. So did “Not correct.”
Cause: the first version of the rule had only “no”, “nope”, “wrong”, “non” and “pas” as no-words. “That’s not right” has none of them, but it has “right”, which is a yes-word. The IVR would have read out the status of an order that the caller had just rejected.
Fix: the snapshot’s rule now has “not”, “incorrect” and “faux” as no-words too, and it still checks for a no before a yes. A unit test in tests/test_ivr.py covers both languages. None of the 60 test calls said “not right”, which is why the replay never found it: test the safety step with answers written on purpose, including negations in both languages.
Consequence in AI work: a confirmation step protects you only if the confirmation itself is understood correctly. Test the safety step as carefully as the main step.
Check your understanding
The IVR hears "Seven nine one four seven" for order 791475. What does it do, and why does it not look the number up?
Show the answer
It finds 79147: 5 digits, too short. It is the first try with a partial number, so it says “I heard 7 9 1, 4 7. Please say the rest of the number.” It does not look up a number that failed the length check, because a wrong lookup could find, and read out, another caller’s order.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the IVR read the order number back in groups, "Order 7 9 8, 0 9 0", and why does it confirm before a return but not before the opening hours?
Show the answer
People check three digits at a time more easily than six, and the pause makes the read-back clear on a phone line. It confirms before a return, because a wrong order number would start a return on the wrong order. That is a wrong action: US$15 in Larkfield’s assumptions. Opening hours need no number and nothing can go wrong, so a confirmation would only waste the caller’s time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the 31 caller turns with an order number, Whisper base got 7 of 13 English numbers right as the IVR runs it, and 10 of 13 without the number prompt. What would you do, and what would you check first?
Show the answer
Try the number state without the prompt for English. But first check that the gap is real. It is one run on 13 turns, so measure on more calls: the hidden IVR calls, or new recordings. Keep everything else the same, and compare containment and wrong orders, not only the number count. French numbers are poor in every setting, so the keypad stays the main way in for French callers.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Turn the caller’s words into digits, then check the length, the check digit and the order system. On the test calls, every wrong number was caught: no caller heard another order.
- Recover in steps. First ask for the rest of a short number, then switch to the keypad. After the second miss, pass the call to a person with everything the IVR heard.
- Read back in groups and wait for a clear yes; check for a no first. Confirm before every action on an order.
- The number prompt did not help Whisper base with English numbers (7 of 13 right as the IVR runs it, 10 without the prompt, one run). French numbers were poor in every setting.
- The replay’s recorded callers cannot answer a new question, so a miss usually ends with a person.
You can now make the IVR safe with numbers and kind when it fails. But is it good? So far, we looked at a few calls at a time. Next, you replay every recorded test call as an automated test, measure the IVR by slice, and listen to its failures like a product team.