Skip to content
Français

Numbers, confirmation and recovery

  • Lesson 26
  • 40 min
  • Module 7: Build a small IVR

Where we are

Our IVR now understands what callers want. Order numbers are still hard: callers pause, say "oh" for zero, and the line adds noise. In this lesson, we get the number right, confirm it, ask again after a miss, and hand over after two.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 26 of 36 · Speech AI: Transcribe, Understand and SpeakNumbers, confirmation and recovery

What you will be able to do

Make the Larkfield IVR safe with order numbers. It must never act on a misheard number, it reads the number back before it acts, and it recovers from a miss without trapping the caller.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook turns the recorded transcripts of the callers’ order numbers into digits. It checks each one (length, check digit, order system), and counts the wrong numbers that pass. Then you write the read-back in groups and a yes-or-no rule. It downloads only small text files and runs no model. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need

  • Your larkspeech folder at the end-of-Module-7 snapshot, with Whisper base.
  • Tested on macOS (Apple M4) with Python 3.12. The Windows commands are the same in PowerShell, but were not run.

You order a pizza by phone. At the end, the person asks for your number. You say “oh-seven-seven, nine-oh-oh, one-two-three-four”. The line is bad and the shop is loud. A good shop does not just write something down. They read it back: “zero seven seven, nine zero zero, one two three four?” You say “yes”. If they got a digit wrong, you hear it, and you correct it.

Three habits make this work. Read it back in groups. Wait for a clear yes. And when something is wrong, ask again in another way (“can you text it to us?”). The IVR needs the same three habits. A wrong order number is worse than no answer, because the caller hears someone else’s order.

The problem: the spoken order number fails most often

Section titled “The problem: the spoken order number fails most often”

On the 60 recorded test calls, the IVR passed 2 of the 12 calls that ask for an order status by speech. It passed 1 of 4 calls where the caller pauses inside the number. It passed 1 of 3 where the caller first says a wrong number, then fixes it. Here are the 12 status-by-speech calls, replayed through the end-of-Module-7 snapshot:

Call Language The caller said The IVR heard Digits found, and the check that failed Outcome
I-001 en eleven, forty-nine, forty-two (114942) “114940.” 114940: check digit wrong a person
I-002 fr un quatre cinq huit huit huit (145888) “Un quatre cinq, 8, 8, 8.” 145888: all checks pass status given
I-003 en five two two nine one two (522912) “Two two nine one two.” 22912: too short a person
I-004 en thirty-nine, eleven, sixty-nine (391169) “Third 9, 11, 69.” 91169: too short a person
I-005 fr un trois six un sept six (136176) “Fortsis, c’est…” no digits a person
I-009 en four seven nine nine zero seven (479907) “479907.” 479907: all checks pass status given
I-010 en three oh three three six two (303362) “Three, both three, three.” 33: too short a person
I-012 fr un sept un quatre un trois (171413) “Un site un 413.” 1413: too short a person
I-006, I-007 en, fr (never asked: the menu answer was not understood) a person
I-008, I-011 fr (never asked: “Français” was not understood) a person

Read the table twice. First, the bad news: the IVR understood only 2 of the 8 order numbers it asked for. Second, the good news: every wrong number was caught. No caller heard the status of someone else’s order. The checks did their job: the IVR is safe, but it is not yet helpful.

The order numbers are counted per caller turn. The IVR test calls hold only the caller’s side, so nothing else can hide an error. On whole conversations, in Module 2, the agent’s read-back could hide a caller’s wrong digit: count the caller’s turns alone.

Idea 1: from words to digits, then three checks

Section titled “Idea 1: from words to digits, then three checks”

The IVR uses the function from Module 4, find_order_number. It turns the caller’s words into digits with the normalisation guide (“thirty-nine” → 39, “oh” between digits → 0). Then it keeps the longest run of digits, and checks it:

Terminal window
python -c "from larkspeech.numbers import find_order_number as f; print(f('Two two nine one two.')); print(f('114940.')); print(f('three oh three three six two')); print(f('Three, both three, three.'))"
('22912', 'too short')
('114940', 'check digit wrong')
('303362', None)
('33', 'too short')

“three oh three three six two” is a valid order number: “oh” became 0. But Whisper wrote “Three, both three, three.”, so the IVR found only “33”. No rule can bring back digits that are not in the text.

Then the IVR looks the number up in the order system. A six-digit number with a right check digit can still be an order that does not exist. Only a number that passes all three checks goes to the next step.

Idea 2: ask for the rest, then another way

Section titled “Idea 2: ask for the rest, then another way”

A failed check is not the end. The IVR has two ways to recover, in this order.

Ask for the rest. When the caller paused in the middle of the number, the IVR heard only the first part. This happens the first time it finds from 2 to 5 digits. It reads back what it heard and asks for the rest: “I heard 7 9 8. Please say the rest of the number.” Then it joins the two parts.

Ask again, in another way. When the number fails a check, the IVR does not ask the same question again. It switches to the keypad: “That number does not look right. Please type it on your keypad, one digit at a time.” A key is never misheard. When the number is not in the order system: “I cannot find that order. Please type the number on your keypad.”

A flow chart. Ask: please say or type your six-digit order number. Heard: words to digits. Fewer than 6 digits, the first time: ask for the rest, I heard 7 4 2, please say the rest, and listen again. Otherwise: check digit and order system OK? Yes: read it back, Order 7 4 2, 1 5 5, is that right? No, or the caller says no to the read-back: ask again another way, please type it on your keypad. A second miss: a person takes over, with what the IVR heard.
How the IVR's order-number state recovers. Each failure leads to a different question; the second miss leads to a person.
Text description of the diagram

The IVR asks for the number, turns the words into digits and checks them. A number that is too short, the first time, leads to “I heard 7 4 2. Please say the rest.” A number that fails the check digit or the order system, or that the caller rejects at the read-back, leads to “Please type it on your keypad”. A second miss leads to a person, who sees what the IVR heard.

A tiny worked example: a number in two parts

Section titled “A tiny worked example: a number in two parts”

Here is the dialogue alone, with no audio, on the words of test call I-049. The caller says the order number in two parts:

Terminal window
python -c "from larkspeech import data as D; from larkspeech.paths import data_dir; from larkspeech.ivr.dialogue import Dialogue; d = Dialogue(D.orders(data_dir(None))); d.start(); [print(a['say']) for e in [('speech', 'English.'), ('speech', 'Where is my order?'), ('speech', 'Seven nine eight.'), ('speech', 'Zero nine zero.'), ('speech', 'Correct.')] for a in d.step(*e) if 'say' in a]"
How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person.
Please say or type your six-digit order number.
I heard 7 9 8. Please say the rest of the number.
Order 7 9 8, 0 9 0. Is that right? Say yes or no, or press 1 for yes and 2 for no.
Your Mistral fan shipped on October 11 and is on its way. Thank you for calling Larkfield. Goodbye.

Step by step: “Seven nine eight.” gives 798, which is too short, so the IVR asks for the rest. “Zero nine zero.” gives 090. Joined, 798090 has 6 digits, a right check digit, and it is in the order system. So the IVR reads it back in groups. “Correct.” is a yes, and the IVR gives the status.

The read-back is the IVR’s last safety check. It says the number in groups of three, “Order 7 9 8, 0 9 0”, because a caller checks three digits at a time more easily than six. Then it waits for a clear answer.

The yes-or-no rule is a small list of words, in both languages:

YES = {"yes", "yeah", "yep", "correct", "right", "that's right", "oui", "exact", "exactement", "c'est ça", "ouais"}
NO = {"no", "nope", "wrong", "not", "incorrect", "non", "pas", "faux"}

The rule checks for a no first: “That’s not right” contains “right”, but it is a no. A “no” takes the caller back to the number, and counts as a miss. No answer, or an answer that is neither yes nor no, asks again: “Please say yes or no, or press 1 for yes and 2 for no.”

Is the read-back worth its 8 seconds? Larkfield counts a wrong action at US$15 and a call to a person at US$4 (both assumptions). For opening hours, no read-back is needed: there is nothing to confirm. For a return, a wrong order number would start a return on the wrong order. So the IVR confirms before every action on an order, and never before a simple answer.

Idea 4: two misses, then a person, with what you know

Section titled “Idea 4: two misses, then a person, with what you know”

After the second miss in the same state, the IVR stops. It says “I am putting you through to a person now. They will see what you told me.” and transfers the call with a reason: no_match, no_input, caller_asked or not_returnable.

The person who answers must not ask everything again. So the call record goes with the call: the language, the intent, and everything the IVR heard. Here is the record when a caller says a wrong number, then types a wrong number too:

{'language': 'en', 'intent': 'order_status', 'order': None, 'outcome': 'handover', 'handoff_reason': 'no_match',
'heard': ['English.', 'Where is my order?', 'seven nine eight zero nine one', '798091'],
'prompts': ['welcome', 'menu', 'number', 'number_retry', 'handover'], 'misses': 2}

The person sees at once: English, an order status, and two tries at a number close to 798090. That person can ask “Is your order 7 9 8, 0 9 0?” instead of “How can I help?”

The last lesson gave Whisper a short prompt in each state. In the number state, it is “My order number is 742155. Seven four two, one five five.” It fixed the language question. Does it help with numbers too?

We measured it. We took every caller turn of the 60 public test calls that says an order number: 31 turns, 13 in English and 18 in French. Whisper base wrote each one four ways. It heard either the whole recorded turn, or the turn as the IVR cuts it (with the VAD and the end of turn). And it worked either without or with the number prompt.

A bar chart of order numbers right, for English (13 turns) and French (18 turns), with four settings. Whole turn, no prompt: English 11, French 2. Whole turn, number prompt: 10 and 3. Cut by the IVR, no prompt: 10 and 1. Cut by the IVR, number prompt, which is the IVR today: 7 and 2.
Whisper base on the 31 caller turns that say an order number, in the 60 public IVR test calls: the order numbers it got right, four ways (one run).
Text description of the diagram

English: on the whole turn without a prompt, Whisper base got 11 of 13 numbers right; with the number prompt, 10. On the turn as the IVR cuts it, 10 without the prompt and 7 with it, which is how the IVR runs today. French: 2, 3, 1 and 2 of 18.

Two lessons come out of it:

  • In English, the number prompt does not help Whisper base. On the turns as the IVR cuts them, it got 7 of 13 right with the prompt and 10 without it. In I-003, the caller said “five two two nine one two”. Without the prompt, Whisper base wrote “5, 2, 2, 9, 1, 2.”; with it, “Two two nine one two.” The first digit was lost.
  • French numbers are hard for Whisper base in every setting: 1 to 3 of 18. The voices’ French and the small model both play a part. Here, the keypad is the real way in.

This is one run on 31 turns, so a difference of one or two turns is noise. A gap of 7 against 10 or 11 in English is worth a test with more calls. The IVR’s settings stay as they are in this module, so that the numbers match from lesson to lesson. Removing the number prompt for English is a good change to try in your own copy.

  1. In larkspeech, with the environment active, replay five calls with spoken order numbers: I-001, I-003 and I-010 (status by speech), I-049 (a pause) and I-054 (a wrong number, then fixed).

    Terminal window
    speech ivr replay --calls I-001,I-003,I-010,I-049,I-054
    IVR replay: 5 recorded test calls, Whisper base, end of turn 0.8 s
    calls passed contained wrong intent to a person s to answer
    all 5 2 40% 0% 60% 61.0
    Failed: I-001 (status_speech: handover, expected status_given), I-003 (status_speech: handover, expected status_given), I-010 (status_speech: handover, expected status_given)
    Wrote outputs/ivr-replay.jsonl (one call record per line)
  2. Read the record of I-003, the second call:

    Terminal window
    python -c "import json; r = [json.loads(x) for x in open('outputs/ivr-replay.jsonl')][1]; print(r['call_id'], r['outcome'], r['handoff_reason']); print(r['heard']); print(r['prompts'])"
    I-003 handover no_input
    ['English.', 'I want to track my order.', 'Two two nine one two.', 'Yes.']
    ['welcome', 'menu', 'number', 'number_rest', 'number_retry', 'handover']

    The IVR heard “Two two nine one two.”, found 5 digits, and asked for the rest. The recorded caller then said “Yes.” That is their next recorded line, the answer to a read-back that never came. A real caller would have said the number again. The replay cannot react, and this lowers the IVR’s score against a real caller.

  3. Check that the checks are safe: in outputs/ivr-replay.jsonl, no call has an order that differs from the expected one.

    Terminal window
    python -c "import json; rows = [json.loads(x) for x in open('outputs/ivr-replay.jsonl')]; print([(r['call_id'], r['order'], r['expected'].get('order')) for r in rows])"
    [('I-001', None, '114942'), ('I-003', None, '522912'), ('I-010', None, '303362'), ('I-049', '798090', '798090'), ('I-054', '603134', '603134')]

    Every order is either the expected number or None: the IVR never kept a wrong number.

Predict: a caller says “seven nine eight zero nine one” (one digit wrong), then types 798091 on the keypad. What does the IVR say after each, and how does the call end? Use the dialogue alone, as in the worked example, with ('keys', '798091') for the keypad.

Check your answer
Terminal window
python -c "from larkspeech import data as D; from larkspeech.paths import data_dir; from larkspeech.ivr.dialogue import Dialogue; d = Dialogue(D.orders(data_dir(None))); d.start(); [print(a.get('say') or a) for e in [('speech', 'English.'), ('speech', 'Where is my order?'), ('speech', 'seven nine eight zero nine one'), ('keys', '798091')] for a in d.step(*e)]; print(d.record())"
How can I help? Say order status, opening hours, or start a return. You can also press 1 for orders, 2 for opening hours, 3 for returns, or 0 for a person.
Please say or type your six-digit order number.
That number does not look right. Please type it on your keypad, one digit at a time.
I am putting you through to a person now. They will see what you told me.
{'transfer': 'no_match'}
{'call_id': None, 'language': 'en', 'intent': 'order_status', 'order': None, 'outcome': 'handover', 'handoff_reason': 'no_match', 'heard': ['English.', 'Where is my order?', 'seven nine eight zero nine one', '798091'], 'prompts': ['welcome', 'menu', 'number', 'number_retry', 'handover'], 'misses': 2}

798091 fails the check digit, so the first try is a miss and the IVR switches to the keypad. The typed number fails again: a second miss, and a person takes over with everything the caller said. The wrong order was never read out.

Common mistake

A read-back that says yes for the caller

Symptom: while writing this lesson, we tried the yes-or-no rule on a few answers a caller might give. “That’s not right.” came back as yes. So did “Not correct.”

Cause: the first version of the rule had only “no”, “nope”, “wrong”, “non” and “pas” as no-words. “That’s not right” has none of them, but it has “right”, which is a yes-word. The IVR would have read out the status of an order that the caller had just rejected.

Fix: the snapshot’s rule now has “not”, “incorrect” and “faux” as no-words too, and it still checks for a no before a yes. A unit test in tests/test_ivr.py covers both languages. None of the 60 test calls said “not right”, which is why the replay never found it: test the safety step with answers written on purpose, including negations in both languages.

Consequence in AI work: a confirmation step protects you only if the confirmation itself is understood correctly. Test the safety step as carefully as the main step.

Check your understanding

The IVR hears "Seven nine one four seven" for order 791475. What does it do, and why does it not look the number up?

Show the answer

It finds 79147: 5 digits, too short. It is the first try with a partial number, so it says “I heard 7 9 1, 4 7. Please say the rest of the number.” It does not look up a number that failed the length check, because a wrong lookup could find, and read out, another caller’s order.

Check your understanding

Why does the IVR read the order number back in groups, "Order 7 9 8, 0 9 0", and why does it confirm before a return but not before the opening hours?

Show the answer

People check three digits at a time more easily than six, and the pause makes the read-back clear on a phone line. It confirms before a return, because a wrong order number would start a return on the wrong order. That is a wrong action: US$15 in Larkfield’s assumptions. Opening hours need no number and nothing can go wrong, so a confirmation would only waste the caller’s time.

Check your understanding

On the 31 caller turns with an order number, Whisper base got 7 of 13 English numbers right as the IVR runs it, and 10 of 13 without the number prompt. What would you do, and what would you check first?

Show the answer

Try the number state without the prompt for English. But first check that the gap is real. It is one run on 13 turns, so measure on more calls: the hidden IVR calls, or new recordings. Keep everything else the same, and compare containment and wrong orders, not only the number count. French numbers are poor in every setting, so the keypad stays the main way in for French callers.

Recap

  • Turn the caller’s words into digits, then check the length, the check digit and the order system. On the test calls, every wrong number was caught: no caller heard another order.
  • Recover in steps. First ask for the rest of a short number, then switch to the keypad. After the second miss, pass the call to a person with everything the IVR heard.
  • Read back in groups and wait for a clear yes; check for a no first. Confirm before every action on an order.
  • The number prompt did not help Whisper base with English numbers (7 of 13 right as the IVR runs it, 10 without the prompt, one run). French numbers were poor in every setting.
  • The replay’s recorded callers cannot answer a new question, so a miss usually ends with a person.

You can now make the IVR safe with numbers and kind when it fails. But is it good? So far, we looked at a few calls at a time. Next, you replay every recorded test call as an automated test, measure the IVR by slice, and listen to its failures like a product team.

Next: Test the IVR like a product