Skip to content
Français

A keypad menu first

  • Lesson 24
  • 40 min
  • Module 7: Build a small IVR

Where we are

We have a map of states and voices for the prompts. Now we build the simplest IVR that works. In this lesson, we make a keypad menu that reads keypad tones, handles silence and wrong keys, and always has a way to a person.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Build the first working version of the Larkfield IVR. It reads the tones of the phone’s keys, asks again after silence or a wrong key, and always lets the caller reach a person.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds a key detector with the Goertzel algorithm and reads the keys of the recorded test calls. Then it writes a small keypad menu as states, and runs the keypad-only test calls through it. Then you add the 0 key for a person. It downloads only a few short audio files. About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need

  • Your larkspeech folder with the voice files and the new requirements from Text-to-speech for prompts.
  • The Whisper base model (Module 4 downloaded it). The keypad needs no model, but the replay command loads it once.
  • Tested on macOS (Apple M4) with Python 3.12. The Windows commands are given but were not run.

Press a key on a phone and listen. You hear a short beep. It is not one note: it is two notes at the same time, like two piano keys pressed together. Each key has its own pair, so the pair tells you the key, even when you cannot see it.

This old trick is why a phone menu works over any phone line. The phone line was built to carry voice, so the keys were made to sound like voice: short tones in the middle of the voice range.

The Larkfield IVR receives the call as a stream of numbers, 8,000 a second. There is no “key 5 was pressed” message in a recorded call. The keys are in the audio, mixed with the line’s noise. Before the menu can react to “press 1 for orders”, it must find the keys in the sound.

Listen to a caller who types their order number on the keypad:

Test call I-017, turn 3, 1.82 seconds, on a VoIP phone line. Keypad tones made by our code (no voice).

Transcript: six keypad tones: 5, 8, 6, 0, 3, 2.

The two notes of each key are called DTMF, for dual-tone multi-frequency: two tones, chosen from a small set of frequencies. One tone comes from a group of four low frequencies (the row of the key), the other from a group of three high frequencies (the column):

1209 Hz 1336 Hz 1477 Hz
697 Hz 1 2 3
770 Hz 4 5 6
852 Hz 7 8 9
941 Hz * 0 #

Key 5 is 770 Hz plus 1336 Hz. Key 0 is 941 Hz plus 1336 Hz.

So the IVR does not need to understand all the sound. It only needs to measure how strong these seven frequencies are. The Goertzel algorithm does exactly that. It is a short loop that measures the strength of one frequency in a piece of audio. It is cheaper than a full spectrum when you need only a few frequencies. The name comes from Gerald Goertzel, who published it in 1958.

The detector cuts the audio into frames of 20 ms (160 samples at 8 kHz). For each frame, it measures the seven frequencies. Here is one frame from the middle of the first key of I-017, as the project’s dtmf.py measures it:

Frequency (Hz) 697 770 852 941 1209 1336 1477
Strength 23 539 24 6 13 555 2

The strongest low tone is 770 Hz and the strongest high tone is 1336 Hz: row 2, column 2, key 5. The detector also checks three rules before it accepts a key:

  1. The two tones carry most of the frame’s energy: at least 60%. Here they carry 98%. Speech spreads its energy over many frequencies, so it fails this rule.
  2. Each tone is at least 4 times stronger than the next one in its group. Here 770 Hz is about 22 times stronger than 852 Hz.
  3. The same key is in two frames in a row (40 ms). A click or a short sound in speech is shorter, so it does not count.
Top: the waveform of six key presses over 1.8 seconds, labelled 5, 8, 6, 0, 3 and 2, with one 20 ms frame of the first key marked in green. Bottom: a bar chart of the strength of seven frequencies in that frame, with the largest as 1: 697 Hz 0.04, 770 Hz 0.97, 852 Hz 0.04, 941 Hz 0.01, 1209 Hz 0.02, 1336 Hz 1.00, 1477 Hz 0.00.
Test call I-017, turn 3, as the IVR hears it. Top: six key presses on the phone line. Bottom: the strength of the seven DTMF frequencies in one 20 ms frame of the first key, measured with the Goertzel algorithm (project file larkspeech/ivr/dtmf.py).
Text description of the diagram

Top: the six key presses are six short blocks of sound, each about 0.1 seconds long, with short gaps between them. The detector read them as 5, 8, 6, 0, 3, 2. Bottom: in one frame of the first key, two bars stand out: 770 Hz (0.97) and 1336 Hz (1.00). The other five bars are 0.04 or less. 770 and 1336 Hz together are key 5.

The detector reads {dt.keys_read} from this turn, the six keys that the caller pressed.

The course’s test phone is a web page that you use at the end of this module. Its keypad on the screen sends a message, {"type": "dtmf", "key": "1"}, instead of a tone. Many phone providers do the same: they send keys as separate events next to the audio. The IVR treats both ways alike: a key is a key.

The menu is the map of states from the first lesson, written in larkspeech/ivr/dialogue.py. The dialogue knows nothing about audio: it gets events (keys, speech or no_input) and returns what to say. So you can test it with plain text, without any sound.

Each key has one meaning in the menu:

KEY_INTENT = {"1": "order_status", "2": "opening_hours", "3": "start_return", "0": "agent"}

Every event goes through one method, step. It checks the two rules that apply in every state first: 0 always reaches a person, and silence is a no-input:

def step(self, kind, value=None):
"""kind: speech (value = the recognised text), keys (value = digits), no_input. Returns actions."""
if self.state == "done":
return []
self.log.append({"state": self.state, "event": kind, "value": value})
if kind == "keys" and value and value[0] == "0" and self.state in ("language", "menu", "number", "confirm"):
self.intent = self.intent or "agent"
return self._transfer("caller_asked")
if kind == "no_input":
retry = {"language": "language_retry", "menu": "menu_retry", "number": "number_retry",
"confirm": "confirm_retry"}[self.state]
return self._miss("no_input", retry)
return getattr(self, f"_on_{self.state}")(kind, value)

The last line calls the method of the current state: _on_language, _on_menu, _on_number or _on_confirm. For keys, the menu state is one dictionary lookup:

def _on_menu(self, kind, value):
intent = KEY_INTENT.get(value[:1]) if kind == "keys" else find_intent(value or "", self.language)
if not intent:
return self._miss("no_match", "menu_retry")
...

The else part, for speech, is the next lesson.

When a state gets no key in time, or a key it does not know, it calls _miss:

def _miss(self, kind, retry_key, **kw):
"""A no-input or a no-match: ask again in a different way once, then hand over."""
self.misses += 1
self.log.append({"state": self.state, "miss": kind})
if self.misses >= 2:
return self._transfer("no_input" if kind == "no_input" else "no_match")
return [self._say(retry_key, **kw)]

The retry prompt is different from the first prompt. The menu says “Say order status, opening hours …”; its retry says “Sorry, I did not get that. Please use your keypad: press 1 for orders …”. When the state changes, the count of misses goes back to 0.

A flow chart. Play the prompt, then listen for keys. If no key comes within 5 seconds, it is a no-input miss. If a key comes: key 0 goes to A person; a key that the state knows goes to the next state; any other key is a no-match miss. After a miss: if it is the second miss in this state, A person takes over; otherwise ask again another way and listen again.
How every state of the keypad menu handles a key, a wrong key and silence.
Text description of the diagram

After the prompt, the state listens. A key within 5 seconds is checked. Key 0 goes straight to a person. A key that the state knows moves the call to the next state. Any other key is a no-match. No key in 5 seconds is a no-input. Both are misses. After the first miss, the state asks again in another way. After the second, a person takes over, and the call record keeps the reason.

Three times control the keypad:

  • No input: 5 seconds after the prompt ends.
  • Between digits: 3 seconds. A caller who stops typing in the middle of an order number has finished, and the digits so far are checked.
  • The number of keys: one key ends the entry in the language, menu and confirm states. In the order-number state, the entry ends after 6 keys, or with #.

These values are a choice, not a law. A caller who reads the number from a crumpled email needs more than 3 seconds between digits. The replay of the last lesson of this module measures what a change does.

A keypad menu is the safest IVR: a key is never misheard the way a word is. The course measured what it does alone. The IVR replayed all 60 public test calls and ignored every spoken word, as a keypad-only menu would:

Tests passed Calls finished without a person (containment)
Keypad only 23 of 60 10%

On Larkfield’s synthetic calls, most callers speak their answers, as many real callers do. A keypad-only menu sends most of them to a person: it is safe, and it does not help much. The notebook shows the other side. On the 7 test calls that use only the keypad, a menu without the 0 key passes 5, and with it, all 7. So the keypad is a strong fallback and a weak first choice. The next lesson adds speech.

Guided practice: the whole IVR in your project

Section titled “Guided practice: the whole IVR in your project”
  1. From the folder that holds larkspeech and nextia-learning-labs, copy the rest of the IVR from the end-of-Module-7 snapshot: the ivr folder, the speech ivr command, its tests and one document.

    macOS and Linux:

    Terminal window
    S=nextia-learning-labs/C41/snapshots/end-of-m07
    cp -R $S/larkspeech/ivr larkspeech/larkspeech/
    cp $S/larkspeech/commands/ivr.py larkspeech/larkspeech/commands/
    cp $S/tests/test_ivr.py larkspeech/tests/
    mkdir -p larkspeech/docs
    cp $S/docs/phone-line-options.md larkspeech/docs/

    Windows PowerShell:

    Terminal window
    $S = "nextia-learning-labs\C41\snapshots\end-of-m07"
    Copy-Item -Recurse -Force "$S\larkspeech\ivr" larkspeech\larkspeech\
    Copy-Item "$S\larkspeech\commands\ivr.py" larkspeech\larkspeech\commands\
    Copy-Item "$S\tests\test_ivr.py" larkspeech\tests\
    New-Item -ItemType Directory -Force larkspeech\docs
    Copy-Item "$S\docs\phone-line-options.md" larkspeech\docs\

    Your project is now the same as the snapshot end-of-m07, except its README.

  2. Go into larkspeech, activate the environment, and run the tests:

    Terminal window
    python -m pytest -q
    ................................. [100%]
    33 passed in 1.69s

    The six new tests in tests/test_ivr.py talk to the dialogue with text and keys. They make keypad tones with dtmf.tones() and read them back, and they replay a fake call with a fake recogniser. They need no download.

  3. Read the options of the replay command:

    Terminal window
    speech ivr replay --help
    usage: speech ivr replay [-h] [--model MODEL] [--end-of-turn END_OF_TURN]
    [--by {kind,line,noise,language}] [--calls CALLS]
    [--voice]
    ...
    --calls CALLS only these call IDs (comma-separated)
    --voice time the prompts with the real voice (slower)
  4. Replay four test calls in which the caller uses the keypad: a status by keypad (I-017), opening hours by keypad (I-033, I-034) and a caller who presses 0 (I-062).

    Terminal window
    speech ivr replay --calls I-017,I-033,I-034,I-062
    IVR replay: 4 recorded test calls, Whisper base, end of turn 0.8 s
    calls passed contained wrong intent to a person s to answer
    all 4 4 75% 0% 25% 44.6
    Wrote outputs/ivr-replay.jsonl (one call record per line)

    All four pass. Three are contained. I-062 is passed to a person, which is what that caller wanted, so it passes too.

  5. Look at the call record of I-017:

    Terminal window
    python -c "import json; r = [json.loads(x) for x in open('outputs/ivr-replay.jsonl')][0]; print(r['outcome'], r['heard']); print(r['prompts'])"
    status_given ['1', '1', '586032', '1']
    ['welcome', 'menu', 'menu_retry', 'number', 'number_retry', 'confirm', 'confirm_retry', 'status']

    The IVR heard the right keys and gave the status. But look at the prompts: after each key press, a retry prompt follows, as if the caller had said nothing. A real caller would hear “Sorry, I did not get that” three times. Keep this in mind: the last lesson of this module finds where it comes from.

Predict: in the menu state, a caller presses 7, then 7 again. What does the IVR say, and how does the call end? Check your prediction with the dialogue alone, no audio:

Terminal window
python -c "from larkspeech import data as D; from larkspeech.paths import data_dir; from larkspeech.ivr.dialogue import Dialogue; d = Dialogue(D.orders(data_dir(None))); d.start(); print(d.step('keys', '1')[0]['key']); print(d.step('keys', '7')[0]['key']); print(d.step('keys', '7')); print(d.record()['outcome'], d.record()['handoff_reason'])"
Check your answer
menu
menu_retry
[{'say': 'I am putting you through to a person now. They will see what you told me.', 'key': 'handover'}, {'transfer': 'no_match'}]
handover no_match

Key 1 chooses English, and the IVR plays the menu. The first 7 is a no-match: the IVR plays the menu retry. The second 7 is the second miss in the menu state. The IVR says that it is putting the caller through, and transfers the call with the reason no_match. The call record keeps that reason, so Grace can count how many callers the menu failed.

Common mistake

Keys that are too short

Symptom: callers with one kind of phone say “I pressed 1, but nothing happened”. Other callers have no problem.

Cause: some phones and lines send very short tones. The detector counts a key only when it is in two 20 ms frames in a row. In the notebook, tones of 40 ms are read and tones of 20 ms are not: a 20 ms tone never fills two whole frames.

Fix: test the detector with tones of the shortest length your phone lines send (the DTMF standard asks for at least 40 ms). If you must accept shorter ones, use shorter frames, and test again that speech does not turn into keys. Where the provider sends keys as separate events, use them.

Consequence in AI work: every detector has a smallest signal it can see. Find it with tests on purpose, before callers find it for you.

Check your understanding

In one 20 ms frame, the strongest low tone is 852 Hz and the strongest high tone is 1477 Hz, each much stronger than the others, and the same in the next frame. Which key is it?

Show the answer

9: 852 Hz is the third row (7, 8, 9) and 1477 Hz is the third column (3, 6, 9). The three rules pass: the two tones carry most of the energy, each one is much stronger than the others in its group, and they last two frames. So the detector reads key 9.

Check your understanding

Why does the dialogue in dialogue.py work with events (keys, speech, no_input) and not with audio?

Show the answer

So that the logic can be tested alone, with text and keys, quickly and without any model or sound. The audio part (finding keys, finding the end of a turn, recognising words) lives in engine.py. Each part can then be tested and changed without the other, and the same dialogue serves the test phone, the replay and the unit tests.

Check your understanding

With speech ignored, the IVR passed 23 of the 60 test calls and contained 10%. What does this tell Larkfield about a keypad-only menu?

Show the answer

It is safe but not enough. Most test callers speak, so a keypad-only menu passes most of them to a person. The keypad is a good fallback: for noisy lines, for numbers, and for callers who prefer it. The way to a person (0) must work in every state. To contain more calls, the IVR must understand speech too.

Recap

  • A keypad tone (DTMF) is two tones at once, one of four low and one of three high frequencies.
  • The Goertzel algorithm measures only those frequencies, frame by frame. A key counts when its two tones dominate a 20 ms frame, in two frames in a row.
  • The dialogue is a state machine that gets events (keys, speech, no-input) and returns what to say. It can be tested with text alone.
  • Every state handles a no-input (5 s of silence) and a no-match (a wrong key): ask again in another way once, then a person. 0 reaches a person everywhere.
  • On Larkfield’s synthetic test calls, a keypad-only menu contained 10%: a strong fallback, a weak first choice.

You can now build a keypad menu that reads keys from the audio and never traps a caller. But most callers would rather say what they need than wait for “press 3”. Next, you let them speak: you find the end of their turn, recognise their words, and turn the words into an intent.

Next: Let callers say what they need