What Larkfield wants from its calls
- Lesson 1
- 25 min
- Module 1: The speech problem and its cost
Where we are
In the machine learning course, every example was a row in a table. Larkfield's help desk also has 4,000 phone calls a month, and nobody can search them. In this lesson, we list the decisions Larkfield wants from its calls and what each kind of mistake costs. First, think of an agent who takes notes during a call.
Lesson 1 of 36 · Speech AI: Transcribe, Understand and SpeakWhat Larkfield wants from its calls
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Turn “we have thousands of recorded calls” into a short list of decisions, each with the cost of a wrong answer. Without that list, a team transcribes everything and still cannot answer the questions that matter.
Before you startWhat to know and to have ready
What you need to know
- What a test set is, and precision and recall, from Machine Learning: From Problem to Reliable Model. This lesson only names them; later lessons use them.
- Nothing about audio. The next lesson explains sound from the start.
What you need on your computer
- Nothing for this lesson. You set up the course project in Set up the speech project, two lessons from now.
An agent with a notepad
Section titled “An agent with a notepad”Think of a help-desk agent on a busy morning. A customer calls. Her garden lights have not arrived. The agent finds the order, says when it shipped, and makes a promise: “I will ask the courier to call you before tomorrow’s delivery.” Then the next call comes in, and the next.
The agent writes a short note after each call, when there is time. Some notes say “lights, not arrived”. Some notes are empty, because the queue was long. The promise about the courier is not in any note.
The next week, the support lead asks two questions. Why did people call last week? Which promises did we make that nobody has kept? Nobody can answer from the notes. The answers are in the calls, but the calls are sound, not text.
Larkfield’s calls
Section titled “Larkfield’s calls”Larkfield is an online shop for home and garden products. Its help desk takes about 4,000 calls a month, in English and in French. About 900 of them come out of hours, when no agent is at work. Every call is recorded, and the caller hears a notice that says so. So the shop has thousands of recordings, and nobody can search them.
Grace leads the support team. She wants to know why people call, which promises agents made, and which calls need a follow-up. She also has a budget for listening: her team leads can spend 60 minutes a day on calls. At one minute a call, that is 60 calls a day. The help desk gets about 130 calls a day (4,000 in a month of 30 days). Most calls are never heard again.
Listen to a call
Section titled “Listen to a call”This is a part of one Larkfield call. Press play: it does not start by itself.
Part of call C-0127, as the phone line recorded it (8,000 samples a second). Both voices are synthetic: Piper voices trained on the LibriTTS-R corpus (the agent, Chloe, is speaker 5401; the caller is speaker 7169). No real person speaks in Larkfield’s calls.
| Time | Who | What they said |
|---|---|---|
| 5.9 s | Caller | Hi, um, good morning. My Pebble path lights haven’t arrived. Please tell me where they are. |
| 9.3 s | Agent | I can help with that. Can I have your order number, please? |
| 12.0 s | Caller | Yes, it’s 114942. |
| 14.0 s | Agent | Thank you. That’s order 114 942. It shipped on the twelfth of March, and it is on its way. |
| 19.6 s | Agent | I will ask the courier to call you before tomorrow’s delivery. |
You understood this call at once. You know why the caller called, and you heard the promise. A computer gets only a list of numbers that describe the sound. To get the same answers, it must turn the sound into words, and the words into a decision.
What goes wrong without it
Section titled “What goes wrong without it”Look at the last line of the call. If nobody acts on that promise, the courier does not call. The customer waits at home, calls again, and is angry. Larkfield counts that as a missed follow-up, and it is one of the most expensive mistakes in this course.
Now multiply. In our scripted calls, every call ends with a promise, because the script says so. Real calls hold fewer. But even if only 1 call in 10 held a promise, 4,000 calls would hold 400 promises a month. Grace’s team leads cannot check them by listening, and nobody else can find them. This is the problem of the course: the information is in the calls, and nobody can use it.
From a call to a record
Section titled “From a call to a record”The idea is simple. For each call, a program writes the agent’s note, the same note every time. We call it a call record: a short, fixed set of fields that says what happened in a call. People can search it, count it and check it.
Here is the record for the call above. These are the labels that we wrote with the call’s script, so they are the right answers.
| Field | Value for call C-0127 | Who uses it |
|---|---|---|
| Transcript | every word, with who said it and when | anyone who searches the calls |
| Reason | order status | Grace: why people call |
| Promise | “I will ask the courier to call you before tomorrow’s delivery.” | Grace: what we must do |
| Next step | courier call | the team that acts on it |
| Recording notice | said, in the exact words | the compliance check |
| Language, line | English, landline | Priya: where the errors are |
A program that writes this record must do several things in order. It must find the parts of the recording where someone speaks. It must turn the speech into text. It must know which words the agent said and which the caller said. Then it must read the text and fill the fields. Speech recognition is the step from sound to text. It is also called ASR, from automatic speech recognition. The step from text to fields is often called speech analytics.
The second want: a line that answers by itself
Section titled “The second want: a line that answers by itself”The 900 out-of-hours calls are a different problem. Today, those callers hear a message and call again the next day. Many of their questions are simple: “Where is my order?”, “When are you open?” A machine could answer them.
Larkfield wants this in two steps. The first step is a small IVR, from interactive voice response. It is a phone menu that understands keypad tones and short spoken answers: “Say what you need, or press 1 for orders.” The second step is a voice agent: a program that listens, answers in full sentences and uses tools. It is for the questions that a menu cannot handle. In this course, you build the IVR and call it. Then you compare it with a voice agent on the same recorded calls.
The diagram follows one call from the ring to the record. Each step is a module of this course.
Text description of the diagram
A call starts when the phone rings. In open hours, a person answers, and the phone system records the agent and the caller on two separate channels. Out of hours, the phone line answers by itself: first a phone menu, then a voice agent (Modules 7 and 8). From there, both kinds of call go through the same steps. The program finds the parts with speech (Module 3). It turns the speech into text and measures how good the text is (Modules 2 and 4). It finds who said each word (Module 5). It reads the text and finds the reason, the promise and the recording notice (Module 6). The result is a call record, with card numbers removed (Module 9). Grace’s team uses the records for follow-ups, checks and reports.
What each mistake costs
Section titled “What each mistake costs”Every step in the diagram can make mistakes. A transcript can miss words. The reason can be wrong. A phone menu can start the wrong action. These mistakes do not cost the same, so the course gives each one a price. The prices are assumptions for the course, not measured numbers. A real team would ask its finance people. Larkfield pays its staff in Canadian dollars, but the course gives every cost in US dollars, the currency of the model and cloud prices that it compares them with.
| Mistake | What happens | Cost (assumption) |
|---|---|---|
| A person handles a call that a machine could handle | 6 minutes of an agent’s time at US$40 an hour | US$4.00 a call |
| The caller hangs up | no answer; the caller tries again, or buys elsewhere | US$8 |
| A machine does the wrong thing | a return started by mistake, or a wrong order status: a second call, a correction and some goodwill | US$15 |
| A missed follow-up | a promise in a call that nobody acts on: a complaint, a second call, sometimes a lost customer | US$25 |
| A team lead listens to a call | US$45 an hour, at most 60 minutes a day | US$0.75 a minute |
A tiny worked example
Section titled “A tiny worked example”Omar, the product manager, owns the out-of-hours line. He asks: “Should the phone line answer as many calls as it can?” Look at the costs in the table before you answer.
Suppose the line answers 100 calls by itself. If all 100 answers are right, Larkfield saves the agent’s time on 100 calls: 100 × US$4.00 = US$400. Now suppose that 10 of the 100 answers are wrong. Those 10 calls do not save US$4.00 each. Each one costs US$15 instead. The line then saves 90 × US$4.00 − 10 × US$15 = US$360 − US$150 = US$210, about half.
A line that answers only the calls it understands, and hands the others to a person, does better. Suppose it answers 70 calls, all right, and hands 30 to a person the next morning. It saves 70 × US$4.00 = US$280, and nobody gets a wrong answer. So the goal is not “answer every call”. The goal is “answer right, and hand over when unsure”. You meet this rule again when you build the IVR.
Who works on this
Section titled “Who works on this”The course follows a small team at Larkfield. You know some of them from the machine learning course.
| Person | Role | What they care about |
|---|---|---|
| Priya | Data scientist | The main guide. She builds the test set, compares the methods and measures them. |
| Grace | Support lead | The cost of a missed reason or follow-up, and her 60 minutes of listening a day. She reads the call records. |
| Amira | Developer | The IVR app: its steps, its prompts, its tests and its call records. |
| Malik | Telephony engineer (new) | Phone lines, sound at 8,000 samples a second, codecs and keypad tones. He knows why a line sounds bad. |
| Tomás | AI engineer | He wants the biggest model for everything. His questions are common mistakes, and they are good questions. |
| Omar | Product manager | The out-of-hours line, the new French region, and what callers feel. |
Synthetic calls, and why
Section titled “Synthetic calls, and why”Every call in this course is synthetic. Our code wrote the scripts. A language model wrote the callers’ own sentences about why they call. Then open text-to-speech voices read the scripts aloud. No real customer, no real call and no real person’s voice is in the data.
We never use real customer calls for practice. Real calls hold voices, names and sometimes card numbers. The people in them agreed to a recording for quality and training, not to a course. A voice is personal data, and you can recognise a person by it.
Synthetic calls also have a cost: they are easier than real speech. The voices are clean and regular. Nobody coughs, nobody talks over the other person, and the accents are the ones the voices learned. You see the effect in Module 4. Whisper is a free speech recognition model from OpenAI, and the course uses it most; Whisper small is one of its sizes. On Larkfield’s synthetic calls, Whisper small gets 3.1% of the English words wrong. On real people reading sentences aloud, from the open LibriSpeech recordings, the same model gets 7.8% wrong. So the course always says “on Larkfield’s synthetic calls” next to a number. The case studies in Module 10 use real recordings.
The call set in numbers
Section titled “The call set in numbers”The course has 340 synthetic conversations between a Larkfield agent and a caller:
- 260 public calls, which you get: 150 for training (their transcripts and labels only), 50 for validation and 60 for testing (with audio).
- 40 hidden test calls, which the course team keeps. They give one honest score at the end.
- 40 calls for the final assignment, in French, from a new region.
Of the 260 public calls, 175 are in English and 85 in French. A call lasts 43.9 seconds on average. There are also 100 test calls for the IVR (60 of them public): only the caller’s side, with speech, keypad tones and silence.
The calls have seven reasons. These are their counts in the 260 public calls:
| Order status | Return or refund | Billing | Damaged item | Product question | Cancel an order | Account change |
|---|---|---|---|---|---|---|
| 70 | 48 | 35 | 33 | 30 | 26 | 18 |
The course map
Section titled “The course map”The course follows the call record from sound to decision, and then turns around: the machine starts to speak.
Text description of the diagram
The course has four big steps after this module. Recognise (Modules 2 to 5): you measure transcripts, find the speech, turn speech into text and find who spoke when. Understand (Module 6): you turn transcripts into the call record. Speak (Modules 7 and 8): you build a phone menu that listens and answers, then a voice agent, and you compare them. Run (Module 9): you run it all in batches and live, remove personal data and watch the system. The case studies of Module 10 repeat the work on open recordings of real people.
The course ends with four case studies on open recordings of real people. Each one takes a real dataset from the question to a recommendation. You need one of them for the certificate.
Case studies in this course
At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.
- Choose a recogniser for two languagesCompare Vosk and Whisper sizes on real English and French speech from an open dataset by speaker group, and recommend one with error, speed and cost.
- Meeting transcripts with speakersDiarize and transcribe real recorded meetings from an open corpus, measure DER and WER, and make action-item notes.
- Spot commands on a small deviceBuild a keyword spotter on an open spoken-commands set, compare a small CNN on spectrograms with Whisper tiny, and measure false alarms per hour.
- An IVR for spoken intentsBuild a menu-and-speech IVR for a bank's common requests, test it with open spoken-intent recordings by speaker group, and compare keyword rules, a classifier and an LLM for the intent step.
The case studies use the whole course, so most learners start them after Module 9.
When speech AI is worth it, and when not
Section titled “When speech AI is worth it, and when not”Speech AI is worth it when there are too many calls to hear, and a decision depends on what people said. Larkfield’s 4,000 calls a month and 60 minutes of listening a day are such a case.
It is not worth it everywhere. A shop with 30 calls a month can listen to all of them. A question that the phone system already answers, such as “how many calls came in on Monday?”, needs no transcript. And a decision that must be right every time, such as a refund to a card, should not rest on a transcript alone. Recognisers make mistakes, and a person or a keypad must confirm.
It also has costs. Computers must run the models, someone must measure them, and recordings of voices must be kept safe and deleted on time. Module 9 puts a price on each.
Common mistake
Transcribe everything first, decide later
Symptom: Tomás says: “Whisper is solved. Let’s run the biggest model on every call, store the text, and then see what we can do with it.” A month later, there are 4,000 transcripts. Grace asks which promises were not kept. Nobody can answer. The transcripts are long, nobody reads them, and nobody knows how many words in them are wrong.
Cause: the team chose the tool before the decision. Without the list of decisions, there is no call record to fill, no test to run and no way to know which errors matter. A transcript that gets an order number wrong is useless for an order-status check. The same mistake in a greeting changes nothing.
Fix: start from the decisions and their costs, as in this lesson. For each decision, write which fields the call record needs and what a wrong field costs. Then choose and measure the methods against those fields, on the same calls.
Consequence in AI work: the team pays for the biggest model on every call, and it may not need it. In Module 4 you measure several sizes of the same model on the same calls, and a small one is often enough.
Check your understanding
Name three decisions that Larkfield wants from its recorded calls. For each one, say which field of the call record it needs.
Show the answer
- Why do people call? It needs the reason of each call. Grace counts the reasons to see what goes wrong in the shop.
- Which promises must we keep? It needs the promise and the next step. A promise that nobody acts on is a missed follow-up.
- Did the agent say the recording notice? It needs a notice field: said exactly, said in other words, or missing. This is a compliance check.
Other good answers: which calls need a follow-up (the next step), and where the transcripts are weak (the language and the line).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict and explain: the out-of-hours line answers 100 calls. In design A, it answers all 100 and 10 answers are wrong. In design B, it answers 70, all right, and hands 30 to a person. With the course's cost assumptions, which design saves more, and why?
Show the answer
Design B saves more. Design A saves 90 × US$4.00 = US$360 on the right answers. But each of the 10 wrong answers costs US$15, so it saves only US$360 − US$150 = US$210. Design B saves 70 × US$4.00 = US$280. The 30 handed-over calls cost nothing extra here. A wrong action costs almost four times as much as a call that a person handles. So a machine should hand over when it is unsure.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: "Whisper small gets only 3.1% of the English words wrong on our calls, so speech recognition is solved for Larkfield." What do you tell him?
Show the answer
The 3.1% is on Larkfield’s synthetic calls, and they are easier than real speech. The voices are clean and regular, nobody talks over the other person, and the accents are only the ones the voices learned. On real people reading sentences (LibriSpeech), the same model gets 7.8% wrong. So the number is a lower bound, not a promise. Also, an average says nothing about the words that matter: an order number with one wrong digit breaks the order-status check. Measure on the slices and the words that matter, and test on real speech too (Module 10).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Larkfield has about 4,000 recorded calls a month and nobody can search them. Its team leads can listen to 60 minutes a day.
- The goal is a call record for every call: transcript, reason, promise, next step and recording notice. Start from these decisions, not from a tool.
- Out of hours, Larkfield wants a phone line that answers by itself. First comes an IVR, a phone menu that understands keypad tones and speech. Then comes a voice agent.
- Mistakes have different costs (assumptions). A call that a person handles costs US$4.00, a hang-up US$8, a wrong action US$15 and a missed follow-up US$25. So a machine must answer right, and hand over when unsure.
- Every call in the course is synthetic and easier than real speech. Quote every number “on Larkfield’s synthetic calls”.
You can now say what Larkfield wants from its calls and what each mistake costs. But a computer does not hear a call the way you do: it gets only numbers. Next, you look at those numbers, and at what a phone line takes away.
Next: Audio as data