Skip to content
Français

What Larkfield wants from its calls

  • Lesson 1
  • 25 min
  • Module 1: The speech problem and its cost

Where we are

In the machine learning course, every example was a row in a table. Larkfield's help desk also has 4,000 phone calls a month, and nobody can search them. In this lesson, we list the decisions Larkfield wants from its calls and what each kind of mistake costs. First, think of an agent who takes notes during a call.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy
Lesson 1 of 36 · Speech AI: Transcribe, Understand and SpeakWhat Larkfield wants from its calls

What you will be able to do

Turn “we have thousands of recorded calls” into a short list of decisions, each with the cost of a wrong answer. Without that list, a team transcribes everything and still cannot answer the questions that matter.

Before you startWhat to know and to have readyAbout 25 minutes

What you need to know

What you need on your computer

Think of a help-desk agent on a busy morning. A customer calls. Her garden lights have not arrived. The agent finds the order, says when it shipped, and makes a promise: “I will ask the courier to call you before tomorrow’s delivery.” Then the next call comes in, and the next.

The agent writes a short note after each call, when there is time. Some notes say “lights, not arrived”. Some notes are empty, because the queue was long. The promise about the courier is not in any note.

The next week, the support lead asks two questions. Why did people call last week? Which promises did we make that nobody has kept? Nobody can answer from the notes. The answers are in the calls, but the calls are sound, not text.

Larkfield is an online shop for home and garden products. Its help desk takes about 4,000 calls a month, in English and in French. About 900 of them come out of hours, when no agent is at work. Every call is recorded, and the caller hears a notice that says so. So the shop has thousands of recordings, and nobody can search them.

Grace leads the support team. She wants to know why people call, which promises agents made, and which calls need a follow-up. She also has a budget for listening: her team leads can spend 60 minutes a day on calls. At one minute a call, that is 60 calls a day. The help desk gets about 130 calls a day (4,000 in a month of 30 days). Most calls are never heard again.

This is a part of one Larkfield call. Press play: it does not start by itself.

Part of call C-0127, as the phone line recorded it (8,000 samples a second). Both voices are synthetic: Piper voices trained on the LibriTTS-R corpus (the agent, Chloe, is speaker 5401; the caller is speaker 7169). No real person speaks in Larkfield’s calls.

Time Who What they said
5.9 s Caller Hi, um, good morning. My Pebble path lights haven’t arrived. Please tell me where they are.
9.3 s Agent I can help with that. Can I have your order number, please?
12.0 s Caller Yes, it’s 114942.
14.0 s Agent Thank you. That’s order 114 942. It shipped on the twelfth of March, and it is on its way.
19.6 s Agent I will ask the courier to call you before tomorrow’s delivery.

You understood this call at once. You know why the caller called, and you heard the promise. A computer gets only a list of numbers that describe the sound. To get the same answers, it must turn the sound into words, and the words into a decision.

Look at the last line of the call. If nobody acts on that promise, the courier does not call. The customer waits at home, calls again, and is angry. Larkfield counts that as a missed follow-up, and it is one of the most expensive mistakes in this course.

Now multiply. In our scripted calls, every call ends with a promise, because the script says so. Real calls hold fewer. But even if only 1 call in 10 held a promise, 4,000 calls would hold 400 promises a month. Grace’s team leads cannot check them by listening, and nobody else can find them. This is the problem of the course: the information is in the calls, and nobody can use it.

The idea is simple. For each call, a program writes the agent’s note, the same note every time. We call it a call record: a short, fixed set of fields that says what happened in a call. People can search it, count it and check it.

Here is the record for the call above. These are the labels that we wrote with the call’s script, so they are the right answers.

Field Value for call C-0127 Who uses it
Transcript every word, with who said it and when anyone who searches the calls
Reason order status Grace: why people call
Promise “I will ask the courier to call you before tomorrow’s delivery.” Grace: what we must do
Next step courier call the team that acts on it
Recording notice said, in the exact words the compliance check
Language, line English, landline Priya: where the errors are

A program that writes this record must do several things in order. It must find the parts of the recording where someone speaks. It must turn the speech into text. It must know which words the agent said and which the caller said. Then it must read the text and fill the fields. Speech recognition is the step from sound to text. It is also called ASR, from automatic speech recognition. The step from text to fields is often called speech analytics.

The second want: a line that answers by itself

Section titled “The second want: a line that answers by itself”

The 900 out-of-hours calls are a different problem. Today, those callers hear a message and call again the next day. Many of their questions are simple: “Where is my order?”, “When are you open?” A machine could answer them.

Larkfield wants this in two steps. The first step is a small IVR, from interactive voice response. It is a phone menu that understands keypad tones and short spoken answers: “Say what you need, or press 1 for orders.” The second step is a voice agent: a program that listens, answers in full sentences and uses tools. It is for the questions that a menu cannot handle. In this course, you build the IVR and call it. Then you compare it with a voice agent on the same recorded calls.

The diagram follows one call from the ring to the record. Each step is a module of this course.

A flow in two rows (one column on a phone). The first row is the call. 1, the phone rings: a customer calls Larkfield's help desk. A decision: open hours? If yes, 2, a person answers: the call is recorded, agent and caller on two channels. If no, 2, the line answers: a phone menu (IVR), then a voice agent, Modules 7 and 8. The second row has the same steps for both kinds of call: 3, find the speech: skip silence and music on hold, Module 3. Then 4, speech to text: a transcript, measured, Modules 2 and 4. Then 5, who said what: agent or caller, with times, Module 5. Then 6, understand: reason, promise, recording notice, Module 6. Then 7, the call record: searchable, with card numbers removed, Module 9. Last, Grace's team: follow-ups, checks, reports.
One call from the ring to the record. In open hours a person answers; out of hours the phone line answers by itself. Both kinds of call end as a call record.
Text description of the diagram

A call starts when the phone rings. In open hours, a person answers, and the phone system records the agent and the caller on two separate channels. Out of hours, the phone line answers by itself: first a phone menu, then a voice agent (Modules 7 and 8). From there, both kinds of call go through the same steps. The program finds the parts with speech (Module 3). It turns the speech into text and measures how good the text is (Modules 2 and 4). It finds who said each word (Module 5). It reads the text and finds the reason, the promise and the recording notice (Module 6). The result is a call record, with card numbers removed (Module 9). Grace’s team uses the records for follow-ups, checks and reports.

Every step in the diagram can make mistakes. A transcript can miss words. The reason can be wrong. A phone menu can start the wrong action. These mistakes do not cost the same, so the course gives each one a price. The prices are assumptions for the course, not measured numbers. A real team would ask its finance people. Larkfield pays its staff in Canadian dollars, but the course gives every cost in US dollars, the currency of the model and cloud prices that it compares them with.

Mistake What happens Cost (assumption)
A person handles a call that a machine could handle 6 minutes of an agent’s time at US$40 an hour US$4.00 a call
The caller hangs up no answer; the caller tries again, or buys elsewhere US$8
A machine does the wrong thing a return started by mistake, or a wrong order status: a second call, a correction and some goodwill US$15
A missed follow-up a promise in a call that nobody acts on: a complaint, a second call, sometimes a lost customer US$25
A team lead listens to a call US$45 an hour, at most 60 minutes a day US$0.75 a minute

Omar, the product manager, owns the out-of-hours line. He asks: “Should the phone line answer as many calls as it can?” Look at the costs in the table before you answer.

Suppose the line answers 100 calls by itself. If all 100 answers are right, Larkfield saves the agent’s time on 100 calls: 100 × US$4.00 = US$400. Now suppose that 10 of the 100 answers are wrong. Those 10 calls do not save US$4.00 each. Each one costs US$15 instead. The line then saves 90 × US$4.00 − 10 × US$15 = US$360 − US$150 = US$210, about half.

A line that answers only the calls it understands, and hands the others to a person, does better. Suppose it answers 70 calls, all right, and hands 30 to a person the next morning. It saves 70 × US$4.00 = US$280, and nobody gets a wrong answer. So the goal is not “answer every call”. The goal is “answer right, and hand over when unsure”. You meet this rule again when you build the IVR.

The course follows a small team at Larkfield. You know some of them from the machine learning course.

Person Role What they care about
Priya Data scientist The main guide. She builds the test set, compares the methods and measures them.
Grace Support lead The cost of a missed reason or follow-up, and her 60 minutes of listening a day. She reads the call records.
Amira Developer The IVR app: its steps, its prompts, its tests and its call records.
Malik Telephony engineer (new) Phone lines, sound at 8,000 samples a second, codecs and keypad tones. He knows why a line sounds bad.
Tomás AI engineer He wants the biggest model for everything. His questions are common mistakes, and they are good questions.
Omar Product manager The out-of-hours line, the new French region, and what callers feel.

Every call in this course is synthetic. Our code wrote the scripts. A language model wrote the callers’ own sentences about why they call. Then open text-to-speech voices read the scripts aloud. No real customer, no real call and no real person’s voice is in the data.

We never use real customer calls for practice. Real calls hold voices, names and sometimes card numbers. The people in them agreed to a recording for quality and training, not to a course. A voice is personal data, and you can recognise a person by it.

Synthetic calls also have a cost: they are easier than real speech. The voices are clean and regular. Nobody coughs, nobody talks over the other person, and the accents are the ones the voices learned. You see the effect in Module 4. Whisper is a free speech recognition model from OpenAI, and the course uses it most; Whisper small is one of its sizes. On Larkfield’s synthetic calls, Whisper small gets 3.1% of the English words wrong. On real people reading sentences aloud, from the open LibriSpeech recordings, the same model gets 7.8% wrong. So the course always says “on Larkfield’s synthetic calls” next to a number. The case studies in Module 10 use real recordings.

The course has 340 synthetic conversations between a Larkfield agent and a caller:

  • 260 public calls, which you get: 150 for training (their transcripts and labels only), 50 for validation and 60 for testing (with audio).
  • 40 hidden test calls, which the course team keeps. They give one honest score at the end.
  • 40 calls for the final assignment, in French, from a new region.

Of the 260 public calls, 175 are in English and 85 in French. A call lasts 43.9 seconds on average. There are also 100 test calls for the IVR (60 of them public): only the caller’s side, with speech, keypad tones and silence.

The calls have seven reasons. These are their counts in the 260 public calls:

Order status Return or refund Billing Damaged item Product question Cancel an order Account change
70 48 35 33 30 26 18

The course follows the call record from sound to decision, and then turns around: the machine starts to speak.

Six boxes in a row (a column on a phone). Start, Module 1: the business problem, audio as data, the project. Recognise, Modules 2 to 5: measure, find the speech, speech to text, who spoke when. Understand, Module 6: reason, summary, promises, notice. Speak, Modules 7 and 8: a phone menu (IVR), then a voice agent. Run, Module 9: batch and streaming, privacy, monitoring. Practise on real speech, Module 10: case studies with open recordings.
The course in four big steps: recognise the speech, understand it, speak back, and run it safely. Module 1 starts, and the case studies end with real speech.
Text description of the diagram

The course has four big steps after this module. Recognise (Modules 2 to 5): you measure transcripts, find the speech, turn speech into text and find who spoke when. Understand (Module 6): you turn transcripts into the call record. Speak (Modules 7 and 8): you build a phone menu that listens and answers, then a voice agent, and you compare them. Run (Module 9): you run it all in batches and live, remove personal data and watch the system. The case studies of Module 10 repeat the work on open recordings of real people.

The course ends with four case studies on open recordings of real people. Each one takes a real dataset from the question to a recommendation. You need one of them for the certificate.

Case studies in this course

At the end of the course, you use what you learn on a realistic project, from the data to the recommendation. Complete at least one for the certificate; the others are optional.

  • Choose a recogniser for two languagesCompare Vosk and Whisper sizes on real English and French speech from an open dataset by speaker group, and recommend one with error, speed and cost.
  • Meeting transcripts with speakersDiarize and transcribe real recorded meetings from an open corpus, measure DER and WER, and make action-item notes.
  • Spot commands on a small deviceBuild a keyword spotter on an open spoken-commands set, compare a small CNN on spectrograms with Whisper tiny, and measure false alarms per hour.
  • An IVR for spoken intentsBuild a menu-and-speech IVR for a bank's common requests, test it with open spoken-intent recordings by speaker group, and compare keyword rules, a classifier and an LLM for the intent step.

The case studies use the whole course, so most learners start them after Module 9.

Speech AI is worth it when there are too many calls to hear, and a decision depends on what people said. Larkfield’s 4,000 calls a month and 60 minutes of listening a day are such a case.

It is not worth it everywhere. A shop with 30 calls a month can listen to all of them. A question that the phone system already answers, such as “how many calls came in on Monday?”, needs no transcript. And a decision that must be right every time, such as a refund to a card, should not rest on a transcript alone. Recognisers make mistakes, and a person or a keypad must confirm.

It also has costs. Computers must run the models, someone must measure them, and recordings of voices must be kept safe and deleted on time. Module 9 puts a price on each.

Common mistake

Transcribe everything first, decide later

Symptom: Tomás says: “Whisper is solved. Let’s run the biggest model on every call, store the text, and then see what we can do with it.” A month later, there are 4,000 transcripts. Grace asks which promises were not kept. Nobody can answer. The transcripts are long, nobody reads them, and nobody knows how many words in them are wrong.

Cause: the team chose the tool before the decision. Without the list of decisions, there is no call record to fill, no test to run and no way to know which errors matter. A transcript that gets an order number wrong is useless for an order-status check. The same mistake in a greeting changes nothing.

Fix: start from the decisions and their costs, as in this lesson. For each decision, write which fields the call record needs and what a wrong field costs. Then choose and measure the methods against those fields, on the same calls.

Consequence in AI work: the team pays for the biggest model on every call, and it may not need it. In Module 4 you measure several sizes of the same model on the same calls, and a small one is often enough.

Check your understanding

Name three decisions that Larkfield wants from its recorded calls. For each one, say which field of the call record it needs.

Show the answer
  • Why do people call? It needs the reason of each call. Grace counts the reasons to see what goes wrong in the shop.
  • Which promises must we keep? It needs the promise and the next step. A promise that nobody acts on is a missed follow-up.
  • Did the agent say the recording notice? It needs a notice field: said exactly, said in other words, or missing. This is a compliance check.

Other good answers: which calls need a follow-up (the next step), and where the transcripts are weak (the language and the line).

Check your understanding

Predict and explain: the out-of-hours line answers 100 calls. In design A, it answers all 100 and 10 answers are wrong. In design B, it answers 70, all right, and hands 30 to a person. With the course's cost assumptions, which design saves more, and why?

Show the answer

Design B saves more. Design A saves 90 × US$4.00 = US$360 on the right answers. But each of the 10 wrong answers costs US$15, so it saves only US$360 − US$150 = US$210. Design B saves 70 × US$4.00 = US$280. The 30 handed-over calls cost nothing extra here. A wrong action costs almost four times as much as a call that a person handles. So a machine should hand over when it is unsure.

Check your understanding

Tomás says: "Whisper small gets only 3.1% of the English words wrong on our calls, so speech recognition is solved for Larkfield." What do you tell him?

Show the answer

The 3.1% is on Larkfield’s synthetic calls, and they are easier than real speech. The voices are clean and regular, nobody talks over the other person, and the accents are only the ones the voices learned. On real people reading sentences (LibriSpeech), the same model gets 7.8% wrong. So the number is a lower bound, not a promise. Also, an average says nothing about the words that matter: an order number with one wrong digit breaks the order-status check. Measure on the slices and the words that matter, and test on real speech too (Module 10).

Recap

  • Larkfield has about 4,000 recorded calls a month and nobody can search them. Its team leads can listen to 60 minutes a day.
  • The goal is a call record for every call: transcript, reason, promise, next step and recording notice. Start from these decisions, not from a tool.
  • Out of hours, Larkfield wants a phone line that answers by itself. First comes an IVR, a phone menu that understands keypad tones and speech. Then comes a voice agent.
  • Mistakes have different costs (assumptions). A call that a person handles costs US$4.00, a hang-up US$8, a wrong action US$15 and a missed follow-up US$25. So a machine must answer right, and hand over when unsure.
  • Every call in the course is synthetic and easier than real speech. Quote every number “on Larkfield’s synthetic calls”.

You can now say what Larkfield wants from its calls and what each mistake costs. But a computer does not hear a call the way you do: it gets only numbers. Next, you look at those numbers, and at what a phone line takes away.

Next: Audio as data