Why did they call?
- Lesson 19
- 40 min
- Module 6: From transcript to decision
Where we are
We now have speaker-labelled transcripts. Grace wants to know why each customer called. In this lesson, we compare keyword rules, a classifier and an LLM on the call reasons, and we see how transcript errors move each result.
Lesson 19 of 36 · Speech AI: Transcribe, Understand and SpeakWhy did they call?
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Give every Larkfield call a reason, such as “order status” or “billing”, from its transcript. Compare three ways to do it: keyword rules, a classifier and an LLM. Then measure how much each one loses when it reads a recogniser’s transcript, with its errors.
NotebookDo this lesson in a notebook
The notebook builds the keyword rules and the classifier of this lesson, and reads the recorded LLM answers. It compares the three on the 60 test calls, on the exact transcripts and on Whisper’s. It needs no model download, no account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Recorded transcripts and their word error rate, from Whisper sizes on a laptop.
- Accuracy, and why the test calls stay apart from the training calls, from the machine learning course: Train, validation and test sets.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your project folder
larkspeechwith the changes of Module 5. In a terminal, in that folder, with.venvactive,pytest -qprints24 passed. - The labs folder
nextia-learning-labsnext tolarkspeech, and the course data inlarkspeech/data, as in Set up the speech project.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands were not run.
A tick box on the call sheet
Section titled “A tick box on the call sheet”Think of a call-centre agent at the end of a call. Before she takes the next call, she ticks one box on a sheet: “Where is my order?”, “Refund”, “Broken item”, “Billing”, and so on. At the end of the month, her team lead counts the ticks. If a third of the calls ask “Where is my order?”, a better tracking email saves a lot of calls.
Larkfield’s agents never had time for the tick box. Larkfield takes about 4,000 calls a month, and nobody counts them. Now we have a transcript of each call. Can a program tick the box?
The problem: one word can fool a program
Section titled “The problem: one word can fool a program”Listen to the start of call C-0106. A caller asks how to set up a product.
Transcript. Caller: “Hello there. The Grow Hub setup isn’t clear, and I’m getting a bit annoyed. How do I connect it properly?” Agent: “Good question. For the Grow Hub, the first step is to charge it fully before you set it up.”
A person ticks “Product question” at once. Now try the simplest program: look for words. “Charge” is a billing word (“you charged me twice”). So a word search can tick “Billing”. The caller never talked about money. The agent said “charge it fully”, about a battery.
Grace, the support lead, needs the right box. A wrong reason sends the call to the wrong team, and the monthly counts tell the wrong story. Tomás, the AI engineer, has an answer: “Send every call to the biggest LLM.” Priya, the data scientist, wants to measure three methods first, on the same test calls.
The idea: a label from the text
Section titled “The idea: a label from the text”The task has a name. Giving each item one label from a fixed list is classification. Here, the item is a call transcript and the list has Larkfield’s seven call reasons:
| Reason | What the caller wants | Test calls |
|---|---|---|
order_status |
Where is my order? | 15 |
return_refund |
Send it back, get my money back | 12 |
cancel_order |
Stop an order | 11 |
billing |
A charge is wrong | 8 |
damaged_item |
It arrived broken | 6 |
product_question |
How does it work? | 5 |
account_change |
Change my address or email | 3 |
Every call in the course data has its true reason as a label. So we can score any method with accuracy: the share of calls where its reason equals the label. We try three methods, from the cheapest to the most expensive.
Method 1: keyword rules
Section titled “Method 1: keyword rules”Keyword rules are a word list for each reason, checked in a fixed order. The first reason with a word in the transcript wins. Here is part of the project’s list:
| Order | Reason | Some of its words |
|---|---|---|
| 1 | cancel_order |
cancel, annuler |
| 2 | billing |
charged, charge, twice, card, facture |
| 3 | damaged_item |
damaged, broken, cassé |
| 4 | return_refund |
return, refund, money back |
| … | … | … |
| 7 | product_question |
how, set up, question |
The rules work in both languages, because the list has English and French words. You can read every decision: “billing, because of the word charge”. But the rules do not know who said the word, or what it means in the sentence.
Method 2: a classifier that learns word weights
Section titled “Method 2: a classifier that learns word weights”A classifier learns from examples instead of a list that we write. The project uses two simple steps:
- TF-IDF turns a transcript into numbers: one number for each word and word pair. A word gets a high number when it is frequent in this call but rare in other calls. “Charge” is in many calls, so it counts less. “Connect” and “setup” are rarer, so they count more.
- Logistic regression learns one weight for each of those numbers, for each reason, from the 150 train calls and their labels. For a new call, it adds up the weights and picks the reason with the highest score.
The classifier sees all the words at once, not only the first match. The text routing course covers such classifiers in depth.
Method 3: an LLM that answers in JSON
Section titled “Method 3: an LLM that answers in JSON”An LLM (large language model) reads the whole transcript and writes the reason. We ask it for structured output: an answer in JSON, a text format with fixed fields, such as "reason": "product_question". Then a program can read the answer without guessing. The LLM application course shows how to define such a schema and check it.
The course used Azure OpenAI chat-small once on every validation and test call, and recorded each answer. You replay the recorded answers, so you need no key and no account. One answer for C-0106:
{ "reason": "product_question", "summary": "The caller asked how to connect the Grow Hub. The agent advised charging it fully before setup and said they would email a picture guide.", "promise": "I will email you the setup guide with pictures today.", "next_step": "send_guide", "recording_notice": true, "follow_up_needed": true}This lesson uses only the reason field. The next lesson checks the other fields.
The tiny example: three methods on one call
Section titled “The tiny example: three methods on one call”Text description of the diagram
On the left is the transcript of call C-0106. The caller asks how to connect the Grow Hub, and the agent says the first step is to charge it fully. Three arrows go from the transcript to three methods. Keyword rules check a word list for each reason in a fixed order, with billing before product question. They find the agent’s word “charge” and answer billing, which is wrong. The classifier has a weight for every word and word pair, learned from 150 train calls, and answers product_question, which is right. The LLM reads the whole call and answers in JSON, from a recorded answer, and says product_question, which is right.
Larkfield: three methods on 60 test calls
Section titled “Larkfield: three methods on 60 test calls”Priya runs the three methods on the 60 test calls. First on the reference: the exact words of each call, from the script. Then on the transcripts of Whisper small, the best recogniser of Module 4, with a WER of 8.6% on the phone audio. Last on Whisper tiny, which has a WER of 20.7%. The LLM answers were recorded for the reference and for Whisper small only.
- Keyword rules
- Classifier
- LLM
Text description of the chart
A column chart with eight columns in three groups, on an axis from 0 to 100%. The colour shows the method. Group “exact”, the reference transcripts: rules 90.0%, classifier 96.7%, LLM 100.0%. Group “small”, Whisper small’s transcripts, exactly the same: rules 90.0%, classifier 96.7%, LLM 100.0%. Group “tiny”, Whisper tiny’s transcripts: rules 81.7%, classifier 95.0%; there is no recorded LLM answer for Whisper tiny.
Three things stand out:
- The order is clear. The LLM is right on every call, the classifier misses 2 calls, and the rules miss 6.
- Whisper small’s errors change nothing. Each method gives the same reason on Whisper small’s text as on the exact words. Whisper small gets the words that decide a reason right: “cancel”, “broken”, “refund”.
- Whisper tiny hurts the rules most. With more word errors, the rules fall from 90.0% to 81.7%, and the classifier only from 96.7% to 95.0%. The rules hang on single words; one misheard word changes their answer. The classifier uses all the words, so one error moves it less.
Be honest: this test is too easy
Section titled “Be honest: this test is too easy”An LLM at 100% looks like the end of the story. It is not. Larkfield’s calls are synthetic, and this set is too easy. Every caller states the reason clearly in the first sentence, in words that a script chose. Real callers talk around the problem, change their mind and give two reasons in one call. On the 50 validation calls, the recorded LLM answers are also all right on the reference, and 49 of 50 on Whisper small.
That one change, call C-0061, is useful. The caller says the fan “isn’t working” and wants her “money back”. The label is return_refund. From Whisper small’s transcript, the LLM said damaged_item. Whisper’s text for the caller’s words was correct, so a misheard word did not cause it. The model read the same request differently. An LLM can give a different answer to almost the same input.
For a harder test, the case study An IVR for spoken intents uses real recordings of people with 14 intents. Use that one before you trust any number on this page for real calls.
Where each method goes wrong
Section titled “Where each method goes wrong”These are all the test calls that a method got wrong, on the reference:
| Call | Language | True reason | Rules | Classifier | LLM |
|---|---|---|---|---|---|
| C-0026 | French | cancel_order |
billing | return_refund | cancel_order |
| C-0173 | French | cancel_order |
cancel_order | order_status | cancel_order |
| C-0053 | French | product_question |
billing | product_question | product_question |
| C-0106 | English | product_question |
billing | product_question | product_question |
| C-0121 | English | product_question |
billing | product_question | product_question |
| C-0133 | English | product_question |
billing | product_question | product_question |
| C-0225 | English | product_question |
billing | product_question | product_question |
The rules make one kind of mistake: five of the five product questions become billing. In each one, the agent’s setup answer says “charge it fully” (in French, “le charger complètement”). In our scripts, the agent gives that answer for every product, even a raised bed: a mistake in our own scripts. But the lesson holds. The rules read the agent’s words as if the caller had said them.
The classifier’s two mistakes are French cancellations. French is 26 of the 60 test calls, but only 43 of the 150 train calls. With fewer French examples, the classifier learned fewer French words. In the notebook, a classifier trained on English calls only gets 23.1% of the French test calls right.
Change one thing: rules on the caller’s words only
Section titled “Change one thing: rules on the caller’s words only”The fix for C-0106 looks easy: run the rules on the caller’s turns only. Priya tries it. C-0106 becomes product_question. But the accuracy on the 60 test calls falls from 90.0% to 83.3%. Many callers never say a listed word. They say “Please stop my Grow Hub order”, and only the agent says “cancel” (“I have cancelled the order”). Without the agent’s words, nine cancellations become product questions. A fix for one call broke others. Always measure a fix on all the test calls, not on the call that you looked at.
What each method costs
Section titled “What each method costs”| Method | Accuracy (test) | Time for each call | Cost for 1,000 calls | Can you explain a decision? |
|---|---|---|---|---|
| Keyword rules | 90.0% | under 1 ms | almost nothing | Yes: the word that matched |
| Classifier | 96.7% | under 1 ms | almost nothing; needs labelled calls | Partly: the word weights |
| LLM (chat-small) | 100.0% | 2.0–2.3 s | US$0.13–0.15 | Only by its answer |
The LLM cost is measured from the recorded answers at the deployment’s price, which can change. At about 4,000 calls a month, it is well under US$1 a month. The real costs of the LLM are elsewhere. The transcript leaves Larkfield’s servers, and the answer takes two seconds. The same call can also get a different answer another time.
Guided practice: add the call reason to the project
Section titled “Guided practice: add the call reason to the project”Add this module’s files to your project. The file understand.py has the rules, the classifier and the reader for the recorded LLM answers. The command speech reason runs all three on a split and scores them.
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Copy the module’s three files from the snapshot
end-of-m06.Terminal window cp ../nextia-learning-labs/C41/snapshots/end-of-m06/larkspeech/understand.py larkspeech/cp ../nextia-learning-labs/C41/snapshots/end-of-m06/larkspeech/commands/understand.py larkspeech/commands/cp ../nextia-learning-labs/C41/snapshots/end-of-m06/tests/test_understand.py tests/Terminal window Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m06\larkspeech\understand.py larkspeech\Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m06\larkspeech\commands\understand.py larkspeech\commands\Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m06\tests\test_understand.py tests\This module needs no new package: scikit-learn came with Module 5.
-
Run the tests:
Terminal window pytest -qYou should see
27 passed: the 24 tests of Module 5 and 3 new tests. -
Score the three methods on the reference transcripts of the test calls:
Terminal window speech reasonThe output on the course’s computer:
Call reason on 60 test calls, transcripts: referencekeyword rules accuracy 90.0%; most common mistake product_question -> billing (5)classifier (TF-IDF) accuracy 96.7%; most common mistake cancel_order -> return_refund (1)LLM (recorded, chat-small) accuracy 100.0% -
Score them again on the recorded transcripts of Whisper small:
Terminal window speech reason --transcripts whisper-smallThe three lines are the same as in step 4. The classifier still learns from the reference transcripts of the 150 train calls.
Your turn: a weaker recogniser
Section titled “Your turn: a weaker recogniser”Run the command on Whisper tiny’s transcripts. Before you run it, predict: which method loses the most? And why is there no LLM line?
speech reason --transcripts whisper-tinyShow the answer
Call reason on 60 test calls, transcripts: whisper-tiny keyword rules accuracy 81.7%; most common mistake product_question -> billing (4) classifier (TF-IDF) accuracy 95.0%; most common mistake cancel_order -> return_refund (1)The rules lose 8.3 points and the classifier 1.7 points. A single misheard keyword changes the rules’ answer. The LLM line is missing because the course recorded LLM answers only for the reference and Whisper small. The command skips a recording that does not exist.
Common mistakes
Section titled “Common mistakes”Common mistake
The classifier scores almost 100% in your test
Symptom: your classifier is right on almost every test call, better than you expected.
Cause: it learned from the same calls that you test it on. It remembers those calls; it did not learn the task. The notebook shows it: trained on the 60 test calls themselves, it scores 100% on them, which means nothing.
Fix: train on the train calls only. Choose the method and its settings on the validation calls. Use the test calls once, at the end.
Common mistake
The rules break on a new product or a new word
Symptom: after a new product launch, many calls get the wrong reason, and the rules’ code did not change.
Cause: the word lists are fixed. A caller who says “it won’t switch on” uses no word from the damaged_item list. A product name that contains “card” or “return” matches the wrong reason.
Fix: measure the rules every month on a fresh sample of labelled calls. Keep the rules as a fast first pass and a baseline that the other methods must beat.
When to use each method
Section titled “When to use each method”- Keyword rules: as the first baseline, when you have no labelled calls yet, or when every decision must be explained. Measure them; never trust them unmeasured.
- A classifier: when you have a few hundred labelled calls, and the reasons stay the same. It is fast, cheap and private: the transcript never leaves your server. It needs examples in every language.
- An LLM: when the calls are varied, when you have few labels, or when you also want a summary from the same answer. Validate its JSON, record its answers, and accept that the transcript goes to a model provider.
On Larkfield’s synthetic calls, all three are good, and the classifier is almost as good as the LLM. On real calls, measure again before you choose.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'The LLM got 100% of the test calls right, so call reasons are solved.' What do you tell him?
Show the answer
The 100% is real, but on Larkfield’s synthetic calls, and this set is too easy. Every caller states the reason clearly in a scripted sentence, and the classifier already gets 96.7%. Real callers talk around the problem and give two reasons in one call. Even here, the LLM changed its answer on one validation call (C-0061) when the input changed only a little. Test on a harder set, such as the spoken-intent case study, before trusting the number.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: you move from Whisper small to Whisper tiny transcripts (WER 8.6% to 20.7%). Which loses more accuracy, the keyword rules or the classifier, and why?
Show the answer
The keyword rules. They decide on single words, so one misheard keyword changes the reason. The classifier adds up weights over all the words, so one error moves its score a little. Measured on the 60 test calls: the rules fell from 90.0% to 81.7%, and the classifier from 96.7% to 95.0%.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: run speech reason --split validation. Which method is worst on the 50 validation calls, what is its most common mistake, and what causes it?
Show the answer
The output is: keyword rules 80.0% (most common mistake product_question → billing, 9), classifier 98.0%, LLM 100.0%. The rules are the worst. Product questions become billing because the agent’s setup answer contains “charge it fully”. “Charge” is on the billing list, and the rules check billing before product questions. The rules read the agent’s words as if the caller had said them.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Giving each call one reason from a fixed list is classification; accuracy is the share of calls with the right reason.
- Keyword rules match the first listed word. A classifier (TF-IDF and logistic regression) learns word weights from labelled calls. An LLM reads the whole call and answers in JSON (structured output).
- On Larkfield’s 60 synthetic test calls: rules 90.0%, classifier 96.7%, recorded chat-small 100.0%, the same on Whisper small’s transcripts. This set is too easy; the spoken-intent case study is a harder test.
- Transcript errors hurt the method that depends on single words most. On Whisper tiny, the rules fell to 81.7%, and the classifier only to 95.0%.
- Your project now has
understand.pyandspeech reason, with 27 tests.
The LLM’s answer had more than a reason: a summary, a promise and a next step. Grace wants those too. Next, we check each field of that call record. A record can say something that the call did not, and a promise can be wrong.
Next: Summaries and follow-ups