Skip to content
Français

Why did they call?

  • Lesson 19
  • 40 min
  • Module 6: From transcript to decision

Where we are

We now have speaker-labelled transcripts. Grace wants to know why each customer called. In this lesson, we compare keyword rules, a classifier and an LLM on the call reasons, and we see how transcript errors move each result.

What you will be able to do

Give every Larkfield call a reason, such as “order status” or “billing”, from its transcript. Compare three ways to do it: keyword rules, a classifier and an LLM. Then measure how much each one loses when it reads a recogniser’s transcript, with its errors.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook builds the keyword rules and the classifier of this lesson, and reads the recorded LLM answers. It compares the three on the 60 test calls, on the exact transcripts and on Whisper’s. It needs no model download, no account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer (only for the guided practice; the notebook needs none of it)

  • Your project folder larkspeech with the changes of Module 5. In a terminal, in that folder, with .venv active, pytest -q prints 24 passed.
  • The labs folder nextia-learning-labs next to larkspeech, and the course data in larkspeech/data, as in Set up the speech project.

Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands were not run.

Think of a call-centre agent at the end of a call. Before she takes the next call, she ticks one box on a sheet: “Where is my order?”, “Refund”, “Broken item”, “Billing”, and so on. At the end of the month, her team lead counts the ticks. If a third of the calls ask “Where is my order?”, a better tracking email saves a lot of calls.

Larkfield’s agents never had time for the tick box. Larkfield takes about 4,000 calls a month, and nobody counts them. Now we have a transcript of each call. Can a program tick the box?

Listen to the start of call C-0106. A caller asks how to set up a product.

Call C-0106, from 8.9 to 19.4 seconds, phone audio (8 kHz). The caller is a synthetic voice: Piper, trained on the LibriTTS-R corpus (speaker 249). The agent is a synthetic voice: Kokoro, voice am_michael. Every Larkfield call is synthetic: open text-to-speech voices read scripts that we wrote.

Transcript. Caller: “Hello there. The Grow Hub setup isn’t clear, and I’m getting a bit annoyed. How do I connect it properly?” Agent: “Good question. For the Grow Hub, the first step is to charge it fully before you set it up.”

A person ticks “Product question” at once. Now try the simplest program: look for words. “Charge” is a billing word (“you charged me twice”). So a word search can tick “Billing”. The caller never talked about money. The agent said “charge it fully”, about a battery.

Grace, the support lead, needs the right box. A wrong reason sends the call to the wrong team, and the monthly counts tell the wrong story. Tomás, the AI engineer, has an answer: “Send every call to the biggest LLM.” Priya, the data scientist, wants to measure three methods first, on the same test calls.

The task has a name. Giving each item one label from a fixed list is classification. Here, the item is a call transcript and the list has Larkfield’s seven call reasons:

Reason What the caller wants Test calls
order_status Where is my order? 15
return_refund Send it back, get my money back 12
cancel_order Stop an order 11
billing A charge is wrong 8
damaged_item It arrived broken 6
product_question How does it work? 5
account_change Change my address or email 3

Every call in the course data has its true reason as a label. So we can score any method with accuracy: the share of calls where its reason equals the label. We try three methods, from the cheapest to the most expensive.

Keyword rules are a word list for each reason, checked in a fixed order. The first reason with a word in the transcript wins. Here is part of the project’s list:

Order Reason Some of its words
1 cancel_order cancel, annuler
2 billing charged, charge, twice, card, facture
3 damaged_item damaged, broken, cassé
4 return_refund return, refund, money back
… … …
7 product_question how, set up, question

The rules work in both languages, because the list has English and French words. You can read every decision: “billing, because of the word charge”. But the rules do not know who said the word, or what it means in the sentence.

Method 2: a classifier that learns word weights

Section titled “Method 2: a classifier that learns word weights”

A classifier learns from examples instead of a list that we write. The project uses two simple steps:

  1. TF-IDF turns a transcript into numbers: one number for each word and word pair. A word gets a high number when it is frequent in this call but rare in other calls. “Charge” is in many calls, so it counts less. “Connect” and “setup” are rarer, so they count more.
  2. Logistic regression learns one weight for each of those numbers, for each reason, from the 150 train calls and their labels. For a new call, it adds up the weights and picks the reason with the highest score.

The classifier sees all the words at once, not only the first match. The text routing course covers such classifiers in depth.

An LLM (large language model) reads the whole transcript and writes the reason. We ask it for structured output: an answer in JSON, a text format with fixed fields, such as "reason": "product_question". Then a program can read the answer without guessing. The LLM application course shows how to define such a schema and check it.

The course used Azure OpenAI chat-small once on every validation and test call, and recorded each answer. You replay the recorded answers, so you need no key and no account. One answer for C-0106:

{
"reason": "product_question",
"summary": "The caller asked how to connect the Grow Hub. The agent advised charging it fully before setup and said they would email a picture guide.",
"promise": "I will email you the setup guide with pictures today.",
"next_step": "send_guide",
"recording_notice": true,
"follow_up_needed": true
}

This lesson uses only the reason field. The next lesson checks the other fields.

The tiny example: three methods on one call

Section titled “The tiny example: three methods on one call”
The transcript of call C-0106 goes to three methods. Keyword rules check a word list in a fixed order, find the agent's word charge and answer billing, which is wrong. The classifier, with word weights from 150 train calls, answers product_question. The LLM reads the whole call, answers in JSON, and says product_question.
One call, three methods. The rules stop at the first word on their list; the other two use the whole transcript.
Text description of the diagram

On the left is the transcript of call C-0106. The caller asks how to connect the Grow Hub, and the agent says the first step is to charge it fully. Three arrows go from the transcript to three methods. Keyword rules check a word list for each reason in a fixed order, with billing before product question. They find the agent’s word “charge” and answer billing, which is wrong. The classifier has a weight for every word and word pair, learned from 150 train calls, and answers product_question, which is right. The LLM reads the whole call and answers in JSON, from a recorded answer, and says product_question, which is right.

Priya runs the three methods on the 60 test calls. First on the reference: the exact words of each call, from the script. Then on the transcripts of Whisper small, the best recogniser of Module 4, with a WER of 8.6% on the phone audio. Last on Whisper tiny, which has a WER of 20.7%. The LLM answers were recorded for the reference and for Whisper small only.

  • Keyword rules
  • Classifier
  • LLM
Call-reason accuracy on Larkfield's 60 synthetic test calls (measured; the LLM answers are recorded), on the exact transcripts and on the transcripts of Whisper small and Whisper tiny. Nothing changes from the reference to Whisper small; Whisper tiny costs the rules 8 points.
Text description of the chart

A column chart with eight columns in three groups, on an axis from 0 to 100%. The colour shows the method. Group “exact”, the reference transcripts: rules 90.0%, classifier 96.7%, LLM 100.0%. Group “small”, Whisper small’s transcripts, exactly the same: rules 90.0%, classifier 96.7%, LLM 100.0%. Group “tiny”, Whisper tiny’s transcripts: rules 81.7%, classifier 95.0%; there is no recorded LLM answer for Whisper tiny.

Three things stand out:

  • The order is clear. The LLM is right on every call, the classifier misses 2 calls, and the rules miss 6.
  • Whisper small’s errors change nothing. Each method gives the same reason on Whisper small’s text as on the exact words. Whisper small gets the words that decide a reason right: “cancel”, “broken”, “refund”.
  • Whisper tiny hurts the rules most. With more word errors, the rules fall from 90.0% to 81.7%, and the classifier only from 96.7% to 95.0%. The rules hang on single words; one misheard word changes their answer. The classifier uses all the words, so one error moves it less.

An LLM at 100% looks like the end of the story. It is not. Larkfield’s calls are synthetic, and this set is too easy. Every caller states the reason clearly in the first sentence, in words that a script chose. Real callers talk around the problem, change their mind and give two reasons in one call. On the 50 validation calls, the recorded LLM answers are also all right on the reference, and 49 of 50 on Whisper small.

That one change, call C-0061, is useful. The caller says the fan “isn’t working” and wants her “money back”. The label is return_refund. From Whisper small’s transcript, the LLM said damaged_item. Whisper’s text for the caller’s words was correct, so a misheard word did not cause it. The model read the same request differently. An LLM can give a different answer to almost the same input.

For a harder test, the case study An IVR for spoken intents uses real recordings of people with 14 intents. Use that one before you trust any number on this page for real calls.

These are all the test calls that a method got wrong, on the reference:

Call Language True reason Rules Classifier LLM
C-0026 French cancel_order billing return_refund cancel_order
C-0173 French cancel_order cancel_order order_status cancel_order
C-0053 French product_question billing product_question product_question
C-0106 English product_question billing product_question product_question
C-0121 English product_question billing product_question product_question
C-0133 English product_question billing product_question product_question
C-0225 English product_question billing product_question product_question

The rules make one kind of mistake: five of the five product questions become billing. In each one, the agent’s setup answer says “charge it fully” (in French, “le charger complètement”). In our scripts, the agent gives that answer for every product, even a raised bed: a mistake in our own scripts. But the lesson holds. The rules read the agent’s words as if the caller had said them.

The classifier’s two mistakes are French cancellations. French is 26 of the 60 test calls, but only 43 of the 150 train calls. With fewer French examples, the classifier learned fewer French words. In the notebook, a classifier trained on English calls only gets 23.1% of the French test calls right.

Change one thing: rules on the caller’s words only

Section titled “Change one thing: rules on the caller’s words only”

The fix for C-0106 looks easy: run the rules on the caller’s turns only. Priya tries it. C-0106 becomes product_question. But the accuracy on the 60 test calls falls from 90.0% to 83.3%. Many callers never say a listed word. They say “Please stop my Grow Hub order”, and only the agent says “cancel” (“I have cancelled the order”). Without the agent’s words, nine cancellations become product questions. A fix for one call broke others. Always measure a fix on all the test calls, not on the call that you looked at.

Method Accuracy (test) Time for each call Cost for 1,000 calls Can you explain a decision?
Keyword rules 90.0% under 1 ms almost nothing Yes: the word that matched
Classifier 96.7% under 1 ms almost nothing; needs labelled calls Partly: the word weights
LLM (chat-small) 100.0% 2.0–2.3 s US$0.13–0.15 Only by its answer

The LLM cost is measured from the recorded answers at the deployment’s price, which can change. At about 4,000 calls a month, it is well under US$1 a month. The real costs of the LLM are elsewhere. The transcript leaves Larkfield’s servers, and the answer takes two seconds. The same call can also get a different answer another time.

Guided practice: add the call reason to the project

Section titled “Guided practice: add the call reason to the project”

Add this module’s files to your project. The file understand.py has the rules, the classifier and the reader for the recorded LLM answers. The command speech reason runs all three on a split and scores them.

  1. Open a terminal in your project folder larkspeech, and activate .venv.

  2. Copy the module’s three files from the snapshot end-of-m06.

    Terminal window
    cp ../nextia-learning-labs/C41/snapshots/end-of-m06/larkspeech/understand.py larkspeech/
    cp ../nextia-learning-labs/C41/snapshots/end-of-m06/larkspeech/commands/understand.py larkspeech/commands/
    cp ../nextia-learning-labs/C41/snapshots/end-of-m06/tests/test_understand.py tests/

    This module needs no new package: scikit-learn came with Module 5.

  3. Run the tests:

    Terminal window
    pytest -q

    You should see 27 passed: the 24 tests of Module 5 and 3 new tests.

  4. Score the three methods on the reference transcripts of the test calls:

    Terminal window
    speech reason

    The output on the course’s computer:

    Call reason on 60 test calls, transcripts: reference
    keyword rules accuracy 90.0%; most common mistake product_question -> billing (5)
    classifier (TF-IDF) accuracy 96.7%; most common mistake cancel_order -> return_refund (1)
    LLM (recorded, chat-small) accuracy 100.0%
  5. Score them again on the recorded transcripts of Whisper small:

    Terminal window
    speech reason --transcripts whisper-small

    The three lines are the same as in step 4. The classifier still learns from the reference transcripts of the 150 train calls.

Run the command on Whisper tiny’s transcripts. Before you run it, predict: which method loses the most? And why is there no LLM line?

Terminal window
speech reason --transcripts whisper-tiny
Show the answer
Call reason on 60 test calls, transcripts: whisper-tiny
keyword rules accuracy 81.7%; most common mistake product_question -> billing (4)
classifier (TF-IDF) accuracy 95.0%; most common mistake cancel_order -> return_refund (1)

The rules lose 8.3 points and the classifier 1.7 points. A single misheard keyword changes the rules’ answer. The LLM line is missing because the course recorded LLM answers only for the reference and Whisper small. The command skips a recording that does not exist.

Common mistake

The classifier scores almost 100% in your test

Symptom: your classifier is right on almost every test call, better than you expected.

Cause: it learned from the same calls that you test it on. It remembers those calls; it did not learn the task. The notebook shows it: trained on the 60 test calls themselves, it scores 100% on them, which means nothing.

Fix: train on the train calls only. Choose the method and its settings on the validation calls. Use the test calls once, at the end.

Common mistake

The rules break on a new product or a new word

Symptom: after a new product launch, many calls get the wrong reason, and the rules’ code did not change.

Cause: the word lists are fixed. A caller who says “it won’t switch on” uses no word from the damaged_item list. A product name that contains “card” or “return” matches the wrong reason.

Fix: measure the rules every month on a fresh sample of labelled calls. Keep the rules as a fast first pass and a baseline that the other methods must beat.

  • Keyword rules: as the first baseline, when you have no labelled calls yet, or when every decision must be explained. Measure them; never trust them unmeasured.
  • A classifier: when you have a few hundred labelled calls, and the reasons stay the same. It is fast, cheap and private: the transcript never leaves your server. It needs examples in every language.
  • An LLM: when the calls are varied, when you have few labels, or when you also want a summary from the same answer. Validate its JSON, record its answers, and accept that the transcript goes to a model provider.

On Larkfield’s synthetic calls, all three are good, and the classifier is almost as good as the LLM. On real calls, measure again before you choose.

Check your understanding

Tomás says: 'The LLM got 100% of the test calls right, so call reasons are solved.' What do you tell him?

Show the answer

The 100% is real, but on Larkfield’s synthetic calls, and this set is too easy. Every caller states the reason clearly in a scripted sentence, and the classifier already gets 96.7%. Real callers talk around the problem and give two reasons in one call. Even here, the LLM changed its answer on one validation call (C-0061) when the input changed only a little. Test on a harder set, such as the spoken-intent case study, before trusting the number.

Check your understanding

Predict: you move from Whisper small to Whisper tiny transcripts (WER 8.6% to 20.7%). Which loses more accuracy, the keyword rules or the classifier, and why?

Show the answer

The keyword rules. They decide on single words, so one misheard keyword changes the reason. The classifier adds up weights over all the words, so one error moves its score a little. Measured on the 60 test calls: the rules fell from 90.0% to 81.7%, and the classifier from 96.7% to 95.0%.

Check your understanding

Practical check: run speech reason --split validation. Which method is worst on the 50 validation calls, what is its most common mistake, and what causes it?

Show the answer

The output is: keyword rules 80.0% (most common mistake product_question → billing, 9), classifier 98.0%, LLM 100.0%. The rules are the worst. Product questions become billing because the agent’s setup answer contains “charge it fully”. “Charge” is on the billing list, and the rules check billing before product questions. The rules read the agent’s words as if the caller had said them.

Recap

  • Giving each call one reason from a fixed list is classification; accuracy is the share of calls with the right reason.
  • Keyword rules match the first listed word. A classifier (TF-IDF and logistic regression) learns word weights from labelled calls. An LLM reads the whole call and answers in JSON (structured output).
  • On Larkfield’s 60 synthetic test calls: rules 90.0%, classifier 96.7%, recorded chat-small 100.0%, the same on Whisper small’s transcripts. This set is too easy; the spoken-intent case study is a harder test.
  • Transcript errors hurt the method that depends on single words most. On Whisper tiny, the rules fell to 81.7%, and the classifier only to 95.0%.
  • Your project now has understand.py and speech reason, with 27 tests.

The LLM’s answer had more than a reason: a summary, a promise and a next step. Grace wants those too. Next, we check each field of that call record. A record can say something that the call did not, and a promise can be wrong.

Next: Summaries and follow-ups