Summaries and follow-ups
- Lesson 20
- 35 min
- Module 6: From transcript to decision
Where we are
We can now give each call a reason. Grace also wants to know what the agent promised and what happens next. In this lesson, we make a structured summary and check it against the transcript, because a summary can say things the call did not.
Lesson 20 of 36 · Speech AI: Transcribe, Understand and SpeakSummaries and follow-ups
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Turn each call into a call record: the reason, a short summary, the promise the agent made, the next step and the recording notice. Then check every record before anybody acts on it: its form, each field against the call, and the promise against Larkfield’s order system.
NotebookDo this lesson in a notebook
The notebook reads the recorded call records of the 50 validation calls and measures each field against the labels. Then it checks the promises against the order system, on the exact transcripts and on Whisper’s, and shows a test that hides a problem. It needs no model download, no account and no key. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The call reasons and the recorded LLM answers in JSON, from Why did they call?.
- Order numbers and their check digit, from Help the model with your words.
What you need on your computer
- Only the notebook. This lesson adds no file to the project.
Tested on macOS with Python 3.12.13, in Jupyter.
The note after the call
Section titled “The note after the call”Think again of the call-centre agent. After each call, she writes a short note in the customer’s file: “Bench not arrived. Promised: the courier will call. Next: ask the courier.” Tomorrow, a colleague reads the note and acts on it. Nobody listens to the call again.
The note is only useful if it is true. If it promises something that the call did not promise, the colleague does the wrong thing. If the agent promised something wrong, the note repeats it, and the mistake travels on.
The problem: a faithful record of a wrong promise
Section titled “The problem: a faithful record of a wrong promise”Listen to call C-0161. A caller asks about a garden bench that has not arrived. The agent looks up the order.
Transcript. Agent: “Thank you. That’s order 260 026. Our system says it was delivered on the eighteenth of April. I will ask the courier to call you before tomorrow’s delivery.”
Did you hear the problem? The order was delivered weeks ago, and the agent promises a call before “tomorrow’s delivery”. This is a mistake in our own scripts: the agent says this promise even when the order was delivered. We kept it, because it is exactly the kind of mistake that a record check must catch.
The LLM wrote this record for the call:
{ "reason": "order_status", "summary": "The customer called because their Harbor bench had not arrived. The system showed the order as delivered on April 18.", "promise": "I will ask the courier to call you before tomorrow's delivery.", "next_step": "courier_call", "recording_notice": true, "follow_up_needed": true}The record is faithful: every field says what the call said. And it is still wrong for Larkfield. Tomorrow, somebody asks a courier to call about a bench that was delivered. The caller still does not know where her bench is. Grace calls this a missed follow-up, and the course’s cost assumption for one is US$25.
The idea: a record, and three checks
Section titled “The idea: a record, and three checks”A call record is a summary with fixed fields, so that a program can read it and route the work. This is the same structured output as in the last lesson; now we use every field. Each field must be checked before somebody acts on it. There are three kinds of check, from cheap to expensive:
- The form. Every field is there. The reason and the next step come from the allowed lists. The notice is
trueorfalse. A program that reads the record would break on a missing field or a new word, such as"reason": "delivery". - The call. Each field agrees with what was said. Is the promise in the agent’s words? Is the next step the one that the promise needs? When we build the system, we measure this against the labels. In production, we compare the record with the transcript.
- The business systems. Each field fits what Larkfield knows. Does the order exist? Does the promise fit the order’s status? An LLM that reads only the transcript cannot know that.
The LLM application course calls the second and third kinds “validating meaning”: a well-formed answer can still be wrong.
The tiny example: one record, three checks
Section titled “The tiny example: one record, three checks”Text description of the diagram
On the left is the transcript of call C-0161. The caller’s bench has not arrived, and her order number is 260026. The agent says the order was delivered on the eighteenth of April. Then the agent promises that the courier will call before tomorrow’s delivery. An arrow goes to the LLM, which gives one answer in JSON. Next is the call record with six fields. The reason is order_status. The summary says that the bench had not arrived and that the system showed it as delivered. The promise is a courier call, the next step is courier_call, and the notice and follow-up fields are true. Three checks read the record. Check 1, the shape: every field is there, and the values come from the lists. Check 2, the call: the promise is in the agent’s words, so it passes. Check 3, the order system: order 260026 is delivered, but the next step is courier_call, so the record is flagged for a person.
Checks 1 and 2 pass, because the record is a good copy of the call. Only check 3 sees that the call itself was wrong.
Larkfield: each field on 60 test calls
Section titled “Larkfield: each field on 60 test calls”Priya measures each field of the recorded records against the labels of the 60 test calls. A free-text promise counts as right when it is similar enough to the agent’s words. Similarity is measured letter by letter, from 0 (nothing alike) to 1 (the same text), and 0.6 or more counts as a match. The next lesson explains this kind of similarity in detail.
- From the exact transcript
- From Whisper small's transcript
Text description of the chart
A column chart with ten columns, two for each field, on an axis from 0 to 100%. The field name is under the first column of each pair. Reason: 100% from the exact transcript and 100% from Whisper small’s. Next step: 100% and 100%. Follow-up needed: 100% and 100%. Recording notice: 100% and 100%. Promise: 83% from the exact transcript (50 of 60) and 78% from Whisper small’s (47 of 60).
The form check found no problem in any of the 220 records (110 calls, two transcripts each). The reason, the next step, the follow-up and the notice agree with the labels on every test call. Remember the last lesson: Larkfield’s synthetic calls are easy. Only the promise is below 100%.
Why the promise “fails”
Section titled “Why the promise “fails””Is the LLM inventing promises? Priya splits the result by language:
| Calls | Promise matches (exact transcript) | Promise matches (Whisper small) |
|---|---|---|
| 34 English test calls | 34 of 34 | 34 of 34 |
| 26 French test calls | 16 of 26 | 13 of 26 |
Every miss is a French call. She reads one, call C-0033 (validation):
| Text | |
|---|---|
| The agent said | Je vous envoie le guide d’installation avec des images par courriel aujourd’hui. |
| The record says | I’ll send you the installation guide with pictures by email today. |
| Similarity | 0.43 |
The meaning is the same. The model translated the promise into English. The prompt asked for a summary in English, and for the promise “in the agent’s words”. The model followed the first part. The check compared letters, so it found a real difference: the language. It did not find an invented promise. On the 110 validation and test calls, the exact transcripts give 22 flagged promises. Priya reads all 22: each one is the agent’s French promise, in English.
That is the honest result of a check: it flags a record, and a person decides why. Two fixes are possible. Ask the model for the promise in the call’s language, and record again. Or compare by meaning, not by letters, for example with a second LLM call. Both cost something, so first count how often the problem happens.
When the record and the facts disagree
Section titled “When the record and the facts disagree”Now the third check. It reads the order number from the transcript with the project’s order-number reader from Module 4, which also tests the check digit. It looks the number up in the order system. It flags a record that promises a courier call for an order that was already delivered. The code, from the notebook:
def order_check(answer, text, language): """'flag' when the record promises a courier call for an order that the system says was delivered.""" number, problem = find_order_number(text, language) if problem or number not in orders: return f"no order to check ({problem or 'not in the system'})" if orders[number]["status"] == "delivered" and answer.get("next_step") == "courier_call": return "flag" return "ok"On the exact transcripts of the 110 validation and test calls, it flags 9 calls:
flagged from the exact transcripts: ['C-0003', 'C-0073', 'C-0265', 'C-0064', 'C-0068', 'C-0074', 'C-0161', 'C-0194', 'C-0288']These are all the calls with the wrong promise, and no other. The LLM itself flagged none of them: in all nine records, the next step is courier_call. The record copied the promise; it did not question it.
The error chain: one digit lost
Section titled “The error chain: one digit lost”In production there is no exact transcript. The check must read Whisper small’s transcript. It flags only 6 of the 9 calls:
flagged from Whisper small: ['C-0003', 'C-0073', 'C-0265', 'C-0064', 'C-0194', 'C-0288']C-0068 said: 540369 Whisper small: ('1540369', 'too long') -> no order to check (too long)C-0074 said: 149849 Whisper small: ('49849', 'too short') -> no order to check (too short)C-0161 said: 260026 Whisper small: ('26026', 'too short') -> no order to check (too short)Text description of the diagram
Five boxes from left to right, joined by arrows. First, the caller says “It’s 260026.” Second, Whisper small writes “It’s 26026”: one digit is lost. Third, the order number check finds only 5 digits. The number is too short, so there is no order to look up. Fourth, the order-system check cannot run. Fifth, the wrong promise, a courier call on a delivered order, stays in the record.
This is the error chain of the module: a misheard word becomes a wrong decision two steps later. Whisper small missed 8 of the 52 order numbers in the test calls (The errors that matter). Here, those misses switch off a check. The WER of the whole call hardly moved.
The fixes are cheap:
- Never treat “no order to check” as “OK”. Send those records to a person, or to a second check.
- Get the order number another way. The IVR of Module 7 asks callers to type it on the keypad. The check digit then catches most typing errors.
What a flag costs
Section titled “What a flag costs”Grace has 60 minutes of team-lead review a day, at US$0.75 a minute (the course’s assumptions). Suppose that checking one flagged record takes 2 minutes (our assumption). On the 110 validation and test calls, the order check flagged 9 records. That is 18 minutes of review, or US$13.50. One missed follow-up costs US$25. Each flag that finds a real problem pays for itself. The French promise flags are different: 22 of the 110 calls, all harmless translations. Fix them in the prompt, not with review minutes.
Common mistakes
Section titled “Common mistakes”Common mistake
You test a check with the labels instead of the transcript
Symptom: in your test, the order check catches every wrong promise. In production, it misses some.
Cause: your test read the order number from the labels, because it was easier. The notebook shows it: with the labels’ order numbers, it flags 9 of 9 calls; with Whisper’s transcripts, 6 of 9. Production has only Whisper’s transcript.
Fix: test every check on the same input that it gets in production: the recognised transcript, with its errors.
Common mistake
The program that reads the record crashes on a new word
Symptom: one day, the routing code fails with a KeyError, or sends calls to nobody.
Cause: the LLM wrote a reason or a next step that is not on the list, or left a field out. The course asked for JSON (JSON mode), not for a fixed schema, so nothing stopped it. In this set, all 220 records had the right form, but that is not a promise for the next 220.
Fix: check the form of every record first, as in check 1. Send a record with a wrong form back to the model once, then to a person. The LLM application course shows this step.
When to use call records
Section titled “When to use call records”Use an LLM call record when people act on calls that nobody has time to listen to again: follow-ups, callbacks, refunds. One answer gives the reason, the summary, the promise and the next step. On Larkfield’s calls, it costs about US$0.13–0.15 for 1,000 calls (recorded at the deployment’s price; prices change).
Do not let the record act alone. A record copies the call, mistakes included. Check its form, compare it with the call, and check it against the systems that know the facts. Send every flag, and every record that a check could not test, to a person.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás says: 'The LLM's record of C-0161 is word for word what the agent said, so it is correct.' What do you tell him?
Show the answer
The record is faithful, but faithful is not the same as right. The agent said the order was delivered. Then the agent promised a courier call before tomorrow’s delivery: a wrong promise, and a mistake in our own scripts. The record copied it, with next step courier_call. Only a check against the order system, which knows that order 260026 is delivered, can flag it. On the 110 validation and test calls, that check flagged 9 such records; the LLM flagged none.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Predict: on Whisper small's transcripts, the promise matches the labels in 78% of the test calls. Does that mean the LLM invented a promise in 22% of the calls?
Show the answer
No. Every miss is a French call, and the English promises match in 34 of 34 calls. The model translated French promises into English, so a letter-by-letter comparison scores them low (C-0033: similarity 0.43, with the same meaning). The check found a difference in language, not an invented promise. Read the flagged records before you decide what the number means.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: in the notebook, run the order-system check on Whisper small's transcripts. How many of the 9 wrong promises does it flag, and what stops it on the others?
Show the answer
It flags 6 of the 9. On C-0068, C-0074 and C-0161, Whisper small added or lost a digit of the order number (“1540369”, “49849”, “26026”). The number has the wrong length, so the check finds no order to look up and cannot run. This is the transcript error that breaks the call record most often here: a misheard order number. The cheapest fixes are to send every “no order to check” record to a person, and to take the order number by keypad.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A call record is a summary with fixed fields (reason, summary, promise, next step, recording notice, follow-up), written by an LLM in JSON.
- Check every record three ways: its form, each field against the call, and the facts against the business systems.
- On Larkfield’s 60 synthetic test calls, four fields agree with the labels on every call. The promise agrees on 83% (exact transcripts) and 78% (Whisper small). Every miss is a French promise that the model wrote in English.
- A record can be faithful and wrong: the order-system check flagged 9 courier promises for delivered orders, which the LLM had copied.
- On Whisper’s transcripts, the check flagged only 6 of the 9: a misheard order number switched it off. Never treat “no order to check” as “OK”.
The record also has a field for the recording notice. Larkfield’s rules require it in every call, so it deserves a better check than an LLM’s yes or no. Next, we search for the notice in the transcript, exactly and with fuzzy match, and in the audio itself.