Skip to content
Français

Summaries and follow-ups

  • Lesson 20
  • 35 min
  • Module 6: From transcript to decision

Where we are

We can now give each call a reason. Grace also wants to know what the agent promised and what happens next. In this lesson, we make a structured summary and check it against the transcript, because a summary can say things the call did not.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Turn each call into a call record: the reason, a short summary, the promise the agent made, the next step and the recording notice. Then check every record before anybody acts on it: its form, each field against the call, and the promise against Larkfield’s order system.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook reads the recorded call records of the 50 validation calls and measures each field against the labels. Then it checks the promises against the order system, on the exact transcripts and on Whisper’s, and shows a test that hides a problem. It needs no model download, no account and no key. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Only the notebook. This lesson adds no file to the project.

Tested on macOS with Python 3.12.13, in Jupyter.

Think again of the call-centre agent. After each call, she writes a short note in the customer’s file: “Bench not arrived. Promised: the courier will call. Next: ask the courier.” Tomorrow, a colleague reads the note and acts on it. Nobody listens to the call again.

The note is only useful if it is true. If it promises something that the call did not promise, the colleague does the wrong thing. If the agent promised something wrong, the note repeats it, and the mistake travels on.

The problem: a faithful record of a wrong promise

Section titled “The problem: a faithful record of a wrong promise”

Listen to call C-0161. A caller asks about a garden bench that has not arrived. The agent looks up the order.

Call C-0161, from 15.9 to 24.7 seconds, phone audio (8 kHz). The agent is a synthetic voice: Piper, trained on the LibriTTS-R corpus (speaker 5401). Every Larkfield call is synthetic: open text-to-speech voices read scripts that we wrote.

Transcript. Agent: “Thank you. That’s order 260 026. Our system says it was delivered on the eighteenth of April. I will ask the courier to call you before tomorrow’s delivery.”

Did you hear the problem? The order was delivered weeks ago, and the agent promises a call before “tomorrow’s delivery”. This is a mistake in our own scripts: the agent says this promise even when the order was delivered. We kept it, because it is exactly the kind of mistake that a record check must catch.

The LLM wrote this record for the call:

{
"reason": "order_status",
"summary": "The customer called because their Harbor bench had not arrived. The system showed the order as delivered on April 18.",
"promise": "I will ask the courier to call you before tomorrow's delivery.",
"next_step": "courier_call",
"recording_notice": true,
"follow_up_needed": true
}

The record is faithful: every field says what the call said. And it is still wrong for Larkfield. Tomorrow, somebody asks a courier to call about a bench that was delivered. The caller still does not know where her bench is. Grace calls this a missed follow-up, and the course’s cost assumption for one is US$25.

A call record is a summary with fixed fields, so that a program can read it and route the work. This is the same structured output as in the last lesson; now we use every field. Each field must be checked before somebody acts on it. There are three kinds of check, from cheap to expensive:

  1. The form. Every field is there. The reason and the next step come from the allowed lists. The notice is true or false. A program that reads the record would break on a missing field or a new word, such as "reason": "delivery".
  2. The call. Each field agrees with what was said. Is the promise in the agent’s words? Is the next step the one that the promise needs? When we build the system, we measure this against the labels. In production, we compare the record with the transcript.
  3. The business systems. Each field fits what Larkfield knows. Does the order exist? Does the promise fit the order’s status? An LLM that reads only the transcript cannot know that.

The LLM application course calls the second and third kinds “validating meaning”: a well-formed answer can still be wrong.

The tiny example: one record, three checks

Section titled “The tiny example: one record, three checks”
The transcript of call C-0161 goes to the LLM, which writes a call record: reason order_status, a summary, the promise that the courier will call before tomorrow's delivery, next step courier_call, recording notice true, follow-up needed true. Three checks read the record. Check 1, the shape: every field is there and the values come from the lists. Check 2, the call: the promise is in the agent's words, yes. Check 3, the order system: order 260026 is delivered, but the next step is courier_call, so the record is flagged for a person.
One call turned into a record, and the three checks. Only the order system knows that the promise is wrong.
Text description of the diagram

On the left is the transcript of call C-0161. The caller’s bench has not arrived, and her order number is 260026. The agent says the order was delivered on the eighteenth of April. Then the agent promises that the courier will call before tomorrow’s delivery. An arrow goes to the LLM, which gives one answer in JSON. Next is the call record with six fields. The reason is order_status. The summary says that the bench had not arrived and that the system showed it as delivered. The promise is a courier call, the next step is courier_call, and the notice and follow-up fields are true. Three checks read the record. Check 1, the shape: every field is there, and the values come from the lists. Check 2, the call: the promise is in the agent’s words, so it passes. Check 3, the order system: order 260026 is delivered, but the next step is courier_call, so the record is flagged for a person.

Checks 1 and 2 pass, because the record is a good copy of the call. Only check 3 sees that the call itself was wrong.

Priya measures each field of the recorded records against the labels of the 60 test calls. A free-text promise counts as right when it is similar enough to the agent’s words. Similarity is measured letter by letter, from 0 (nothing alike) to 1 (the same text), and 0.6 or more counts as a match. The next lesson explains this kind of similarity in detail.

  • From the exact transcript
  • From Whisper small's transcript
Each field of chat-small's recorded call records against the labels of Larkfield's 60 synthetic test calls. Four fields agree on every call; the promise agrees on 83% and 78%.
Text description of the chart

A column chart with ten columns, two for each field, on an axis from 0 to 100%. The field name is under the first column of each pair. Reason: 100% from the exact transcript and 100% from Whisper small’s. Next step: 100% and 100%. Follow-up needed: 100% and 100%. Recording notice: 100% and 100%. Promise: 83% from the exact transcript (50 of 60) and 78% from Whisper small’s (47 of 60).

The form check found no problem in any of the 220 records (110 calls, two transcripts each). The reason, the next step, the follow-up and the notice agree with the labels on every test call. Remember the last lesson: Larkfield’s synthetic calls are easy. Only the promise is below 100%.

Is the LLM inventing promises? Priya splits the result by language:

Calls Promise matches (exact transcript) Promise matches (Whisper small)
34 English test calls 34 of 34 34 of 34
26 French test calls 16 of 26 13 of 26

Every miss is a French call. She reads one, call C-0033 (validation):

Text
The agent said Je vous envoie le guide d’installation avec des images par courriel aujourd’hui.
The record says I’ll send you the installation guide with pictures by email today.
Similarity 0.43

The meaning is the same. The model translated the promise into English. The prompt asked for a summary in English, and for the promise “in the agent’s words”. The model followed the first part. The check compared letters, so it found a real difference: the language. It did not find an invented promise. On the 110 validation and test calls, the exact transcripts give 22 flagged promises. Priya reads all 22: each one is the agent’s French promise, in English.

That is the honest result of a check: it flags a record, and a person decides why. Two fixes are possible. Ask the model for the promise in the call’s language, and record again. Or compare by meaning, not by letters, for example with a second LLM call. Both cost something, so first count how often the problem happens.

Now the third check. It reads the order number from the transcript with the project’s order-number reader from Module 4, which also tests the check digit. It looks the number up in the order system. It flags a record that promises a courier call for an order that was already delivered. The code, from the notebook:

def order_check(answer, text, language):
"""'flag' when the record promises a courier call for an order that the system says was delivered."""
number, problem = find_order_number(text, language)
if problem or number not in orders:
return f"no order to check ({problem or 'not in the system'})"
if orders[number]["status"] == "delivered" and answer.get("next_step") == "courier_call":
return "flag"
return "ok"

On the exact transcripts of the 110 validation and test calls, it flags 9 calls:

flagged from the exact transcripts: ['C-0003', 'C-0073', 'C-0265', 'C-0064', 'C-0068', 'C-0074', 'C-0161', 'C-0194', 'C-0288']

These are all the calls with the wrong promise, and no other. The LLM itself flagged none of them: in all nine records, the next step is courier_call. The record copied the promise; it did not question it.

In production there is no exact transcript. The check must read Whisper small’s transcript. It flags only 6 of the 9 calls:

flagged from Whisper small: ['C-0003', 'C-0073', 'C-0265', 'C-0064', 'C-0194', 'C-0288']
C-0068 said: 540369 Whisper small: ('1540369', 'too long') -> no order to check (too long)
C-0074 said: 149849 Whisper small: ('49849', 'too short') -> no order to check (too short)
C-0161 said: 260026 Whisper small: ('26026', 'too short') -> no order to check (too short)
Five boxes in a chain. The caller says: It's 260026. Whisper small writes: It's 26026, one digit lost. The order number check finds 5 digits, too short, so there is no order to look up. The order-system check cannot run. The wrong promise, a courier call on a delivered order, stays in the record.
How one misheard digit in call C-0161 lets a wrong promise through. Each step works as designed; the error moves from step to step.
Text description of the diagram

Five boxes from left to right, joined by arrows. First, the caller says “It’s 260026.” Second, Whisper small writes “It’s 26026”: one digit is lost. Third, the order number check finds only 5 digits. The number is too short, so there is no order to look up. Fourth, the order-system check cannot run. Fifth, the wrong promise, a courier call on a delivered order, stays in the record.

This is the error chain of the module: a misheard word becomes a wrong decision two steps later. Whisper small missed 8 of the 52 order numbers in the test calls (The errors that matter). Here, those misses switch off a check. The WER of the whole call hardly moved.

The fixes are cheap:

  • Never treat “no order to check” as “OK”. Send those records to a person, or to a second check.
  • Get the order number another way. The IVR of Module 7 asks callers to type it on the keypad. The check digit then catches most typing errors.

Grace has 60 minutes of team-lead review a day, at US$0.75 a minute (the course’s assumptions). Suppose that checking one flagged record takes 2 minutes (our assumption). On the 110 validation and test calls, the order check flagged 9 records. That is 18 minutes of review, or US$13.50. One missed follow-up costs US$25. Each flag that finds a real problem pays for itself. The French promise flags are different: 22 of the 110 calls, all harmless translations. Fix them in the prompt, not with review minutes.

Common mistake

You test a check with the labels instead of the transcript

Symptom: in your test, the order check catches every wrong promise. In production, it misses some.

Cause: your test read the order number from the labels, because it was easier. The notebook shows it: with the labels’ order numbers, it flags 9 of 9 calls; with Whisper’s transcripts, 6 of 9. Production has only Whisper’s transcript.

Fix: test every check on the same input that it gets in production: the recognised transcript, with its errors.

Common mistake

The program that reads the record crashes on a new word

Symptom: one day, the routing code fails with a KeyError, or sends calls to nobody.

Cause: the LLM wrote a reason or a next step that is not on the list, or left a field out. The course asked for JSON (JSON mode), not for a fixed schema, so nothing stopped it. In this set, all 220 records had the right form, but that is not a promise for the next 220.

Fix: check the form of every record first, as in check 1. Send a record with a wrong form back to the model once, then to a person. The LLM application course shows this step.

Use an LLM call record when people act on calls that nobody has time to listen to again: follow-ups, callbacks, refunds. One answer gives the reason, the summary, the promise and the next step. On Larkfield’s calls, it costs about US$0.13–0.15 for 1,000 calls (recorded at the deployment’s price; prices change).

Do not let the record act alone. A record copies the call, mistakes included. Check its form, compare it with the call, and check it against the systems that know the facts. Send every flag, and every record that a check could not test, to a person.

Check your understanding

Tomás says: 'The LLM's record of C-0161 is word for word what the agent said, so it is correct.' What do you tell him?

Show the answer

The record is faithful, but faithful is not the same as right. The agent said the order was delivered. Then the agent promised a courier call before tomorrow’s delivery: a wrong promise, and a mistake in our own scripts. The record copied it, with next step courier_call. Only a check against the order system, which knows that order 260026 is delivered, can flag it. On the 110 validation and test calls, that check flagged 9 such records; the LLM flagged none.

Check your understanding

Predict: on Whisper small's transcripts, the promise matches the labels in 78% of the test calls. Does that mean the LLM invented a promise in 22% of the calls?

Show the answer

No. Every miss is a French call, and the English promises match in 34 of 34 calls. The model translated French promises into English, so a letter-by-letter comparison scores them low (C-0033: similarity 0.43, with the same meaning). The check found a difference in language, not an invented promise. Read the flagged records before you decide what the number means.

Check your understanding

Practical check: in the notebook, run the order-system check on Whisper small's transcripts. How many of the 9 wrong promises does it flag, and what stops it on the others?

Show the answer

It flags 6 of the 9. On C-0068, C-0074 and C-0161, Whisper small added or lost a digit of the order number (“1540369”, “49849”, “26026”). The number has the wrong length, so the check finds no order to look up and cannot run. This is the transcript error that breaks the call record most often here: a misheard order number. The cheapest fixes are to send every “no order to check” record to a person, and to take the order number by keypad.

Recap

  • A call record is a summary with fixed fields (reason, summary, promise, next step, recording notice, follow-up), written by an LLM in JSON.
  • Check every record three ways: its form, each field against the call, and the facts against the business systems.
  • On Larkfield’s 60 synthetic test calls, four fields agree with the labels on every call. The promise agrees on 83% (exact transcripts) and 78% (Whisper small). Every miss is a French promise that the model wrote in English.
  • A record can be faithful and wrong: the order-system check flagged 9 courier promises for delivered orders, which the LLM had copied.
  • On Whisper’s transcripts, the check flagged only 6 of the 9: a misheard order number switched it off. Never treat “no order to check” as “OK”.

The record also has a field for the recording notice. Larkfield’s rules require it in every call, so it deserves a better check than an LLM’s yes or no. Next, we search for the notice in the transcript, exactly and with fuzzy match, and in the audio itself.

Next: Compliance phrases and keyword spotting