Batch and streaming
- Lesson 33
- 35 min
- Module 9: Run it in production, safely
Where we are
We chose a design for the phone line and a pipeline for recorded calls. Now they must run every day. In this lesson, we compare a batch queue with a streaming path, and the cost per audio hour on a CPU, a GPU and a hosted API.
Lesson 33 of 36 · Speech AI: Transcribe, Understand and SpeakBatch and streaming
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Run speech every day, in two ways. Recorded calls go into a batch queue: a worker transcribes them one by one, after the call. The phone line needs a streaming path: it must decide while the caller speaks. Measure the speed of the batch path, turn it into a cost per audio hour, and compare it with a GPU and with hosted services.
NotebookDo this lesson in a notebook
The notebook computes the real-time factor and the cost per audio hour from the course’s recorded runs. It builds a small queue worker that can stop and start again without doing a call twice. Then it reads one IVR call in 200 ms pieces, as a streaming path does, and finds where each caller turn ends. It downloads Whisper tiny once (75 MB) and needs no account. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The real-time factor and the cost per audio hour, from Whisper sizes on a laptop.
- The latency budget of a voice turn, from The voice-agent pipeline.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- Your folder
larkspeechwith the files of Menu, agent or both?.pytest -qprints37 passed.
Tested on macOS with Python 3.12.13. The commands are the same on Windows, except where a tab shows PowerShell; they were not run there.
Two kinds of note-taker
Section titled “Two kinds of note-taker”Think of two people who turn speech into text. The first one gets a box of recorded meetings every evening. She types them up during the night, one after another. Nobody waits for her. If she stops at midnight, she continues in the morning with the next tape.
The second one is a live interpreter on a phone call. He must answer within a second or two, or the caller thinks the line is dead. He never gets the whole conversation at once. He hears it in small pieces, and he must decide when the other person has finished.
Larkfield has both jobs:
- Recorded calls. About 4,000 calls a month are recorded (with consent). Grace wants a transcript, a reason and a summary for each one by the next morning. Nobody waits on the phone.
- The phone line. Out of hours, about 900 callers a month talk to the IVR or the voice agent. Each answer must come back in about one to two seconds.
The problem: one design does not fit both
Section titled “The problem: one design does not fit both”Try to use one design for both jobs, and one of them fails.
Use the batch design on the phone line, and the caller waits for the end of the call before anything happens. That is useless. Even with a streaming design, the wait is long. In The voice-agent pipeline, the median wait from “caller stops” to “first sound back” was 2.10 seconds on Larkfield’s synthetic calls.
Use the streaming design for recorded calls, and you pay for speed that nobody needs. A streaming server must be ready at the busiest minute of the day. Most of the time, it waits. And a crash in the middle of the night loses the calls that were in progress, unless something remembers which calls are done.
The idea: a queue for recordings, small pieces for the line
Section titled “The idea: a queue for recordings, small pieces for the line”The diagram shows the two paths side by side.
Text description of the diagram
Two paths. Batch, after the call: the call ends, and the recording goes into an inbox folder. A queue holds the calls that wait for their turn. A worker transcribes one call after another. A results file holds the redacted text of each call, kept by call ID. Streaming, during the call: the caller speaks, and the audio arrives in small pieces. Every 200 ms, the voice activity detector checks if the caller still speaks. After 0.8 seconds of silence, the turn ends, and the recogniser reads it. The IVR answers in about 1 to 2 seconds.
Some words for the rest of the lesson:
- A queue is a list of jobs that wait for their turn. Here, each job is one recorded call.
- A worker is the program that takes the next job from the queue and does it.
- Batch means “many jobs together, done later”. Nobody waits for one job.
- Streaming means the audio arrives in small pieces while the person speaks, and the program works on each piece at once.
- Latency is how long one person waits for one answer. Throughput is how much work is done in an hour. Batch cares about throughput; streaming cares about latency.
Why the queue remembers what it did
Section titled “Why the queue remembers what it did”Larkfield’s worker writes one line to a results file after each call, with the call’s file name. When it starts, it reads that file and skips every call that is already there. So a worker that stops at call 300 of 500 starts again at call 301. Nothing is done twice, and nothing is lost. This property has a name: the worker is safe to run again (engineers also say idempotent).
A tiny worked example: from seconds to dollars
Section titled “A tiny worked example: from seconds to dollars”The course ran the batch worker on three test calls with Whisper base. The numbers below are from the course’s snapshot check.
| Step | Value |
|---|---|
| Audio in the inbox | 3 calls, 2.3 minutes (137 s) |
| Compute time of the worker | 3.4 s |
| Real-time factor = compute time ÷ audio time | 3.4 ÷ 137 = 0.025 |
| One hour of audio takes | 0.025 × 60 minutes = 1.5 minutes |
| CPU server price (assumption: 4 vCPUs, 16 GB, on demand) | US$0.17 an hour |
| Cost per audio hour = real-time factor × server price | 0.025 × 0.17 = US$0.0042 |
The real-time factor (RTF) says how much faster than real time the work runs. 0.025 means 40 times faster. The cost per audio hour assumes one transcription at a time, and a server that is busy all the time. Both are true for a queue: the worker never waits while calls are in the inbox.
Larkfield’s month
Section titled “Larkfield’s month”How many audio hours does Larkfield have? Our synthetic calls last 44 seconds on average, but real calls are longer. A call handled by a person costs 6 minutes of agent time in the course’s assumptions. We use that as an upper bound for the audio: 4,000 calls × 6 minutes = 400 audio hours a month. This is our assumption, not a measurement.
With Whisper base on the CPU server, that is 400 × US$0.0051 = US$2.04 a month of compute. One server finishes a month of audio in about 12 hours (400 × 0.030). The night is long enough.
The streaming path: you pay to be ready
Section titled “The streaming path: you pay to be ready”The IVR of Module 7 already streams. It reads the call in 20 ms frames. Every 200 ms, Silero VAD looks at the recent audio. After 0.8 seconds without speech, the turn has ended. Then Whisper base reads only that turn, in about 0.24 seconds on the course’s computer. The voice agent of Module 8 adds about 1 second for the LLM’s first word.
A streaming server costs money even when nobody calls. It must be ready for the busiest minute. Each call needs its recogniser at once, not tonight. The course assumes US$0.002 of compute for each IVR call: recognition and voice on the same CPU server, rounded up. For 900 out-of-hours calls a month, that is US$1.80. The bigger cost is the server that waits. One CPU server for a whole month is about 730 hours × US$0.17 = US$124, whatever the number of calls.
Live captions are a third kind of streaming. The recogniser writes words while the person speaks and corrects them when it hears more (“for” becomes “four one one one”). The course did not build or measure live captions. They need a recogniser that can return partial results, and the corrections must not confuse the reader.
Cost per audio hour by option
Section titled “Cost per audio hour by option”The table puts the options side by side. The local rows use the course’s measured real-time factors (test calls, 4 threads, no VAD) and the CPU price assumption. The hosted rows are vendor claims: list prices on the vendors’ pages, pay as you go. The course did not measure them. Prices change: check the current page before you decide.
| Option | WER on our test calls | Cost per audio hour | Where the audio goes |
|---|---|---|---|
| Whisper tiny, CPU | 20.7% | US$0.0033 | stays on Larkfield’s server |
| Whisper base, CPU | 13.7% | US$0.0051 | stays on Larkfield’s server |
| Whisper small, CPU | 8.6% | US$0.013 | stays on Larkfield’s server |
| Whisper medium, CPU (first 20 calls) | 6.9% | US$0.033 | stays on Larkfield’s server |
| Azure AI Speech, batch | not measured | US$0.18 (vendor claim) | to the vendor’s region |
| Amazon Transcribe, batch | not measured | US$0.36 (vendor claim) | to the vendor’s region |
| Amazon Transcribe, streaming | not measured | US$0.60 (vendor claim) | to the vendor’s region |
| Azure AI Speech, real time | not measured | US$1.00 (vendor claim) | to the vendor’s region |
The price pages are Azure AI Speech pricing and Amazon Transcribe pricing. For Larkfield’s assumed 400 audio hours a month, the chart shows the bill.
- Whisper on a CPU server (measured speed × assumed price)
- Azure AI Speech (vendor claim)
- Amazon Transcribe (vendor claim)
Text description of the chart
A column chart of the monthly cost for 400 audio hours. Whisper on a CPU server costs US$1.32 with tiny and US$2.04 with base. It costs US$5.08 with small and US$13.12 with medium. Azure AI Speech batch: US$72. Amazon Transcribe batch: US$144. Amazon Transcribe streaming: US$240. Azure AI Speech real time: US$400. The hosted streaming prices are about twice the batch prices of the same vendor.
Three things to see:
- Streaming costs more than batch at both vendors, for the same audio. The vendor must keep a model ready for you, as Larkfield would.
- Local compute is cheap at this size. The bill is a few dollars. The real costs are elsewhere: a person to run the server, updates, and the time to build the queue.
- The hosted rows buy other things: no server to run, more languages, and often better accuracy on real speech. They also send the audio to another company. The lesson A hosted speech API explains what to ask about that.
And a GPU?
Section titled “And a GPU?”The course did not measure a cloud GPU server, and it quotes no GPU price. It did measure the Apple GPU of the laptop. There, whisper.cpp base ran at a real-time factor of 0.012, against 0.020 on the CPU. The WER was the same within noise. A GPU server costs much more per hour than a CPU server. It pays only when it stays busy: a large batch, a big model, or many calls at once. Use the same formula with your own price. The cost per audio hour is the real-time factor on that machine × its price per hour.
Guided practice: run the batch queue
Section titled “Guided practice: run the batch queue”This module adds five files to your project. Copy them all now. Two of them are for this lesson: queue.py (the worker) and commands/batch.py (the command speech batch). The worker already removes card numbers from the text with redact.py. The next lesson explains that file.
-
Open a terminal in your project folder
larkspeech, and activate.venv. -
Copy the five files of the snapshot
end-of-m09.Terminal window E=../nextia-learning-labs/C41/snapshots/end-of-m09cp $E/larkspeech/queue.py $E/larkspeech/redact.py larkspeech/cp $E/larkspeech/commands/batch.py $E/larkspeech/commands/privacy.py larkspeech/commands/cp $E/tests/test_privacy.py tests/Terminal window $E = "..\nextia-learning-labs\C41\snapshots\end-of-m09"Copy-Item $E\larkspeech\queue.py, $E\larkspeech\redact.py larkspeech\Copy-Item $E\larkspeech\commands\batch.py, $E\larkspeech\commands\privacy.py larkspeech\commands\Copy-Item $E\tests\test_privacy.py tests\This module needs no new package.
-
Run the tests:
Terminal window pytest -qYou should see
41 passed: the 37 tests of Module 8 and 4 new tests. -
Make an inbox with three recorded test calls:
Terminal window mkdir inboxcp data/calls/phone/C-0245.wav data/calls/phone/C-0128.wav data/calls/phone/C-0225.wav inbox/Terminal window New-Item -ItemType Directory inboxCopy-Item data\calls\phone\C-0245.wav, data\calls\phone\C-0128.wav, data\calls\phone\C-0225.wav inbox\ -
Run the worker on the inbox:
Terminal window speech batch inboxOne output on the course’s computer:
3 calls (2.3 min of audio) in 2.9 s: real-time factor 0.021; 0 done beforeCost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0036Wrote outputs/batch-results.jsonl (card numbers removed from the text)Your numbers will differ. Three runs on the course’s computer gave real-time factors from 0.021 to 0.032. The snapshot check gave 0.025.
-
Run the same command again:
Terminal window speech batch inboxNothing new in inbox (3 done before). Results: outputs/batch-results.jsonlThe worker read its results file and skipped the three calls.
-
Add one more call to the inbox, and run the worker again:
Terminal window cp data/calls/phone/C-0052.wav inbox/speech batch inboxOn Windows, copy the file with
Copy-Item data\calls\phone\C-0052.wav inbox\.1 calls (0.7 min of audio) in 1.1 s: real-time factor 0.025; 3 done beforeCost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0043Wrote outputs/batch-results.jsonl (card numbers removed from the text)Only the new call was transcribed.
Your turn: a bigger model in the queue
Section titled “Your turn: a bigger model in the queue”Delete the results file, and run the worker with Whisper small. Before you run it, predict: how much more does an audio hour cost?
Show the answer
Delete outputs/batch-results.jsonl (on Windows: Remove-Item outputs\batch-results.jsonl), then:
speech batch inbox --model small4 calls (3.0 min of audio) in 11.0 s: real-time factor 0.061; 0 done beforeCost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0104Wrote outputs/batch-results.jsonl (card numbers removed from the text)Whisper small costs about 2.5 to 3 times as much as Whisper base, about US$0.01 an audio hour. For Larkfield’s assumed month, that is still only about US$4. Whisper small makes far fewer errors: 8.6% against 13.7% on the test calls. For the batch path, the bigger model is worth its price. The first run downloads the model (about 480 MB). The download is not part of the compute time.
Common mistakes
Section titled “Common mistakes”Common mistake
The real-time factor changes every time you measure it
Symptom: you run the same command three times. You get real-time factors of 0.032, 0.022 and 0.021. With Whisper tiny, one run was even slower than Whisper base.
Cause: three short calls are only a few seconds of compute. The first call may warm up the model, and other programs on the computer take the CPU for a moment. Small numbers move a lot.
Fix: measure on at least an hour of audio, run it twice, and keep the second run. The course’s table used the 60 test calls (47 minutes of audio) for this reason.
Common mistake
The streaming server is sized for the average
Symptom: the out-of-hours line works in tests, but on a busy evening callers hear long silences.
Cause: 900 calls a month is only about one call an hour on average. But calls come in groups: after a delivery problem, ten callers may ring in the same minute. Each one needs a recogniser at once.
Fix: size the streaming path for the busiest minute that you expect, with some spare room. Test it with several replayed calls at the same time. The course tested one call at a time only.
When to use each path
Section titled “When to use each path”- Batch with a queue: recordings, nightly reports, search and compliance checks. Pick the most accurate model you can afford: here, Whisper small costs a few dollars a month.
- Streaming: the IVR, the voice agent, live captions. Pick for latency first: the IVR of Module 7 uses Whisper base, which reads a turn in about 0.24 seconds.
- Both, from one call: the phone line streams during the call, and the recording goes into the batch queue afterwards for a better transcript. This is Larkfield’s plan.
- Hosted: when you do not want to run servers, or need languages and accuracy that the local models lack. Read the price page and the data terms first.
Knowledge check
Section titled “Knowledge check”Check your understanding
The worker transcribed 3 calls (2.3 minutes of audio) in 3.4 seconds on a server that costs US$0.17 an hour. Compute the real-time factor and the cost per audio hour. Show the steps.
Show the answer
2.3 minutes is about 137 seconds. The real-time factor is 3.4 ÷ 137 = 0.025. One hour of audio then takes 0.025 hours of compute: 1.5 minutes. The cost per audio hour is 0.025 × US$0.17 = US$0.0042. This assumes a server that is busy all the time. That is true for a queue with calls waiting.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Tomás says: 'Hosted streaming costs US$0.60 an audio hour and our CPU costs US$0.005. So a streaming server of our own is more than 100 times cheaper.' What is wrong with his comparison?
Show the answer
The US$0.005 is a batch number. It assumes a server that is busy all the time. A streaming server must be ready for the busiest minute, and it waits most of the day. So you pay for all its hours: about US$124 a month for one CPU server, whatever the number of calls. For 900 out-of-hours calls, that is far more than the compute itself. Also, the hosted price is a vendor claim that can change. It includes things that a local server does not: no server to run, and more languages. The comparison must use the same kind of work and the full monthly cost.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Practical check: with three calls in the inbox and their results already written, you add C-0052 and run speech batch inbox. What does the worker print in its first line, and why?
Show the answer
It prints 1 calls (0.7 min of audio) in … s: real-time factor …; 3 done before. The worker reads outputs/batch-results.jsonl first. Three file names are already there, so it skips those calls and transcribes only C-0052. This makes the worker safe to run again after a crash.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Batch: a queue of recorded calls and a worker that does one after another. It cares about throughput and cost. It writes a results file and skips the calls it has done, so it is safe to run again.
- Streaming: the audio arrives in small pieces, and the program decides at once. It cares about latency, and the server must be ready for the busiest minute.
- Cost per audio hour = real-time factor × server price per hour. The batch worker: 0.025 × US$0.17 = US$0.0042 on Larkfield’s synthetic calls.
- For an assumed 400 audio hours a month, Whisper on a CPU costs a few dollars. Hosted batch lists US$72 to US$144 a month, and hosted streaming more (vendor claims; prices change).
The queue now writes a transcript of every call. Those transcripts and recordings hold names, order numbers and sometimes a card number that a caller read aloud. Next, we remove them from the text and from the audio.
Next: Personal data in voice