Skip to content
Français

Batch and streaming

  • Lesson 33
  • 35 min
  • Module 9: Run it in production, safely

Where we are

We chose a design for the phone line and a pipeline for recorded calls. Now they must run every day. In this lesson, we compare a batch queue with a streaming path, and the cost per audio hour on a CPU, a GPU and a hosted API.

What you will be able to do

Run speech every day, in two ways. Recorded calls go into a batch queue: a worker transcribes them one by one, after the call. The phone line needs a streaming path: it must decide while the caller speaks. Measure the speed of the batch path, turn it into a cost per audio hour, and compare it with a GPU and with hosted services.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook computes the real-time factor and the cost per audio hour from the course’s recorded runs. It builds a small queue worker that can stop and start again without doing a call twice. Then it reads one IVR call in 200 ms pieces, as a streaming path does, and finds where each caller turn ends. It downloads Whisper tiny once (75 MB) and needs no account. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the guided practice; the notebook needs none of it)

Tested on macOS with Python 3.12.13. The commands are the same on Windows, except where a tab shows PowerShell; they were not run there.

Think of two people who turn speech into text. The first one gets a box of recorded meetings every evening. She types them up during the night, one after another. Nobody waits for her. If she stops at midnight, she continues in the morning with the next tape.

The second one is a live interpreter on a phone call. He must answer within a second or two, or the caller thinks the line is dead. He never gets the whole conversation at once. He hears it in small pieces, and he must decide when the other person has finished.

Larkfield has both jobs:

  • Recorded calls. About 4,000 calls a month are recorded (with consent). Grace wants a transcript, a reason and a summary for each one by the next morning. Nobody waits on the phone.
  • The phone line. Out of hours, about 900 callers a month talk to the IVR or the voice agent. Each answer must come back in about one to two seconds.

Try to use one design for both jobs, and one of them fails.

Use the batch design on the phone line, and the caller waits for the end of the call before anything happens. That is useless. Even with a streaming design, the wait is long. In The voice-agent pipeline, the median wait from “caller stops” to “first sound back” was 2.10 seconds on Larkfield’s synthetic calls.

Use the streaming design for recorded calls, and you pay for speed that nobody needs. A streaming server must be ready at the busiest minute of the day. Most of the time, it waits. And a crash in the middle of the night loses the calls that were in progress, unless something remembers which calls are done.

The idea: a queue for recordings, small pieces for the line

Section titled “The idea: a queue for recordings, small pieces for the line”

The diagram shows the two paths side by side.

Batch, after the call: the call ends and the recording goes into the inbox; a queue holds the calls that wait their turn; a worker transcribes one call after another; a results file holds the redacted text, kept by call ID. Streaming, during the call: the caller speaks and the audio arrives in small pieces; the VAD checks every 200 ms whether the caller is still speaking; after 0.8 s of silence the turn ends and is recognised; the IVR answers in about 1 to 2 seconds.
Batch and streaming. The batch path cares about cost and never losing a call; the streaming path cares about the wait.
Text description of the diagram

Two paths. Batch, after the call: the call ends, and the recording goes into an inbox folder. A queue holds the calls that wait for their turn. A worker transcribes one call after another. A results file holds the redacted text of each call, kept by call ID. Streaming, during the call: the caller speaks, and the audio arrives in small pieces. Every 200 ms, the voice activity detector checks if the caller still speaks. After 0.8 seconds of silence, the turn ends, and the recogniser reads it. The IVR answers in about 1 to 2 seconds.

Some words for the rest of the lesson:

  • A queue is a list of jobs that wait for their turn. Here, each job is one recorded call.
  • A worker is the program that takes the next job from the queue and does it.
  • Batch means “many jobs together, done later”. Nobody waits for one job.
  • Streaming means the audio arrives in small pieces while the person speaks, and the program works on each piece at once.
  • Latency is how long one person waits for one answer. Throughput is how much work is done in an hour. Batch cares about throughput; streaming cares about latency.

Larkfield’s worker writes one line to a results file after each call, with the call’s file name. When it starts, it reads that file and skips every call that is already there. So a worker that stops at call 300 of 500 starts again at call 301. Nothing is done twice, and nothing is lost. This property has a name: the worker is safe to run again (engineers also say idempotent).

A tiny worked example: from seconds to dollars

Section titled “A tiny worked example: from seconds to dollars”

The course ran the batch worker on three test calls with Whisper base. The numbers below are from the course’s snapshot check.

Step Value
Audio in the inbox 3 calls, 2.3 minutes (137 s)
Compute time of the worker 3.4 s
Real-time factor = compute time ÷ audio time 3.4 ÷ 137 = 0.025
One hour of audio takes 0.025 × 60 minutes = 1.5 minutes
CPU server price (assumption: 4 vCPUs, 16 GB, on demand) US$0.17 an hour
Cost per audio hour = real-time factor × server price 0.025 × 0.17 = US$0.0042

The real-time factor (RTF) says how much faster than real time the work runs. 0.025 means 40 times faster. The cost per audio hour assumes one transcription at a time, and a server that is busy all the time. Both are true for a queue: the worker never waits while calls are in the inbox.

How many audio hours does Larkfield have? Our synthetic calls last 44 seconds on average, but real calls are longer. A call handled by a person costs 6 minutes of agent time in the course’s assumptions. We use that as an upper bound for the audio: 4,000 calls × 6 minutes = 400 audio hours a month. This is our assumption, not a measurement.

With Whisper base on the CPU server, that is 400 × US$0.0051 = US$2.04 a month of compute. One server finishes a month of audio in about 12 hours (400 × 0.030). The night is long enough.

The IVR of Module 7 already streams. It reads the call in 20 ms frames. Every 200 ms, Silero VAD looks at the recent audio. After 0.8 seconds without speech, the turn has ended. Then Whisper base reads only that turn, in about 0.24 seconds on the course’s computer. The voice agent of Module 8 adds about 1 second for the LLM’s first word.

A streaming server costs money even when nobody calls. It must be ready for the busiest minute. Each call needs its recogniser at once, not tonight. The course assumes US$0.002 of compute for each IVR call: recognition and voice on the same CPU server, rounded up. For 900 out-of-hours calls a month, that is US$1.80. The bigger cost is the server that waits. One CPU server for a whole month is about 730 hours × US$0.17 = US$124, whatever the number of calls.

Live captions are a third kind of streaming. The recogniser writes words while the person speaks and corrects them when it hears more (“for” becomes “four one one one”). The course did not build or measure live captions. They need a recogniser that can return partial results, and the corrections must not confuse the reader.

The table puts the options side by side. The local rows use the course’s measured real-time factors (test calls, 4 threads, no VAD) and the CPU price assumption. The hosted rows are vendor claims: list prices on the vendors’ pages, pay as you go. The course did not measure them. Prices change: check the current page before you decide.

Option WER on our test calls Cost per audio hour Where the audio goes
Whisper tiny, CPU 20.7% US$0.0033 stays on Larkfield’s server
Whisper base, CPU 13.7% US$0.0051 stays on Larkfield’s server
Whisper small, CPU 8.6% US$0.013 stays on Larkfield’s server
Whisper medium, CPU (first 20 calls) 6.9% US$0.033 stays on Larkfield’s server
Azure AI Speech, batch not measured US$0.18 (vendor claim) to the vendor’s region
Amazon Transcribe, batch not measured US$0.36 (vendor claim) to the vendor’s region
Amazon Transcribe, streaming not measured US$0.60 (vendor claim) to the vendor’s region
Azure AI Speech, real time not measured US$1.00 (vendor claim) to the vendor’s region

The price pages are Azure AI Speech pricing and Amazon Transcribe pricing. For Larkfield’s assumed 400 audio hours a month, the chart shows the bill.

  • Whisper on a CPU server (measured speed × assumed price)
  • Azure AI Speech (vendor claim)
  • Amazon Transcribe (vendor claim)
What 400 audio hours a month would cost (assumption: 4,000 calls of 6 minutes). Local rows: measured real-time factor on Larkfield's synthetic test calls × US$0.17 an hour. Hosted rows: list prices, not measured; prices change. Live: streaming or real-time transcription.
Text description of the chart

A column chart of the monthly cost for 400 audio hours. Whisper on a CPU server costs US$1.32 with tiny and US$2.04 with base. It costs US$5.08 with small and US$13.12 with medium. Azure AI Speech batch: US$72. Amazon Transcribe batch: US$144. Amazon Transcribe streaming: US$240. Azure AI Speech real time: US$400. The hosted streaming prices are about twice the batch prices of the same vendor.

Three things to see:

  • Streaming costs more than batch at both vendors, for the same audio. The vendor must keep a model ready for you, as Larkfield would.
  • Local compute is cheap at this size. The bill is a few dollars. The real costs are elsewhere: a person to run the server, updates, and the time to build the queue.
  • The hosted rows buy other things: no server to run, more languages, and often better accuracy on real speech. They also send the audio to another company. The lesson A hosted speech API explains what to ask about that.

The course did not measure a cloud GPU server, and it quotes no GPU price. It did measure the Apple GPU of the laptop. There, whisper.cpp base ran at a real-time factor of 0.012, against 0.020 on the CPU. The WER was the same within noise. A GPU server costs much more per hour than a CPU server. It pays only when it stays busy: a large batch, a big model, or many calls at once. Use the same formula with your own price. The cost per audio hour is the real-time factor on that machine × its price per hour.

This module adds five files to your project. Copy them all now. Two of them are for this lesson: queue.py (the worker) and commands/batch.py (the command speech batch). The worker already removes card numbers from the text with redact.py. The next lesson explains that file.

  1. Open a terminal in your project folder larkspeech, and activate .venv.

  2. Copy the five files of the snapshot end-of-m09.

    Terminal window
    E=../nextia-learning-labs/C41/snapshots/end-of-m09
    cp $E/larkspeech/queue.py $E/larkspeech/redact.py larkspeech/
    cp $E/larkspeech/commands/batch.py $E/larkspeech/commands/privacy.py larkspeech/commands/
    cp $E/tests/test_privacy.py tests/

    This module needs no new package.

  3. Run the tests:

    Terminal window
    pytest -q

    You should see 41 passed: the 37 tests of Module 8 and 4 new tests.

  4. Make an inbox with three recorded test calls:

    Terminal window
    mkdir inbox
    cp data/calls/phone/C-0245.wav data/calls/phone/C-0128.wav data/calls/phone/C-0225.wav inbox/
  5. Run the worker on the inbox:

    Terminal window
    speech batch inbox

    One output on the course’s computer:

    3 calls (2.3 min of audio) in 2.9 s: real-time factor 0.021; 0 done before
    Cost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0036
    Wrote outputs/batch-results.jsonl (card numbers removed from the text)

    Your numbers will differ. Three runs on the course’s computer gave real-time factors from 0.021 to 0.032. The snapshot check gave 0.025.

  6. Run the same command again:

    Terminal window
    speech batch inbox
    Nothing new in inbox (3 done before). Results: outputs/batch-results.jsonl

    The worker read its results file and skipped the three calls.

  7. Add one more call to the inbox, and run the worker again:

    Terminal window
    cp data/calls/phone/C-0052.wav inbox/
    speech batch inbox

    On Windows, copy the file with Copy-Item data\calls\phone\C-0052.wav inbox\.

    1 calls (0.7 min of audio) in 1.1 s: real-time factor 0.025; 3 done before
    Cost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0043
    Wrote outputs/batch-results.jsonl (card numbers removed from the text)

    Only the new call was transcribed.

Delete the results file, and run the worker with Whisper small. Before you run it, predict: how much more does an audio hour cost?

Show the answer

Delete outputs/batch-results.jsonl (on Windows: Remove-Item outputs\batch-results.jsonl), then:

Terminal window
speech batch inbox --model small
4 calls (3.0 min of audio) in 11.0 s: real-time factor 0.061; 0 done before
Cost per audio hour on a CPU server at US$0.17/hour (assumption): US$0.0104
Wrote outputs/batch-results.jsonl (card numbers removed from the text)

Whisper small costs about 2.5 to 3 times as much as Whisper base, about US$0.01 an audio hour. For Larkfield’s assumed month, that is still only about US$4. Whisper small makes far fewer errors: 8.6% against 13.7% on the test calls. For the batch path, the bigger model is worth its price. The first run downloads the model (about 480 MB). The download is not part of the compute time.

Common mistake

The real-time factor changes every time you measure it

Symptom: you run the same command three times. You get real-time factors of 0.032, 0.022 and 0.021. With Whisper tiny, one run was even slower than Whisper base.

Cause: three short calls are only a few seconds of compute. The first call may warm up the model, and other programs on the computer take the CPU for a moment. Small numbers move a lot.

Fix: measure on at least an hour of audio, run it twice, and keep the second run. The course’s table used the 60 test calls (47 minutes of audio) for this reason.

Common mistake

The streaming server is sized for the average

Symptom: the out-of-hours line works in tests, but on a busy evening callers hear long silences.

Cause: 900 calls a month is only about one call an hour on average. But calls come in groups: after a delivery problem, ten callers may ring in the same minute. Each one needs a recogniser at once.

Fix: size the streaming path for the busiest minute that you expect, with some spare room. Test it with several replayed calls at the same time. The course tested one call at a time only.

  • Batch with a queue: recordings, nightly reports, search and compliance checks. Pick the most accurate model you can afford: here, Whisper small costs a few dollars a month.
  • Streaming: the IVR, the voice agent, live captions. Pick for latency first: the IVR of Module 7 uses Whisper base, which reads a turn in about 0.24 seconds.
  • Both, from one call: the phone line streams during the call, and the recording goes into the batch queue afterwards for a better transcript. This is Larkfield’s plan.
  • Hosted: when you do not want to run servers, or need languages and accuracy that the local models lack. Read the price page and the data terms first.

Check your understanding

The worker transcribed 3 calls (2.3 minutes of audio) in 3.4 seconds on a server that costs US$0.17 an hour. Compute the real-time factor and the cost per audio hour. Show the steps.

Show the answer

2.3 minutes is about 137 seconds. The real-time factor is 3.4 ÷ 137 = 0.025. One hour of audio then takes 0.025 hours of compute: 1.5 minutes. The cost per audio hour is 0.025 × US$0.17 = US$0.0042. This assumes a server that is busy all the time. That is true for a queue with calls waiting.

Check your understanding

Tomás says: 'Hosted streaming costs US$0.60 an audio hour and our CPU costs US$0.005. So a streaming server of our own is more than 100 times cheaper.' What is wrong with his comparison?

Show the answer

The US$0.005 is a batch number. It assumes a server that is busy all the time. A streaming server must be ready for the busiest minute, and it waits most of the day. So you pay for all its hours: about US$124 a month for one CPU server, whatever the number of calls. For 900 out-of-hours calls, that is far more than the compute itself. Also, the hosted price is a vendor claim that can change. It includes things that a local server does not: no server to run, and more languages. The comparison must use the same kind of work and the full monthly cost.

Check your understanding

Practical check: with three calls in the inbox and their results already written, you add C-0052 and run speech batch inbox. What does the worker print in its first line, and why?

Show the answer

It prints 1 calls (0.7 min of audio) in … s: real-time factor …; 3 done before. The worker reads outputs/batch-results.jsonl first. Three file names are already there, so it skips those calls and transcribes only C-0052. This makes the worker safe to run again after a crash.

Recap

  • Batch: a queue of recorded calls and a worker that does one after another. It cares about throughput and cost. It writes a results file and skips the calls it has done, so it is safe to run again.
  • Streaming: the audio arrives in small pieces, and the program decides at once. It cares about latency, and the server must be ready for the busiest minute.
  • Cost per audio hour = real-time factor × server price per hour. The batch worker: 0.025 × US$0.17 = US$0.0042 on Larkfield’s synthetic calls.
  • For an assumed 400 audio hours a month, Whisper on a CPU costs a few dollars. Hosted batch lists US$72 to US$144 a month, and hosted streaming more (vendor claims; prices change).

The queue now writes a transcript of every call. Those transcripts and recordings hold names, order numbers and sometimes a card number that a caller read aloud. Next, we remove them from the text and from the audio.

Next: Personal data in voice