Whisper sizes on a laptop
- Lesson 11
- 40 min
- Module 4: Speech recognition: compare the families
Where we are
We know how Vosk and Whisper turn sound into text. Whisper comes in many sizes, and Tomás wants the biggest. In this lesson, we run Whisper sizes on a laptop CPU and compare error by slice with speed, memory and cost. First, think of a shop that must choose a delivery van.
Lesson 11 of 36 · Speech AI: Transcribe, Understand and SpeakWhisper sizes on a laptop
1. The speech problem and its cost
2. Measure before you choose
3. Find the speech: voice activity detection
4. Speech recognition: compare the families
5. Who spoke when: diarization and timing
6. From transcript to decision
7. Build a small IVR
8. From IVR to voice agent
9. Run it in production, safely
What you will be able to do
Choose a Whisper size for Larkfield’s calls with evidence: its errors on the slices that matter, next to its speed, memory and cost on an ordinary computer. “The biggest, to be safe” is a choice too, and it has a price.
NotebookDo this lesson in a notebook
The notebook scores the recorded transcripts of Vosk small and Whisper tiny, base and small on the test calls, by slice. It draws error against speed, and runs Whisper tiny on three calls on your computer. Its setup cells install faster-whisper and download the files it needs (about 80 MB with the model). About 40 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- How Whisper’s encoder and decoder work, from From sound to text.
- The real-time factor, from Read the result in Set up the speech project, and WER by slice, from Build a test set by slice.
What you need
- Your
larkspeechfolder at the end of Module 3, with its.venvenvironment and the course data, and the labs foldernextia-learning-labsnext to it. - About 600 MB of free disk for the Whisper tiny, base and small models (downloaded on first use) and an internet connection the first time.
- Tested on macOS (Apple M4, 16 GB) with Python 3.12 and faster-whisper 1.2.1. The Windows commands are given but were not run.
Choosing a delivery van
Section titled “Choosing a delivery van”A small shop needs a delivery van. A big van carries more boxes, so it seems the safe choice. But it costs more to buy, uses more fuel on every trip, and does not fit in narrow streets. If the shop delivers twenty small parcels a day, a small van does the same job for less. The right question is not “which van is biggest?” but “which is the smallest van that carries our load?”
A speech model is the same. A bigger model makes fewer errors, but it needs more memory and more time for every minute of audio. Larkfield must find the smallest model that is accurate enough on its own calls.
The problem: “use the biggest, to be safe”
Section titled “The problem: “use the biggest, to be safe””Tomás, the AI engineer, says: “Speech recognition is solved. Use the biggest Whisper, and we never think about it again.” Whisper comes in sizes, from tiny (39 million parameters) to medium (769 million) and larger, as From sound to text showed.
Priya, the data scientist, measured the sizes on the same laptop and the same test calls. Medium was too slow to run on all 60 calls, so she ran it on the first 20. Compare on those 20 calls:
| Size | WER on the first 20 test calls | Real-time factor | Peak memory |
|---|---|---|---|
| tiny | 24.6% | 0.019 | 554 MB |
| base | 15.7% | 0.030 | 784 MB |
| small | 10.7% | 0.075 | 1,267 MB |
| medium | 6.9% | 0.19 | 1,851 MB |
Medium makes fewer errors. It also takes about 2.5 times as long as small for every minute of audio. Is that worth it? We cannot answer without a target, and without the slices.
Two numbers for every model: error and speed
Section titled “Two numbers for every model: error and speed”You met the real-time factor (RTF) in the setup lesson: compute time divided by audio time. An RTF of 0.075 means 7.5 seconds of compute for 100 seconds of audio. Smaller is faster. Below 1, the model is faster than the audio plays.
Every model now has two numbers: its error (WER) and its speed (RTF). One model is better than another only if it is better on both, or if the gain on one is worth the loss on the other. The team must decide what is “good enough” before it looks at the results. Otherwise, it picks the target that makes its favourite model win.
Priya and Grace agree on two targets for batch transcription of recorded calls:
- WER at most 10% on the phone test calls, after the normalisation guide.
- RTF at most 0.1, so that one CPU server transcribes an hour of calls in 6 minutes or less.
These targets are the team’s decision, not a law. A phone line that answers callers needs a stricter speed target, as Module 8 shows.
A tiny worked example: one month of calls
Section titled “A tiny worked example: one month of calls”Larkfield takes about 4,000 calls a month, about 49 hours of audio. The compute time for a month is the audio time × the RTF:
| Size | RTF | Compute for 49 hours of audio | Cost at US$0.17 an hour (assumption) |
|---|---|---|---|
| tiny | 0.019 | 49 × 0.019 = 0.9 hours | US$0.16 |
| base | 0.030 | 49 × 0.030 = 1.5 hours | US$0.25 |
| small | 0.075 | 49 × 0.075 = 3.7 hours | US$0.62 |
| medium | 0.19 | 49 × 0.19 = 9.3 hours | US$1.58 |
The price per hour of a CPU server (4 cores, 16 GB) is an assumption from the course’s cost file. At Larkfield’s volume, even medium costs less than two dollars of compute a month. So for batch work, speed is not about money here. It matters when the volume is a hundred times larger, or when a caller is waiting for an answer.
Error against speed: the picture
Section titled “Error against speed: the picture”Here is every recogniser that Priya measured, on the 60 phone test calls. The shaded corner is where both targets are met.
Text description of the diagram
Each point is one recogniser. Further left is faster, lower is more accurate. The shaded corner, below 10% WER and left of an RTF of 0.1, meets both targets. Only Whisper small is inside it: 8.6% at an RTF of 0.075, and 8.2% at 0.087 with float32 weights. Whisper tiny (20.7%) and base (13.7%) are faster but make too many errors. Medium is more accurate (6.9%, on 20 calls only) but slower than the target, at 0.19. Vosk small is both slower and less accurate than Whisper base. The whisper.cpp points are green for the CPU and orange for the Apple GPU. They are the same tiny and base models, run by another program: faster, with about the same errors.
So, for batch transcription, Whisper small is the smallest model that meets both targets. Medium is better on errors, but it is outside the speed target. Choosing it would be choosing a bigger van “to be safe”.
Why the same model runs at different speeds
Section titled “Why the same model runs at different speeds”8-bit weights: int8 quantisation
Section titled “8-bit weights: int8 quantisation”A model’s parameters are numbers. They are usually stored as 32-bit floating-point numbers (float32). Quantisation stores each one with fewer bits. With int8, each one is an 8-bit whole number, plus a scale. It is like writing prices to the nearest cent instead of to the nearest thousandth of a cent: a little less exact, and much smaller.
The course uses int8 for every Whisper run. For Whisper small, Priya measured both:
| Weights | WER | RTF | Peak memory |
|---|---|---|---|
| float32 | 8.2% | 0.087 | 2.21 GB |
| int8 | 8.6% | 0.075 | 1.27 GB |
The WER is close: float32 makes 26 fewer word errors in 7,334 words, or 0.4 points. The memory is almost half. That is why the course project runs faster-whisper with int8 weights.
- int8 weights
- float32 weights
Text description of the chart
Five columns. Whisper tiny needs 554 MB, base 784 MB and small 1,267 MB with int8 weights. Small with float32 weights needs 2,214 MB, more than medium with int8 weights (1,851 MB). On a 16 GB laptop, all of them fit. On a small cloud server that also runs other programs, the float32 version could be the one that does not fit.
whisper.cpp and the Apple GPU
Section titled “whisper.cpp and the Apple GPU”faster-whisper is one program that runs Whisper’s models. whisper.cpp is another, written in C++. Priya ran the same tiny and base models in whisper.cpp twice: on the CPU, and on the laptop’s Apple GPU (Metal). whisper.cpp uses the GPU by default when it finds one.
| Model | faster-whisper, CPU | whisper.cpp, CPU | whisper.cpp, Apple GPU |
|---|---|---|---|
| tiny | 20.7%, RTF 0.019 | 21.7%, RTF 0.011 | 22.8%, RTF 0.008 |
| base | 13.7%, RTF 0.030 | 15.3%, RTF 0.020 | 15.2%, RTF 0.012 |
The errors are close: the model is the same, and small differences come from how each program searches and rounds. The speed depends on the program and on the hardware it finds. A speed measured on one machine is a fact about that machine. Measure on the computer you will use.
Slices: where the overall number hides the problem
Section titled “Slices: where the overall number hides the problem”The overall WER mixes English and French, three lines and three kinds of background. Here are the two biggest slices, by language:
| Size | English (34 calls) | French (26 calls) |
|---|---|---|
| tiny | 7.0% | 37.0% |
| base | 5.2% | 23.8% |
| small | 3.1% | 15.1% |
On English calls, even tiny is under the 10% target. On French calls, even small is above it. So “Whisper small meets the target” is true on average, and false for every French caller. The last lesson of this module looks at the languages, the lines and the noise.
The line itself
Section titled “The line itself”Each call also has a wideband version, at 16,000 samples a second. The same models on the same calls make fewer errors there: tiny 17.6% against 20.7% on the phone, base 12.4% against 13.7%, small 7.7% against 8.6%. The phone line costs accuracy for every size. If Larkfield could get wideband recordings from its phone provider, every model would improve at no compute cost.
Our calls are easier than real calls
Section titled “Our calls are easier than real calls”Here is an honest surprise. Many people say “a model scores 6% on audiobooks and 24% on the phone”. On Larkfield’s English calls, that did not happen for Whisper. Whisper small has a WER of only 3.1% on them.
As The errors that matter showed, the same Whisper small makes 7.8% errors on real read sentences from LibriSpeech (8.1% through our phone line). That is more than twice the error on our English calls. Our synthetic callers read clean scripts. So every number in this course is “on Larkfield’s synthetic calls”, and a real call set would score worse. The gap still appears where a model is weak: Vosk small has 24.3% on the English phone calls, and Whisper tiny 37.0% on the French ones.
Priya also kept 40 calls hidden until the end. She ran Whisper small on them once: WER 10.7% (English 4.6%, French 17.1%). That is higher than the 8.6% of the test calls, and just above the 10% target. The French calls make most of the errors. We rebuilt the hidden calls, and their French calls now use a newer version of one French voice. That is why the score is a little above the target. The new name of the agent has nothing to do with it. Still, a choice made on the test calls has only a small margin.
Guided practice: run Whisper sizes yourself
Section titled “Guided practice: run Whisper sizes yourself”So far, every number came from recorded transcripts. Now you add the transcribe command to your project and run Whisper on your own computer.
-
In a terminal, go to your folder
larkspeechand activate the environment (source .venv/bin/activate, or.venv\Scripts\Activate.ps1on Windows). Copy the new command file from the snapshotend-of-m04into your project.macOS and Linux:
Terminal window cp ../nextia-learning-labs/C41/snapshots/end-of-m04/larkspeech/commands/transcribe.py larkspeech/commands/Windows PowerShell:
Terminal window Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\larkspeech\commands\transcribe.py larkspeech\commands\The
speechcommand finds its subcommands by their files, so nothing else changes. -
Check that the command is there:
Terminal window speech transcribe --helpusage: speech transcribe [-h] [--model {tiny,base,small,medium}][--split SPLIT] [--kind {phone,wideband}][--limit LIMIT] [--vad] [--prompt][--language {en,fr}]options:-h, --help show this help message and exit--model {tiny,base,small,medium}--split SPLIT--kind {phone,wideband}--limit LIMIT only the first N calls (0 for all)--vad cut the silences first (Silero VAD)--prompt give Whisper the Larkfield vocabulary--language {en,fr} only calls in this language -
Run Whisper tiny on the first 5 test calls. The first run downloads the model (about 75 MB).
Terminal window speech transcribe --model tiny --limit 5whisper-tiny on 5 test calls (phone): WER 36.0%, CER 19.7%, order numbers wrong 2 of 4, real-time factor 0.022Wrote outputs/transcripts-whisper-tiny.jsonlThe command transcribes each call, saves the transcripts in
outputs/, and scores them with the same guide asspeech score. Five calls are a small sample: one bad call moves the WER a lot. Your numbers can differ a little from these; the note after the steps explains why. The first run also prints download progress (Fetching 6 files) and a warning about unauthenticated requests to Hugging Face. You can ignore both. -
Run base and small on the same 5 calls. Small downloads about 480 MB the first time.
Terminal window speech transcribe --model base --limit 5speech transcribe --model small --limit 5whisper-base on 5 test calls (phone): WER 22.6%, CER 11.3%, order numbers wrong 2 of 4, real-time factor 0.063Wrote outputs/transcripts-whisper-base.jsonlwhisper-small on 5 test calls (phone): WER 11.3%, CER 5.0%, order numbers wrong 0 of 4, real-time factor 0.085Wrote outputs/transcripts-whisper-small.jsonl
Your real-time factors will differ: they depend on your processor and on what else the computer is doing. On a free cloud notebook, they are often 2 to 5 times larger. Even on one computer, 5 calls give an unsteady speed. A second run of base on the same calls gave an RTF of 0.102 here, while other programs were busy. The recorded run of all 60 calls gave 0.030. The WER should be the same or very close, because the model and its settings are pinned. Our second run gave 23.1% instead of 22.6%: the program’s arithmetic on several CPU threads can change a word now and then. Measure speed on many calls, on a quiet computer.
Your turn: the wideband version
Section titled “Your turn: the wideband version”Predict: on the wideband files of the same 5 calls, will Whisper tiny’s WER go down? By how much? Then run it with --kind wideband.
Check your answer
speech transcribe --model tiny --limit 5 --kind widebandwhisper-tiny on 5 test calls (wideband): WER 22.9%, CER 11.1%, order numbers wrong 2 of 4, real-time factor 0.028Wrote outputs/transcripts-whisper-tiny.jsonlOn all 60 test calls, tiny goes from 20.7% on the phone to 17.6% on wideband. On 5 calls, the change can be larger or smaller: compare with the full table, not with your 5 calls.
Common mistake
Comparing models on different calls
Symptom: Tomás puts two numbers side by side: Whisper medium 6.9%, Whisper small 8.6%. “Medium only gains 1.7 points,” he says. Next week he says the opposite.
Cause: medium ran on the first 20 test calls only, and small on all 60. The first 20 calls are harder than the average: on those same 20 calls, small has 10.7%, so medium gains 3.8 points there. The two numbers in the first sentence measure different calls.
Fix: compare models only on the same calls, with the same normalisation guide. If one model ran on fewer calls, score the others on those calls too, and say so.
Consequence in AI work: a comparison on different test data can reverse a decision. The course’s comparison table keeps one test split for every method for this reason.
Check your understanding
Whisper small has a WER of 8.6% and an RTF of 0.075. Whisper medium has 6.9% (on 20 calls) and an RTF of 0.19. With the targets WER ≤ 10% and RTF ≤ 0.1, which do you choose for batch transcription, and what would make you change your mind?
Show the answer
Whisper small: it is the smallest model that meets both targets. Medium misses the speed target, and its error gain was measured on 20 calls only. You would reconsider if a slice that matters missed its target with small, for example French calls at 15.1%, and medium fixed it on the same calls. Then you would need to measure medium on all of them, and weigh its speed.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Your RTF for Whisper tiny is 0.06 on your laptop, and the lesson says 0.019. Is your setup broken?
Show the answer
Probably not. The RTF depends on the computer: its processor, the number of threads, and whether the program uses a GPU. Check that the WER is the same or very close, because the model and settings are pinned. If the WER matches, your setup is right and your computer is slower. Also make sure you did not time the first run, which includes the model download.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does the course run Whisper with int8 weights? Give the numbers for Whisper small.
Show the answer
int8 stores each parameter in 8 bits instead of 32. For Whisper small, the WER is close (8.6% against 8.2% with float32, 26 more word errors in 7,334 words). The RTF is lower (0.075 against 0.087), and the peak memory is almost half (1.27 GB against 2.21 GB). So int8 costs a little accuracy here and saves memory and time.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Every model has two numbers: its error (WER) and its speed (RTF). Agree on targets for both before you look at the results.
- On Larkfield’s 60 synthetic phone test calls, Whisper small (8.6%, RTF 0.075) is the smallest model that meets WER ≤ 10% and RTF ≤ 0.1. Medium is more accurate but slower, and was measured on 20 calls only.
- int8 weights change the WER only a little and halve the memory. The speed also depends on the program (faster-whisper or whisper.cpp) and the hardware it finds.
- The overall number hides the slices: small has 3.1% on English calls and 15.1% on French calls.
- Our synthetic calls are easier than real speech: Whisper small has 7.8% on real LibriSpeech sentences and 3.1% on our English calls.
- Compare models only on the same calls.
You can now choose a Whisper size with evidence, and run it on your laptop. But Larkfield could also pay a company to transcribe its calls. Next, you look at what a hosted speech API adds, what it costs, and where the callers’ audio goes.
Next: A hosted speech API