Skip to content
Français

Whisper sizes on a laptop

  • Lesson 11
  • 40 min
  • Module 4: Speech recognition: compare the families

Where we are

We know how Vosk and Whisper turn sound into text. Whisper comes in many sizes, and Tomás wants the biggest. In this lesson, we run Whisper sizes on a laptop CPU and compare error by slice with speed, memory and cost. First, think of a shop that must choose a delivery van.

Lesson 11 of 36 · Speech AI: Transcribe, Understand and SpeakWhisper sizes on a laptop

What you will be able to do

Choose a Whisper size for Larkfield’s calls with evidence: its errors on the slices that matter, next to its speed, memory and cost on an ordinary computer. “The biggest, to be safe” is a choice too, and it has a price.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook scores the recorded transcripts of Vosk small and Whisper tiny, base and small on the test calls, by slice. It draws error against speed, and runs Whisper tiny on three calls on your computer. Its setup cells install faster-whisper and download the files it needs (about 80 MB with the model). About 40 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need

  • Your larkspeech folder at the end of Module 3, with its .venv environment and the course data, and the labs folder nextia-learning-labs next to it.
  • About 600 MB of free disk for the Whisper tiny, base and small models (downloaded on first use) and an internet connection the first time.
  • Tested on macOS (Apple M4, 16 GB) with Python 3.12 and faster-whisper 1.2.1. The Windows commands are given but were not run.

A small shop needs a delivery van. A big van carries more boxes, so it seems the safe choice. But it costs more to buy, uses more fuel on every trip, and does not fit in narrow streets. If the shop delivers twenty small parcels a day, a small van does the same job for less. The right question is not “which van is biggest?” but “which is the smallest van that carries our load?”

A speech model is the same. A bigger model makes fewer errors, but it needs more memory and more time for every minute of audio. Larkfield must find the smallest model that is accurate enough on its own calls.

The problem: “use the biggest, to be safe”

Section titled “The problem: “use the biggest, to be safe””

Tomás, the AI engineer, says: “Speech recognition is solved. Use the biggest Whisper, and we never think about it again.” Whisper comes in sizes, from tiny (39 million parameters) to medium (769 million) and larger, as From sound to text showed.

Priya, the data scientist, measured the sizes on the same laptop and the same test calls. Medium was too slow to run on all 60 calls, so she ran it on the first 20. Compare on those 20 calls:

Size WER on the first 20 test calls Real-time factor Peak memory
tiny 24.6% 0.019 554 MB
base 15.7% 0.030 784 MB
small 10.7% 0.075 1,267 MB
medium 6.9% 0.19 1,851 MB

Medium makes fewer errors. It also takes about 2.5 times as long as small for every minute of audio. Is that worth it? We cannot answer without a target, and without the slices.

Two numbers for every model: error and speed

Section titled “Two numbers for every model: error and speed”

You met the real-time factor (RTF) in the setup lesson: compute time divided by audio time. An RTF of 0.075 means 7.5 seconds of compute for 100 seconds of audio. Smaller is faster. Below 1, the model is faster than the audio plays.

Every model now has two numbers: its error (WER) and its speed (RTF). One model is better than another only if it is better on both, or if the gain on one is worth the loss on the other. The team must decide what is “good enough” before it looks at the results. Otherwise, it picks the target that makes its favourite model win.

Priya and Grace agree on two targets for batch transcription of recorded calls:

  • WER at most 10% on the phone test calls, after the normalisation guide.
  • RTF at most 0.1, so that one CPU server transcribes an hour of calls in 6 minutes or less.

These targets are the team’s decision, not a law. A phone line that answers callers needs a stricter speed target, as Module 8 shows.

Larkfield takes about 4,000 calls a month, about 49 hours of audio. The compute time for a month is the audio time × the RTF:

Size RTF Compute for 49 hours of audio Cost at US$0.17 an hour (assumption)
tiny 0.019 49 × 0.019 = 0.9 hours US$0.16
base 0.030 49 × 0.030 = 1.5 hours US$0.25
small 0.075 49 × 0.075 = 3.7 hours US$0.62
medium 0.19 49 × 0.19 = 9.3 hours US$1.58

The price per hour of a CPU server (4 cores, 16 GB) is an assumption from the course’s cost file. At Larkfield’s volume, even medium costs less than two dollars of compute a month. So for batch work, speed is not about money here. It matters when the volume is a hundred times larger, or when a caller is waiting for an answer.

Here is every recogniser that Priya measured, on the 60 phone test calls. The shaded corner is where both targets are met.

A scatter chart. Across: the real-time factor on a log scale from 0.005 to 0.3; left is faster. Up: WER on the phone test calls from 0 to 35%. A shaded corner below 10% WER and left of RTF 0.1 is labelled accurate enough and fast enough. Points: Vosk small at RTF 0.085 and 31.2%. Whisper tiny at 0.019 and 20.7%, base at 0.030 and 13.7%, small at 0.075 and 8.6%, small with float32 weights at 0.087 and 8.2%. Whisper medium, an open circle, at 0.19 and 6.9%, measured on 20 calls only. whisper.cpp tiny on the CPU at 0.011 and 21.7%, base on the CPU at 0.020 and 15.3%. whisper.cpp tiny on the Apple GPU at 0.008 and 22.8%, base on the Apple GPU at 0.012 and 15.2%. Only Whisper small, in both forms, is inside the corner.
Error against speed for every recogniser, on Larkfield's 60 synthetic phone test calls (one laptop, Apple M4). The shaded corner meets both targets. Circles: Whisper with faster-whisper; diamond: small with float32 weights; triangles: whisper.cpp (green on the CPU, orange on the Apple GPU); square: Vosk. *Medium was run on the first 20 test calls only.
Text description of the diagram

Each point is one recogniser. Further left is faster, lower is more accurate. The shaded corner, below 10% WER and left of an RTF of 0.1, meets both targets. Only Whisper small is inside it: 8.6% at an RTF of 0.075, and 8.2% at 0.087 with float32 weights. Whisper tiny (20.7%) and base (13.7%) are faster but make too many errors. Medium is more accurate (6.9%, on 20 calls only) but slower than the target, at 0.19. Vosk small is both slower and less accurate than Whisper base. The whisper.cpp points are green for the CPU and orange for the Apple GPU. They are the same tiny and base models, run by another program: faster, with about the same errors.

So, for batch transcription, Whisper small is the smallest model that meets both targets. Medium is better on errors, but it is outside the speed target. Choosing it would be choosing a bigger van “to be safe”.

Why the same model runs at different speeds

Section titled “Why the same model runs at different speeds”

A model’s parameters are numbers. They are usually stored as 32-bit floating-point numbers (float32). Quantisation stores each one with fewer bits. With int8, each one is an 8-bit whole number, plus a scale. It is like writing prices to the nearest cent instead of to the nearest thousandth of a cent: a little less exact, and much smaller.

The course uses int8 for every Whisper run. For Whisper small, Priya measured both:

Weights WER RTF Peak memory
float32 8.2% 0.087 2.21 GB
int8 8.6% 0.075 1.27 GB

The WER is close: float32 makes 26 fewer word errors in 7,334 words, or 0.4 points. The memory is almost half. That is why the course project runs faster-whisper with int8 weights.

  • int8 weights
  • float32 weights
Peak memory while transcribing the test calls (measured on one laptop). int8 unless the column says float32.
Text description of the chart

Five columns. Whisper tiny needs 554 MB, base 784 MB and small 1,267 MB with int8 weights. Small with float32 weights needs 2,214 MB, more than medium with int8 weights (1,851 MB). On a 16 GB laptop, all of them fit. On a small cloud server that also runs other programs, the float32 version could be the one that does not fit.

faster-whisper is one program that runs Whisper’s models. whisper.cpp is another, written in C++. Priya ran the same tiny and base models in whisper.cpp twice: on the CPU, and on the laptop’s Apple GPU (Metal). whisper.cpp uses the GPU by default when it finds one.

Model faster-whisper, CPU whisper.cpp, CPU whisper.cpp, Apple GPU
tiny 20.7%, RTF 0.019 21.7%, RTF 0.011 22.8%, RTF 0.008
base 13.7%, RTF 0.030 15.3%, RTF 0.020 15.2%, RTF 0.012

The errors are close: the model is the same, and small differences come from how each program searches and rounds. The speed depends on the program and on the hardware it finds. A speed measured on one machine is a fact about that machine. Measure on the computer you will use.

Slices: where the overall number hides the problem

Section titled “Slices: where the overall number hides the problem”

The overall WER mixes English and French, three lines and three kinds of background. Here are the two biggest slices, by language:

Size English (34 calls) French (26 calls)
tiny 7.0% 37.0%
base 5.2% 23.8%
small 3.1% 15.1%

On English calls, even tiny is under the 10% target. On French calls, even small is above it. So “Whisper small meets the target” is true on average, and false for every French caller. The last lesson of this module looks at the languages, the lines and the noise.

Each call also has a wideband version, at 16,000 samples a second. The same models on the same calls make fewer errors there: tiny 17.6% against 20.7% on the phone, base 12.4% against 13.7%, small 7.7% against 8.6%. The phone line costs accuracy for every size. If Larkfield could get wideband recordings from its phone provider, every model would improve at no compute cost.

Here is an honest surprise. Many people say “a model scores 6% on audiobooks and 24% on the phone”. On Larkfield’s English calls, that did not happen for Whisper. Whisper small has a WER of only 3.1% on them.

As The errors that matter showed, the same Whisper small makes 7.8% errors on real read sentences from LibriSpeech (8.1% through our phone line). That is more than twice the error on our English calls. Our synthetic callers read clean scripts. So every number in this course is “on Larkfield’s synthetic calls”, and a real call set would score worse. The gap still appears where a model is weak: Vosk small has 24.3% on the English phone calls, and Whisper tiny 37.0% on the French ones.

Priya also kept 40 calls hidden until the end. She ran Whisper small on them once: WER 10.7% (English 4.6%, French 17.1%). That is higher than the 8.6% of the test calls, and just above the 10% target. The French calls make most of the errors. We rebuilt the hidden calls, and their French calls now use a newer version of one French voice. That is why the score is a little above the target. The new name of the agent has nothing to do with it. Still, a choice made on the test calls has only a small margin.

Guided practice: run Whisper sizes yourself

Section titled “Guided practice: run Whisper sizes yourself”

So far, every number came from recorded transcripts. Now you add the transcribe command to your project and run Whisper on your own computer.

  1. In a terminal, go to your folder larkspeech and activate the environment (source .venv/bin/activate, or .venv\Scripts\Activate.ps1 on Windows). Copy the new command file from the snapshot end-of-m04 into your project.

    macOS and Linux:

    Terminal window
    cp ../nextia-learning-labs/C41/snapshots/end-of-m04/larkspeech/commands/transcribe.py larkspeech/commands/

    Windows PowerShell:

    Terminal window
    Copy-Item ..\nextia-learning-labs\C41\snapshots\end-of-m04\larkspeech\commands\transcribe.py larkspeech\commands\

    The speech command finds its subcommands by their files, so nothing else changes.

  2. Check that the command is there:

    Terminal window
    speech transcribe --help
    usage: speech transcribe [-h] [--model {tiny,base,small,medium}]
    [--split SPLIT] [--kind {phone,wideband}]
    [--limit LIMIT] [--vad] [--prompt]
    [--language {en,fr}]
    options:
    -h, --help show this help message and exit
    --model {tiny,base,small,medium}
    --split SPLIT
    --kind {phone,wideband}
    --limit LIMIT only the first N calls (0 for all)
    --vad cut the silences first (Silero VAD)
    --prompt give Whisper the Larkfield vocabulary
    --language {en,fr} only calls in this language
  3. Run Whisper tiny on the first 5 test calls. The first run downloads the model (about 75 MB).

    Terminal window
    speech transcribe --model tiny --limit 5
    whisper-tiny on 5 test calls (phone): WER 36.0%, CER 19.7%, order numbers wrong 2 of 4, real-time factor 0.022
    Wrote outputs/transcripts-whisper-tiny.jsonl

    The command transcribes each call, saves the transcripts in outputs/, and scores them with the same guide as speech score. Five calls are a small sample: one bad call moves the WER a lot. Your numbers can differ a little from these; the note after the steps explains why. The first run also prints download progress (Fetching 6 files) and a warning about unauthenticated requests to Hugging Face. You can ignore both.

  4. Run base and small on the same 5 calls. Small downloads about 480 MB the first time.

    Terminal window
    speech transcribe --model base --limit 5
    speech transcribe --model small --limit 5
    whisper-base on 5 test calls (phone): WER 22.6%, CER 11.3%, order numbers wrong 2 of 4, real-time factor 0.063
    Wrote outputs/transcripts-whisper-base.jsonl
    whisper-small on 5 test calls (phone): WER 11.3%, CER 5.0%, order numbers wrong 0 of 4, real-time factor 0.085
    Wrote outputs/transcripts-whisper-small.jsonl

Your real-time factors will differ: they depend on your processor and on what else the computer is doing. On a free cloud notebook, they are often 2 to 5 times larger. Even on one computer, 5 calls give an unsteady speed. A second run of base on the same calls gave an RTF of 0.102 here, while other programs were busy. The recorded run of all 60 calls gave 0.030. The WER should be the same or very close, because the model and its settings are pinned. Our second run gave 23.1% instead of 22.6%: the program’s arithmetic on several CPU threads can change a word now and then. Measure speed on many calls, on a quiet computer.

Predict: on the wideband files of the same 5 calls, will Whisper tiny’s WER go down? By how much? Then run it with --kind wideband.

Check your answer
Terminal window
speech transcribe --model tiny --limit 5 --kind wideband
whisper-tiny on 5 test calls (wideband): WER 22.9%, CER 11.1%, order numbers wrong 2 of 4, real-time factor 0.028
Wrote outputs/transcripts-whisper-tiny.jsonl

On all 60 test calls, tiny goes from 20.7% on the phone to 17.6% on wideband. On 5 calls, the change can be larger or smaller: compare with the full table, not with your 5 calls.

Common mistake

Comparing models on different calls

Symptom: Tomás puts two numbers side by side: Whisper medium 6.9%, Whisper small 8.6%. “Medium only gains 1.7 points,” he says. Next week he says the opposite.

Cause: medium ran on the first 20 test calls only, and small on all 60. The first 20 calls are harder than the average: on those same 20 calls, small has 10.7%, so medium gains 3.8 points there. The two numbers in the first sentence measure different calls.

Fix: compare models only on the same calls, with the same normalisation guide. If one model ran on fewer calls, score the others on those calls too, and say so.

Consequence in AI work: a comparison on different test data can reverse a decision. The course’s comparison table keeps one test split for every method for this reason.

Check your understanding

Whisper small has a WER of 8.6% and an RTF of 0.075. Whisper medium has 6.9% (on 20 calls) and an RTF of 0.19. With the targets WER ≤ 10% and RTF ≤ 0.1, which do you choose for batch transcription, and what would make you change your mind?

Show the answer

Whisper small: it is the smallest model that meets both targets. Medium misses the speed target, and its error gain was measured on 20 calls only. You would reconsider if a slice that matters missed its target with small, for example French calls at 15.1%, and medium fixed it on the same calls. Then you would need to measure medium on all of them, and weigh its speed.

Check your understanding

Your RTF for Whisper tiny is 0.06 on your laptop, and the lesson says 0.019. Is your setup broken?

Show the answer

Probably not. The RTF depends on the computer: its processor, the number of threads, and whether the program uses a GPU. Check that the WER is the same or very close, because the model and settings are pinned. If the WER matches, your setup is right and your computer is slower. Also make sure you did not time the first run, which includes the model download.

Check your understanding

Why does the course run Whisper with int8 weights? Give the numbers for Whisper small.

Show the answer

int8 stores each parameter in 8 bits instead of 32. For Whisper small, the WER is close (8.6% against 8.2% with float32, 26 more word errors in 7,334 words). The RTF is lower (0.075 against 0.087), and the peak memory is almost half (1.27 GB against 2.21 GB). So int8 costs a little accuracy here and saves memory and time.

Recap

  • Every model has two numbers: its error (WER) and its speed (RTF). Agree on targets for both before you look at the results.
  • On Larkfield’s 60 synthetic phone test calls, Whisper small (8.6%, RTF 0.075) is the smallest model that meets WER ≤ 10% and RTF ≤ 0.1. Medium is more accurate but slower, and was measured on 20 calls only.
  • int8 weights change the WER only a little and halve the memory. The speed also depends on the program (faster-whisper or whisper.cpp) and the hardware it finds.
  • The overall number hides the slices: small has 3.1% on English calls and 15.1% on French calls.
  • Our synthetic calls are easier than real speech: Whisper small has 7.8% on real LibriSpeech sentences and 3.1% on our English calls.
  • Compare models only on the same calls.

You can now choose a Whisper size with evidence, and run it on your laptop. But Larkfield could also pay a company to transcribe its calls. Next, you look at what a hosted speech API adds, what it costs, and where the callers’ audio goes.

Next: A hosted speech API