Running Speech Models Locally: ASR and TTS on Hardware You Already Own

Start here

Any machine from the last five years runs good speech recognition and good speech synthesis today. ggml-large-v3-turbo-q5_0.bin is 574,041,195 bytes. A Piper voice is 63,201,294 bytes. Kokoro's 8-bit ONNX export is 86,033,585 bytes. Nothing here requires a GPU purchase, and for most of it a GPU is optional rather than merely affordable. This is the most accessible category in local AI and the least written about, largely because there is no hardware upsell in it.

RTF, and why every figure you've read is unusable

Tokens per second is the wrong metric for speech. What you want is the real-time factor: processing time divided by audio duration. RTF 0.1 means ten seconds of audio processed in one second. Below 1.0 you keep up with a live stream; above 1.0 you fall behind forever. The Open ASR Leaderboard reports the reciprocal, RTFx, defined in its code as sum(audio_length_s) / sum(transcription_time_s) — total audio over total wall time, summed across the whole dataset.

That aggregation is where the trouble starts. The leaderboard's own repo specifies the hardware as 1x H200 (141 GB), and run_whisper.sh sets BATCH_SIZE=64. So nvidia/parakeet-tdt-0.6b-v2 scoring RTFx 3386.02 against openai/whisper-large-v3 at RTFx 145.51 is a real, reproducible, honestly-published result that tells you nothing about your laptop. It describes 64 utterances in flight on a 141 GB datacenter card. Your dictation app runs one utterance at a time on a chip with a fan.

The blog figures are worse, because the leaderboard at least publishes its methodology. "Kokoro runs at RTF 0.02" with no GPU named, no text length, and no batch size is not a measurement.

For contrast, F5-TTS publishes a table with actual conditions — a single L20, 26 prompt/text pairs, 16 NFE steps — and the result is instructive: RTF 0.0402 offline under TRT-LLM versus 0.1467 offline under PyTorch. Same model, same GPU, same audio. 3.6x purely from runtime choice. Whatever RTF you read somewhere, the runtime probably moved it more than the model did.

Whisper's 30-second window

Whisper doesn't process your audio. It processes exactly 30 seconds, always. pad_or_trim in whisper/audio.py forces every input to N_SAMPLES = 480000, which becomes N_FRAMES = 3000 mel frames, which the stride-2 convolutions halve into the encoder's max_source_positions: 1500. A three-second voice memo is padded with silence to thirty and costs the identical encoder pass.

This is why short-clip RTF is structurally bad and why push-to-talk dictation feels sluggish on fast hardware. It also explains what the "fast Whisper" variants actually did. Compare the configs: large-v3 is 32 encoder and 32 decoder layers; large-v3-turbo is 32 encoder and 4; distil-large-v3.5 is 32 encoder and 2. All three left the encoder untouched.

So the encoder is the floor, and the leaderboard shows where it sits. Turbo, with 4 decoder layers, scores RTFx 200.19. Distil-large-v3.5, with half that decoder, scores 202.03. Halving the decoder again bought about 1%. Nothing is left to win on that side.

Parakeet's advantage isn't only that it is 627,057,286 parameters against Whisper's ~1.54 billion. It's a Fast Conformer with a TDT head that consumes variable-length audio with no 30-second padding at all. That difference survives on your hardware even though the 23x does not.

What to download for ASR

Every size below is the exact byte count from the HuggingFace API, verified this session.

Parakeet's checkpoint ships in fp32 — 627,057,286 params × 4 bytes accounts for essentially the whole file — so it halves if your runtime loads it in bf16. And distil-whisper/distil-large-v3.5 publishes only an fp32 model.safetensors at 3,025,686,376 bytes, larger than fp16 large-v3 despite being the smaller model. Check the dtype, not the reputation.

On accuracy, the leaderboard's English short-form average WER puts parakeet-tdt-0.6b-v2 at 6.05 and whisper-large-v3 at 7.44. The striking row is parakeet-tdt_ctc-110m: 7.49 WER at 0.11B parameters, indistinguishable from large-v3 at fourteen times the size. But this is English short-form academic audio. Whisper's tokenizer carries 100 languages; Parakeet v3 covers 25 European ones. If you need Japanese or Arabic, the comparison is over before it starts.

Pick this way: English-only batch transcription with a GPU, use Parakeet. Multilingual or Mac, use whisper.cpp with turbo. Python pipeline needing word timestamps and VAD, use faster-whisper.

What to download for TTS

Chatterbox Nano is the cautionary tale. Resemble advertises it as a 110M-parameter package, but the runnable model is three files: t3_nano_v1.safetensors at 869,899,204 bytes, s3gen.safetensors at 1,056,484,620, and ve.safetensors at 5,695,784. That is 1.799 GiB on disk, and s3gen is the same size in every Chatterbox variant, so "Nano" saves nothing there. Resident footprint and headline parameter count are different questions, and in multi-component pipelines the headline is usually the smallest piece.

Two licensing facts matter more than anyone admits: F5-TTS is CC-BY-NC-4.0, so no commercial use, and every Chatterbox output carries Resemble's PerTh neural watermark by default. Kokoro at Apache-2.0 and Piper's MIT voices are the unencumbered options.

If you were about to install Coqui: coqui-ai/TTS was last pushed 2024-08-16 and the company is gone. The maintained fork is idiap/coqui-ai-TTS, last pushed 2026-06-10. Install that one.

Streaming latency is the metric RTF hides

For anything conversational, RTF is nearly irrelevant and time to first audio chunk is everything. A model at RTF 0.1 that synthesizes a whole paragraph before emitting a sample feels far worse than one at RTF 0.4 that starts talking in 200 ms. F5-TTS reports 253 ms average latency at concurrency 2 in client-server mode, again on an L20 under TensorRT-LLM.

We have not measured first-chunk latency on consumer hardware and won't publish a number we didn't take. What we can tell you is where to look: Kokoro and Piper are non-autoregressive and start fast; Chatterbox, XTTS, and F5-TTS generate speech tokens sequentially and pay a prefill before the first sample exists.

Setup

whisper.cpp, the right default on Apple Silicon:

./build/bin/whisper-cli -m models/ggml-large-v3-turbo-q5_0.bin -f audio.wav -l en -t 8

Flash attention now defaults to on (bool flash_attn = true in the CLI source); -nfa disables it. Thread count defaults to min(4, hardware_concurrency), almost certainly too low, so set -t explicitly. Sampling defaults to best_of 5 and beam_size 5; -bo 1 trades a little accuracy for real speed. Pass -l en when you know the language, since auto-detect costs an extra decode pass.

faster-whisper, for Python pipelines:

from faster_whisper import WhisperModel, BatchedInferencePipeline
model = WhisperModel("turbo", device="cuda", compute_type="int8_float16")
batched = BatchedInferencePipeline(model=model)
segments, info = batched.transcribe("audio.mp3", batch_size=16, vad_filter=True)

Use compute_type="int8" on CPU. vad_filter=True routes audio through Silero VAD and is the highest-leverage option here: skipping silence cuts wall time on real recordings more than any model swap, because Whisper otherwise burns full 30-second encoder passes on nothing.

For Parakeet on long files, switch the attention mode or you will hit memory limits. NVIDIA documents up to 24 minutes with full attention on an A100 80GB, versus about 3 hours with local attention:

asr_model.change_attention_model(self_attention_model="rel_pos_local_attn", att_context_size=[256, 256])

TTS is pip install piper-tts (GPL-3.0 engine, MIT voices; the old rhasspy/piper repo now lives at OHF-Voice/piper1-gpl under the Open Home Foundation) or pip install kokoro plus a system espeak-ng, which Kokoro needs for grapheme-to-phoneme on out-of-dictionary words. Kokoro outputs 24 kHz; Piper's medium voices output 22.05 kHz.

What we don't know

We have not benchmarked these models on consumer hardware, so this page contains no RTF figure of our own. Everything above is an exact file size, an architectural fact from a config or source file, or someone else's benchmark with their conditions attached. A speech benchmark without a GPU name, a batch size, and an audio length isn't a weak measurement. It isn't a measurement.