Hushpen benchmarks

How we test speech to text apps and models

Suraj Mandal · 19 November 2026 · See the results

Abstract

We ran 10 speech to text setups on one Mac: Hushpen with 5 speech models, Handy 0.10.1, FluidVoice 1.6.13, MacWhisper 15.11, VoiceInk 2.27 and Buzz 1.4.5. Each one wrote out the same 36 clips from TED talks in 19 English accents. We scored every transcript word by word against the talk's human-made captions. Buzz got the most words right, 95.1 out of 100, and Hushpen + Whisper Turbo got 95.0. FluidVoice was the fastest, 0.37 s for one minute of audio. Hushpen + Parakeet v3 took 0.57 s with 93.6 words right. In the full Hushpen app, its text appeared 0.48 s after the speaker stopped (0.46-0.51 s, 3 runs). This page gives the full method, the rules that kept the runs fair, the limits, and the steps to repeat the test.

1Setup

Method
Clips36 scored clips from TED talks, 25 to 30 s each, about 1 minute into each talk, with no laughter or applause inside. The full list with links is below. No audio is published.
Accents19 English accents, 2 speakers each. Filipino has one speaker. Russian has one scored speaker: on Evgeny Morozov's talk the captions run about 5 s ahead of the sound, so that clip was transcribed but not scored.
Truth textThe talks' human-made English captions. Auto captions were never used. Each clip starts and ends on caption edges. The captions leave out some "uh"s and repeated words, so every model loses a few points for words the speaker really said.
ScoreBoth sides go through the Whisper English text normalizer (lowercase, no punctuation, numbers and spellings made the same, fillers such as "um" removed). Then word error rate with jiwer over all words. Words right is 100 minus the word error rate.
SpeedMedian of 3 runs for each length. Every app ran with its model already loaded, except Buzz: its command line starts the app and loads the model for every file, about 13 s, and its times include that. Each app's row below says what its time includes.
Wait after you stop talkingHushpen only. The full app on this Mac, default settings, cue sounds off. A 26.6 s TED clip with 1 s of silence in front was played into the app at real speed while the dictation key was held. The clock ran from the end of speech (taken from the clip, not from the app) until the last text was in a TextEdit document in the background. The text in TextEdit matched the app's history exactly. Median of 3 runs with the model loaded; load time measured on its own. Other apps were not timed this way, so their rows show a dash.
This MacApple M4 Pro, 14 cores, 24 GB, macOS. Metal GPU on for every app.
Linux machinesHushpen only, CPU only, 3 threads: two Arm Neoverse-N1 servers with 4 cores, and an AMD Ryzen 3 2200G desktop with 4 cores. Accuracy there came out within 1 point of the Mac.
HushpenBuilt from master on 19 November 2026. A small bench program calls Hushpen's own speech loader and transcribe call with the app's defaults: Silero speech detection, Auto threads, language Detect. Nemotron Unified ran through the live text engine with the recording fed in as fast as possible.
FluidVoice 1.6.13The installed app's local API: a WAV file sent to POST /v1/transcribe on loopback. Default model, Parakeet TDT v3 (Multilingual).
Handy 0.10.1The official Apple silicon release, command line --transcribe-file, built-in transcribe-cpp on Metal. Model Parakeet Unified English 0.6B Q8_0, the first choice in its English catalog.
MacWhisper 15.11The free version, through its bundled mw command line tool, with the app hidden. Model Whisper Large v3 Turbo (WhisperKit, Core ML), the model the app recommended for this Mac. Its Parakeet v3 model needs a Pro license. Time is the wall clock of one mw call, model already loaded.
VoiceInk 2.27Files added to the app's Transcribe queue in the background, then Start pressed once. Model Whisper Large v3 Turbo from its model list, whisper.cpp engine. Time runs from Start to the saved result, then from one saved result to the next. The model stays loaded, so model load is not in the time. Language left at the app default, Auto.
Buzz 1.4.5Its command line with the window hidden: Buzz add --hide-gui, whisper.cpp engine, Whisper Large v3 Turbo with the Core ML encoder. Buzz cannot keep a model loaded this way, so every time includes the app start and model load, about 13 s.
Other appsSuperwhisper, Wispr Flow, Aqua Voice, Willow Voice and Spokenly were not tested in this round. Their own claims are on the results page.

2Keeping the runs fair

  1. Same clips for every app. Every setup got the same 37 WAV files, 16 kHz mono, cut once by a script. No app saw a different copy.
  2. Default settings. No custom words, no prompt and no language hint beyond each app's default.
  3. Nothing tuned on the test clips. The clips were picked by fixed rules (about 1 minute into each talk, whole caption lines, no laughter or applause) before any app ran. No setting was changed after we saw a score.
  4. No best-of picks. Accuracy is one run per clip, scored as it came out.
  5. Warm model, load time left out. The model is loaded before the clock starts. Load time is measured on its own and not added to the speed numbers.
  6. Median of 3 runs. Each speed number is the middle of 3 runs on the same file, so one slow run does not change the result.
  7. One scoring path. Every transcript and every caption goes through the same text normalizer and the same word error rate code.

3Scoring

The score answers one question: of the words the speaker said, how many did the app get right?

  1. Normalize both sides. The Whisper English text normalizer makes text lowercase, removes punctuation, writes numbers and spellings the same way, and removes fillers such as "um".
  2. Word error rate. jiwer counts the words that were swapped, dropped or added, over all clips together, and divides by the number of caption words.
  3. Words right. 100 minus the word error rate. 95 words right means about 5 mistakes in every 100 words.
  4. Speed. Seconds to write out one minute of audio, from a 1 minute file. The speed files are the clips joined into one recording, then cut to 5 s, 15 s, 30 s, 1 min, 2 min and 5 min, and looped to 10 and 30 min.

4Results

The full tables, with each accent and with long recordings, are on the results page.

Words right and speed for each setup
SetupWords right out of 100Higher is betterText appears after you stop talkingLower is betterWrite out a 1 minute recordingLower is better
Buzz 1.4.5Whisper Large v3 Turbo95.1—15.36 s
HushpenWhisper Large v3 Turbo Q595.01.69 s1.68-1.69 s, 3 runs2.76 s
MacWhisper 15.11Whisper Large v3 Turbo95.0—4.95 s
Handy 0.10.1Parakeet Unified English 0.6B Q894.7—0.50 s
HushpenNemotron Unified ASR 0.6B Q894.20.30 s0.29-0.33 s, 3 runs7.00 s
HushpenParakeet v3 Q493.60.48 s0.46-0.51 s, 3 runs0.57 s
FluidVoice 1.6.13Parakeet TDT v393.5—0.37 s
VoiceInk 2.27Whisper Large v3 Turbo92.1—4.34 s
HushpenWhisper Small Q575.4—1.51 s
HushpenWhisper Base Q571.5—0.66 s

5Limits

  • Paid apps were not tested. Superwhisper, Wispr Flow, Aqua Voice, Willow Voice and Spokenly have no numbers here.
  • The captions skip some words. TED captions leave out some "uh"s and repeated words, so every setup loses a few points for words the speaker really said.
  • One clip was not scored. On Evgeny Morozov's talk the captions run about 5 s ahead of the sound. That clip was written out but not scored.
  • Each app is timed at its own edge. Hushpen time is its transcribe call. FluidVoice time runs from sending the file to its local API to getting the text back. Handy time is the inference time its command line tool reports.
  • Only Hushpen was timed from when you stop talking. The other apps have no end to end time here. Their speed is the time to write out a file.
  • One machine. All Mac numbers come from one Apple M4 Pro with 24 GB. Other Macs will be faster or slower.
  • Nemotron Unified is timed through live text. Hushpen runs it in its live text engine, which writes the text while you speak, so its speed is not a file speed.

6Repeat the test

You need a Mac or PC, Python 3, ffmpeg and yt-dlp. The clip list is in the appendix below. The audio is not published, so you cut it yourself from the talks.

  1. Install the tools.pip install jiwer whisper-normalizer yt-dlp
  2. Download a talk and its human captions.yt-dlp -x --audio-format wav --write-subs --sub-langs en --sub-format vtt -o talk "<YouTube link>"
  3. Cut the clip at the times in the appendix, as 16 kHz mono.ffmpeg -ss <start> -to <end> -i talk.wav -ac 1 -ar 16000 -c:a pcm_s16le clip.wav
  4. Save the caption lines between the same times as the truth text.Name it after the clip, such as clip.truth.txt.
  5. Write out each clip with the app's default settings.In Hushpen, open Manual, drop the WAV files in, and export TXT next to each clip, such as clip.txt.
  6. Score all clips together with the script below.python3 score.py
# score.py: words right out of 100 over every clip in this folder
from pathlib import Path
import jiwer
from whisper_normalizer.english import EnglishTextNormalizer

n = EnglishTextNormalizer()
clips = sorted(p.stem for p in Path(".").glob("*.wav"))
truth = [n(Path(c + ".truth.txt").read_text()) for c in clips]
heard = [n(Path(c + ".txt").read_text()) for c in clips]
print(round(100 - 100 * jiwer.wer(truth, heard), 1))

For speed, open the app and load the model first. Then time one file 3 times and keep the middle time.

AThe clips

Each link opens the talk on YouTube at the clip start.

Clips used in the benchmark
AccentSpeakerClip
AmericanBrené Brown1:03 to 1:29
AmericanSimon Sinek1:01 to 1:31
ArabicRana el Kaliouby1:05 to 1:31
ArabicWael Ghonim1:02 to 1:30
AustralianPeter Singer1:02 to 1:28
AustralianRachel Botsman1:02 to 1:29
BrazilianEduardo Paes1:10 to 1:35
BrazilianMiguel Nicolelis1:02 to 1:29
BritishJamie Oliver1:00 to 1:26
BritishKen Robinson1:55 to 2:24
ChineseEric X. Li1:01 to 1:28
ChineseYang Lan1:30 to 1:55
FilipinoMaria Ressa1:03 to 1:31
FrenchEsther Duflo1:02 to 1:27
FrenchMatthieu Ricard2:39 to 3:07
GermanRalph Langner1:00 to 1:27
GermanStefan Sagmeister1:01 to 1:27
IndianArunachalam Muruganantham1:01 to 1:27
IndianSugata Mitra1:02 to 1:27
IrishBono1:03 to 1:30
IrishMary Robinson1:00 to 1:28
JapaneseShigeru Ban1:15 to 1:42
JapaneseTakaharu Tezuka1:10 to 1:36
KoreanHyeonseo Lee1:03 to 1:30
KoreanJoseph Kim1:01 to 1:29
NigerianChimamanda Ngozi Adichie1:37 to 2:02
NigerianNgozi Okonjo-Iweala1:01 to 1:26
PakistaniKhalida Brohi1:02 to 1:27
PakistaniZiauddin Yousafzai1:02 to 1:29
RussianEvgeny Morozov (not scored: captions out of sync)1:01 to 1:28
RussianGarry Kasparov1:15 to 1:43
ScottishEvelyn Glennie1:00 to 1:26
ScottishGordon Brown1:00 to 1:28
South AfricanNeil Turok1:04 to 1:34
South AfricanToby Shapshak1:02 to 1:27
Spanish or Latin AmericanLuis von Ahn1:30 to 1:56
Spanish or Latin AmericanPia Mancini1:02 to 1:30