Hushpen benchmarks
How we test speech to text apps and models
Suraj Mandal · 19 November 2026 · See the results
Abstract
We ran 10 speech to text setups on one Mac: Hushpen with 5 speech models, Handy 0.10.1, FluidVoice 1.6.13, MacWhisper 15.11, VoiceInk 2.27 and Buzz 1.4.5. Each one wrote out the same 36 clips from TED talks in 19 English accents. We scored every transcript word by word against the talk's human-made captions. Buzz got the most words right, 95.1 out of 100, and Hushpen + Whisper Turbo got 95.0. FluidVoice was the fastest, 0.37 s for one minute of audio. Hushpen + Parakeet v3 took 0.57 s with 93.6 words right. In the full Hushpen app, its text appeared 0.48 s after the speaker stopped (0.46-0.51 s, 3 runs). This page gives the full method, the rules that kept the runs fair, the limits, and the steps to repeat the test.
1Setup
| Clips | 36 scored clips from TED talks, 25 to 30 s each, about 1 minute into each talk, with no laughter or applause inside. The full list with links is below. No audio is published. |
|---|---|
| Accents | 19 English accents, 2 speakers each. Filipino has one speaker. Russian has one scored speaker: on Evgeny Morozov's talk the captions run about 5 s ahead of the sound, so that clip was transcribed but not scored. |
| Truth text | The talks' human-made English captions. Auto captions were never used. Each clip starts and ends on caption edges. The captions leave out some "uh"s and repeated words, so every model loses a few points for words the speaker really said. |
| Score | Both sides go through the Whisper English text normalizer (lowercase, no punctuation, numbers and spellings made the same, fillers such as "um" removed). Then word error rate with jiwer over all words. Words right is 100 minus the word error rate. |
| Speed | Median of 3 runs for each length. Every app ran with its model already loaded, except Buzz: its command line starts the app and loads the model for every file, about 13 s, and its times include that. Each app's row below says what its time includes. |
| Wait after you stop talking | Hushpen only. The full app on this Mac, default settings, cue sounds off. A 26.6 s TED clip with 1 s of silence in front was played into the app at real speed while the dictation key was held. The clock ran from the end of speech (taken from the clip, not from the app) until the last text was in a TextEdit document in the background. The text in TextEdit matched the app's history exactly. Median of 3 runs with the model loaded; load time measured on its own. Other apps were not timed this way, so their rows show a dash. |
| This Mac | Apple M4 Pro, 14 cores, 24 GB, macOS. Metal GPU on for every app. |
| Linux machines | Hushpen only, CPU only, 3 threads: two Arm Neoverse-N1 servers with 4 cores, and an AMD Ryzen 3 2200G desktop with 4 cores. Accuracy there came out within 1 point of the Mac. |
| Hushpen | Built from master on 19 November 2026. A small bench program calls Hushpen's own speech loader and transcribe call with the app's defaults: Silero speech detection, Auto threads, language Detect. Nemotron Unified ran through the live text engine with the recording fed in as fast as possible. |
| FluidVoice 1.6.13 | The installed app's local API: a WAV file sent to POST /v1/transcribe on loopback. Default model, Parakeet TDT v3 (Multilingual). |
| Handy 0.10.1 | The official Apple silicon release, command line --transcribe-file, built-in transcribe-cpp on Metal. Model Parakeet Unified English 0.6B Q8_0, the first choice in its English catalog. |
| MacWhisper 15.11 | The free version, through its bundled mw command line tool, with the app hidden. Model Whisper Large v3 Turbo (WhisperKit, Core ML), the model the app recommended for this Mac. Its Parakeet v3 model needs a Pro license. Time is the wall clock of one mw call, model already loaded. |
| VoiceInk 2.27 | Files added to the app's Transcribe queue in the background, then Start pressed once. Model Whisper Large v3 Turbo from its model list, whisper.cpp engine. Time runs from Start to the saved result, then from one saved result to the next. The model stays loaded, so model load is not in the time. Language left at the app default, Auto. |
| Buzz 1.4.5 | Its command line with the window hidden: Buzz add --hide-gui, whisper.cpp engine, Whisper Large v3 Turbo with the Core ML encoder. Buzz cannot keep a model loaded this way, so every time includes the app start and model load, about 13 s. |
| Other apps | Superwhisper, Wispr Flow, Aqua Voice, Willow Voice and Spokenly were not tested in this round. Their own claims are on the results page. |
2Keeping the runs fair
- Same clips for every app. Every setup got the same 37 WAV files, 16 kHz mono, cut once by a script. No app saw a different copy.
- Default settings. No custom words, no prompt and no language hint beyond each app's default.
- Nothing tuned on the test clips. The clips were picked by fixed rules (about 1 minute into each talk, whole caption lines, no laughter or applause) before any app ran. No setting was changed after we saw a score.
- No best-of picks. Accuracy is one run per clip, scored as it came out.
- Warm model, load time left out. The model is loaded before the clock starts. Load time is measured on its own and not added to the speed numbers.
- Median of 3 runs. Each speed number is the middle of 3 runs on the same file, so one slow run does not change the result.
- One scoring path. Every transcript and every caption goes through the same text normalizer and the same word error rate code.
3Scoring
The score answers one question: of the words the speaker said, how many did the app get right?
- Normalize both sides. The Whisper English text normalizer makes text lowercase, removes punctuation, writes numbers and spellings the same way, and removes fillers such as "um".
- Word error rate. jiwer counts the words that were swapped, dropped or added, over all clips together, and divides by the number of caption words.
- Words right. 100 minus the word error rate. 95 words right means about 5 mistakes in every 100 words.
- Speed. Seconds to write out one minute of audio, from a 1 minute file. The speed files are the clips joined into one recording, then cut to 5 s, 15 s, 30 s, 1 min, 2 min and 5 min, and looped to 10 and 30 min.
4Results
The full tables, with each accent and with long recordings, are on the results page.
| Setup | Words right out of 100Higher is better | Text appears after you stop talkingLower is better | Write out a 1 minute recordingLower is better |
|---|---|---|---|
| Buzz 1.4.5Whisper Large v3 Turbo | 95.1 | — | 15.36 s |
| HushpenWhisper Large v3 Turbo Q5 | 95.0 | 1.69 s1.68-1.69 s, 3 runs | 2.76 s |
| MacWhisper 15.11Whisper Large v3 Turbo | 95.0 | — | 4.95 s |
| Handy 0.10.1Parakeet Unified English 0.6B Q8 | 94.7 | — | 0.50 s |
| HushpenNemotron Unified ASR 0.6B Q8 | 94.2 | 0.30 s0.29-0.33 s, 3 runs | 7.00 s |
| HushpenParakeet v3 Q4 | 93.6 | 0.48 s0.46-0.51 s, 3 runs | 0.57 s |
| FluidVoice 1.6.13Parakeet TDT v3 | 93.5 | — | 0.37 s |
| VoiceInk 2.27Whisper Large v3 Turbo | 92.1 | — | 4.34 s |
| HushpenWhisper Small Q5 | 75.4 | — | 1.51 s |
| HushpenWhisper Base Q5 | 71.5 | — | 0.66 s |
5Limits
- Paid apps were not tested. Superwhisper, Wispr Flow, Aqua Voice, Willow Voice and Spokenly have no numbers here.
- The captions skip some words. TED captions leave out some "uh"s and repeated words, so every setup loses a few points for words the speaker really said.
- One clip was not scored. On Evgeny Morozov's talk the captions run about 5 s ahead of the sound. That clip was written out but not scored.
- Each app is timed at its own edge. Hushpen time is its transcribe call. FluidVoice time runs from sending the file to its local API to getting the text back. Handy time is the inference time its command line tool reports.
- Only Hushpen was timed from when you stop talking. The other apps have no end to end time here. Their speed is the time to write out a file.
- One machine. All Mac numbers come from one Apple M4 Pro with 24 GB. Other Macs will be faster or slower.
- Nemotron Unified is timed through live text. Hushpen runs it in its live text engine, which writes the text while you speak, so its speed is not a file speed.
6Repeat the test
You need a Mac or PC, Python 3, ffmpeg and yt-dlp. The clip list is in the appendix below. The audio is not published, so you cut it yourself from the talks.
- Install the tools.
pip install jiwer whisper-normalizer yt-dlp - Download a talk and its human captions.
yt-dlp -x --audio-format wav --write-subs --sub-langs en --sub-format vtt -o talk "<YouTube link>" - Cut the clip at the times in the appendix, as 16 kHz mono.
ffmpeg -ss <start> -to <end> -i talk.wav -ac 1 -ar 16000 -c:a pcm_s16le clip.wav - Save the caption lines between the same times as the truth text.Name it after the clip, such as clip.truth.txt.
- Write out each clip with the app's default settings.In Hushpen, open Manual, drop the WAV files in, and export TXT next to each clip, such as clip.txt.
- Score all clips together with the script below.
python3 score.py
# score.py: words right out of 100 over every clip in this folder
from pathlib import Path
import jiwer
from whisper_normalizer.english import EnglishTextNormalizer
n = EnglishTextNormalizer()
clips = sorted(p.stem for p in Path(".").glob("*.wav"))
truth = [n(Path(c + ".truth.txt").read_text()) for c in clips]
heard = [n(Path(c + ".txt").read_text()) for c in clips]
print(round(100 - 100 * jiwer.wer(truth, heard), 1))
For speed, open the app and load the model first. Then time one file 3 times and keep the middle time.
AThe clips
Each link opens the talk on YouTube at the clip start.
| Accent | Speaker | Clip |
|---|---|---|
| American | Brené Brown | 1:03 to 1:29 |
| American | Simon Sinek | 1:01 to 1:31 |
| Arabic | Rana el Kaliouby | 1:05 to 1:31 |
| Arabic | Wael Ghonim | 1:02 to 1:30 |
| Australian | Peter Singer | 1:02 to 1:28 |
| Australian | Rachel Botsman | 1:02 to 1:29 |
| Brazilian | Eduardo Paes | 1:10 to 1:35 |
| Brazilian | Miguel Nicolelis | 1:02 to 1:29 |
| British | Jamie Oliver | 1:00 to 1:26 |
| British | Ken Robinson | 1:55 to 2:24 |
| Chinese | Eric X. Li | 1:01 to 1:28 |
| Chinese | Yang Lan | 1:30 to 1:55 |
| Filipino | Maria Ressa | 1:03 to 1:31 |
| French | Esther Duflo | 1:02 to 1:27 |
| French | Matthieu Ricard | 2:39 to 3:07 |
| German | Ralph Langner | 1:00 to 1:27 |
| German | Stefan Sagmeister | 1:01 to 1:27 |
| Indian | Arunachalam Muruganantham | 1:01 to 1:27 |
| Indian | Sugata Mitra | 1:02 to 1:27 |
| Irish | Bono | 1:03 to 1:30 |
| Irish | Mary Robinson | 1:00 to 1:28 |
| Japanese | Shigeru Ban | 1:15 to 1:42 |
| Japanese | Takaharu Tezuka | 1:10 to 1:36 |
| Korean | Hyeonseo Lee | 1:03 to 1:30 |
| Korean | Joseph Kim | 1:01 to 1:29 |
| Nigerian | Chimamanda Ngozi Adichie | 1:37 to 2:02 |
| Nigerian | Ngozi Okonjo-Iweala | 1:01 to 1:26 |
| Pakistani | Khalida Brohi | 1:02 to 1:27 |
| Pakistani | Ziauddin Yousafzai | 1:02 to 1:29 |
| Russian | Evgeny Morozov (not scored: captions out of sync) | 1:01 to 1:28 |
| Russian | Garry Kasparov | 1:15 to 1:43 |
| Scottish | Evelyn Glennie | 1:00 to 1:26 |
| Scottish | Gordon Brown | 1:00 to 1:28 |
| South African | Neil Turok | 1:04 to 1:34 |
| South African | Toby Shapshak | 1:02 to 1:27 |
| Spanish or Latin American | Luis von Ahn | 1:30 to 1:56 |
| Spanish or Latin American | Pia Mancini | 1:02 to 1:30 |