Parakeet vs Apple's Speech API vs MOSS: Benchmark Round 2

The result, up front

NVIDIA's Parakeet TDT v2, running as the int8 CoreML model a real app would ship, is the most accurate on-device English engine we have measured: 2.01% WER on LibriSpeech test-clean and 3.40% on test-other. But the clean-speech race is now astonishingly tight. Three engines with completely different architectures, a speech transducer from NVIDIA, a 0.9B audio language model from Fudan University's OpenMOSS lab, and Apple's system engine, landed within 0.11 points of each other. Noisy speech is where they separate.

Lower is better: WER is word error rate, the percentage of words an engine substitutes, drops, or invents. Same corpus as round 1: LibriSpeech test-clean (2,620 utterances of clean read speech) and test-other (2,939 harder, noisier utterances), all 5,559 run in full on the same Apple M2 Pro (32GB, macOS 26.5.1), scored by the same normalizer and the same scoring code.

Why a round 2

When we published the first round of this benchmark, the most common response, on Hacker News and in every developer community that discussed it, was some version of: Whisper is not the state of the art anymore, benchmark Parakeet. Fair. And a Hacker News commenter pointed us at MOSS-Transcribe-Diarize, a model released days earlier that does transcription and speaker diarization jointly in one pass.

So this round adds three engines, measured on the identical 5,559 utterances with the identical scoring pipeline, which makes every number here directly comparable to round 1. As before, every raw transcript is downloadable below, so you can rescore us with your own normalizer if you disagree with ours.

Parakeet: the new accuracy champion, with an asterisk nobody prints

Parakeet TDT 0.6B v2 wins this benchmark. On clean speech its lead over Apple's engine is 0.11 points, a photo finish. On noisy speech the lead is real: 3.40% vs 4.56%, about a 25% reduction in errors. If your only criterion is English accuracy on difficult audio, Parakeet v2 is the strongest thing you can run on-device on a Mac today.

Here is the asterisk. NVIDIA's published numbers for this model are 1.69% and 3.19%, measured on the reference GPU implementation. We measured 2.01% and 3.40%, because we benchmarked what an app can actually ship: the int8-quantized CoreML conversion (via the open-source FluidAudio package, the same one Inscribe already uses for speaker diarization). The difference, roughly +0.3 on clean and +0.2 on noisy, is the cost of quantization plus the CoreML port. We could not find that number published anywhere, and it changes the story: the shippable Parakeet's lead over Apple's built-in engine is about a third of what the published numbers suggest.

Parakeet v3, the multilingual version covering 25 European languages, trades English accuracy for that coverage: 2.51% clean / 4.28% other. Interestingly it still edges out Apple's engine on noisy speech while losing on clean, and it beats Whisper Small, which covers more languages, everywhere.

MOSS: a different kind of contender

MOSS-Transcribe-Diarize is not another speech recognizer. It is a 0.9B-parameter end-to-end audio language model that transcribes and diarizes at the same time: its output is a timestamped transcript with speaker labels like [S01] and [S02] already attached, across 50+ languages, in a single pass over up to 90 minutes of audio.

On pure transcription accuracy it landed at 2.07% clean / 4.68% other. Statistically indistinguishable from Apple's SpeechAnalyzer on clean speech, slightly behind on noisy speech, and ahead of every Whisper model, while also doing a job no other engine in this table attempts.

We probed the diarization with a controlled test: six utterances from three different LibriSpeech speakers stitched into one 62-second file, so the true transcript and the true speaker turns are known exactly. MOSS scored 1.75% WER on the words and attributed all 13 of its output segments to the correct speaker, including correctly re-identifying each speaker when they returned 30+ seconds later. That is a small, easy test, clean speech with no crosstalk, but it is the hard part of diarization done right where plenty of production pipelines get it wrong. A proper multi-speaker meeting benchmark (AMI corpus, real crosstalk, measured DER) is round 3.

The bug we found: MOSS goes silent when speech starts at zero

Benchmarks find bugs. Round 1 found one in our own app; this round found one in how MOSS handles trimmed audio, and it is worth documenting because anyone evaluating this model will hit it.

In our first full pass, MOSS returned a completely empty transcript for 182 of the 2,620 clean clips, deterministically, on perfectly good audio. The pattern took some finding: LibriSpeech clips are trimmed tight, so speech starts at exactly t=0, and MOSS, trained on long-form recordings where nobody speaks in the first instant, responds to some of these clips by emitting its end-of-output token immediately. Appending silence at the end made it worse: 426 additional clips went mute. Prepending it fixed every clip we tested.

The published numbers use a uniform protocol: one second of silence prepended to every clip, which adds no words and cannot flatter a transcript, with a documented retry ladder (0.5s, 2s, 0s) for stragglers. Only 7 of 5,559 clips needed the ladder; zero ended up empty. And to verify the padding did not change transcription quality itself: on the clips that transcribed in both the naive and the padded runs, corpus WER moved by 0.05 points. The padding fixed the silence bug and nothing else. Both runs are in the downloadable data.

The practical read for anyone shipping MOSS: real-world recordings, which start with a breath or a click or half a second of room tone, are its happy path. Tightly trimmed clips are not.

Methodology, and why you can check it

Everything from round 1's methodology carries over: same corpus, corpus-level WER (total errors over total reference words, not averaged per-utterance), same text normalizer mirroring OpenAI's English normalizer, failures counted as 100% WER rather than dropped. The round 1 harness reproduced OpenAI's published Whisper WERs within +0.11 to +0.42 on all six measurements, which validates the corpus handling and scoring that all of these numbers, old and new, flow through.

What round 2 adds:

• Parakeet ran inside the same XCTest harness as every round 1 engine, via FluidAudio (pinned revision 19600a48), models parakeet-tdt-0.6b-v2/v3-coreml with the int8 encoder, the only precision the package ships for these models. Fresh decoder state per utterance, warm-up clip excluded from measurement.
• MOSS ran on the same machine through mlx-audio 0.4.5 (the first release supporting it) on Apple's MLX framework, since it is a Python model with no Swift path today. To keep the scoring identical rather than approximately similar, we compiled our Swift normalizer and scorer into a standalone tool and scored MOSS's raw output with byte-identical code to every other row in the table.
• Speaker tags and timestamps were stripped before scoring MOSS ([S01], [12.4] and the like are output framing, not recognized words). The raw tagged output is preserved in the download for anyone who wants to study the diarization.

The raw data

• summary.json - all round 2 measurements plus protocol details, machine-readable (2KB)
• raw-transcripts-parakeet-v2.json.gz - all 5,559 utterances (534KB)
• raw-transcripts-parakeet-v3.json.gz - all 5,559 utterances (535KB)
• raw-transcripts-moss.json.gz - all 5,559 utterances, including the tagged speaker/timestamp output (627KB)
• round 1 data - Apple SpeechAnalyzer, SFSpeechRecognizer, and Whisper

Limitations

• English only, still. Parakeet v3 covers 25 languages and MOSS claims 50+; we measured none of them. Multilingual accuracy is a different benchmark.
• Read audiobook speech, not meetings. MOSS in particular is built for long multi-speaker audio, so a read-speech corpus arguably understates it. The AMI meeting benchmark is the follow-up.
• No speed table. All engines ran far faster than real time (both Parakeets transcribed the full 10.5-hour corpus in minutes), but our timing was collected on a machine that was doing other work, so we are not publishing precise per-engine speed claims.
• Parakeet is int8 CoreML. That is the point, it is what apps ship, but it means our numbers are not directly comparable to GPU leaderboard entries, as the quantization-tax section quantifies.
• MOSS is not app-shippable on Apple platforms today. Python plus MLX works for desktop pipelines and tools; there is no supported Swift/CoreML path we could measure.
• One machine. M2 Pro, macOS 26.5.1. Accuracy should transfer across Apple Silicon.

What we are doing with this

Inscribe's Auto engine keeps preferring Apple's SpeechAnalyzer for English. That may look odd under a table where it sits third, so here is the reasoning: on clean speech the top three are separated by 0.11 points, roughly one differently-heard word per thousand; SpeechAnalyzer costs users zero megabytes of download against Parakeet's 443MB; and it is the engine Apple maintains and improves with the OS. A 0.11-point clean-speech lead does not buy a half-gigabyte download in a consumer app. Parakeet's real edge, noisy audio, is exactly what the round 3 meeting benchmark will measure properly, and if the gap holds there, our answer may change. That is what these benchmarks are for.

MOSS we decided not to adopt, at least for now, and the reasoning is worth showing because it is the same cost-benefit anyone shipping a transcription app faces. On transcription accuracy MOSS is a wash with the system engine, so the entire case for it rests on diarization. What it would cost: a 1.7GB model download, about 2.7GB of memory at inference (workable on a Mac, hostile on an iPhone), and a runtime we cannot ship today, since MOSS runs through Python and MLX with no Swift or CoreML path. It also cannot transcribe live: MOSS needs the complete recording before it produces anything (its streaming mode streams output text, not audio input), so live captions would stay on a separate engine anyway and MOSS could only ever replace the post-recording half of the pipeline. What we would get: speaker labeling that beat our controlled probe, measured against a current pipeline (system transcription plus a dedicated ~35MB diarization model) whose real-meeting accuracy we have not benchmarked either. Paying three and a half gigabytes of download-plus-memory for an unquantified improvement is not a decision, it is a bet. So round 3 is the decision: both systems on the AMI meeting corpus, real crosstalk, measured DER. If MOSS wins dramatically, we will revisit and say so here. If it wins narrowly, the 35MB pipeline keeps its job.

Related reading

• Round 1: Apple's new Speech API vs Whisper
• Best offline transcription apps
• Private transcription apps