How Good is Gemma 4 at ASR Tasks? Benchmarking Against the Open ASR Leaderboard

Gemma 4 does audio. I ran all three sizes, E2B (2B), E4B (4B), and the new 12B, against all eight Open ASR Leaderboard short-form datasets, batched. Performance is decent on clean speech and unreliable on harder inputs, so it’s useful as a general-purpose capability and for understanding. Don’t swap it in for a dedicated ASR model. The surprise: scaling reverses at the top. The 12B, the biggest, is the worst of the three.

3.05%

Best WER

E4B · LS Clean

7.6%

E4B mean WER

7 datasets, ex-AMI

265×

E2B throughput

LS Clean · batched

7.4%

12B refusal rate

vs 0.14% for E2B/E4B

Setup

Harness: gemma4-audio, jiwer for WER/CER/MER/WIL.¹
Backend: vLLM, bf16, batched (16 for E2B/E4B, 8 for the 12B). RTFx is throughput: total audio / wall clock. The 12B needed a from-source vLLM build and a fix to its batched-audio path.²
Hardware: RTX 6000 Pro Blackwell, 96 GB.
Prompt:

Transcribe the following speech segment in its original language.

Text before audio, the model card’s order. Audio-first ordering and extra instruction bullets both inflate the 12B’s refusals; this stripped-down prompt roughly quarters them.³

WER across the suite

Dataset / Model	Cohere	Parakeet 0.6B v2	Phi-4 MM	Whisper lg-v3	Whisper base.en	Gemma 4 E2B	Gemma 4 E4B	Gemma 4 12B
AMI	8.13	11.16	11.09	15.95	21.13	20.81	18.95	104
Earnings22	10.86	11.15	10.16	11.29	15.09	15.44	13.98	50.40
GigaSpeech	9.34	9.74	9.33	10.02	12.83	11.81	11.11	23.91
LS Clean	1.25	1.69	1.69	2.01	4.25	3.70	3.05	3.85
LS Other	2.37	3.19	3.82	3.91	10.35	8.69	7.61	14.75
SPGISpeech	3.08	2.17	3.06	2.94	4.26	5.47	4.86	54.04
TED-LIUM	2.49	3.38	2.94	3.86	4.87	4.41	3.85	8.62
VoxPopuli	5.87	5.95	6.04	9.54	9.76	9.09	8.54	9.41

WER (%) across the Open ASR short-form suite, lower is better. Competitor numbers from the Open ASR Leaderboard (Apr 2026); Gemma 4 E2B, E4B and 12B are our runs on vLLM (batched, text-first prompt, RTX 6000 Pro Blackwell 96 GB), Gemma columns outlined. E4B is the strongest of the three and E2B trails it by ~10–15%; both stay within a few points of the dedicated models on clean read speech. The 12B keeps pace on LS Clean but degrades sharply on everything harder (AMI 104%, Earnings22 50%, SPGISpeech 54%). At the top, more scale stops helping.

E4B stays within 2–3 WER points of the leaders on clean read speech. The gap widens on noisy spontaneous audio. On LS Clean specifically, E4B (3.05%) beats Whisper base.en (4.25%). A 4B generalist holding its own against a 74M CTC model isn’t bad. E2B trails E4B by ~10–15% everywhere; more parameters help.

Then the 12B. Fine on LS Clean (3.85%), off a cliff everywhere else: 104% on AMI, 50% on Earnings22, 54% on SPGISpeech.⁴ A 12B losing to a 2B on clean financial speech by 10×. At the top, more parameters stop helping.

Speed vs accuracy

Model	WER (%)	RTFx (×)	Source
Cohere	1.25	525	Open ASR Leaderboard
Parakeet 0.6B v2	1.69	3386	Open ASR Leaderboard
Phi-4 MM	1.69	151	Open ASR Leaderboard
Whisper lg-v3	2.01	146	Open ASR Leaderboard
Whisper med.en	3.02	182	Open ASR Leaderboard
Whisper small.en	3.05	269	Open ASR Leaderboard
Whisper base.en	4.25	321	Open ASR Leaderboard
Whisper tiny.en	5.66	348	Open ASR Leaderboard
Gemma 4 E2B	3.7	265	Gemma 4 (this run)
Gemma 4 E4B	3.05	178	Gemma 4 (this run)
Gemma 4 12B	3.85	61	Gemma 4 (this run)

LibriSpeech test-clean WER vs RTFx, log x-axis. Both axes are apples-to-apples: competitor RTFx is batched leaderboard throughput, and our Gemma runs are batched too (throughput RTFx = total audio / wall clock) on an RTX 6000 Pro Blackwell (96 GB). They trail dedicated ASR models on raw speed, and the 12B is both the slowest and the least accurate of the three, despite being the largest.

Competitor RTFx and ours are both batched. E2B runs 265× real-time on LS Clean, E4B 178×. Short of Parakeet’s 3,386×, but usable. The 12B trails at 61×: more tokens per clip, most of them wrong.

Error composition

Dataset	Substitutions (%)	Insertions (%)	Deletions (%)
AMI	9.2	2.98	6.76
Earnings22	6.97	3.3	3.71
GigaSpeech	6.31	1.86	2.94
VoxPopuli	3.74	2.2	2.59
LS Other	5.87	0.61	1.12
SPGISpeech	2.39	1.85	0.63
TED-LIUM	2.15	0.62	1.07
LS Clean	2.29	0.24	0.52

Error composition for Gemma 4 E4B by dataset, ordered worst to best WER. Bars show each error type as % of reference words. The insertion rate still climbs on noisy, spontaneous speech (AMI 3%, Earnings22 3%), the signature of a language-model prior overriding weak acoustic evidence, but the lean text-first prompt keeps it modest; audio-first ordering pushes AMI past 20%. The 12B is the outlier: its AMI insertion rate is 64%, off this chart.

Insertions climb on noisy data: the LM prior filling in when the audio is hard to decode. E4B: 0.2% on LS Clean, 3% on AMI.⁵ The 12B is a different animal. AMI insertion rate 64%, pure hallucination. On short or hard clips it flips, stops inserting and starts refusing.

Where Gemma 4 actually breaks

Duration bucket	E2B WER (%)	E4B WER (%)	12B WER (%)	n
<1s	49.2	41.9	667.4	6232
1–2s	38.2	34.2	91	4623
2–3s	21.8	19.9	52.7	4357
3–4s	17.4	15.5	37	3942
4–6s	10.2	9.2	40	12181
6–8s	7.5	6.7	42.3	16350
8–11s	6.9	6.2	42.1	20366
11–15s	6.5	5.9	43.5	13989
15–20s	8.5	7.6	18	801
20–30s	11.5	10.8	13	332
30s+	15	14.4	26.4	39

Mean WER (%, log scale) by audio-duration bucket, across all 8 Open ASR short-form datasets (93k+ samples). The U-shape persists, but the story is the model size: switching to the model card's text-first prompt holds the small models together at the short end (sub-1s E2B is 49%; audio-first ordering produces 2,203% on the same clips), while the 12B is worse at every single bucket: 667% on sub-1s clips and still ~42% even on the comfortable 6–15s range. The largest model degrades the most.

Bucket every sample by audio duration and you get a U-shape. Over ~3s is fine. Under that it slips. Sub-1s is the cliff.

Sub-1s, E2B: 49% mean WER. Audio-first ordering would make it 2,203%.
Sub-1s, E4B: 42%.
Sub-1s, 12B: 667%.

The 12B is worse at every bucket, never under ~40% even at a comfortable 6–15s. More parameters help, except at the top.

The 12B’s failure mode

The 12B doesn’t mis-hear so much as stop transcribing. 7% of its outputs are refusals⁶ (E2B and E4B: 0.14%):

I’m sorry, but I cannot fulfill this request. I am a text-based AI and do not have the ability to listen to or process audio.

From a model that was handed audio. The rest is greedy loops (comment on your revenue growth → “come on, come on, come on…”) and paraphrase in place of verbatim transcription. The extra scale bought a chatbot, not a transcriber.

Raw data

Dataset	E2B (2B)	E4B (4B)	12B	Upstream license
AMI	csv · json	csv · json	csv · json	CC BY 4.0
Earnings22	csv · json	csv · json	csv · json	CC BY-SA 4.0
GigaSpeech	csv · json	csv · json	csv · json	Apache 2.0
LS Clean	csv · json	csv · json	csv · json	CC BY 4.0
LS Other	csv · json	csv · json	csv · json	CC BY 4.0
VoxPopuli	csv · json	csv · json	csv · json	CC0

Per-sample metrics. CSV: one row per utterance with sample id, WER, CER, MER, WIL, substitutions/insertions/deletions, latency, RTFx, audio duration. JSON has the run config and corpus-level metrics. References and hypotheses are stripped to avoid redistributing upstream text; originals are recoverable by sample id from each dataset. SPGISpeech (Kensho research-only) and TED-LIUM (CC BY-NC-ND 3.0) are excluded from redistribution, but aggregate metrics from those runs still appear in the charts above.

Footnotes

Word error rate: (substitutions + insertions + deletions) divided by the number of words in the reference, computed with jiwer. CER, MER, and WIL are the character-error, match-error, and word-information-lost variants the same harness records. ↩
vLLM’s stock build mishandled the 12B’s batched-audio path, so this run used a from-source build with a patch to it. The exact build lives in the gemma4-audio harness. ↩
“Audio-first” puts the audio ahead of the text instruction in the prompt. That ordering sharply raises the 12B’s refusal rate and, on sub-1-second clips, sends E2B’s WER to 2,203%. Every number here uses text-first ordering, the order given on the model card. ↩
WER isn’t capped at 100%. Insertions count as errors, so a model that emits more words than the reference contains — here, hallucinated runs of text — pushes the rate past 100%. ↩
AMI is far-field, multi-speaker meeting audio recorded on distant microphones — the hardest set in the suite. Every model degrades on it and the 12B collapses, which is why the mean WER above is quoted over the other seven datasets. ↩
Refusals are outputs that decline the task instead of transcribing — the stock “I’m sorry, but I cannot fulfill this request” reply quoted below and its close variants. ↩