Ming 3 Audio Family Quantized to NVFP4

I published NVFP4¹ quants of inclusionAI’s Ming 3.0 Audio Tiny and Flash. Both handle speech recognition, audio understanding, and reasoning in English and Chinese.

Flash has 126B total parameters (239 GiB), which doesn’t really fit on my RTX PRO 6000 Blackwell (96 GiB) for audio-to-text inference.

Getting it into four bits

I used NVIDIA Model Optimizer for W4A4² post-training quantization. Calibration used 2,053 audio and text examples with mean squared error. Only the routed experts are quantized; attention, routers, shared experts, and audio components keep their original precision.

Tiny ended up keeping seven routed layers in BF16³ after quality checks, with the other 16 in NVFP4. Flash quantizes all 40 routed layers. Some Flash experts barely saw calibration tokens, so I used Model Optimizer’s pooled activation scales, with separate weight scales for each expert.⁴

Inside Ming 3 Audio

TinyFlash

Hover to unfold · click to explore

Ming 3 AudioContains changed projections

Audio and text enter the Thinker together.

Getting there meant calibrating layer by layer, offloading source weights, auditing the export, and adapting the vLLM-Omni serving path to load the mixed precision correctly. The repository includes the recipes, calibration selection, export checks, and Docker workflows for quantization, serving, and evaluation.

Benchmarks

Like the Gemma 4 benchmark, I wanted actual measurements. I ran Tiny BF16, Tiny NVFP4, and Flash NVFP4 through eight full benchmark splits.

All scores below are percentages. Lower WER/CER and higher accuracy are better.⁵ Alongside our three local runs, I included Seed2.0 Lite, Qwen3.8-Omni Flash, and Gemini 3.8 Flash from inclusionAI’s published comparisons.⁶

Dataset / Model	Tiny BF16 (local)	Tiny NVFP4 (local)	Flash NVFP4 (local)	Flash (published)	Seed2.0 Lite (published)	Qwen3.8-Omni Flash (published)	Gemini 3.8 Flash (published)
FLEURS English	3.29	3.44	2.92	2.86	2.75	3.58	3.29
FLEURS Chinese	3.52	3.43	3.45	2.92	4.89	3.76	7.31
LibriSpeech clean	1.42	1.44	1.20	1.20	1.07	1.25	1.96
LibriSpeech other	3.04	3.07	2.45	2.40	2.17	2.64	4.46
MMAU mini	76.70	77.30	78.20	82.00	77.20	82.30	78.20
MMAR	63.90	65.20	69.50	74.40	77.70	78.60	81.00
MMSU	68.04	67.14	77.80	78.82	82.10	80.84	83.39
MMLU-Pro	61.43	60.16	75.24	80.00	—	87.70	—

Tiny’s ASR⁷ scores stayed close to BF16; MMLU-Pro lost about 1.3 percentage points. Its measured throughput was also roughly unchanged: text gained 3.2%, audio-to-text changed by −0.1%. The clear win was memory, from 15.97 to 9.50 GiB of Thinker model allocation.⁸

The biggest debugging detour was Chinese FLEURS. Tiny NVFP4 initially scored 8.45% CER. Normalizing number forms and some parenthetical text brought the same saved predictions to 3.43%. Even original Tiny scored 8.54% under the strict scorer. Much of that apparent gap came from how we counted errors.⁹

The weights and model cards are on Hugging Face: Tiny and Flash. The quantization code is open source under MIT.

Footnotes

1. NVIDIA’s four-bit floating-point format. ↩
2. Four-bit weights (W4) and four-bit activations (A4). ↩
3. Bfloat16, a 16-bit floating-point format. ↩
4. Pooling shares activation-scale inputs across experts. It does not mean every expert was well represented in the calibration data; the raw coverage counts are preserved in the reproduction workflow. ↩
5. WER is word error rate; CER is character error rate. Both count substitutions, deletions, and insertions relative to the reference transcript. ↩
6. Published scores come from inclusionAI’s ASR, audio, and text tables, not local runs; protocols differ. Asterisks mark self-evaluations; dashes mean unreported. Qwen’s audio scores are non-thinking; its text mode and published MMAU splits are unspecified (ours is test-mini). ↩
7. Automatic speech recognition: transcribing speech into text. ↩
8. Three-trial medians at client concurrency 512 for text and 1,024 for audio. Model allocation excludes the key-value (KV) cache, workspaces, CUDA graphs, and other allocations. Workload settings and measurements. ↩
9. The table uses UltraEval-Audio text normalization with character-level scoring for Chinese, Whisper normalization for English, and explicit answer parsing for multiple choice. These datasets overlap earlier local development and checkpoint selection. The full report preserves unchanged scorer outputs and explains the remaining gaps. These are local measurements, not an exact reproduction of inclusionAI’s results. ↩