# Ming 3 Audio Family Quantized to NVFP4

I published NVFP4[^1] quants of inclusionAI's **Ming 3.0 Audio [Tiny](https://huggingface.co/twangodev/Ming-3.0-audio-tiny-NVFP4) and [Flash](https://huggingface.co/twangodev/Ming-3.0-audio-flash-NVFP4)**. Both handle speech recognition, audio understanding, and reasoning in English and Chinese.

Flash has 126B total parameters (239 GiB), which doesn't really fit on my RTX PRO 6000 Blackwell (96 GiB) for audio-to-text inference.

## Getting it into four bits

I used [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer) for W4A4[^2] post-training quantization. Calibration used 2,053 audio and text examples with mean squared error. Only the routed experts are quantized; attention, routers, shared experts, and audio components keep their original precision.

Tiny ended up keeping seven routed layers in BF16[^3] after quality checks, with the other 16 in NVFP4. Flash quantizes all 40 routed layers. Some Flash experts barely saw calibration tokens, so I used Model Optimizer's pooled activation scales, with separate weight scales for each expert.[^4]
Quantization scope: audio → encoder → projector, and text → embeddings, feed the Thinker. The text head predicts tokens; the optional, separate Talker consumes text and generates latents for AudioVAE to decode into speech.

Only routed-expert gate_proj, up_proj, and down_proj linears use NVFP4 weights and activations (W4A4). Attention, routers, shared experts, dense layers, embeddings, norms, text head, audio components, and caches retain source precision. The optional speech path was not evaluated.

| Model | Decoder layers | Dense layers | NVFP4 expert layers | BF16 expert layers | Experts per MoE layer |
| --- | --- | --- | --- | --- | --- |
| Tiny | 24 | 1 | 16 | 7 | 128 |
| Flash | 42 | 2 | 40 | 0 | 512 |

Tiny retains routed-expert layers 5, 10, 15, 18, 20, 21, 22 in BF16 (zero-based). Eight routed experts are selected per token in either model. Each MoE layer also has one shared expert. Tiny uses MLA every fourth layer; Flash every sixth; other layers use KDA. All attention remains at source precision.

**Kimi Delta Attention:** Instead of storing every past token as a key/value pair, KDA updates a matrix memory for each head. Queries read that memory; keys and values correct it. The attention weights and state were not quantized. [Implementation](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/modeling_bailing_moe_v3.py).

**Multi-head latent attention:** Compress the key/value input through a shared low-rank representation, reconstruct per-head keys and values, add rotary position information, and run causal softmax attention. [Implementation](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/modeling_bailing_moe_v3.py).

**Grouped top-8 router:** Score every expert, use the learned routing bias to choose groups and experts, then weight the chosen experts using the original sigmoid scores. Router arithmetic stays float32. [Implementation](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/modeling_bailing_moe_v3.py).

**Optional speech output:** The official Talker receives text, normalizes it, and generates continuous audio latents. AudioVAE decodes them into a waveform. This is a separate speech model, not a direct projection of Thinker hidden states. [Implementation](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/talker/inference.py).

**Auxiliary next-token predictor:** Both checkpoints include one additional MTP layer after the main decoder stack. It combines shifted token embeddings with hidden states. The audio wrapper supplies inputs_embeds, so this auxiliary branch is skipped in our evaluated path and its weights are preserved. [Implementation](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/modeling_bailing_moe_v3.py).

All decoder layers are shown. The single 3D scene supports nested inspection of audio features, encoder and projector, text embeddings, KDA and MLA, norms and residuals, routing and expert projections, text generation, Talker, AudioVAE, and the auxiliary MTP branch. MTP is unchanged and skipped by the evaluated audio wrapper. Operation-specific shapes unfold on hover and pin on click: layer stacks, tapered linear projections, weight matrices, convolution windows, expert banks, recurrent state, and arithmetic operators. Moving dots illustrate signal direction, not live activations or measured expert routing. Collapsed overview block volumes are proportional to original parameter counts (not quantized storage or active parameters), including auxiliary MTP and optional Talker/AudioVAE weights; non-learned buffers and the external speaker encoder are excluded. Inner grids and exploded spacing are schematic. [Quantization recipes and scope](https://github.com/twangodev/ming3-nvfp4/blob/main/docs/reproduction.md).

Getting there meant calibrating layer by layer, offloading source weights, auditing the export, and adapting the vLLM-Omni serving path to load the mixed precision correctly. The [repository](https://github.com/twangodev/ming3-nvfp4) includes the recipes, calibration selection, export checks, and Docker workflows for quantization, serving, and evaluation.

## Benchmarks

Like the [Gemma 4 benchmark](/writing/gemma4-asr-benchmark), I wanted actual measurements. I ran Tiny BF16, Tiny NVFP4, and Flash NVFP4 through eight full benchmark splits.

All scores below are percentages. Lower WER/CER and higher accuracy are better.[^5] Alongside our three local runs, I included **[Seed2.0 Lite](https://seed.bytedance.com/en/seed2), [Qwen3.8-Omni Flash](https://docs.qwencloud.com/developer-guides/speech/multimodal-speech), and [Gemini 3.8 Flash](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash)** from inclusionAI's published comparisons.[^6]
Scores (%); local NVFP4 columns outlined. FLEURS Chinese uses CER; other speech rows use WER (lower is better). MMAU, MMAR, MMSU, and MMLU-Pro use accuracy (higher is better). Published columns are references, not matched local runs. — means unreported in the cited tables. Published MMAU splits are unspecified; our run uses test-mini.

| Dataset / Model | Tiny BF16 (local) | Tiny NVFP4 (local) | Flash NVFP4 (local) | Flash (published) | Seed2.0 Lite (published) | Qwen3.8-Omni Flash (published) | Gemini 3.8 Flash (published) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| FLEURS English | 3.29 | 3.44 | 2.92 | 2.86 | 2.75 | 3.58 | 3.29 |
| FLEURS Chinese | 3.52 | 3.43 | 3.45 | 2.92 | 4.89 | 3.76 | 7.31 |
| LibriSpeech clean | 1.42 | 1.44 | 1.20 | 1.20 | 1.07 | 1.25 | 1.96 |
| LibriSpeech other | 3.04 | 3.07 | 2.45 | 2.40 | 2.17 | 2.64 | 4.46 |
| MMAU mini | 76.70 | 77.30 | 78.20 | 82.00 | 77.20 | 82.30 | 78.20 |
| MMAR | 63.90 | 65.20 | 69.50 | 74.40 | 77.70 | 78.60 | 81.00 |
| MMSU | 68.04 | 67.14 | 77.80 | 78.82 | 82.10 | 80.84 | 83.39 |
| MMLU-Pro | 61.43 | 60.16 | 75.24 | 80.00 | — | 87.70 | — |

Tiny's ASR[^7] scores stayed close to BF16; MMLU-Pro lost about 1.3 percentage points. Its measured throughput was also roughly unchanged: text gained 3.2%, audio-to-text changed by −0.1%. The clear win was memory, from **15.97 to 9.50 GiB** of Thinker model allocation.[^8]

The biggest debugging detour was Chinese FLEURS. Tiny NVFP4 initially scored **8.45% CER**. Normalizing number forms and some parenthetical text brought the same saved predictions to **3.43%**. Even original Tiny scored 8.54% under the strict scorer. Much of that apparent gap came from how we counted errors.[^9]

The weights and model cards are on Hugging Face: [Tiny](https://huggingface.co/twangodev/Ming-3.0-audio-tiny-NVFP4) and [Flash](https://huggingface.co/twangodev/Ming-3.0-audio-flash-NVFP4). The quantization code is [open source under MIT](https://github.com/twangodev/ming3-nvfp4).

## Notes

[^1]: NVIDIA's four-bit floating-point format.

[^2]: Four-bit weights (W4) and four-bit activations (A4).

[^3]: Bfloat16, a 16-bit floating-point format.

[^4]: Pooling shares activation-scale inputs across experts. It does not mean every expert was well represented in the calibration data; the raw coverage counts are preserved in the [reproduction workflow](https://github.com/twangodev/ming3-nvfp4/blob/main/docs/reproduction.md).

[^5]: WER is word error rate; CER is character error rate. Both count substitutions, deletions, and insertions relative to the reference transcript.

[^6]: Published scores come from inclusionAI's [ASR](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/assets/benchmark-asr.png), [audio](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/assets/benchmark-audio-intelligence.png), and [text](https://huggingface.co/inclusionAI/Ming-3.0-audio-tiny/blob/3c9578f641b6745abf2d1239dc8419726f37c0e2/assets/benchmark-text.png) tables, not local runs; protocols differ. Asterisks mark self-evaluations; dashes mean unreported. Qwen's audio scores are non-thinking; its text mode and published MMAU splits are unspecified (ours is test-mini).

[^7]: Automatic speech recognition: transcribing speech into text.

[^8]: Three-trial medians at client concurrency 512 for text and 1,024 for audio. Model allocation excludes the key-value (KV) cache, workspaces, CUDA graphs, and other allocations. [Workload settings and measurements](https://huggingface.co/twangodev/Ming-3.0-audio-tiny-NVFP4/blob/main/evals/results/performance.csv).

[^9]: The table uses UltraEval-Audio text normalization with character-level scoring for Chinese, Whisper normalization for English, and explicit answer parsing for multiple choice. These datasets overlap earlier local development and checkpoint selection. The [full report](https://github.com/twangodev/ming3-nvfp4/blob/main/docs/results/full-suite-parity.md) preserves unchanged scorer outputs and explains the remaining gaps. These are local measurements, not an exact reproduction of inclusionAI's results.