NVIDIA Speeds Up MoE Decoding on DGX Spark
NVIDIA stacked three optimizations to speed up batch-size-1 decoding of Inclusion AI's Ling-3.0-flash on DGX Spark, and warned that one combination can quietly leave the setup slower than baseline.
NVIDIA detailed three ways to speed up mixture-of-experts inference on DGX Spark, using Inclusion AI's Ling-3.0-flash-fp4 as the test model. The model has 124B total parameters and 5.1B active, and each token is routed to eight of 512 experts. Profiling at batch size 1 showed decoding was memory-bound: most of the time went to reading weights in the MoE layers and in the 157,184-token output projection (LM head).
The baseline used the official cookbook's W4A8 setup and ran at 27.71 ms per output token (36.09 tok/s). The gains stacked up:
- Humming MXFP4 MoE kernels. These replace FlashInfer and compile kernels at runtime through NVRTC for the model's exact shapes. Each kernel builds in seconds, versus tens of minutes for FlashInfer's CUTLASS path. Throughput rose to 39.45 tok/s (+9.3%).
- Online FP8 LM head. Quantizing the BF16 output layer to FP8 halves the weight reads for that layer. With Humming, throughput reached 43.94 tok/s (+21.7%).
- MTP speculative decoding. This uses the model's built-in NEXTN layer as the draft model, with three draft steps and a four-token verification window. One verification step cost about 2.26× a normal decode step, mostly because checking four tokens can activate up to 32 experts instead of eight.
According to a summary accompanying the post, the full stack delivered a 42.5% real-world decode speedup on GSM8K with no detectable accuracy loss.
NVIDIA also flagged a silent failure. When FP8 LM-head quantization and MTP are both on, the draft model has to share the whole LM-head module, not just its weight tensor. Without that, the draft reads FP8 weights as if they were BF16, accept length falls to 1.00, and throughput ends up below a configuration with no optimizations. Code and deployment instructions are in the Ling cookbook.