NVIDIA explainer weighs dense models against MoE
NVIDIA has published a developer-blog primer on dense versus Mixture-of-Experts architectures, using its Nemotron 3.5 Lightning model to argue that sparsity, not parameter count, now sets serving throughput and memory cost.
The post is an explainer rather than an announcement: no new benchmark result, model release, or performance claim accompanies it.
Its framing device is Nemotron 3.5 Lightning, a ~30B-total-parameter model that activates roughly 3B parameters per token. NVIDIA walks through the mechanics — an MoE layer replaces the single shared feed-forward network in each decoder layer with multiple expert FFNs (8, 64, or 128), fronted by a learned router that assigns each token to the top-k scorers. Routing is re-decided at every layer, and tokens still pass through the full attention path, so a quoted "3B active" figure includes attention and embedding weights. Expert specialization, the post notes, tends toward syntax and token-type patterns rather than subject matter. Many current MoEs, including Mistral Small 4, also run a shared expert every token passes through.
The load-bearing argument is that MoE decouples memory from compute: all experts must sit in GPU memory, so VRAM is paid up front while FLOPs scale only with the experts that fire — which also leaves less room for KV cache than a comparable dense model. Lightning complicates that further with Mamba-2 layers carrying a constant-size recurrent state instead of a growing cache, plus speculative decoding.
A comparison table sets Lightning against Gemma 4 31B at similar total size, with NVIDIA reporting non-overlapping output-speed ranges across GPU providers. The blog concedes MoE's latency edge compresses at high concurrency, where routing and memory movement bite.