NVIDIA claims 30x per-megawatt gain for Vera Rubin
NVIDIA says Vera Rubin NVL72 delivers up to 30x GB300's agentic throughput per megawatt at the same 288 GB per GPU, and credits hardware-software co-design rather than added memory capacity.
NVIDIA argues in a technical blog post that the limit on AI inference is now how much useful work each byte of high-bandwidth memory (HBM) delivers, not how much HBM a GPU has. Per-GPU HBM capacity stayed at 288 GB from GB300 to Rubin, while bandwidth rose to 19.2 TB/s.
The main new figure is NVIDIA's own. On SemiAnalysis AgentX, a benchmark that replays recorded coding-agent sessions, the company says Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. NVIDIA has not published independent verification of that result.
Most of the post covers techniques NVIDIA has described before:
- NVFP4: Applied to weights, the 4-bit format shrinks model footprint about 3.5x versus FP16 and 1.8x versus FP8. NVIDIA reports up to 4x FP8 throughput at the same interactivity. As a KV-cache format, NVFP4 cuts cache size by up to 50% versus FP8.
- Wide expert parallelism in TensorRT-LLM: Spreading mixture-of-experts weights across more GPUs frees HBM for KV cache. On GB200 NVL72, EP32 delivers up to 1.8x more output tokens per second per GPU than EP8 at 100 tokens per second per user.
- Dynamo: The serving layer handles KV-aware routing and runs prefill and decode on separate GPU pools to avoid recomputing cached prefixes.
NVIDIA says each Rubin GPU has 3.0 TB/s of NVLink 6 bandwidth, giving about 20.7 TB of HBM4 per rack. The material available here cuts off before the last two of the five techniques, which the signal metadata names as the Vera CPU and CMX storage.