Baseten reports 3x inference speedups with DFlash
Baseten says its DFlash speculative decoding gets about 3x speedups on Qwen3-8B, past the roughly 2x that EAGLE-style autoregressive drafting usually tops out at.
Baseten reported benchmark results for its implementation of DFlash, a speculative decoding method introduced in February 2026. The company says it is about 50% faster than its own EAGLE setup and 10–30% faster than vLLM's DFlash backend.
Speculative decoding has a small draft model propose tokens, and the target model then checks them in parallel. EAGLE drafts one token per forward pass, so errors build up from pass to pass, and Baseten says this tends to cap speedups at about 2x. DFlash uses bidirectional attention to unmask a whole block of 8–16 tokens in a single pass. It conditions on hidden states taken from 5–6 evenly spaced layers of the target model. Each DFlash pass can be 2–4x slower than an EAGLE pass, but Baseten says one pass still takes less time than EAGLE's entire draft phase. The time saved also leaves room for a deeper draft model.
On GSM8k with Qwen3-8B on one B200 GPU at concurrency 16, Baseten measured these results against a TRT-LLM baseline with no speculative decoding:
- Baseten DFlash: 3.04x throughput (654 tokens/s mean) and 2.84x lower latency
- vLLM DFlash: 2.77x throughput and 2.16x lower latency
- Baseten EAGLE: 2.12x throughput
Baseten also ran MATH-500 and NVIDIA's Nemotron post-training dataset. It says its implementation led on all three benchmarks, but the excerpt includes full numbers only for GSM8k. Baseten left SGLang out of the comparison because SGLang often produced looping outputs, which pushed up its acceptance rates and throughput. After those samples were filtered out, Baseten says SGLang was slightly slower than vLLM.
The results come from Baseten and have not been independently verified. Baseten has not said whether the gains hold for larger target models.