NVIDIA publishes speculative decoding tuning guidelines
NVIDIA's developer blog has posted the third part of its AI model co-design series, laying out formulas and five guidelines for picking speculative decoding draft length and draft mechanism without losing accuracy.
The post frames speculative decoding — a small draft model proposes tokens, the larger target model verifies them in a single parallel pass — as a lever for raising arithmetic intensity without raising concurrency. Because only tokens the target accepts are kept, output matches standard decoding unless acceptance criteria are deliberately relaxed.
NVIDIA gives an explicit speedup expression in terms of draft length (D), acceptance length (AL), batch size, target verification time and drafting latency, and argues the optimization target is the (D, AL, T_draft) tuple, with the optimal D varying along the Pareto frontier.
Three guidelines appear in the excerpt. First: raise D to push linear-layer GEMMs into the compute-bound region without adding KV cache pressure — for a representative 6144x6144 expert GEMM, D=7 reaches compute-bound at one-eighth the batch size of D=0, which the authors say matters more as MoE models get sparser. Second: when attention dominates decode time, set D = 128/G − 1, where G is query heads per KV head; charts for G=8 and G=32 at 32K and 128K KV lengths show G=32 saturating at lower D. Third: if exceeding that, choose D such that G × (1 + D) is a multiple of 128, the benchmarked kernel's tile size, to avoid partial-tile waste.
The available text cuts off mid-sentence while weighing FFN versus attention runtime; the remaining guidelines are not visible.