NVIDIA details when encoder disaggregation pays off

NVIDIA has published a guide to encode-prefill-decode (EPD) disaggregation in Dynamo, its open source distributed inference framework, arguing that splitting the vision encoder off from LLM prefill and decode delivers up to 5x faster time to first token (TTFT) and 7x faster end-to-end response — but only under specific conditions.

In aggregated serving, vision encoding, prefill, and decode share one worker and one scheduling domain. That works when media is a small share of the workload. As requests carry more images or video, vision transformer (ViT) encoding can run hundreds of milliseconds, delaying a request's own prefill and forcing text-only requests to queue behind multimodal ones. Dynamo separates encoder and PD workers, passing embeddings between them via the NVIDIA Inference Transfer Library (NIXL), so each stage batches and scales independently.

The post is explicit about limits. Isolated encode work must be large enough to offset coordination and embedding-transfer overhead. Gains concentrate in image-heavy prompts, short-to-medium output lengths, and small, MoE, or low-precision models, where the ViT-to-LLM compute ratio is higher; longer output sequence lengths preserve TTFT gains but erode end-to-end gains, and large dense models benefit less.

On homogeneous clusters, NVIDIA recommends colocated encoder workers over a dedicated GPU, which it says leaves the encoder GPU underutilized. Disaggregated placement is pitched for mixed fleets: its test setup paired two RTX 6000D GPUs for encoding with four GB200 GPUs for prefill and decode. Benchmarks used Qwen3.5 122B A10B NVFP4; the supplied excerpt cuts off before full results.