Dynamo-Triton adds multi-GPU TensorRT serving
NVIDIA's Dynamo-Triton 26.07 turns on TensorRT multi-device inference, letting one served model span up to eight GPUs and cutting a Cosmos 3 Nano video generation from 156.6 seconds to 34.2.
NVIDIA Dynamo-Triton release 26.07 enables the multi-device inference path in its TensorRT backend, letting a single KIND_MODEL instance own several GPUs, create per-rank TensorRT execution contexts, CUDA streams, and NCCL communicators, and launch the ranks together per request. Clients call one named model over a gRPC endpoint rather than coordinating GPU ranks themselves. The underlying TensorRT multi-device capability, which runs one network across multiple GPUs using NCCL-backed collectives, is fully supported starting with TensorRT 11.0.
NVIDIA demonstrates the integration on Cosmos 3 Nano video generation, serving the 36-layer denoising transformer while Diffusers continues to handle prompts, latents, classifier-free guidance, scheduling, VAE decode, and postprocessing. Ulysses context parallelism spreads the workload's 44,160 video tokens across up to eight GPUs; at CP8 each rank handles 5,520 video tokens outside attention, with the 2,992-token text path replicated. The distributed graph is compiled into each TensorRT plan ahead of deployment — Triton configuration activates it but does not convert a single-device engine.
On one eight-GPU system at 1280×720, 189 frames and 35 denoising steps, with one warm-up and five measured generations, end-to-end latency fell from 156.595 seconds on one GPU to 34.183 seconds on eight, a 4.58x speedup. Transformer RPC time dropped 6.09x and its share of runtime fell from 93.4% to 70.2%; non-RPC work stayed at roughly 10.2–10.5 seconds throughout. Model loading and mp4 encoding were excluded from timings.