NVIDIA shows how to serve HSTU recommenders on Dynamo-Triton
NVIDIA has published a workflow for deploying HSTU generative recommenders on Dynamo-Triton, using ahead-of-time compilation and GPU KV caching to cut latency by up to 5.93x.
NVIDIA Dynamo-Triton, the renamed Triton Inference Server, now supports a complete inference workflow for Hierarchical Sequential Transduction Unit (HSTU) generative recommenders. The workflow lives in NVIDIA's recsys-examples repository, and NVIDIA describes it in a new technical blog post.
Generative recommenders treat recommendation as sequence modeling over user behavior. Contexts, items and actions become tokens in a single event stream. This makes serving costly: long user histories, large embedding tables and repeated requests from the same user can lead to redundant computation.
The workflow combines several parts:
- PyTorch AOTI. The model is exported with
torch.exportand compiled ahead of time into a.pt2package that a native C++ runtime can load, which reduces Python overhead. - KV caching. A KVCacheManager keeps a paged KV table in GPU memory, evicts entries in an LRU-style way and adds a host-storage tier with a FlexKV-backed backend. Cache-aware custom ops handle lookup, allocation, append and offload.
- Embeddings. DynamicEmb tables are paired with NV Embedding Cache, which keeps popular embeddings on the GPU and the full table in CPU memory.
- Validation. Artifacts can be checked in both Python and native C++ before they are deployed.
NVIDIA says the model doesn't need to be rewritten for a separate runtime.
NVIDIA tested at dynamic batch size 8 on an RTX PRO 6000 Blackwell Workstation Edition GPU. It reports best-case speedups of 4.47x for a three-layer HSTU model and 5.93x for an eight-layer model. Both figures assume a 100% GPU KV-cache hit rate and are measured against the same AOTI setup without caching. The excerpt gives no results for lower hit rates or comparisons with other serving stacks.