NVIDIA brings TensorRT Edge-LLM CLI to DGX Spark
NVIDIA's new trtedge command-line tool runs edge LLMs on DGX Spark in a few commands, so developers can build on Spark with the same inference runtime they deploy on Jetson Thor and DRIVE AGX Thor.
NVIDIA published a tutorial for trtedge, a command-line interface for TensorRT Edge-LLM, its lightweight C++ inference runtime for embedded workloads. The CLI downloads a selected model and builds TensorRT engines optimized for DGX Spark. NVIDIA also said DGX Spark is now available in a 64GB configuration.
The pitch is a single runtime from prototype to deployment. Developers can test and fine-tune models on Spark, then deploy on NVIDIA Jetson Thor, DRIVE AGX Thor, and IGX Thor. The tutorial uses Nemotron-3.5 Lightning (30B-A3B, NVFP4), a text-only model that is about a 21GB download. It covers an interactive terminal chat (trtedge chat) and a local OpenAI-compatible API server (trtedge serve). Requirements are CUDA 13, TensorRT 10.16.1.11 or a later 10.x release, and Python 3.10–3.12. NVIDIA tested with TensorRT Edge-LLM 0.11.1.
The server can also run DSpark speculative decoding, in which a draft model proposes several tokens and the main model verifies them in a single pass. NVIDIA measured on DGX Spark 64GB with three draft tokens. Mean generation speed rose from about 73 tokens/s to:
- 108.5 tokens/s on SpeedBench 1K
- 105.3 tokens/s on SpeedBench 2K
- 126.9 tokens/s on HumanEval prompts
That works out to roughly 1.45–1.73x. Median time to first token was essentially unchanged, at about 98–330ms depending on workload. Peak system memory with DSpark was 27.9GB, including the OS.
All figures are NVIDIA's own. The post gives no independent benchmarks, pricing for the 64GB Spark, or list of supported models beyond Nemotron.