NVIDIA runs MLPerf edge agentic 6.4x faster
NVIDIA says its TensorRT Edge-LLM runtime completed MLPerf Inference v6.1's Edge Agentic workload on a single Jetson AGX Thor in 24 minutes, 6.4x faster than the benchmark's llama.cpp reference run.
In MLPerf Inference v6.1's Edge Agentic benchmark, NVIDIA's TensorRT Edge-LLM ran Qwen3.6-27B on a single Jetson AGX Thor Developer Kit — 128 GB unified memory, MAXN power mode, SingleStream — and finished all 1,007 turns of the performance workload in 24 minutes 36 seconds. NVIDIA reports 52.33 output tokens per second, a 247.12 ms median time to first token, 14.68 ms median time per output token, and 87.94% overall accuracy on the benchmark's BFCL v4 accuracy phase (single-turn, reasoning off).
The 6.4x figure is measured against MLCommons' own llama.cpp reference run on the same Jetson hardware, which uses Q4_K_M quantization and takes 2 hours 37 minutes. It is a comparison against an unoptimized baseline, not a competing optimized submission.
The gains come from three stacked optimizations. NVFP4 quantization covers weights and activations including the LM head, with FP8 KV cache, cutting the DRAM bandwidth bottleneck that dominates low-batch edge decoding. Prefix cache reuse — including recurrent and partial-KV-page state, since Qwen3.6 is a hybrid architecture — serves roughly 96% of prompt tokens from hot cache, prefilling only ~0.5M of 13.6M prompt tokens across the run. Tree-based multi-token prediction (8 draft steps, top-2 candidates per depth, 16-node verification tree) adds an estimated ~40% decoding gain over linear MTP with 3 draft steps.
The workload replays 20 recorded software-engineering agent trajectories, with input length growing to roughly 23.5K tokens. The implementation ships in the TensorRT Edge-LLM 0.9.1 release branch.