NVIDIA benchmark fails a third of passing patches
NVIDIA has published SWE-Serve, a 53-task benchmark built on merged SGLang pull requests, which found that roughly one in three agent patches passing conventional checks fails once a real server runs.
NVIDIA has released SWE-Serve, a benchmark that measures whether AI coding agents can make working changes to inference-serving software, built with input from the SGLang team as launch partner.
The benchmark converts 83 merged SGLang pull requests into 53 executable tasks across six inference-engineering families: speculative and advanced decoding (14), model and backend enablement (12), kernels/quantization/performance (8), serving APIs and runtime correctness (8), caching and runtime state (7), and distributed execution and scheduling (4). Twelve tasks run on CPU; 41 use a single NVIDIA H100. Each agent gets an instruction and a containerized SGLang checkout from before the target change, and passes only if a hidden verifier succeeds. Reference solutions are substantial: the median modifies 553 lines across seven files.
The headline finding comes from 19 tasks that start a real server and test the patch through a live serving interface. Across 627 patches, the pass rate was 45.9% under the full verifier but 69.4% with live-serving tests excluded — 147 patches flipped from fail to pass. On one Gemma 4 MoE task, 16 of 33 patches cleared every other check but failed at least one live-serving test.
Tasks spanning more than one runtime domain scored 47.7% versus 69.0% for single-domain tasks. NVIDIA evaluated 11 models and 31 model-effort configurations using mini-swe-agent under closed-book conditions, three runs each; mean pass@1 ranged from 34.6% to 75.5%.
This first release does not cover other inference engines, multi-GPU execution, or multi-node serving.