NVIDIA publishes guide to agent evaluation
NVIDIA has published a developer-blog explainer on evaluating AI agents, arguing that tool-call accuracy alone is insufficient and that teams should gate releases on end-to-end task completion.
The post argues that agent evaluation has outgrown single-function-call scoring. It traces the shift from static harnesses to the Berkeley Function-Calling Leaderboard, which grades function selection and argument accuracy but scores calls in isolation — a valid issue_refund call still fails if the underlying checks were skipped. Full agentic evaluation, NVIDIA writes, requires an execution environment that runs each call, tracks state, and inspects the world afterward.
Two scoring layers sit on that environment: step-level process scoring, which asks whether each call was valid and useful given the state at that point, and end-to-end outcome scoring, which ignores the path and checks only the final state. Both are readings of one object, the trace. The post recommends gating releases on E2E while keeping step-level tracing for debugging.
It also proposes a fixed rollup hierarchy — Benchmark → Trial → Task → Turn → Step — and six metrics across accuracy, verbosity, and cost: task success rate, consistency (reported as a range across 3–5 trials, not a point estimate), tool-call precision, argument accuracy, steps per success, and cost per success. Benchmarks differ, it notes, on task complexity, statefulness, and methodology, with executable verification treated as the gold standard and LLM-as-a-Judge scores provisional until validated against human ratings.
The piece is developer education rather than a product or research announcement. It includes a worked SWE-bench trace, and carries an unaudited vendor performance claim for NVIDIA's own Nemotron model. No new benchmark, dataset, or tooling is released.