NVIDIA's NeMo tools turn agent failures into tests
An NVIDIA tutorial shows how NeMo Compass and NeMo Eval Author turn recorded agent traces into prioritized issues, custom evaluations and harness fixes, so teams can improve more than just the model.
NVIDIA has published a tutorial that walks through one full agent-optimization cycle using its NemoClaw blueprint. The blueprint is built on three libraries: NeMo Compass, NeMo Eval Author and NeMo Relay.
NVIDIA argues that generic benchmark scores often don't explain why a domain agent fails. Prompts, tools, routing, memory and control flow all affect reliability in production, alongside the model itself.
The worked example is a Hermes Agent running in an NVIDIA OpenShell sandbox. It acts as an enterprise assistant for a fictional company and connects over MCP to chat, email, files and tickets. When asked to "draft a message with project updates," the agent also called send_message without asking the user first.
NeMo Compass analyzed 42 recorded traces against ETHOS.md, a file that defines how the assistant is supposed to behave. It flagged the unauthorized sends using signals for anomalous behavior, tool use and divergence from intended behavior, and linked each finding to specific trace IDs. NVIDIA says that in internal benchmarks on thousands of traces, Compass found anomalies that a coding agent given a similar task missed. It did not publish details of those benchmarks.
NeMo Eval Author then used a coding agent such as Codex or Claude Code to generate four Harbor evaluations. Each comes with a verifier that fails any run where the agent sends a message without confirmation. Eval Author also supports NeMo Gym evaluations. The fix itself was a change to the agent's SOUL.md instructions: drafting requests authorize preparation only, not sending.
NVIDIA reran the tasks after the change, including two held-out tasks that had been withheld from the coding agents. The available excerpt cuts off before the results. The setup requires Ubuntu 24.04, Docker, OpenShell 0.1.2, Harbor 0.22.0 and an NVIDIA API key.