NVIDIA details agent tracing with NeMo Relay
A new NVIDIA tutorial uses NeMo Relay to trace Hermes Agent runs, showing developers how to find retries, errors and wasted tokens that a passing success check does not reveal.
NVIDIA has published a technical tutorial on NeMo Relay, its observability layer for recording model and tool execution in AI agents. The tutorial uses Hermes Agent, which integrates NeMo Relay natively. Its main argument is that a correct final answer can hide an inefficient path. A failed search might trigger another search, or a truncated file read might lead to a second fetch of the same content. Both add latency and token cost.
Hermes maps its sessions, turns, model calls and tool calls onto NeMo Relay's scope hierarchy. Relay records start and end events with their timing and parent-child links. It exports three formats:
- ATOF (Agent Trajectory Observability Format): a JSONL event log for debugging and auditing individual events.
- ATIF (Agent Trajectory Interchange Format): a step-by-step JSON record of the agent's actions, used for review and evaluation.
- OpenTelemetry with OpenInference: spans that tools such as Arize Phoenix can read.
NVIDIA notes that an ATIF tool request shows only what the model asked to run. To confirm what actually happened, developers need to check the matching ATOF start and end events.
The tutorial's first example runs a Python script in a Docker sandbox that has no network access. The second is a file-and-web research task viewed in Phoenix. The examples use NVIDIA Nemotron 3.5 Lightning, Hermes 0.21.1 and NeMo Relay 0.8.3. The code is in the NVIDIA/nemoclaw-community repository.
NVIDIA warns that traces can contain prompts, tool arguments and file paths, so they should be reviewed before sharing. A "Hermes ToolPerf" case study applies the same method to test changes to the agent harness over repeated runs. Its results were not in the available excerpt.