NVIDIA open-sources its AI factory SRE agents
NVIDIA has released the multi-agent tool its own SREs use on DGX SuperPOD clusters as an experimental developer example, giving GPU-cluster operators a read-only, MCP-based template for tracing failures across layers.
NVIDIA detailed in a technical blog post its AI Factory Operations Agent. The company built the system to cut manual work for the site reliability engineers who run its internal DGX SuperPOD clusters, and is now offering it as a developer example. The problem it targets is familiar to operators: when training performance drops, SREs have to check separate dashboards for compute, network, hardware and scheduling to find out why.
The framework is built on NVIDIA's NemoClaw blueprint. A central runtime receives SRE questions through a chat interface and sends each one to the relevant domain agent. Those agents connect to infrastructure through OpenClaw plugins and MCP servers, and each server exposes a narrow set of approved tools. Inference runs on locally served Nemotron open models, so the whole stack can run on-premises. NVIDIA says it runs on its internal DGX H100, B200 and GB300 production clusters.
The internal deployment includes six domain agents: job recovery, hardware, observability, research, Kubernetes and cluster management. Teams can also connect their own MCP servers and APIs.
Security comes in layers. Helm deploys the stack as a Kubernetes control-plane application. NVIDIA OpenShell creates a separate sandbox for each session to run shell commands. Cluster inspection goes through a server-side kubectl tool that uses read-only credentials, and the sandbox itself never gets kubectl or cluster credentials. In NVIDIA's example, the agent can read pods and configmaps but can't read secrets or create or modify pods.
NVIDIA lists five lessons from production use: give agents clear boundaries, start read-only to build SRE trust, correlate signals across layers, define MCP tools precisely, and show the agent's reasoning. The company says it measures root-cause-analysis accuracy and how often SREs agree with the agents, but the post doesn't report either figure.