NVIDIA open-sources GPU cluster readiness controller

NVIDIA has released the NVIDIA Cluster Readiness Engine (NVCRE), an open source Kubernetes controller that certifies GPU clusters by running real distributed workloads against them and naming the specific nodes that fail.

The premise, described in a post on NVIDIA's developer blog, is that standard health checks miss failures that only appear under load: a single slow GPU, a link that degrades in production, or a misconfiguration that routes traffic over a slower path can leave a 512-GPU training job underperforming for hours. On Slurm, running a real test workload takes one srun command; Kubernetes has no equivalent, requiring hand-written GPU and RDMA resource requests, fabric-matched NCCL settings, shared-memory volumes, and gang scheduling.

NVCRE exposes three custom resource definitions under nvcre.nvidia.com/v1alpha1 — Certification, Workflow, and Job — so runs are driven with kubectl and managed through GitOps. The built-in catalog covers three domains: five NCCL communication variants (all-reduce, all-gather, all-to-all, loopback, and loopback across NVSwitch), the DCGM level-4 diagnostic suite, and NeMo pretraining with Nemotron 5 at 8B and 56B parameters. The controller detects GPU architecture and cloud platform to derive GPUs per node, NCCL environment, and networking defaults.

Pass/fail criteria are written in Common Expression Language and evaluated against measured metrics; no thresholds ship by default. A job that completes but misses its target gets a separate ValidationFailed condition. A testScale field selects grouping — intra-node, intra-rack by nvidia.com/gpu.clique topology label, full-scale, or an adaptive diagnose mode.

The post does not give a repository link, version, or adoption figures.