NVIDIA open-sources automated training recovery service
NVIDIA has released NVIDIA Recovery, an open-source service that detects failed distributed training jobs and restarts them from checkpoints across a cluster, aiming to recover lost GPU time without manual work.
NVIDIA described NVIDIA Recovery, a workload-level service that automates recovery for training jobs across a whole cluster. It is part of NVIDIA DSX OS, the software portfolio for the company's DSX AI Factory Platform. The release is under the Apache 2.0 license, and IBM Research and Verda are named as early adopters.
The service detects interruptions using application logs, GPU metrics and scheduler signals. These include NCCL timeouts, unresponsive nodes and stragglers. It attributes each failure to the resources involved, excludes those nodes, and restarts affected jobs from checkpoints on healthy capacity. The post focuses on how it integrates with and deploys on Slurm.
NVIDIA presents it as a cluster-level layer that sits between existing tools. Break-fix systems such as NVSentinel handle node health. Libraries such as NVRx handle in-job recovery and hang detection. Slurm handles scheduling. The components have no hard dependencies on each other, so sites with existing health workflows can adopt Recovery separately.
NVIDIA reported production figures from one training cluster between July 9 and Oct. 9. Over that period, Recovery monitored nearly 200 jobs per day and recorded about 8,550 anomaly detections. Mean detection latency was 69 seconds, and the mean time to trigger a recovery action was 94 seconds. NVIDIA notes that the 94-second figure ends when the automated action fires. Scheduling and checkpoint restoration add more time before training actually resumes.
To show why scale matters, the post cites a Meta study of 24,000 A100 GPUs. In that study, hardware failures hit fewer than 1% of jobs, but those jobs accounted for 19% of GPU runtime.