NVIDIA cuts the cost of bitwise-deterministic pretraining
NVIDIA says Megatron Core can now run bitwise-deterministic pretraining at trillion-parameter scale with little slowdown, so expensive runs can be replayed, debugged and resumed exactly instead of only approximately.
NVIDIA has published a technical post on bitwise-deterministic pretraining in Megatron Core, using a trillion-parameter Nemotron model as its case study. NVIDIA says overhead was about 2% on 2,432 GPUs. That figure matters because, as the post notes, even a small slowdown at this scale can cost thousands of GPU-days.
Megatron Core aims for two guarantees. First, two runs launched from the same initial state stay bitwise identical at every step. Second, a run that saves and restores checkpoints stays identical to one that was never interrupted. Both hold only when data order, architecture, recipe, parallelism, software, runtime settings and hardware are all fixed.
NVIDIA lists four uses: replaying loss spikes to isolate their cause, resuming training without changing the numerical trajectory, spotting checkpoint corruption or hardware faults through unexpected mismatches, and separating the effect of an intended change from normal run-to-run variation.
The post warns that printed loss values are too coarse to confirm determinism. Two runs can print the same rounded loss while their gradients differ in the lower-order bits. Instead, it recommends fingerprinting metrics such as loss, load-balancing loss, multi-token prediction loss and gradient norm at every step.
For debugging, a workflow proposed in Megatron-LM PR #7262 records per-rank tensor fingerprints using torch.hash_tensor. Engineers narrow the search from the whole iteration down to the training phase, module, operation and finally kernel. The goal is to find the first point where inputs match but outputs differ. For MXFP8 and NVFP4 tensors, both the encoded values and the scale buffers are hashed.
The available excerpt ends before NVIDIA explains how the overhead was reduced.