Configure training to detect failures and recover.
What it does
Guides developers through configuring fault recovery, slow-rank detection, graceful shutdown, and checkpoint recovery for distributed training.
When to use it
Use when enabling resiliency features or investigating training hangs, straggler detection failures, or broken restarts.
How to use it
Ask how to configure training resiliency; it returns configuration instructions, shell commands, and code.
What you need · 5
Requires approximately version 0.5.0 for fault tolerance, straggler detection, in-process restart, and local checkpointing.
The recommended fault-tolerance workflow requires Slurm.
The recommended fault-tolerance plugin workflow imports nemo_run and launches the training task through it.
The experimental in-process restart feature requires PyTorch 2.5.1 or newer.
Access · 0
No additional actions listed.