Scale training across nodes and diagnose failed launches.
What it does
Convert single-node launches into multi-node Slurm job scripts and diagnose distributed training failures.
When to use it
Use when writing or converting Slurm scripts, scaling to multiple nodes, or investigating NCCL, launch, and multi-node training failures.
How to use it
Give it a project's single-node launch or multi-node failure; it returns shell commands and configuration guidance for conversion or debugging.
What you provide
- An existing project
Uses
Good to know
- cannot be undone: nemo_experiments
What you need · 8
Running the generated jobs requires a Slurm cluster with an account, a batch partition, and container support.
The container needs uv for dependency synchronization and Python launches; the files identify a container without uv as a failure.
Access · 5
This skill
nemo_experiments
Deletes existing training experiments (irreversible)
Weights & Biases
Logs in to Weights & Biases