Configure sharded training and check that it works.
What it does
Guides you through configuring and verifying Megatron FSDP training and diagnosing configuration-related memory or performance problems.
When to use it
Use when replacing DDP with FSDP-based data parallelism or investigating an OOM or regression associated with an FSDP configuration change.
How to use it
Give it a Megatron-Bridge project and an FSDP configuration question; it returns configuration instructions, code, and verification commands.
What you provide
- An existing project
What you need · 4
The performance harness and functional verification commands require Python; no version is specified.
The functional verification command uses the torch.distributed.run module to launch two workers.
The functional verification command runs the existing Megatron FSDP test through pytest.
Running the prescribed functional smoke test requires two GPUs, selected as CUDA devices 0 and 1.
Access · 0
No additional actions listed.