pytorch-fsdp2
orchestra-research898 installs
Train models too large for one GPU with PyTorch.
What it does
Adds distributed sharding and checkpointing to PyTorch training scripts so large models can train across GPUs.
When to use it
Use when a model exceeds one GPU's memory or needs DTensor-based sharding, including composition with tensor parallelism through DeviceMesh.
How to use it
Give it a training script to adapt; it adds FSDP2 initialization, sharding, optimizer setup, and distributed checkpointing.
What you provide
- An existing project
Latest review
What you need · 2
Requires the torch package.
The training setup requires distinct CUDA GPUs for the distributed ranks.
Access · 0
No additional actions listed.