moe-training
orchestra-research894 installs
Train larger models with fewer active experts.
What it does
Guides the implementation and training of sparse mixture-of-experts models with routing, load balancing, and expert parallelism.
When to use it
Use when training larger models with limited compute, scaling model capacity, or specializing experts for different domains, tasks, or languages.
How to use it
Give it a model-training task in a project; it provides mixture-of-experts implementation and training configuration.
What you provide
- An existing project
Latest review
What you need · 1
The model implementation imports torch and its neural-network module.
Access · 1
This skill
deepspeed
Execute
Installs DeepSpeed for MoE training