deepspeedai/DeepSpeed
DeepSpeed: optimization library for training
DeepSpeed is an optimization deep learning library ensuring efficient distributed training and inference. Supports ZeRO, MoE and parallelism for training models with hundreds of billions of parameters.
// KEY FEATURES
- Distributed training of models up to 530B parameters
- Support for ZeRO, MoE and 3D-parallelism
- Integration with Transformers, Accelerate and Lightning
- Installation via pip, independent of CUDA/PyTorch versions
// DEEP DIVE: HOW IT WORKS
DeepSpeed is an optimization deep learning library simplifying distributed training and model inference. It includes ZeRO, ZeRO-Infinity, 3D-parallelism, Ulysses Sequence Parallelism and DeepSpeed-MoE for scaling to hundreds of billions of parameters. Integrates with Transformers, Accelerate, Lightning, MosaicML, Determined and MMEngine frameworks.
The repository supports building on hardware NVIDIA, AMD, CPU, Intel Gaudi, Intel XPU and Huawei Ascend NPU, making it universal for different servers. pip-based installation is not tied to specific PyTorch or CUDA versions, simplifying the start of projects of any scale.
Recommended for training and inference of large models requiring efficient memory and computational resources. Project office hours take place on the last Tuesday of the month via Zoom, and the latest news cover optimizer updates, offloading and new parallelism methods.
// USE CASES
Best for: To developers and researchers working with large language models
- Train models up to 530B parameters on multiple GPUs
- Integrate LLM into projects based on Transformers
- Scale training with ZeRO-Infinity and MoE
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram