Making large AI models cheaper, faster and more accessible

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

🌸 Run large language models at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

A GPipe implementation in PyTorch

Paddle Distributed Training Examples. 飞桨分布式训练示例 Resnet Bert GPT MOE DataParallel ModelParallel PipelineParallel HybridParallel AutoParallel Zero Sharding Recompute GradientMerge Offload AMP DGC LocalSGD Wide&Deep

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.

A curated list of awesome projects and papers for distributed training or inference

An Efficient Pipelined Data Parallel Approach for Training Large Model

FTPipe and related pipeline model parallelism research.

Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks

Development of Project HPGO | Hybrid Parallelism Global Orchestration

Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)

pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism