AIPython★ 472⑂ 66 forksApache-2.0
leyten/shard
Shard: Distributed Inference
Shard implements pipeline parallelism for running large language models across multiple GPU nodes, allowing inference without needing to place the entire model in the memory of a single device.
// KEY FEATURES
- Pipeline parallelism of model layers across GPUs on different machines.
- Automatic tensor splitting and assembly, minimizing data-transfer delays.
- Support for Hugging Face Transformers and dynamic batching to increase throughput.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram