AIPython★ 38
weicj/vLLM-2080Ti-Definitive
vLLM for RTX 2080Ti
Optimized vLLM runtime for two RTX 2080 Ti 22 GB GPUs with NVLink, enabling local inference of 27B and 31B parameter models with >100 tokens/sec throughput and up to 262K token context.
// KEY FEATURES
- Support for two RTX 2080 Ti 22 GB GPUs via NVLink to combine memory and compute power.
- Run local 27B/31B inference model with >100 tokens/sec throughput per request.
- Native context up to 262K tokens without segmentation, enabling long-prompt handling.
- Optimized vLLM kernel with kernel fusion and int8/float16 support for maximum performance.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram