R_REDDYX.XYZ
AIPython★ 38

weicj/vLLM-2080Ti-Definitive

vLLM for RTX 2080Ti

Optimized vLLM runtime for two RTX 2080 Ti 22 GB GPUs with NVLink, enabling local inference of 27B and 31B parameter models with >100 tokens/sec throughput and up to 262K token context.

// KEY FEATURES

  • Support for two RTX 2080 Ti 22 GB GPUs via NVLink to combine memory and compute power.
  • Run local 27B/31B inference model with >100 tokens/sec throughput per request.
  • Native context up to 262K tokens without segmentation, enabling long-prompt handling.
  • Optimized vLLM kernel with kernel fusion and int8/float16 support for maximum performance.
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

Llama2 Assembly★ 38Personal AI agent with memory★ 38Self-hosted LLM gateway★ 38Inference Agent Tokenmaxxing★ 38Dual-Clock Euler Sampler★ 38Persistent Agent Memory Store★ 38

← FULL CATALOG

weicj/vLLM-2080Ti-Definitive — vLLM for RTX 2080Ti