AIPython★ 159
huawei-csl/KVarN
KVarN: KV-Cache Quantization
KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, activated by one flag.
// KEY FEATURES
- Native KV-cache quantization for vLLM without external dependencies
- 3-5x increase in available context compared to baseline
- Throughput above FP16 while preserving FP16 accuracy
- Enabled with one flag; no calibration required
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram