AIPython★ 1,254⑂ 200 forksApache-2.0
NVIDIA/kvpress
KVPRESS: LLM CACHE COMPRESSION
Library for easy compression of LLM KV‑cache, accelerating generation and reducing GPU memory consumption.
// KEY FEATURES
- Automatic KV‑cache compression without loss of accuracy
- Compatible with HuggingFace, vLLM and other engines
- Minimalist API: a single function compress_kvcache
#inference#kv-cache#kv-cache-compression#large-language-models#llm#long-context#python#pytorch#transformers
Open on GitHub →New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram