AIPython★ 2,041⑂ 240 forksApache-2.0
ovg-project/kvcached
KVCACHED: ACCELERATE GPU CACHE
Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.
// KEY FEATURES
- Dynamic GPU memory allocation among tasks
- Compatible with popular frameworks PyTorch and TensorFlow
- Minimal latency for cache read/write
#elastic-kvcache#gpu-mutiplexing#gpu-sharing#inference-engine#kvcache#kvcache-optimization#kvcached#llm#llm-framework#llm-inference#llm-serving#ollama
Open on GitHub →New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram