Kv Cache: open-source projects on GitHub
16 repositories on Kv Cache, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
LMCache: LLM Acceleration
The fastest KV cache layer for large language models, boosting performance and reducing latency.
★ 8,292 · PythonAI Infrastructure Book
Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.
★ 7,748 · PythonHyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonAWESOME MODEL QUANTIZATION RESOURCES
A curated list of articles, benchmarks, surveys, and tools for neural network quantization. Covers low-bit networks, LLMs, multimodal and generative models, vector and la
★ 2,469ZERO-TO-SGLANG: LLM INFERENCE TRAINING
The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.
★ 2,136 · PythonKVCache-Factory: Cache Compression for LLM
Unified KV-cache compression methods for autoregressive models. Memory savings and faster inference in one package.
★ 1,394 · PythonKVPRESS: LLM CACHE COMPRESSION
Library for easy compression of LLM KV‑cache, accelerating generation and reducing GPU memory consumption.
★ 1,254 · PythonCACHE-TO-CACHE: DIRECT LLM EXCHANGE
Implementation of the Cache-to-Cache algorithm for direct semantic exchange between large language models. Enables transferring KV‑cache without deserialization speeding
★ 778 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustMIXING RECURSIONS FOR LLM
The Mixture-of-Recursions project offers dynamic recursion depth selection per token in a transformer, enabling adaptive compute allocation. This reduces KV‑cache costs a
★ 651 · PythonFLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · PythonKV-CACHE OPTIMIZATION FOR LLM
Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.
★ 426 · PythonTAIR-KVCACHE: ACCELERATING LLM INFERENCE
A high-performance KV caching system from Alibaba Cloud that accelerates LLM inference.
★ 283 · C++PEGAFLOW KV CACHE FOR LLM
Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL
★ 221 · RustGLYD: LOSSLESS AI COMPRESSION
Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma
★ 200 · RustKVarN: KV-Cache Quantization
KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act
★ 159 · PythonNew projects on this topic in Telegram
Telegram →