R_REDDYX.XYZ

Kv Cache: open-source projects on GitHub

16 repositories on Kv Cache, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094LLM Inference 76Cuda 48Vllm 69Inference 95LLM Serving 24Pytorch 98GPU 48Qwen 106Large Language Models 154Inference Engine 18Deepseek 229Quantization 34Qwen3 30
LMCache/LMCache

LMCache: LLM Acceleration

The fastest KV cache layer for large language models, boosting performance and reducing latency.

★ 8,292 · Python
bojieli/ai-infra-book

AI Infrastructure Book

Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.

★ 7,748 · Python
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
AI-Efficiency/Awesome-Model-Quantization

AWESOME MODEL QUANTIZATION RESOURCES

A curated list of articles, benchmarks, surveys, and tools for neural network quantization. Covers low-bit networks, LLMs, multimodal and generative models, vector and la

★ 2,469
datawhalechina/zero-to-sglang

ZERO-TO-SGLANG: LLM INFERENCE TRAINING

The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.

★ 2,136 · Python
Zefan-Cai/KVCache-Factory

KVCache-Factory: Cache Compression for LLM

Unified KV-cache compression methods for autoregressive models. Memory savings and faster inference in one package.

★ 1,394 · Python
NVIDIA/kvpress

KVPRESS: LLM CACHE COMPRESSION

Library for easy compression of LLM KV‑cache, accelerating generation and reducing GPU memory consumption.

★ 1,254 · Python
thu-nics/C2C

CACHE-TO-CACHE: DIRECT LLM EXCHANGE

Implementation of the Cache-to-Cache algorithm for direct semantic exchange between large language models. Enables transferring KV‑cache without deserialization speeding

★ 778 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
raymin0223/mixture_of_recursions

MIXING RECURSIONS FOR LLM

The Mixture-of-Recursions project offers dynamic recursion depth selection per token in a transformer, enabling adaptive compute allocation. This reduces KV‑cache costs a

★ 651 · Python
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
jjiantong/Awesome-KV-Cache-Optimization

KV-CACHE OPTIMIZATION FOR LLM

Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.

★ 426 · Python
alibaba/tair-kvcache

TAIR-KVCACHE: ACCELERATING LLM INFERENCE

A high-performance KV caching system from Alibaba Cloud that accelerates LLM inference.

★ 283 · C++
novitalabs/pegaflow

PEGAFLOW KV CACHE FOR LLM

Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL

★ 221 · Rust
surya-koritala/Glyd

GLYD: LOSSLESS AI COMPRESSION

Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma

★ 200 · Rust
huawei-csl/KVarN

KVarN: KV-Cache Quantization

KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act

★ 159 · Python

New projects on this topic in Telegram

Telegram →
Kv Cache — 16 open-source GitHub projects | REDDYX AI