R_REDDYX.XYZ
AIC++★ 487⑂ 61 forksApache-2.0

localai-org/vllm.cpp

VLLM ON C++: POWERFUL SOLUTION

vllm.cpp is an open-source C++ engine, fully compatible with vLLM: continuous batching, paged KV-cache, GGUF support, RadixAttention and cache-aware scheduling. 447★, built by the community for fast LLM inference.

// KEY FEATURES

  • Continuous batching increases throughput when processing LLM requests.
  • Paged KV-cache reduces memory fragmentation and enables very long contexts.
  • GGUF format and RadixAttention support provide fast model loading and accelerated self-attention.
  • Cache-aware scheduler optimizes GPU usage, lowering latency and boosting inference efficiency.
#continous-batching#cpp20#llm#mlx#paged-attention#radixattention#vllm
Open on GitHub →

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram
← Full catalog·Full index

// SIMILAR REPOSITORIES

Development Sandbox★ 487ALPHASIFT - AI Engine for Stocks★ 487AUTOMATING SCIENTIFIC PAPERS IN LATEX★ 487Production OCR on Rust★ 487R3BL OPEN CORE TUI IN RUST★ 487Grounded Reasoning for Autonomous Driving★ 487

← FULL CATALOG

localai-org/vllm.cpp — Vllm ON C++: Powerful Solution