AIC++★ 487⑂ 61 forksApache-2.0
localai-org/vllm.cpp
VLLM ON C++: POWERFUL SOLUTION
vllm.cpp is an open-source C++ engine, fully compatible with vLLM: continuous batching, paged KV-cache, GGUF support, RadixAttention and cache-aware scheduling. 447★, built by the community for fast LLM inference.
// KEY FEATURES
- Continuous batching increases throughput when processing LLM requests.
- Paged KV-cache reduces memory fragmentation and enables very long contexts.
- GGUF format and RadixAttention support provide fast model loading and accelerated self-attention.
- Cache-aware scheduler optimizes GPU usage, lowering latency and boosting inference efficiency.
New repositories every 30 minutes
REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.
Join on Telegram