Vllm: open-source projects on GitHub
69 repositories on Vllm, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
FunASR — Speech Recognition
Industrial speech recognition tool: 170× realtime, 50+ languages, diarization, emotion detection, streaming, and OpenAI‑compatible API.
★ 16,235 · PythonAI-RESEARCH-SKILLS: 9.7K STARS
A skill library for AI agents: turns Claude Code, Codex, and Gemini into full-fledged AI researchers. From Orchestra Research.
★ 14,571 · TeXXINFERENCE - ANY LLM ONE LINE
Replace GPT with any open model by changing just one line of code. Run LLM, multimodal, and speech models via a single production-ready API.
★ 9,318 · PythonLMCache: LLM Acceleration
The fastest KV cache layer for large language models, boosting performance and reducing latency.
★ 8,292 · Pythontiny-llm - BUILD MINI LLM M1
Course on deploying LLM on Apple Silicon: build your own tiny vLLM + Qwen in just minutes.
★ 5,744 · PythonUltraRAG: RAG pipeline framework
Low-code MCP framework for creating complex and innovative RAG pipelines.
★ 5,549 · PythonMOONCAKE - KIMI LLM PLATFORM, 5400+ STARS
Mooncake is the backend platform for Kimi from Moonshot AI. High-performance open-source LLM service in C++.
★ 5,429 · C++SPARROW: POWERFUL DATA EXTRACTION
Sparrow extracts structured data, executes instructions, and builds agent workflows based on ML, LLM, and Vision LLM.
★ 5,160 · PythonGPU cluster manager
Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.
★ 5,041 · PythonLLAMA-SWAP - RELIABLE MODEL REPLACEMENT
Utility for seamless model replacement on local servers (llama.cpp, vLLM, etc.)
★ 4,328 · GoIntelligent System Router
An intelligent router for managing mixed models in the cloud, data centers, and at the edge.
★ 4,182 · GoCASCADEFLOW SPEEDS UP AI AGENTS
Cascading runtime for AI agents that optimizes cost, latency, quality, and policy inside the agent loop.
★ 3,926 · PythonFASTDEPLOY — FAST LLM DEPLOYMENT
High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐
★ 3,691 · PythonRAMALAMA EASY AI LAUNCH
RamaLama is an open Python tool that lets you quickly run any AI models in containers and use them for inference in production.
★ 2,872 · PythonHyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonVLLM-ASCEND: ASCEND PLUGIN ON GITHUB
Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!
★ 2,156 · C++KVCACHED: ACCELERATE GPU CACHE
Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.
★ 2,041 · PythonVLLM-METAL: ACCELERATING LLM ON APPLE SILICON
vllm-metal is a community‑maintained plugin that brings Metal‑based acceleration to vLLM on Apple Silicon. It uses MLX to run LLMs on macOS with low latency and high thro
★ 1,997 · PythonSTARTLUX‑DECISION: AI DECISIONS
Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference
★ 1,745 · PythonANYJEV TURNS LLM INTO JEV-MODEL
AnyJev converts any LLM into a Jev‑style decision model, outputting typed decisions and real probabilities without requiring retraining. The project is actively developed
★ 1,609 · PythonKUBEAI – Integral Inference Operator
AI Inference Operator for Kubernetes: the simplest way to deploy ML models in production. Supports VLM, LLM, embeddings, and STT.
★ 1,291 · GoEAGLE-4 Speculative Decoding
Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.
★ 1,287 · Pythonvllm-mlx LLM Server
OpenAI and Anthropic compatible server for Apple Silicon that runs LLMs and vision models with continuous batching and MCP tool calling, achieving 400+ tokens/s.
★ 1,260 · PythonTINY VLLM: BUILD YOUR LLM
A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.
★ 1,194 · C++LLM Inference Optimization Roadmap
A 10‑week plan to study the service and accelerate large language models, covering vLLM, SGLang, quantization, and speculative decoding practices.
★ 1,179 · HTMLInferenceX – Inference Benchmark
Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.
★ 1,016 · PythonMULTIMODAL DECISION MODEL
Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.
★ 918 · PythonSPARK-X2.6: AGENTS ON DEVICE
Open Spark-x2.5 model series focused on agentic capabilities. Runs locally on device without cloud. 814 stars on GitHub.
★ 911LOCAL AI OCR
Offline OCR tool based on DeepSeek-OCR for recognizing text from images and PDF directly on your computer.
★ 890 · PythonHALO: FRAMEWORK FOR LLM TRAINING
Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an
★ 862 · PythonBAMBOOAI AI DATA ANALYST
BambooAI — AI data analyst that performs transparent reproducible analysis step by step in Python, verifying the evidence behind its conclusions. Supports many models, in
★ 846 · PythonLLMCORD: Discord as LLM Interface
Connect any model via OpenAI-compatible API to Discord and chat with AI right in chat.
★ 843 · PythonFREE TPU FOR QWEN3.8-27B
The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor
★ 801 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustSMG - Universal LLM Gateway
Engine-agnostic LLM gateway in Rust with full OpenAI and Anthropic API compatibility, gRPC pipeline, and KV-cache caching.
★ 723 · RustNODETOOL: YOUR AI WORKSPACE
Open workspace for creative AI: create, edit, and share models right in the browser.
★ 637 · TypeScriptRUN ANY AGENT ON ANY MODEL
llmman lets you run any AI‑agent on any model stored as an OCI image. It supports Claude Code, Codex, llama.cpp, MLX and other popular LLM frameworks.
★ 590 · RustLocal-First AI Creation Workstation
A local-first full-stack AI workstation with chat, comics, drama, writing, and learning modules, supporting vLLM, llama.cpp, and Transformers for multi-engine inference w
★ 535 · PythonLLM Serving in Kubernetes
OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.
★ 527 · GoQWEN3.6-27B ULTIMATE VERSION
Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.
★ 495 · PythonVLLM ON C++: POWERFUL SOLUTION
vllm.cpp is an open-source C++ engine, fully compatible with vLLM: continuous batching, paged KV-cache, GGUF support, RadixAttention and cache-aware scheduling. 447★, bui
★ 487 · C++WORKER-VLLM: POWERFUL LLM DEPLOYMENT
Runpod worker template for deploying LLM via vLLM — quick start and scalable operation.
★ 474 · PythonONECOMPRESSION: Lossless LLM Compression
The Python package OneCompression simplifies compression of large language models while preserving accuracy and speeding up inference.
★ 445 · PythonMATRIXHUB: AI Hub – Open Source
Self-hosted platform for AI models with Hugging Face compatibility and vLLM/SGLang acceleration. Stars: 246 ⭐
★ 367 · GoRUN LLM LOCALLY ON ANY HARDWARE
TinyLLM lets you run a local LLM and chatbot on consumer hardware. The JavaScript project uses llama-cpp-python for inference without GPU. 352 stars on GitHub.
★ 357 · JavaScriptACCELERATE LLM WITH CACHE
The unified-cache-management project enables storing and reusing KV Cache to accelerate large language models.
★ 356 · PythonXINFER - FAST LLM INFERENCE
Xinfer — ultrafast LLM inference in pure Rust, without PyTorch and Python. Enables running models with minimal latency.
★ 354 · RustVERL-OMNI: REINFORCEMENT LEARNING
Framework for RL training of diffusion and multimodal models. 347 stars on GitHub.
★ 347 · PythonAPEX AUTONOMOUS PENTEST AGENT
Apex — autonomous AI agents for offensive pentesting directly in the terminal. Fast, deep, no manual labor.
★ 326 · TypeScriptOpen Full-Duplex Voice Models
Sophon-Realtime v1 provides open full-duplex voice models that understand and express tone. Tag and Flash run on one GPU via vLLM-Omni; end-to-end version in preview.
★ 324 · PythonSMG - Universal LLM Gateway
SMG — engine‑agnostic LLM gateway on Rust with full compatibility with OpenAI and Anthropic API.
★ 322 · RustLOCAL LLM FOR HOME ASSISTANT
LLM integration in Home Assistant for local OpenAI-compatible servers (llamacpp, vllm, etc.)
★ 313 · PythonFLAMA: ONE LINE FOR AI API
Framework for fast deployment of AI models as API with built-in chat and MCP support.
★ 307 · PythonGPU BY TURN WITHOUT CRASHES
Berth leases the GPU to chat, image, and voice engines in turn. Two processes never load together and crash each other. Single Go binary, no dependencies.
★ 297 · GoBUILD YOUR AI INTRANET
Autonomous grid unites your computers into a single infrastructure for running AI models: training, fine‑tuning and inference. Supports distributed computing, fine‑tuning
★ 278 · PythonHYPERLOOM AUTO-OPTIMIZATION LLM ON AMD
Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.
★ 271 · PythonTORCHSPEC SPECULATIVE DECODING
PyTorch library for training speculative decoding models, accelerating text generation.
★ 268 · PythonLLMKUBE: LLM IN KUBERNETES
LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi
★ 266 · GoModel-Agnostic Quantization Framework
Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.
★ 254 · C++PEGAFLOW KV CACHE FOR LLM
Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL
★ 221 · RustNew projects on this topic in Telegram
Telegram →