R_REDDYX.XYZ

Vllm: open-source projects on GitHub

69 repositories on Vllm, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Inference 95Sglang 24LLM Inference 76Qwen 106Ollama 249Openai 690Mlx 47Deepseek 229Llama Cpp 56Cuda 48LLM Serving 24Anthropic 452Local LLM 107
modelscope/FunASR

FunASR — Speech Recognition

Industrial speech recognition tool: 170× realtime, 50+ languages, diarization, emotion detection, streaming, and OpenAI‑compatible API.

★ 16,235 · Python
Orchestra-Research/AI-Research-SKILLs

AI-RESEARCH-SKILLS: 9.7K STARS

A skill library for AI agents: turns Claude Code, Codex, and Gemini into full-fledged AI researchers. From Orchestra Research.

★ 14,571 · TeX
xorbitsai/inference

XINFERENCE - ANY LLM ONE LINE

Replace GPT with any open model by changing just one line of code. Run LLM, multimodal, and speech models via a single production-ready API.

★ 9,318 · Python
LMCache/LMCache

LMCache: LLM Acceleration

The fastest KV cache layer for large language models, boosting performance and reducing latency.

★ 8,292 · Python
skyzh/tiny-llm

tiny-llm - BUILD MINI LLM M1

Course on deploying LLM on Apple Silicon: build your own tiny vLLM + Qwen in just minutes.

★ 5,744 · Python
OpenBMB/UltraRAG

UltraRAG: RAG pipeline framework

Low-code MCP framework for creating complex and innovative RAG pipelines.

★ 5,549 · Python
kvcache-ai/Mooncake

MOONCAKE - KIMI LLM PLATFORM, 5400+ STARS

Mooncake is the backend platform for Kimi from Moonshot AI. High-performance open-source LLM service in C++.

★ 5,429 · C++
katanaml/sparrow

SPARROW: POWERFUL DATA EXTRACTION

Sparrow extracts structured data, executes instructions, and builds agent workflows based on ML, LLM, and Vision LLM.

★ 5,160 · Python
gpustack/gpustack

GPU cluster manager

Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.

★ 5,041 · Python
mostlygeek/llama-swap

LLAMA-SWAP - RELIABLE MODEL REPLACEMENT

Utility for seamless model replacement on local servers (llama.cpp, vLLM, etc.)

★ 4,328 · Go
vllm-project/semantic-router

Intelligent System Router

An intelligent router for managing mixed models in the cloud, data centers, and at the edge.

★ 4,182 · Go
lemony-ai/cascadeflow

CASCADEFLOW SPEEDS UP AI AGENTS

Cascading runtime for AI agents that optimizes cost, latency, quality, and policy inside the agent loop.

★ 3,926 · Python
PaddlePaddle/FastDeploy

FASTDEPLOY — FAST LLM DEPLOYMENT

High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐

★ 3,691 · Python
containers/ramalama

RAMALAMA EASY AI LAUNCH

RamaLama is an open Python tool that lets you quickly run any AI models in containers and use them for inference in production.

★ 2,872 · Python
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
vllm-project/vllm-ascend

VLLM-ASCEND: ASCEND PLUGIN ON GITHUB

Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!

★ 2,156 · C++
ovg-project/kvcached

KVCACHED: ACCELERATE GPU CACHE

Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.

★ 2,041 · Python
vllm-project/vllm-metal

VLLM-METAL: ACCELERATING LLM ON APPLE SILICON

vllm-metal is a community‑maintained plugin that brings Metal‑based acceleration to vLLM on Apple Silicon. It uses MLX to run LLMs on macOS with low latency and high thro

★ 1,997 · Python
StartLuxLabs/StartLux-Decision

STARTLUX‑DECISION: AI DECISIONS

Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference

★ 1,745 · Python
nokia-applied-research/AnyJev

ANYJEV TURNS LLM INTO JEV-MODEL

AnyJev converts any LLM into a Jev‑style decision model, outputting typed decisions and real probabilities without requiring retraining. The project is actively developed

★ 1,609 · Python
kubeai-project/kubeai

KUBEAI – Integral Inference Operator

AI Inference Operator for Kubernetes: the simplest way to deploy ML models in production. Supports VLM, LLM, embeddings, and STT.

★ 1,291 · Go
eagle-hq/EAGLE-4

EAGLE-4 Speculative Decoding

Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.

★ 1,287 · Python
waybarrios/vllm-mlx

vllm-mlx LLM Server

OpenAI and Anthropic compatible server for Apple Silicon that runs LLMs and vision models with continuous batching and MCP tool calling, achieving 400+ tokens/s.

★ 1,260 · Python
jmaczan/tiny-vllm

TINY VLLM: BUILD YOUR LLM

A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.

★ 1,194 · C++
patchy631/time-to-first-token

LLM Inference Optimization Roadmap

A 10‑week plan to study the service and accelerate large language models, covering vLLM, SGLang, quantization, and speculative decoding practices.

★ 1,179 · HTML
SemiAnalysisAI/InferenceX

InferenceX – Inference Benchmark

Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.

★ 1,016 · Python
Valen-Team/Valen

MULTIMODAL DECISION MODEL

Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.

★ 918 · Python
XHToken/Spark-X2.6

SPARK-X2.6: AGENTS ON DEVICE

Open Spark-x2.5 model series focused on agentic capabilities. Runs locally on device without cloud. 814 stars on GitHub.

★ 911
th1nhhdk/local_ai_ocr

LOCAL AI OCR

Offline OCR tool based on DeepSeek-OCR for recognizing text from images and PDF directly on your computer.

★ 890 · Python
whitecircle/halo

HALO: FRAMEWORK FOR LLM TRAINING

Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an

★ 862 · Python
pgalko/BambooAI

BAMBOOAI AI DATA ANALYST

BambooAI — AI data analyst that performs transparent reproducible analysis step by step in Python, verifying the evidence behind its conclusions. Supports many models, in

★ 846 · Python
jakobdylanc/llmcord

LLMCORD: Discord as LLM Interface

Connect any model via OpenAI-compatible API to Discord and chat with AI right in chat.

★ 843 · Python
ARahim3/kaggle-tpu-lab

FREE TPU FOR QWEN3.8-27B

The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor

★ 801 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
smg-project/smg

SMG - Universal LLM Gateway

Engine-agnostic LLM gateway in Rust with full OpenAI and Anthropic API compatibility, gRPC pipeline, and KV-cache caching.

★ 723 · Rust
nodetool-ai/nodetool

NODETOOL: YOUR AI WORKSPACE

Open workspace for creative AI: create, edit, and share models right in the browser.

★ 637 · TypeScript
llmmanorg/llmman

RUN ANY AGENT ON ANY MODEL

llmman lets you run any AI‑agent on any model stored as an OCI image. It supports Claude Code, Codex, llama.cpp, MLX and other popular LLM frameworks.

★ 590 · Rust
Yzw202011/OmniSpace

Local-First AI Creation Workstation

A local-first full-stack AI workstation with chat, comics, drama, writing, and learning modules, supporting vLLM, llama.cpp, and Transformers for multi-engine inference w

★ 535 · Python
ome-projects/ome

LLM Serving in Kubernetes

OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.

★ 527 · Go
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

QWEN3.6-27B ULTIMATE VERSION

Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.

★ 495 · Python
localai-org/vllm.cpp

VLLM ON C++: POWERFUL SOLUTION

vllm.cpp is an open-source C++ engine, fully compatible with vLLM: continuous batching, paged KV-cache, GGUF support, RadixAttention and cache-aware scheduling. 447★, bui

★ 487 · C++
runpod-workers/worker-vllm

WORKER-VLLM: POWERFUL LLM DEPLOYMENT

Runpod worker template for deploying LLM via vLLM — quick start and scalable operation.

★ 474 · Python
FujitsuResearch/OneCompression

ONECOMPRESSION: Lossless LLM Compression

The Python package OneCompression simplifies compression of large language models while preserving accuracy and speeding up inference.

★ 445 · Python
matrixhub-ai/matrixhub

MATRIXHUB: AI Hub – Open Source

Self-hosted platform for AI models with Hugging Face compatibility and vLLM/SGLang acceleration. Stars: 246 ⭐

★ 367 · Go
jasonacox/TinyLLM

RUN LLM LOCALLY ON ANY HARDWARE

TinyLLM lets you run a local LLM and chatbot on consumer hardware. The JavaScript project uses llama-cpp-python for inference without GPU. 352 stars on GitHub.

★ 357 · JavaScript
ModelEngine-Group/unified-cache-management

ACCELERATE LLM WITH CACHE

The unified-cache-management project enables storing and reusing KV Cache to accelerate large language models.

★ 356 · Python
guoqingbao/xinfer

XINFER - FAST LLM INFERENCE

Xinfer — ultrafast LLM inference in pure Rust, without PyTorch and Python. Enables running models with minimal latency.

★ 354 · Rust
verl-project/verl-omni

VERL-OMNI: REINFORCEMENT LEARNING

Framework for RL training of diffusion and multimodal models. 347 stars on GitHub.

★ 347 · Python
pensarai/apex

APEX AUTONOMOUS PENTEST AGENT

Apex — autonomous AI agents for offensive pentesting directly in the terminal. Fast, deep, no manual labor.

★ 326 · TypeScript
adventists-ai/sophon-realtime

Open Full-Duplex Voice Models

Sophon-Realtime v1 provides open full-duplex voice models that understand and express tone. Tag and Flash run on one GPU via vLLM-Omni; end-to-end version in preview.

★ 324 · Python
lightseekorg/smg

SMG - Universal LLM Gateway

SMG — engine‑agnostic LLM gateway on Rust with full compatibility with OpenAI and Anthropic API.

★ 322 · Rust
skye-harris/hass_local_openai_llm

LOCAL LLM FOR HOME ASSISTANT

LLM integration in Home Assistant for local OpenAI-compatible servers (llamacpp, vllm, etc.)

★ 313 · Python
vortico/flama

FLAMA: ONE LINE FOR AI API

Framework for fast deployment of AI models as API with built-in chat and MCP support.

★ 307 · Python
T-Crypt/berth

GPU BY TURN WITHOUT CRASHES

Berth leases the GPU to chat, image, and voice engines in turn. Two processes never load together and crash each other. Single Go binary, no dependencies.

★ 297 · Go
autonomous-ai/autonomous-grid

BUILD YOUR AI INTRANET

Autonomous grid unites your computers into a single infrastructure for running AI models: training, fine‑tuning and inference. Supports distributed computing, fine‑tuning

★ 278 · Python
AMD-AGI/Hyperloom

HYPERLOOM AUTO-OPTIMIZATION LLM ON AMD

Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.

★ 271 · Python
lightseekorg/TorchSpec

TORCHSPEC SPECULATIVE DECODING

PyTorch library for training speculative decoding models, accelerating text generation.

★ 268 · Python
defilantech/LLMKube

LLMKUBE: LLM IN KUBERNETES

LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi

★ 266 · Go
huawei-csl/MACaQ

Model-Agnostic Quantization Framework

Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.

★ 254 · C++
novitalabs/pegaflow

PEGAFLOW KV CACHE FOR LLM

Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL

★ 221 · Rust

New projects on this topic in Telegram

Telegram →
Vllm — 69 open-source GitHub projects | REDDYX AI