LLM Serving: open-source projects on GitHub
24 repositories on LLM Serving, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
High-performance LLM engine
A high-performance and memory-efficient engine for inference and serving large language models.
★ 79,783 · PythonRay: AI Compute Engine
Ray is a distributed engine for AI computing, combining a runtime core and a set of libraries to accelerate ML workloads.
★ 42,516 · PythonOPENLLM: RUN LLMS AS OPENAI API
OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and read
★ 12,607 · PythonBENTOML PATH TO AI API
Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.
★ 8,663 · PythonAI Infrastructure Book
Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.
★ 7,748 · PythonGPU cluster manager
Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.
★ 5,041 · PythonFASTDEPLOY — FAST LLM DEPLOYMENT
High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐
★ 3,691 · PythonCHITU: ACCELERATING LLMS IN PYTHON
High-performance framework for fast inference of large language models in Python. Provides flexibility, efficiency, and ease of deployment.
★ 3,123 · PythonVLLM-ASCEND: ASCEND PLUGIN ON GITHUB
Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!
★ 2,156 · C++KVCACHED: ACCELERATE GPU CACHE
Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.
★ 2,041 · PythonLLM Inference on AWS Trainium
Open-source LLM inference engine for AWS Trainium featuring NKI megakernels, 1M-token context, and OpenAI-compatible API.
★ 1,585 · PythonRTP-LLM ENHANCED LLM INFERENCE
High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.
★ 1,400 · CudaFAST LAUNCH QWEN3.8 ON AMD
The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference
★ 1,349 · ShellPARALLAX: BUILD AI CLUSTER
Parallax lets you quickly deploy a distributed inference service on any infrastructure.
★ 1,301 · PythonMOSEC - MAXIMUM ML PRODUCTIVITY
Fast framework for serving ML models with dynamic batching and CPU/GPU pipelines.
★ 904 · PythonHELIX: Private AI Agent Fleet
Flagship fleet of AI agents, each gets its own GPU-accelerated desktop. Claude, Codex, Gemini on a private stack.
★ 831 · GoFREE TPU FOR QWEN3.8-27B
The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor
★ 801 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustFLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · PythonEXPERIMENTAL INFERENCE LIBRARY
resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25
★ 432 · PythonKV-CACHE OPTIMIZATION FOR LLM
Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.
★ 426 · PythonFLAMA: ONE LINE FOR AI API
Framework for fast deployment of AI models as API with built-in chat and MCP support.
★ 307 · PythonLLM Prefix Cache Simulator
Simulates prefix caching mechanisms for LLM inference to analyze cache hit rates and memory efficiency.
★ 76 · C++SparkDuet: on-demand model library
The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and
★ 0 · PythonNew projects on this topic in Telegram
Telegram →