Model Serving: open-source projects on GitHub
18 repositories on Model Serving, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
High-performance LLM engine
A high-performance and memory-efficient engine for inference and serving large language models.
★ 79,783 · PythonBENTOML PATH TO AI API
Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.
★ 8,663 · PythonEfficient Multimodal Inference
Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.
★ 4,863 · PythonOLARES — YOUR OWN DATA CLOUD
Olares — an open personal cloud platform written in Go for complete control over your data.
★ 4,558 · GoLightLLM - Fast LLM Inference
A lightweight and fast framework for inference and serving of large language models in Python. Simple scaling and high performance.
★ 4,092 · PythonUniversal FedML Platform
A universal library for distributed training, model serving, and federated learning with a cross-cloud scheduler.
★ 4,046 · PythonCHITU: ACCELERATING LLMS IN PYTHON
High-performance framework for fast inference of large language models in Python. Provides flexibility, efficiency, and ease of deployment.
★ 3,123 · PythonVLLM-ASCEND: ASCEND PLUGIN ON GITHUB
Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!
★ 2,156 · C++SGLANG-OMNI: OMNI-SERVICE FOR AI
SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.
★ 1,791 · PythonRTP-LLM ENHANCED LLM INFERENCE
High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.
★ 1,400 · CudaMOSEC - MAXIMUM ML PRODUCTIVITY
Fast framework for serving ML models with dynamic batching and CPU/GPU pipelines.
★ 904 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustML Models Compiler to C99
An AOT compiler that converts XGBoost, LightGBM, scikit-learn, CatBoost, and ONNX models into native C99 code, achieving inference speedups of up to 336× compared to Pyth
★ 680 · PythonLLM Serving in Kubernetes
OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.
★ 527 · GoOpenLake: GPU Storage
Ultra-efficient storage for GPU workloads. Deliver data at lightning speed.
★ 385 · RustPREDIKIT - CONNECT MODELS TO AGENTS
Predikit connects ML models with AI agents, simplifying integration and accelerating development of intelligent systems.
★ 382 · PythonFLAMA: ONE LINE FOR AI API
Framework for fast deployment of AI models as API with built-in chat and MCP support.
★ 307 · PythonLive Multi-Camera SAM Tracking
Serves multiple cameras with one SAM model per GPU, achieving 4x memory reduction and faster inference than original implementations.
★ 221 · PythonNew projects on this topic in Telegram
Telegram →