R_REDDYX.XYZ

Model Serving: open-source projects on GitHub

18 repositories on Model Serving, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Inference 95LLM Serving 24Pytorch 98Machine Learning 190MLOps 47GPU 48Deepseek 229GPT 153Llama 106Qwen 106Deep Learning 105Vllm 69Cuda 48
vllm-project/vllm

High-performance LLM engine

A high-performance and memory-efficient engine for inference and serving large language models.

★ 79,783 · Python
bentoml/BentoML

BENTOML PATH TO AI API

Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.

★ 8,663 · Python
vllm-project/vllm-omni

Efficient Multimodal Inference

Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.

★ 4,863 · Python
beclab/Olares

OLARES — YOUR OWN DATA CLOUD

Olares — an open personal cloud platform written in Go for complete control over your data.

★ 4,558 · Go
ModelTC/LightLLM

LightLLM - Fast LLM Inference

A lightweight and fast framework for inference and serving of large language models in Python. Simple scaling and high performance.

★ 4,092 · Python
FedML-AI/FedML

Universal FedML Platform

A universal library for distributed training, model serving, and federated learning with a cross-cloud scheduler.

★ 4,046 · Python
thu-pacman/chitu

CHITU: ACCELERATING LLMS IN PYTHON

High-performance framework for fast inference of large language models in Python. Provides flexibility, efficiency, and ease of deployment.

★ 3,123 · Python
vllm-project/vllm-ascend

VLLM-ASCEND: ASCEND PLUGIN ON GITHUB

Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!

★ 2,156 · C++
sgl-project/sglang-omni

SGLANG-OMNI: OMNI-SERVICE FOR AI

SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.

★ 1,791 · Python
alibaba/rtp-llm

RTP-LLM ENHANCED LLM INFERENCE

High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.

★ 1,400 · Cuda
mosecorg/mosec

MOSEC - MAXIMUM ML PRODUCTIVITY

Fast framework for serving ML models with dynamic batching and CPU/GPU pipelines.

★ 904 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
kossisoroyce/timber

ML Models Compiler to C99

An AOT compiler that converts XGBoost, LightGBM, scikit-learn, CatBoost, and ONNX models into native C99 code, achieving inference speedups of up to 336× compared to Pyth

★ 680 · Python
ome-projects/ome

LLM Serving in Kubernetes

OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.

★ 527 · Go
openlake-project/openlake

OpenLake: GPU Storage

Ultra-efficient storage for GPU workloads. Deliver data at lightning speed.

★ 385 · Rust
Tejas-TA/predikit

PREDIKIT - CONNECT MODELS TO AGENTS

Predikit connects ML models with AI agents, simplifying integration and accelerating development of intelligent systems.

★ 382 · Python
vortico/flama

FLAMA: ONE LINE FOR AI API

Framework for fast deployment of AI models as API with built-in chat and MCP support.

★ 307 · Python
aryamandarda/tamserve

Live Multi-Camera SAM Tracking

Serves multiple cameras with one SAM model per GPU, achieving 4x memory reduction and faster inference than original implementations.

★ 221 · Python

New projects on this topic in Telegram

Telegram →
Model Serving — 18 open-source GitHub projects | REDDYX AI