R_REDDYX.XYZ

LLM Serving: open-source projects on GitHub

24 repositories on LLM Serving, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094LLM Inference 76Inference 95Qwen 106Deepseek 229Model Serving 18Vllm 69Pytorch 98Cuda 48Llama 106Machine Learning 190MLOps 47GPU 48Openai API 86
vllm-project/vllm

High-performance LLM engine

A high-performance and memory-efficient engine for inference and serving large language models.

★ 79,783 · Python
ray-project/ray

Ray: AI Compute Engine

Ray is a distributed engine for AI computing, combining a runtime core and a set of libraries to accelerate ML workloads.

★ 42,516 · Python
bentoml/OpenLLM

OPENLLM: RUN LLMS AS OPENAI API

OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and read

★ 12,607 · Python
bentoml/BentoML

BENTOML PATH TO AI API

Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.

★ 8,663 · Python
bojieli/ai-infra-book

AI Infrastructure Book

Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.

★ 7,748 · Python
gpustack/gpustack

GPU cluster manager

Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.

★ 5,041 · Python
PaddlePaddle/FastDeploy

FASTDEPLOY — FAST LLM DEPLOYMENT

High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐

★ 3,691 · Python
thu-pacman/chitu

CHITU: ACCELERATING LLMS IN PYTHON

High-performance framework for fast inference of large language models in Python. Provides flexibility, efficiency, and ease of deployment.

★ 3,123 · Python
vllm-project/vllm-ascend

VLLM-ASCEND: ASCEND PLUGIN ON GITHUB

Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!

★ 2,156 · C++
ovg-project/kvcached

KVCACHED: ACCELERATE GPU CACHE

Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.

★ 2,041 · Python
RunanywhereAI/trnrt

LLM Inference on AWS Trainium

Open-source LLM inference engine for AWS Trainium featuring NKI megakernels, 1M-token context, and OpenAI-compatible API.

★ 1,585 · Python
alibaba/rtp-llm

RTP-LLM ENHANCED LLM INFERENCE

High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.

★ 1,400 · Cuda
peonist-ai/halogen-flash-server

FAST LAUNCH QWEN3.8 ON AMD

The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference

★ 1,349 · Shell
GradientHQ/parallax

PARALLAX: BUILD AI CLUSTER

Parallax lets you quickly deploy a distributed inference service on any infrastructure.

★ 1,301 · Python
mosecorg/mosec

MOSEC - MAXIMUM ML PRODUCTIVITY

Fast framework for serving ML models with dynamic batching and CPU/GPU pipelines.

★ 904 · Python
helixml/helix

HELIX: Private AI Agent Fleet

Flagship fleet of AI agents, each gets its own GPU-accelerated desktop. Claude, Codex, Gemini on a private stack.

★ 831 · Go
ARahim3/kaggle-tpu-lab

FREE TPU FOR QWEN3.8-27B

The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor

★ 801 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
ekzhang/resin

EXPERIMENTAL INFERENCE LIBRARY

resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25

★ 432 · Python
jjiantong/Awesome-KV-Cache-Optimization

KV-CACHE OPTIMIZATION FOR LLM

Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.

★ 426 · Python
vortico/flama

FLAMA: ONE LINE FOR AI API

Framework for fast deployment of AI models as API with built-in chat and MCP support.

★ 307 · Python
cacheMon/PrefixSim

LLM Prefix Cache Simulator

Simulates prefix caching mechanisms for LLM inference to analyze cache hit rates and memory efficiency.

★ 76 · C++
zorost/sparkduet

SparkDuet: on-demand model library

The project provides a single OpenAI‑compatible endpoint for running DeepSeek and Qwen models. It serves them via two vLLM lanes (TP=2) on an NVIDIA DGX Spark cluster and

★ 0 · Python

New projects on this topic in Telegram

Telegram →
LLM Serving — 24 open-source GitHub projects | REDDYX AI