Inference: open-source projects on GitHub
95 repositories on Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
High-performance LLM engine
A high-performance and memory-efficient engine for inference and serving large language models.
★ 79,783 · PythonYOLOV5: OBJECT DETECTION ON PYTORCH
Computer vision model for object detection, instance segmentation, and image classification. Runs on PyTorch, supports export to CoreML and ONNX.
★ 58,278 · Pythonwhisper.cpp: C/C++ port of Whisper
High-performance port of the OpenAI Whisper speech recognition model in C/C++ for local and cross-platform use.
★ 49,828 · C++DeepSpeed: optimization library for training
DeepSpeed is an optimization deep learning library ensuring efficient distributed training and inference. Supports ZeRO, MoE and parallelism for training models with hund
★ 43,332 · PythonFRAMEWORK FOR CHEAPER LARGE MODEL TRAINING
Library for distributed training and inference of large language models. Supports Zero, tensor, and pipeline parallelism strategies to reduce GPU costs.
★ 41,422 · PythonCROSS-PLATFORM ON-DEVICE ML FRAMEWORK
MediaPipe is Google's framework for running ML on-device. Supports mobile platforms, web, desktop, and IoT without sending data to the cloud.
★ 37,717 · C++SGLang: LLM Framework
SGLang is a high-performance framework for serving large language and multimodal models.
★ 27,854 · PythonFAST SPEECH TRANSCRIPTION
Faster Whisper — an accelerated version of the Whisper model for speech recognition, built on CTranslate2. It supports quantized models and provides real‑time transcripti
★ 26,758 · PythonHIGH-PERFORMANCE NEURAL NETWORK INFERENCE FRAMEWORK FOR MOBILE
ncnn is a neural network inference framework with no third-party dependencies for mobile, embedded, and desktop platforms. Supports CPU and Vulkan GPU backends, includes
★ 24,068 · C++ML-ENGINEERING OPEN BOOK
Open book on ML engineering with practical materials and examples in Python.
★ 19,717 · PythonFreeToken: Local Model Run
FreeToken lets you run large AI models on your PC, delivering data‑center‑class speed and efficiency without needing cloud resources.
★ 17,197 · PythonOPTIMIZATION AND DEPLOYMENT OF AI MODELS
Toolkit for optimizing and deploying deep learning models. Supports conversion from PyTorch, TensorFlow, ONNX and inference on CPU, GPU, and Intel NPU.
★ 11,656 · C++Local AI SDK Set
Production-ready toolkit for running AI models on a local device.
★ 10,350 · C++OUMI EASY LLM TUNING
OUMI lets you quickly fine‑tune (SFT/RL), evaluate, and deploy Qwen, Gemma, and any open agentic LLM/VLM without extra complexity.
★ 9,433 · PythonXINFERENCE - ANY LLM ONE LINE
Replace GPT with any open model by changing just one line of code. Run LLM, multimodal, and speech models via a single production-ready API.
★ 9,318 · PythonLMCache: LLM Acceleration
The fastest KV cache layer for large language models, boosting performance and reducing latency.
★ 8,292 · PythonMLX-Powered Laya Decisions
A native MLX runtime for Laya typed decision models that runs short decisions in 7–14 ms on an M3 Max. It avoids text generation, PyTorch, and cloud APIs.
★ 7,102 · PythonMOONCAKE - KIMI LLM PLATFORM, 5400+ STARS
Mooncake is the backend platform for Kimi from Moonshot AI. High-performance open-source LLM service in C++.
★ 5,429 · C++GPU cluster manager
Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.
★ 5,041 · PythonEfficient Multimodal Inference
Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.
★ 4,863 · PythonUNIFIED FRAMEWORK FOR ACCELERATED VIDEO GENERATION
FastVideo is a unified framework for post-training and inference of accelerated video generation. It supports distillation, sparse attention, and real-time via Dreamverse
★ 4,831 · PythonONE DNN: ACCELERATING AI ON CPU
oneDNN provides high-performance AI kernel implementations for x64, AArch64, RISC‑V, and Intel GPUs. It accelerates PyTorch, TensorFlow, OpenVINO, and ONNX Runtime using
★ 4,092 · C++CSGHUB — PLATFORM FOR LLM
CSGHub — a new open‑source platform from OpenCSG for managing LLMs, datasets and agents. Supports on‑premise and SaaS deployments, compatible with the Python SDK of Huggi
★ 4,082 · VueFASTDEPLOY — FAST LLM DEPLOYMENT
High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐
★ 3,691 · PythonLLM inference acceleration on Kubernetes
Achieve state of the art inference performance with modern accelerators on Kubernetes
★ 3,247 · ShellCreate Videos on Your PC
FreeVideo lets you generate videos using the MiniMax H3 model on modest hardware, requiring as little as 8 GB VRAM and 16 GB RAM, and automatically selects the best accel
★ 3,094 · PythonNATIVE LLM SERVER FOR MAC
LLM server compatible with OpenAI+Anthropic, no Python required. Includes MLX Core macOS: chat, agent, and tool calling.
★ 2,472 · ZigUNIFIED GATEWAY FOR AI SERVICES
agent-router is an Envoy Gateway‑based gateway that provides unified access to various generative AI services. Implemented in Go, it runs in Kubernetes and adheres to CNC
★ 2,404 · GoFast Local AI Engine
Rapid-MLX is the fastest local AI engine for Apple Silicon, running 4.2× faster than Ollama and providing a cached first-token time of just 0.08s.
★ 2,185 · PythonVLLM-ASCEND: ASCEND PLUGIN ON GITHUB
Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!
★ 2,156 · C++SUPERLINKED INFERENCE ENGINE
Open-source server for embedding inference, reranking, and data extraction. 2000+ stars on GitHub.
★ 2,029 · PythonANY-LLM: One Interface for All LLMs
Python library for unified interaction with LLM providers through a single interface.
★ 2,010 · PythonSGLANG-OMNI: OMNI-SERVICE FOR AI
SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.
★ 1,791 · PythonSTARTLUX‑DECISION: AI DECISIONS
Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference
★ 1,745 · PythonAI-GATEWAY: UNIFIED GATEWAY FOR AI
The ai-gateway project provides unified access to various generative AI services via Envoy Gateway, written in Go.
★ 1,717 · GoTinyML LLM for cheap boards
Running a language model with 1B parameters on a $10 device with 256 MB RAM in C.
★ 1,614 · CUZU: Powerful Engine for AI Models
High-performance inference engine for AI models in Rust. Fast, reliable, efficient.
★ 1,586 · RustLLM Inference on AWS Trainium
Open-source LLM inference engine for AWS Trainium featuring NKI megakernels, 1M-token context, and OpenAI-compatible API.
★ 1,585 · PythonBUILD YOUR AI COMPUTER
The project helps you build your personal AI computer with open-source code and support for autonomous agents.
★ 1,577RTP-LLM ENHANCED LLM INFERENCE
High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.
★ 1,400 · CudaEMBEDANYTHING - POWERFUL INFERENCE ON RUST
EmbedAnything is a high‑performance, modular, and memory‑safe framework for inference, ingestion, and indexing on Rust.
★ 1,324 · Rustvllm-mlx LLM Server
OpenAI and Anthropic compatible server for Apple Silicon that runs LLMs and vision models with continuous batching and MCP tool calling, achieving 400+ tokens/s.
★ 1,260 · PythonKVPRESS: LLM CACHE COMPRESSION
Library for easy compression of LLM KV‑cache, accelerating generation and reducing GPU memory consumption.
★ 1,254 · PythonLLMGATEWAY: UNIFIED API FOR ALL LLM
Manage requests to different LLM providers through a single unified interface. 1.2K ⭐ on GitHub.
★ 1,252 · TypeScriptAGENT MESH: NETWORK FRAMEWORK FOR AI AGENTS
Agent Mesh implements the A2A protocol for exchanging messages between autonomous agents on Go. The framework supports Model Context Protocol, provides tools for building
★ 1,201 · GoTINY VLLM: BUILD YOUR LLM
A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.
★ 1,194 · C++RUST RUNTIME FOR LLM INFERENCE AND EDITING
larql — Rust runtime to run, serve, and edit models. VINDEX3 format makes models executable and inspectable. 1173⭐ on GitHub.
★ 1,175 · RustFrom Instructions to Inference
A C reading path from one instruction to production inference, using primary sources and runnable benchmarks for each section.
★ 1,160 · CRUN MODELS LARGER THAN VRAM
inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.
★ 1,108 · C++Running Laya in Node.js
The project provides TypeScript bindings to run the open Laya decision model, compatible with Jev System-1, via ONNX Runtime in Node.js.
★ 1,074 · TypeScriptWhich local LLM to choose?
Find a local model with the best performance on your hardware based on real benchmarks, not parameter count. One command — instant launch.
★ 947 · PythonBIGMOEONEDGE - SUPER MOE ON PHONE
The project enables running 284 B Mixture‑of‑Experts models on a device with 12 GB of RAM, using only CPU and llama.cpp without loss of accuracy.
★ 855 · C++ML from First Principles
Educational resource teaching machine learning concepts from first principles, covering inference, CUDA, deep learning, and reinforcement learning with Python implementat
★ 842 · PythonSWIFTLM: SWIFT LLM ON APPLE
Native LLM inference server in Swift with MLX, OpenAI‑compatible API, SSD streaming for MoE >100B and TurboQuant KV‑cache compression.
★ 799 · SwiftResidual Memory Context Compaction
Remory trains a residual memory module to compact long contexts into shorter representations for transformer models.
★ 746 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustMLX Studio: Main JANG_Q Tool
Image generation and editing, chat and code in one place with support for OpenClaw and Anthropic API.
★ 727ML Models Compiler to C99
An AOT compiler that converts XGBoost, LightGBM, scikit-learn, CatBoost, and ONNX models into native C99 code, achieving inference speedups of up to 336× compared to Pyth
★ 680 · PythonTENSORSHARP - POWERFUL .NET LLM
A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.
★ 679 · C#Browser LLM Device Pool
Run large open‑source LLMs in the browser by linking multiple devices into a peer‑to‑peer compute pool, enabling chat and coding agents without any server.
★ 649 · JavaScriptNew projects on this topic in Telegram
Telegram →