LLM Inference: open-source projects on GitHub
76 repositories on LLM Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Ray: AI Compute Engine
Ray is a distributed engine for AI computing, combining a runtime core and a set of libraries to accelerate ML workloads.
★ 42,516 · PythonGITLEAKS - FIND SECRETS IN CODE
Powerful scanner for secret leaks in Git repositories. Protect your keys and tokens!
★ 27,428 · Golitgpt: Powerful LLMs
High-performance language models with ready-to-use recipes for pretraining, fine-tuning, and scaling. 13K+ stars!
★ 13,387 · PythonOPENLLM: RUN LLMS AS OPENAI API
OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and read
★ 12,607 · PythonKIMI K3: 2.78 Trillion Parameters
Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.
★ 12,109 · COPTIMIZATION AND DEPLOYMENT OF AI MODELS
Toolkit for optimizing and deploying deep learning models. Supports conversion from PyTorch, TensorFlow, ONNX and inference on CPU, GPU, and Intel NPU.
★ 11,656 · C++BENTOML PATH TO AI API
Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.
★ 8,663 · PythonLMDeploy: Easy LLM Deployment
A toolkit for compressing, deploying, and serving LLMs. ⭐ 7873 | Python
★ 7,873 · PythonAI Infrastructure Book
Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.
★ 7,748 · PythonTURBO-FIELDFARE - GEMMA 4 ON MACBOOK
Run Gemma 4 26B‑A4B on any MacBook with an M‑series chip using under 2 GB of RAM. Fast and with no extra resources.
★ 7,391 · SwiftPLANO: AI PROXY FOR AGENTS
AI-native proxy and data plane for agent applications with orchestration and security.
★ 6,567 · RustAI FRAME FOR DEVICES
This project offers low-bit quant kernels and runtime for consumer devices on Android/ARM. It solves edge-ai tasks with minimal resources.
★ 6,192 · C++GPU cluster manager
Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.
★ 5,041 · PythonEfficient AI for Mobile Devices
An AI engine with minimal latency for smartphones and wearable devices.
★ 4,949 · COPTILLM — SPEEDING LLM INFERENCE
optillm — a proxy accelerator for LLMs that reduces latency and saves resources during inference.
★ 4,343 · PythonLemonade SDK
Lemonade helps users run local AI applications by optimizing LLM performance on their GPU and NPU. Join our Discord!
★ 4,059 · C++GENERATIVE AI MICROSERVICE WORKFLOW
A collection of generative AI examples optimized for accelerated infrastructure and microservices architecture. Ready-to-use Jupyter notebooks for a quick start.
★ 4,037 · Jupyter NotebookAI‑ENGINEER HQ: SCIENCE CORE
The project contains Jupyter notebooks with methods for training models, RAG pipelines on LangChain and data structure algorithms for AI agents. It allows quickly creatin
★ 3,780 · Jupyter NotebookPROJECT MAYA: LOCAL LLM ENGINE
Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,
★ 2,898 · C++HyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonMTPLX SUPERFAST LAUNCH
MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a
★ 2,706 · PythonLocal AI Engine
Splash lets you run machine learning models directly on Apple‑silicon Macs, using the full power of CPU and GPU without relying on the cloud.
★ 2,284 · PythonTENSORFOLD: FAST LLM DECODING ON MAC.
TensorFold provides exact and fast LLM decoding on Apple Silicon via MLX, offering an OpenAI-compatible API. The Python project has already earned 526 stars and supports
★ 2,172 · PythonZERO-TO-SGLANG: LLM INFERENCE TRAINING
The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.
★ 2,136 · PythonKVCACHED: ACCELERATE GPU CACHE
Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.
★ 2,041 · PythonLOCAL OPEN-SOURCE AI AGENT
Atomic-Chat is a private AI engine for running LLM locally without internet, fully offline on your computer. ⭐ 967 stars on GitHub.
★ 2,008 · TypeScriptNanocoder — Local Assistant
Beautiful local coding agent for the terminal, built by the community for the community ⚒
★ 1,957 · TypeScriptNEURON-AI PHP FRAMEWORK
A PHP framework for building AI agents that connects LLM, vector databases, and memory.
★ 1,951 · PHPBETA9 - Lightning GPU Output
Ultrafast serverless GPU inference with sandboxes and background jobs, written in Go for instant scaling.
★ 1,664 · GoAwesome AI Web Search
A collection of the best web tools where AI helps search the web faster and more accurately.
★ 1,519 · HTMLLEARNCOPILOT: AI Proof Assistant
The project uses large language models as a copilot to assist in proving theorems in Lean.
★ 1,345 · C++EAGLE-4 Speculative Decoding
Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.
★ 1,287 · PythonTINY VLLM: BUILD YOUR LLM
A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.
★ 1,194 · C++LLM Inference Optimization Roadmap
A 10‑week plan to study the service and accelerate large language models, covering vLLM, SGLang, quantization, and speculative decoding practices.
★ 1,179 · HTMLAWESOME-HOSTING TOP CHEAP HOSTINGS
A curated selection of the best hostings, sorted by the price of the minimal plan. Quickly find the most cost-effective option for your project.
★ 970JEVOS: BINARY DECISIONS ON LAPTOP
Jevos is an open-source Jev alternative for yes/no decisions. Runs fully locally on CPU via FastAPI. Data never leaves your device.
★ 963 · PythonPURE C++ CHAT BOTS
Implementation of several models for real-time communication directly on your computer. Works on CPU and GPU.
★ 942 · C++LLM.SWIFT - LOCAL ACCESS TO LLM
Simple library for working with large language models locally on all Apple platforms.
★ 884 · C++FREE TPU FOR QWEN3.8-27B
The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor
★ 801 · PythonPEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustMLX-DSPARK: 4X LLM ACCELERATION
Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.
★ 740 · PythonSLOTSTREAM LAUNCHES QWEN3.8 ON MAC
Run Qwen3.8‑Flash‑Next (125B MoE, 104 GB at 4‑bit) on Mac by streaming experts from SSD on demand. Implemented with MLX + Swift, it offers an Ollama‑compatible API.
★ 673 · SwiftOLLITERT LLM SERVER ANDROID
OlliteRT turns your Android smartphone into a local inference server compatible with the OpenAI API. It works completely offline, ensuring privacy and open-source code in
★ 633 · KotlinRKLLAMA — LOCAL AI ON ROCKCHIP
Ollama alternative for Rockchip NPU. Run AI and Deep Learning models directly on Rockchip devices with NPU optimization.
★ 627 · PythonRUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC
Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent
★ 534 · C++LLM Serving in Kubernetes
OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.
★ 527 · GoAI Surveys on Steroids
edsl — a framework for creating AI surveys and experiments with thousands of AI agents. Ideal for social sciences and marketing.
★ 519 · PythonFLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · PythonCOMPLETE LLM INFERENCE GUIDE
A collection of best practices, tools, and examples for efficient inference of large language models using TypeScript.
★ 481 · TypeScriptKV-CACHE OPTIMIZATION FOR LLM
Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.
★ 426 · PythonLLM Inference Engineering
Step-by-step training on inference engineering techniques for large language models: KV cache, PagedAttention, batching, and frameworks vLLM, SGLang.
★ 412 · MarkdownZERAIX: LOCAL AI DESKTOP
Local open-source environment for running AI models on device. Speeds up inference without cloud.
★ 405 · TypeScriptCONTEXT-ENGINE CONTEXT COMPRESSION
Context-Engine MCP compresses agent prompts, boosting LLM efficiency without losing meaning. It supports Python and is ready for integration.
★ 402 · PythonLLAMAMPERE: LLM ACCELERATION ON RTX 3090
llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.
★ 380 · C++LLM REASONING VIA Z3 THEOREM PROVING
Project combines LLM and Z3 for formal reasoning verification. Supports SMT2 and JSON DSL backends. 377 stars on GitHub.
★ 377 · PythonMATRIXHUB: AI Hub – Open Source
Self-hosted platform for AI models with Hugging Face compatibility and vLLM/SGLang acceleration. Stars: 246 ⭐
★ 367 · GoECOLOGIT: ENERGY ASSESSMENT
EcoLogits measures energy consumption and carbon footprint when using generative AI models via API. It collects CO₂ emission metrics and offers recommendations to reduce
★ 364 · PythonMONOCLE - GenAI App Tracing
Framework for tracing GenAI application code in Python. Easily track the execution of your AI applications.
★ 346 · PythonPMETAL: THE ACCEL LLM ON APPLE SILICON
PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele
★ 333 · RustLLM-TPU - RUN AI ON BM1684X
Run generative AI models on Sophgo BM1684X/BM1688 with high speed and low power consumption.
★ 323 · C++New projects on this topic in Telegram
Telegram →