R_REDDYX.XYZ

LLM Inference: open-source projects on GitHub

76 repositories on LLM Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Vllm 69Cuda 48Qwen 106Local LLM 107LLM Serving 24Machine Learning 190Self Hosted 359Speculative Decoding 23Large Language Models 154Llms 136Quantization 34Generative AI 192Apple Silicon 56
ray-project/ray

Ray: AI Compute Engine

Ray is a distributed engine for AI computing, combining a runtime core and a set of libraries to accelerate ML workloads.

★ 42,516 · Python
gitleaks/gitleaks

GITLEAKS - FIND SECRETS IN CODE

Powerful scanner for secret leaks in Git repositories. Protect your keys and tokens!

★ 27,428 · Go
Lightning-AI/litgpt

litgpt: Powerful LLMs

High-performance language models with ready-to-use recipes for pretraining, fine-tuning, and scaling. 13K+ stars!

★ 13,387 · Python
bentoml/OpenLLM

OPENLLM: RUN LLMS AS OPENAI API

OpenLLM allows running any open-source LLMs (Llama, Qwen, DeepSeek) as OpenAI-compatible APIs with a single command. Includes chat UI, modern inference backends, and read

★ 12,607 · Python
FareedKhan-dev/kimi-k3-in-c

KIMI K3: 2.78 Trillion Parameters

Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.

★ 12,109 · C
openvinotoolkit/openvino

OPTIMIZATION AND DEPLOYMENT OF AI MODELS

Toolkit for optimizing and deploying deep learning models. Supports conversion from PyTorch, TensorFlow, ONNX and inference on CPU, GPU, and Intel NPU.

★ 11,656 · C++
bentoml/BentoML

BENTOML PATH TO AI API

Serve models and AI applications in minutes: API, queues, multi‑model pipelines, and LLM services.

★ 8,663 · Python
InternLM/lmdeploy

LMDeploy: Easy LLM Deployment

A toolkit for compressing, deploying, and serving LLMs. ⭐ 7873 | Python

★ 7,873 · Python
bojieli/ai-infra-book

AI Infrastructure Book

Open‑source Chinese book on AI infrastructure, covering LLM inference and training system design with quantitative analysis, plus PDF, code tools, and experiments.

★ 7,748 · Python
drumih/turbo-fieldfare

TURBO-FIELDFARE - GEMMA 4 ON MACBOOK

Run Gemma 4 26B‑A4B on any MacBook with an M‑series chip using under 2 GB of RAM. Fast and with no extra resources.

★ 7,391 · Swift
katanemo/plano

PLANO: AI PROXY FOR AGENTS

AI-native proxy and data plane for agent applications with orchestration and security.

★ 6,567 · Rust
cactus-compute/engine

AI FRAME FOR DEVICES

This project offers low-bit quant kernels and runtime for consumer devices on Android/ARM. It solves edge-ai tasks with minimal resources.

★ 6,192 · C++
gpustack/gpustack

GPU cluster manager

Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.

★ 5,041 · Python
cactus-compute/cactus

Efficient AI for Mobile Devices

An AI engine with minimal latency for smartphones and wearable devices.

★ 4,949 · C
algorithmicsuperintelligence/optillm

OPTILLM — SPEEDING LLM INFERENCE

optillm — a proxy accelerator for LLMs that reduces latency and saves resources during inference.

★ 4,343 · Python
lemonade-sdk/lemonade

Lemonade SDK

Lemonade helps users run local AI applications by optimizing LLM performance on their GPU and NPU. Join our Discord!

★ 4,059 · C++
NVIDIA/GenerativeAIExamples

GENERATIVE AI MICROSERVICE WORKFLOW

A collection of generative AI examples optimized for accelerated infrastructure and microservices architecture. Ready-to-use Jupyter notebooks for a quick start.

★ 4,037 · Jupyter Notebook
hemansnation/AI-Engineer-Headquarters

AI‑ENGINEER HQ: SCIENCE CORE

The project contains Jupyter notebooks with methods for training models, RAG pipelines on LangChain and data structure algorithms for AI agents. It allows quickly creatin

★ 3,780 · Jupyter Notebook
mw00/project-maya

PROJECT MAYA: LOCAL LLM ENGINE

Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,

★ 2,898 · C++
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
youssofal/MTPLX

MTPLX SUPERFAST LAUNCH

MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a

★ 2,706 · Python
incoai/splash

Local AI Engine

Splash lets you run machine learning models directly on Apple‑silicon Macs, using the full power of CPU and GPU without relying on the cloud.

★ 2,284 · Python
ashhart/TensorFold

TENSORFOLD: FAST LLM DECODING ON MAC.

TensorFold provides exact and fast LLM decoding on Apple Silicon via MLX, offering an OpenAI-compatible API. The Python project has already earned 526 stars and supports

★ 2,172 · Python
datawhalechina/zero-to-sglang

ZERO-TO-SGLANG: LLM INFERENCE TRAINING

The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.

★ 2,136 · Python
ovg-project/kvcached

KVCACHED: ACCELERATE GPU CACHE

Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.

★ 2,041 · Python
AtomicBot-ai/Atomic-Chat

LOCAL OPEN-SOURCE AI AGENT

Atomic-Chat is a private AI engine for running LLM locally without internet, fully offline on your computer. ⭐ 967 stars on GitHub.

★ 2,008 · TypeScript
Nano-Collective/nanocoder

Nanocoder — Local Assistant

Beautiful local coding agent for the terminal, built by the community for the community ⚒

★ 1,957 · TypeScript
neuron-core/neuron-ai

NEURON-AI PHP FRAMEWORK

A PHP framework for building AI agents that connects LLM, vector databases, and memory.

★ 1,951 · PHP
beam-cloud/beta9

BETA9 - Lightning GPU Output

Ultrafast serverless GPU inference with sandboxes and background jobs, written in Go for instant scaling.

★ 1,664 · Go
felladrin/awesome-ai-web-search

Awesome AI Web Search

A collection of the best web tools where AI helps search the web faster and more accurately.

★ 1,519 · HTML
lean-dojo/LeanCopilot

LEARNCOPILOT: AI Proof Assistant

The project uses large language models as a copilot to assist in proving theorems in Lean.

★ 1,345 · C++
eagle-hq/EAGLE-4

EAGLE-4 Speculative Decoding

Official inference code for EAGLE-4, a speculative decoding method that accelerates LLM generation by drafting tokens via feature-level extrapolation.

★ 1,287 · Python
jmaczan/tiny-vllm

TINY VLLM: BUILD YOUR LLM

A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.

★ 1,194 · C++
patchy631/time-to-first-token

LLM Inference Optimization Roadmap

A 10‑week plan to study the service and accelerate large language models, covering vLLM, SGLang, quantization, and speculative decoding practices.

★ 1,179 · HTML
dalisoft/awesome-hosting

AWESOME-HOSTING TOP CHEAP HOSTINGS

A curated selection of the best hostings, sorted by the price of the minimal plan. Quickly find the most cost-effective option for your project.

★ 970
feder-cr/jev

JEVOS: BINARY DECISIONS ON LAPTOP

Jevos is an open-source Jev alternative for yes/no decisions. Runs fully locally on CPU via FastAPI. Data never leaves your device.

★ 963 · Python
foldl/chatllm.cpp

PURE C++ CHAT BOTS

Implementation of several models for real-time communication directly on your computer. Works on CPU and GPU.

★ 942 · C++
eastriverlee/LLM.swift

LLM.SWIFT - LOCAL ACCESS TO LLM

Simple library for working with large language models locally on all Apple platforms.

★ 884 · C++
ARahim3/kaggle-tpu-lab

FREE TPU FOR QWEN3.8-27B

The kaggle-tpu-lab project runs Qwen3.8-27B (bf16) on a free Kaggle TPU, providing an OpenAI‑compatible API. It supports a 262k token context and ~130 tokens/s speed, wor

★ 801 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
ARahim3/mlx-dspark

MLX-DSPARK: 4X LLM ACCELERATION

Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.

★ 740 · Python
carloslfu/slotstream

SLOTSTREAM LAUNCHES QWEN3.8 ON MAC

Run Qwen3.8‑Flash‑Next (125B MoE, 104 GB at 4‑bit) on Mac by streaming experts from SSD on demand. Implemented with MLX + Swift, it offers an Ollama‑compatible API.

★ 673 · Swift
NightMean/OlliteRT

OLLITERT LLM SERVER ANDROID

OlliteRT turns your Android smartphone into a local inference server compatible with the OpenAI API. It works completely offline, ensuring privacy and open-source code in

★ 633 · Kotlin
NotPunchnox/rkllama

RKLLAMA — LOCAL AI ON ROCKCHIP

Ollama alternative for Rockchip NPU. Run AI and Deep Learning models directly on Rockchip devices with NPU optimization.

★ 627 · Python
Indras-Mirror/Xolotl

RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC

Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent

★ 534 · C++
ome-projects/ome

LLM Serving in Kubernetes

OME is an operator for running LLMs in Kubernetes with smart GPU scheduling and support for SGLang, vLLM, TensorRT-LLM, and Triton.

★ 527 · Go
expectedparrot/edsl

AI Surveys on Steroids

edsl — a framework for creating AI surveys and experiments with thousands of AI agents. Ideal for social sciences and marketing.

★ 519 · Python
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
modular/llm-inference-handbook

COMPLETE LLM INFERENCE GUIDE

A collection of best practices, tools, and examples for efficient inference of large language models using TypeScript.

★ 481 · TypeScript
jjiantong/Awesome-KV-Cache-Optimization

KV-CACHE OPTIMIZATION FOR LLM

Overview of KV-Cache optimization methods for efficient serving of large language models. A systematic approach to inference speedup.

★ 426 · Python
amitshekhariitbhu/llm-inference-engineering

LLM Inference Engineering

Step-by-step training on inference engineering techniques for large language models: KV cache, PagedAttention, batching, and frameworks vLLM, SGLang.

★ 412 · Markdown
zeraix/zeraix

ZERAIX: LOCAL AI DESKTOP

Local open-source environment for running AI models on device. Speeds up inference without cloud.

★ 405 · TypeScript
Context-Engine-AI/Context-Engine

CONTEXT-ENGINE CONTEXT COMPRESSION

Context-Engine MCP compresses agent prompts, boosting LLM efficiency without losing meaning. It supports Python and is ready for integration.

★ 402 · Python
JakeATX/llamAmpere

LLAMAMPERE: LLM ACCELERATION ON RTX 3090

llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.

★ 380 · C++
DebarghaG/proofofthought

LLM REASONING VIA Z3 THEOREM PROVING

Project combines LLM and Z3 for formal reasoning verification. Supports SMT2 and JSON DSL backends. 377 stars on GitHub.

★ 377 · Python
matrixhub-ai/matrixhub

MATRIXHUB: AI Hub – Open Source

Self-hosted platform for AI models with Hugging Face compatibility and vLLM/SGLang acceleration. Stars: 246 ⭐

★ 367 · Go
mlco2/ecologits

ECOLOGIT: ENERGY ASSESSMENT

EcoLogits measures energy consumption and carbon footprint when using generative AI models via API. It collects CO₂ emission metrics and offers recommendations to reduce

★ 364 · Python
monocle2ai/monocle

MONOCLE - GenAI App Tracing

Framework for tracing GenAI application code in Python. Easily track the execution of your AI applications.

★ 346 · Python
Epistates/pmetal

PMETAL: THE ACCEL LLM ON APPLE SILICON

PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele

★ 333 · Rust
sophgo/LLM-TPU

LLM-TPU - RUN AI ON BM1684X

Run generative AI models on Sophgo BM1684X/BM1688 with high speed and low power consumption.

★ 323 · C++

New projects on this topic in Telegram

Telegram →
LLM Inference — 76 open-source GitHub projects | REDDYX AI