Inference Engine: open-source projects on GitHub
18 repositories on Inference Engine, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
KIMI K3: 2.78 Trillion Parameters
Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.
★ 12,109 · CUniversal FedML Platform
A universal library for distributed training, model serving, and federated learning with a cross-cloud scheduler.
★ 4,046 · PythonMTPLX SUPERFAST LAUNCH
MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a
★ 2,706 · PythonZERO-TO-SGLANG: LLM INFERENCE TRAINING
The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.
★ 2,136 · PythonKVCACHED: ACCELERATE GPU CACHE
Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.
★ 2,041 · PythonRUN LLM LOCALLY WITH NOBODYWHO
NobodyWho lets you run LLM locally on any device thanks to an efficient inference engine built in Rust.
★ 1,677 · RustFAST LAUNCH QWEN3.8 ON AMD
The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference
★ 1,349 · ShellRUN LARGE LLMS LOCALLY
PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.
★ 1,037 · C++PEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustMLX-DSPARK: 4X LLM ACCELERATION
Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.
★ 740 · PythonMLX Studio: Main JANG_Q Tool
Image generation and editing, chat and code in one place with support for OpenClaw and Anthropic API.
★ 727JANAS: FAST LLM INFERENCE!!
janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU
★ 712 · CTENSORSHARP - POWERFUL .NET LLM
A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.
★ 679 · C#FLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · Pythonaudio.cpp WebUI C++
Full‑featured web UI for audio model inference in pure C++ using ggml. Supports TTS, ASR/STT, VAD, speaker diarization, voice conversion, and music generation without Pyt
★ 431 · C++PIE — MODEL SERVING ON RUST
Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.
★ 240 · RustPULSAR — ULTRAFAST MOE INFERENCE
Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.
★ 238 · RustVLLM INFERENCE: LOOPED TRANSFORMERS
vllm-rlt — vLLM-style inference engine for looped transformers. Implements loop-level scheduling and execution via composable stages. 207 stars on GitHub.
★ 227 · PythonNew projects on this topic in Telegram
Telegram →