R_REDDYX.XYZ

Inference Engine: open-source projects on GitHub

18 repositories on Inference Engine, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094LLM Inference 76Local LLM 107Qwen 106Mixture Of Experts 20Speculative Decoding 23Cuda 48Moe 23LLM Serving 24Openai API 86Qwen3 30Gguf 47Inference 95Machine Learning 190
FareedKhan-dev/kimi-k3-in-c

KIMI K3: 2.78 Trillion Parameters

Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.

★ 12,109 · C
FedML-AI/FedML

Universal FedML Platform

A universal library for distributed training, model serving, and federated learning with a cross-cloud scheduler.

★ 4,046 · Python
youssofal/MTPLX

MTPLX SUPERFAST LAUNCH

MTPLX lets you run Qwen 3.8 Flash Next and Qwen 3.8 27B on Mac at 125 tokens/sec in OpenCode. Thanks to MTP speculative decoding on Apple Silicon, results stay accurate a

★ 2,706 · Python
datawhalechina/zero-to-sglang

ZERO-TO-SGLANG: LLM INFERENCE TRAINING

The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.

★ 2,136 · Python
ovg-project/kvcached

KVCACHED: ACCELERATE GPU CACHE

Virtualized elastic KV‑cache for dynamic GPU sharing, accelerating LLM inference and reducing latency.

★ 2,041 · Python
nobodywho-ooo/nobodywho

RUN LLM LOCALLY WITH NOBODYWHO

NobodyWho lets you run LLM locally on any device thanks to an efficient inference engine built in Rust.

★ 1,677 · Rust
peonist-ai/halogen-flash-server

FAST LAUNCH QWEN3.8 ON AMD

The project provides scripts for instant deployment of Qwen3.8-Flash-Next on AMD Strix Halo (gfx1151) GPU via HIP. All dependencies are built automatically, and inference

★ 1,349 · Shell
simplexai-labs/PolyStrata

RUN LARGE LLMS LOCALLY

PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.

★ 1,037 · C++
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
ARahim3/mlx-dspark

MLX-DSPARK: 4X LLM ACCELERATION

Portable MLX port of DeepSeek DSpark and z‑lab DFlash for accelerating LLM decoding on Apple Silicon up to 4× without quality loss.

★ 740 · Python
jjang-ai/mlxstudio

MLX Studio: Main JANG_Q Tool

Image generation and editing, chat and code in one place with support for OpenClaw and Anthropic API.

★ 727
prabanta-dev/janas

JANAS: FAST LLM INFERENCE!!

janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU

★ 712 · C
zhongkaifu/TensorSharp

TENSORSHARP - POWERFUL .NET LLM

A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.

★ 679 · C#
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
kigner/audio.cpp-webui

audio.cpp WebUI C++

Full‑featured web UI for audio model inference in pure C++ using ggml. Supports TTS, ASR/STT, VAD, speaker diarization, voice conversion, and music generation without Pyt

★ 431 · C++
pie-project/pie

PIE — MODEL SERVING ON RUST

Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.

★ 240 · Rust
giannisanni/pulsar

PULSAR — ULTRAFAST MOE INFERENCE

Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.

★ 238 · Rust
InferMatrixLab/vllm-rlt

VLLM INFERENCE: LOOPED TRANSFORMERS

vllm-rlt — vLLM-style inference engine for looped transformers. Implements loop-level scheduling and execution via composable stages. 207 stars on GitHub.

★ 227 · Python

New projects on this topic in Telegram

Telegram →
Inference Engine — 18 open-source GitHub projects | REDDYX AI