Inference: open-source projects on GitHub
95 repositories on Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Survive LLM Chaos
Open-source Go tool for fine-tuning, building, and deploying LLMs with a single command.
★ 545 · GoDISLLM INFERENCE ON MOJO
Runs diffusion LLMs (LLaDA, Dream) locally on CPU without GPU. Written in Mojo with GGUF format support. 289 stars on GitHub.
★ 540 · MojoEXPERIMENTAL INFERENCE LIBRARY
resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25
★ 432 · PythonDS4 on Spark: 3X Speedup
One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.
★ 415 · ShellMagiCompiler - Free Optimization for Models
MagiCompiler is a plug‑and‑play Python compiler that automatically adds free optimizations for model inference and training.
★ 337 · PythonFLAMA: ONE LINE FOR AI API
Framework for fast deployment of AI models as API with built-in chat and MCP support.
★ 307 · PythonGPU BY TURN WITHOUT CRASHES
Berth leases the GPU to chat, image, and voice engines in turn. Two processes never load together and crash each other. Single Go binary, no dependencies.
★ 297 · GoTURBO LLM: OFFLINE WITH AI
Run any local LLM with automatic GPU tuning, a web UI, and an OpenAI/Anthropic‑compatible API.
★ 296 · TypeScriptBUILD YOUR AI INTRANET
Autonomous grid unites your computers into a single infrastructure for running AI models: training, fine‑tuning and inference. Supports distributed computing, fine‑tuning
★ 278 · PythonPROFILER FOR AI AGENTS AND LLM
Graphsignal Profiler is a tool for debugging and monitoring inference. It helps find performance bottlenecks in deep neural networks and AI agents. Perfect for working wi
★ 276 · PythonKOTLIN AND AI: PRACTICAL EXAMPLES
A collection of Kotlin examples with Spring AI, LangChain4j, and Jupyter notebooks for practical AI study.
★ 272 · Jupyter NotebookHYPERLOOM AUTO-OPTIMIZATION LLM ON AMD
Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.
★ 271 · PythonLLMKUBE: LLM IN KUBERNETES
LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi
★ 266 · GoUNIFIED GATEWAY TO AI MODELS
llm7.io provides a single API gateway to dozens of AI models from OpenAI, Anthropic, Cohere and others. A unified interface lets you switch between providers, ensures fal
★ 262 · TypeScriptAuto-Tune LLM Serving Speed
Automatically optimizes local LLM serving speed via hill-climbing across decode, speculative decoding, prefill, KV cache, and startup metrics on any hardware.
★ 249 · PythonPIE — MODEL SERVING ON RUST
Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.
★ 240 · RustAMD Strix Halo Inference Engine
Kyojin runs ExLlamaV3 on AMD Strix Halo (ROCm gfx1151) to execute 300B‑parameter Mixture‑of‑Experts models on a single 128 GB mini PC.
★ 238 · PythonPURE-RUST ENGINE FOR QWEN GGUF
RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.
★ 231 · RustCAMELID: LOCAL AI ON RUST
Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful
★ 221 · RustPEGAFLOW KV CACHE FOR LLM
Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL
★ 221 · RustLive Multi-Camera SAM Tracking
Serves multiple cameras with one SAM model per GPU, achieving 4x memory reduction and faster inference than original implementations.
★ 221 · PythonON-DEVICE AI SDK KOTLIN
Llamatik is an on-device AI SDK for Kotlin Multiplatform (Android, iOS, Desktop, JVM, WASM). Supports LLM inference, Speech-to-Text, image generation, Agent Framework and
★ 220 · KotlinANUBIS: BENCHMARK LLM ON APPLE SILICON
anubis-oss — Swift utility for local LLM launch and benchmarking on Apple Silicon. It measures speed and accuracy of GPU inference, comparing results with CPU baseline an
★ 217 · SwiftAI IN BROWSER WITHOUT SERVER
Library of providers for Vercel AI SDK runs models directly in the browser. Simplifies client-side AI integration and unifies work across vendors. 205 stars on GitHub.
★ 207 · TypeScriptUltra-Fast Qwen3-TTS
A library for instant speech synthesis based on Qwen3, ensuring time to first audio under 50 ms at a load of 10 requests per second.
★ 192 · PythonOn-Demand Model Alignment
Aligns LLMs instantly per request for verified users or always-on mode. Integrates with Hermes, OpenClaw, and custom harnesses.
★ 126 · PythonLLM TCO Calculator
Bindwidth estimates hardware requirements and TCO for running LLMs locally, using real data to calculate required GPU/CPU and energy consumption, depreciation, and licens
★ 109 · JavaScriptAI Inference Gateway
Synapass is a Go‑based AI inference gateway that routes requests across multiple LLM providers, applies policies, caches responses, and provides observability.
★ 98 · Go125B MoE Inference on RTX
Run Qwen3.8-Flash-Next (125B MoE) with 1M context on one or two RTX 3080 10GB cards. Tuned, benchmarked fork of Strata.
★ 71 · C++Self-hosted LLM gateway
A single Go binary turns your Mac and Linux machines into a private inference cluster featuring multi‑machine routing, sharding via llama.cpp‑RPC, and support for the Ope
★ 38 · GoInference Agent Tokenmaxxing
Agentic-in/inferoa is a framework for creating agents optimized for inference with Tokenmaxxing technology, providing high efficiency when working with large language mod
★ 38 · TypeScriptDVLT on CUDA without dependencies
The project ports NVIDIA DVLT to pure CUDA/C++, allowing images to be fed in and a 3D point cloud together with camera poses to be obtained. Everything is packed into a s
★ 31 · CudaPure Rust LFM2.5-8B-A1B
CPU-only implementation of the LiquidAI LFM2.5-8B-A1B model in pure Rust. Enables text inference without GPU and external libraries.
★ 31 · RustOpen Decision Model API
Provides an open decision model exposing Jev's API for typed answers (yes/no, choice, score, multi) with probabilities from forward passes, using Qwen3.5-4B fine‑tuned wi
★ 25 · PythonMEMRA - AI Accelerator for RTX
Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.
★ 6 · RustNew projects on this topic in Telegram
Telegram →