R_REDDYX.XYZ

Inference: open-source projects on GitHub

95 repositories on Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Vllm 69Cuda 48Pytorch 98Qwen 106Machine Learning 190Gguf 47Openai API 86GPU 48Mlx 47Local LLM 107Llama Cpp 56Deep Learning 105Llama 106
kaito-project/aikit

Survive LLM Chaos

Open-source Go tool for fine-tuning, building, and deploying LLMs with a single command.

★ 545 · Go
vladimirrott/mojo-dllm

DISLLM INFERENCE ON MOJO

Runs diffusion LLMs (LLaDA, Dream) locally on CPU without GPU. Written in Mojo with GGUF format support. 289 stars on GitHub.

★ 540 · Mojo
ekzhang/resin

EXPERIMENTAL INFERENCE LIBRARY

resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25

★ 432 · Python
Entrpi/ds4-on-spark

DS4 on Spark: 3X Speedup

One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.

★ 415 · Shell
SandAI-org/MagiCompiler

MagiCompiler - Free Optimization for Models

MagiCompiler is a plug‑and‑play Python compiler that automatically adds free optimizations for model inference and training.

★ 337 · Python
vortico/flama

FLAMA: ONE LINE FOR AI API

Framework for fast deployment of AI models as API with built-in chat and MCP support.

★ 307 · Python
T-Crypt/berth

GPU BY TURN WITHOUT CRASHES

Berth leases the GPU to chat, image, and voice engines in turn. Two processes never load together and crash each other. Single Go binary, no dependencies.

★ 297 · Go
mohitsoni48/TurboLLM

TURBO LLM: OFFLINE WITH AI

Run any local LLM with automatic GPU tuning, a web UI, and an OpenAI/Anthropic‑compatible API.

★ 296 · TypeScript
autonomous-ai/autonomous-grid

BUILD YOUR AI INTRANET

Autonomous grid unites your computers into a single infrastructure for running AI models: training, fine‑tuning and inference. Supports distributed computing, fine‑tuning

★ 278 · Python
graphsignal/graphsignal

PROFILER FOR AI AGENTS AND LLM

Graphsignal Profiler is a tool for debugging and monitoring inference. It helps find performance bottlenecks in deep neural networks and AI agents. Perfect for working wi

★ 276 · Python
Kotlin/Kotlin-AI-Examples

KOTLIN AND AI: PRACTICAL EXAMPLES

A collection of Kotlin examples with Spring AI, LangChain4j, and Jupyter notebooks for practical AI study.

★ 272 · Jupyter Notebook
AMD-AGI/Hyperloom

HYPERLOOM AUTO-OPTIMIZATION LLM ON AMD

Hyperloom — agentic system that automatically optimizes LLM workloads on AMD GPUs by selecting kernels and tuning parameters for peak performance without manual setup.

★ 271 · Python
defilantech/LLMKube

LLMKUBE: LLM IN KUBERNETES

LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi

★ 266 · Go
chigwell/llm7.io

UNIFIED GATEWAY TO AI MODELS

llm7.io provides a single API gateway to dozens of AI models from OpenAI, Anthropic, Cohere and others. A unified interface lets you switch between providers, ensures fal

★ 262 · TypeScript
sybil-solutions/inference-optimisation-skill

Auto-Tune LLM Serving Speed

Automatically optimizes local LLM serving speed via hill-climbing across decode, speculative decoding, prefill, KV cache, and startup metrics on any hardware.

★ 249 · Python
pie-project/pie

PIE — MODEL SERVING ON RUST

Programmable engine for high-performance LLM and diffusion model serving. Written in Rust, supports batching, KV caching, and dynamic adapter loading.

★ 240 · Rust
Yamz-Labs/kyojin

AMD Strix Halo Inference Engine

Kyojin runs ExLlamaV3 on AMD Strix Halo (ROCm gfx1151) to execute 300B‑parameter Mixture‑of‑Experts models on a single 128 GB mini PC.

★ 238 · Python
RustingStudio/RustingStrata

PURE-RUST ENGINE FOR QWEN GGUF

RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.

★ 231 · Rust
timtoole02/Camelid

CAMELID: LOCAL AI ON RUST

Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful

★ 221 · Rust
novitalabs/pegaflow

PEGAFLOW KV CACHE FOR LLM

Pegaflow — high‑performance KV‑cache storage for LLM inference. Supports offloading to GPU, caching on SSD, and cross‑node exchange via RDMA. Compatible with vLLM and SGL

★ 221 · Rust
aryamandarda/tamserve

Live Multi-Camera SAM Tracking

Serves multiple cameras with one SAM model per GPU, achieving 4x memory reduction and faster inference than original implementations.

★ 221 · Python
llamatik/Llamatik

ON-DEVICE AI SDK KOTLIN

Llamatik is an on-device AI SDK for Kotlin Multiplatform (Android, iOS, Desktop, JVM, WASM). Supports LLM inference, Speech-to-Text, image generation, Agent Framework and

★ 220 · Kotlin
uncSoft/anubis-oss

ANUBIS: BENCHMARK LLM ON APPLE SILICON

anubis-oss — Swift utility for local LLM launch and benchmarking on Apple Silicon. It measures speed and accuracy of GPU inference, comparing results with CPU baseline an

★ 217 · Swift
jakobhoeg/browser-ai

AI IN BROWSER WITHOUT SERVER

Library of providers for Vercel AI SDK runs models directly in the browser. Simplifies client-side AI integration and unifies work across vendors. 205 stars on GitHub.

★ 207 · TypeScript
nari-labs/nari-qwen3-tts

Ultra-Fast Qwen3-TTS

A library for instant speech synthesis based on Qwen3, ensuring time to first audio under 50 ms at a load of 10 requests per second.

★ 192 · Python
Aevonix/FlashAlign

On-Demand Model Alignment

Aligns LLMs instantly per request for verified users or always-on mode. Integrates with Hermes, OpenClaw, and custom harnesses.

★ 126 · Python
juxhinr/bindwidth

LLM TCO Calculator

Bindwidth estimates hardware requirements and TCO for running LLMs locally, using real data to calculate required GPU/CPU and energy consumption, depreciation, and licens

★ 109 · JavaScript
ShadowSafin/Synapass

AI Inference Gateway

Synapass is a Go‑based AI inference gateway that routes requests across multiple LLM providers, applies policies, caches responses, and provides observability.

★ 98 · Go
DevDock-AI/Speedy

125B MoE Inference on RTX

Run Qwen3.8-Flash-Next (125B MoE) with 1M context on one or two RTX 3080 10GB cards. Tuned, benchmarked fork of Strata.

★ 71 · C++
hadihonarvar/flock

Self-hosted LLM gateway

A single Go binary turns your Mac and Linux machines into a private inference cluster featuring multi‑machine routing, sharding via llama.cpp‑RPC, and support for the Ope

★ 38 · Go
agentic-in/inferoa

Inference Agent Tokenmaxxing

Agentic-in/inferoa is a framework for creating agents optimized for inference with Tokenmaxxing technology, providing high efficiency when working with large language mod

★ 38 · TypeScript
yassa9/dvlt.cu

DVLT on CUDA without dependencies

The project ports NVIDIA DVLT to pure CUDA/C++, allowing images to be fed in and a 3D point cloud together with camera poses to be obtained. Everything is packed into a s

★ 31 · Cuda
maximecb/bebelm

Pure Rust LFM2.5-8B-A1B

CPU-only implementation of the LiquidAI LFM2.5-8B-A1B model in pure Rust. Enables text inference without GPU and external libraries.

★ 31 · Rust
Jwuthri/SelfJev

Open Decision Model API

Provides an open decision model exposing Jev's API for typed answers (yes/no, choice, score, multi) with probabilities from forward passes, using Qwen3.5-4B fine‑tuned wi

★ 25 · Python
avifenesh/memra

MEMRA - AI Accelerator for RTX

Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.

★ 6 · Rust

New projects on this topic in Telegram

Telegram →
Inference — page 2 | REDDYX AI