R_REDDYX.XYZ

Inference: open-source projects on GitHub

95 repositories on Inference, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Vllm 69Cuda 48Pytorch 98Qwen 106Machine Learning 190Gguf 47Openai API 86GPU 48Mlx 47Local LLM 107Llama Cpp 56Deep Learning 105Llama 106
vllm-project/vllm

High-performance LLM engine

A high-performance and memory-efficient engine for inference and serving large language models.

★ 79,783 · Python
ultralytics/yolov5

YOLOV5: OBJECT DETECTION ON PYTORCH

Computer vision model for object detection, instance segmentation, and image classification. Runs on PyTorch, supports export to CoreML and ONNX.

★ 58,278 · Python
ggml-org/whisper.cpp

whisper.cpp: C/C++ port of Whisper

High-performance port of the OpenAI Whisper speech recognition model in C/C++ for local and cross-platform use.

★ 49,828 · C++
deepspeedai/DeepSpeed

DeepSpeed: optimization library for training

DeepSpeed is an optimization deep learning library ensuring efficient distributed training and inference. Supports ZeRO, MoE and parallelism for training models with hund

★ 43,332 · Python
hpcaitech/ColossalAI

FRAMEWORK FOR CHEAPER LARGE MODEL TRAINING

Library for distributed training and inference of large language models. Supports Zero, tensor, and pipeline parallelism strategies to reduce GPU costs.

★ 41,422 · Python
google-ai-edge/mediapipe

CROSS-PLATFORM ON-DEVICE ML FRAMEWORK

MediaPipe is Google's framework for running ML on-device. Supports mobile platforms, web, desktop, and IoT without sending data to the cloud.

★ 37,717 · C++
sgl-project/sglang

SGLang: LLM Framework

SGLang is a high-performance framework for serving large language and multimodal models.

★ 27,854 · Python
SYSTRAN/faster-whisper

FAST SPEECH TRANSCRIPTION

Faster Whisper — an accelerated version of the Whisper model for speech recognition, built on CTranslate2. It supports quantized models and provides real‑time transcripti

★ 26,758 · Python
Tencent/ncnn

HIGH-PERFORMANCE NEURAL NETWORK INFERENCE FRAMEWORK FOR MOBILE

ncnn is a neural network inference framework with no third-party dependencies for mobile, embedded, and desktop platforms. Supports CPU and Vulkan GPU backends, includes

★ 24,068 · C++
stas00/ml-engineering

ML-ENGINEERING OPEN BOOK

Open book on ML engineering with practical materials and examples in Python.

★ 19,717 · Python
FlashML-org/FreeToken

FreeToken: Local Model Run

FreeToken lets you run large AI models on your PC, delivering data‑center‑class speed and efficiency without needing cloud resources.

★ 17,197 · Python
openvinotoolkit/openvino

OPTIMIZATION AND DEPLOYMENT OF AI MODELS

Toolkit for optimizing and deploying deep learning models. Supports conversion from PyTorch, TensorFlow, ONNX and inference on CPU, GPU, and Intel NPU.

★ 11,656 · C++
RunanywhereAI/runanywhere-sdks

Local AI SDK Set

Production-ready toolkit for running AI models on a local device.

★ 10,350 · C++
oumi-ai/oumi

OUMI EASY LLM TUNING

OUMI lets you quickly fine‑tune (SFT/RL), evaluate, and deploy Qwen, Gemma, and any open agentic LLM/VLM without extra complexity.

★ 9,433 · Python
xorbitsai/inference

XINFERENCE - ANY LLM ONE LINE

Replace GPT with any open model by changing just one line of code. Run LLM, multimodal, and speech models via a single production-ready API.

★ 9,318 · Python
LMCache/LMCache

LMCache: LLM Acceleration

The fastest KV cache layer for large language models, boosting performance and reducing latency.

★ 8,292 · Python
mizorewww/laya-mlx

MLX-Powered Laya Decisions

A native MLX runtime for Laya typed decision models that runs short decisions in 7–14 ms on an M3 Max. It avoids text generation, PyTorch, and cloud APIs.

★ 7,102 · Python
kvcache-ai/Mooncake

MOONCAKE - KIMI LLM PLATFORM, 5400+ STARS

Mooncake is the backend platform for Kimi from Moonshot AI. High-performance open-source LLM service in C++.

★ 5,429 · C++
gpustack/gpustack

GPU cluster manager

Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.

★ 5,041 · Python
vllm-project/vllm-omni

Efficient Multimodal Inference

Framework for efficient inference of multimodal models. Unifies text, images, and audio in a single generation system.

★ 4,863 · Python
hao-ai-lab/FastVideo

UNIFIED FRAMEWORK FOR ACCELERATED VIDEO GENERATION

FastVideo is a unified framework for post-training and inference of accelerated video generation. It supports distillation, sparse attention, and real-time via Dreamverse

★ 4,831 · Python
uxlfoundation/oneDNN

ONE DNN: ACCELERATING AI ON CPU

oneDNN provides high-performance AI kernel implementations for x64, AArch64, RISC‑V, and Intel GPUs. It accelerates PyTorch, TensorFlow, OpenVINO, and ONNX Runtime using

★ 4,092 · C++
OpenCSGs/csghub

CSGHUB — PLATFORM FOR LLM

CSGHub — a new open‑source platform from OpenCSG for managing LLMs, datasets and agents. Supports on‑premise and SaaS deployments, compatible with the Python SDK of Huggi

★ 4,082 · Vue
PaddlePaddle/FastDeploy

FASTDEPLOY — FAST LLM DEPLOYMENT

High-performance toolkit for inference and deployment of LLMs and VLMs based on PaddlePaddle. 3691 ⭐

★ 3,691 · Python
llm-d/llm-d

LLM inference acceleration on Kubernetes

Achieve state of the art inference performance with modern accelerators on Kubernetes

★ 3,247 · Shell
FlashML-org/FreeVideo

Create Videos on Your PC

FreeVideo lets you generate videos using the MiniMax H3 model on modest hardware, requiring as little as 8 GB VRAM and 16 GB RAM, and automatically selects the best accel

★ 3,094 · Python
ddalcu/mlx-serve

NATIVE LLM SERVER FOR MAC

LLM server compatible with OpenAI+Anthropic, no Python required. Includes MLX Core macOS: chat, agent, and tool calling.

★ 2,472 · Zig
theagentrouter/agent-router

UNIFIED GATEWAY FOR AI SERVICES

agent-router is an Envoy Gateway‑based gateway that provides unified access to various generative AI services. Implemented in Go, it runs in Kubernetes and adheres to CNC

★ 2,404 · Go
raullenchai/Rapid-MLX

Fast Local AI Engine

Rapid-MLX is the fastest local AI engine for Apple Silicon, running 4.2× faster than Ollama and providing a cached first-token time of just 0.08s.

★ 2,185 · Python
vllm-project/vllm-ascend

VLLM-ASCEND: ASCEND PLUGIN ON GITHUB

Community plugin for vLLM on Ascend hardware from the community, written in C++. Gaining stars — already 2156!

★ 2,156 · C++
superlinked/sie

SUPERLINKED INFERENCE ENGINE

Open-source server for embedding inference, reranking, and data extraction. 2000+ stars on GitHub.

★ 2,029 · Python
mozilla-ai/any-llm

ANY-LLM: One Interface for All LLMs

Python library for unified interaction with LLM providers through a single interface.

★ 2,010 · Python
sgl-project/sglang-omni

SGLANG-OMNI: OMNI-SERVICE FOR AI

SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.

★ 1,791 · Python
StartLuxLabs/StartLux-Decision

STARTLUX‑DECISION: AI DECISIONS

Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference

★ 1,745 · Python
envoyproxy/ai-gateway

AI-GATEWAY: UNIFIED GATEWAY FOR AI

The ai-gateway project provides unified access to various generative AI services via Envoy Gateway, written in Go.

★ 1,717 · Go
RightNow-AI/picolm

TinyML LLM for cheap boards

Running a language model with 1B parameters on a $10 device with 256 MB RAM in C.

★ 1,614 · C
trymirai/uzu

UZU: Powerful Engine for AI Models

High-performance inference engine for AI models in Rust. Fast, reliable, efficient.

★ 1,586 · Rust
RunanywhereAI/trnrt

LLM Inference on AWS Trainium

Open-source LLM inference engine for AWS Trainium featuring NKI megakernels, 1M-token context, and OpenAI-compatible API.

★ 1,585 · Python
autonomous-ai/autonomous-computer

BUILD YOUR AI COMPUTER

The project helps you build your personal AI computer with open-source code and support for autonomous agents.

★ 1,577
alibaba/rtp-llm

RTP-LLM ENHANCED LLM INFERENCE

High-performance LLM inference engine from Alibaba, written in CUDA, enables fast model operation across diverse applications.

★ 1,400 · Cuda
StarlightSearch/EmbedAnything

EMBEDANYTHING - POWERFUL INFERENCE ON RUST

EmbedAnything is a high‑performance, modular, and memory‑safe framework for inference, ingestion, and indexing on Rust.

★ 1,324 · Rust
waybarrios/vllm-mlx

vllm-mlx LLM Server

OpenAI and Anthropic compatible server for Apple Silicon that runs LLMs and vision models with continuous batching and MCP tool calling, achieving 400+ tokens/s.

★ 1,260 · Python
NVIDIA/kvpress

KVPRESS: LLM CACHE COMPRESSION

Library for easy compression of LLM KV‑cache, accelerating generation and reducing GPU memory consumption.

★ 1,254 · Python
theopenco/llmgateway

LLMGATEWAY: UNIFIED API FOR ALL LLM

Manage requests to different LLM providers through a single unified interface. 1.2K ⭐ on GitHub.

★ 1,252 · TypeScript
google/agentmesh

AGENT MESH: NETWORK FRAMEWORK FOR AI AGENTS

Agent Mesh implements the A2A protocol for exchanging messages between autonomous agents on Go. The framework supports Model Context Protocol, provides tools for building

★ 1,201 · Go
jmaczan/tiny-vllm

TINY VLLM: BUILD YOUR LLM

A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.

★ 1,194 · C++
chrishayuk/larql

RUST RUNTIME FOR LLM INFERENCE AND EDITING

larql — Rust runtime to run, serve, and edit models. VINDEX3 format makes models executable and inspectable. 1173⭐ on GitHub.

★ 1,175 · Rust
usamahz/cpu-performance-engineering

From Instructions to Inference

A C reading path from one instruction to production inference, using primary sources and runnable benchmarks for each section.

★ 1,160 · C
wici-ai/inferssd

RUN MODELS LARGER THAN VRAM

inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.

★ 1,108 · C++
receptron/laya

Running Laya in Node.js

The project provides TypeScript bindings to run the open Laya decision model, compatible with Jev System-1, via ONNX Runtime in Node.js.

★ 1,074 · TypeScript
Andyyyy64/whichllm

Which local LLM to choose?

Find a local model with the best performance on your hardware based on real benchmarks, not parameter count. One command — instant launch.

★ 947 · Python
Helldez/BigMoeOnEdge

BIGMOEONEDGE - SUPER MOE ON PHONE

The project enables running 284 B Mixture‑of‑Experts models on a device with 12 GB of RAM, using only CPU and llama.cpp without loss of accuracy.

★ 855 · C++
TrenTorch/TrenTorch

ML from First Principles

Educational resource teaching machine learning concepts from first principles, covering inference, CUDA, deep learning, and reinforcement learning with Python implementat

★ 842 · Python
SharpAI/SwiftLM

SWIFTLM: SWIFT LLM ON APPLE

Native LLM inference server in Swift with MLX, OpenAI‑compatible API, SSD streaming for MoE >100B and TurboQuant KV‑cache compression.

★ 799 · Swift
fudan-generative-vision/Remory

Residual Memory Context Compaction

Remory trains a residual memory module to compact long contexts into shorter representations for transformer models.

★ 746 · Python
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
jjang-ai/mlxstudio

MLX Studio: Main JANG_Q Tool

Image generation and editing, chat and code in one place with support for OpenClaw and Anthropic API.

★ 727
kossisoroyce/timber

ML Models Compiler to C99

An AOT compiler that converts XGBoost, LightGBM, scikit-learn, CatBoost, and ONNX models into native C99 code, achieving inference speedups of up to 336× compared to Pyth

★ 680 · Python
zhongkaifu/TensorSharp

TENSORSHARP - POWERFUL .NET LLM

A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.

★ 679 · C#
Nehanth/pooled

Browser LLM Device Pool

Run large open‑source LLMs in the browser by linking multiple devices into a peer‑to‑peer compute pool, enabling chat and coding agents without any server.

★ 649 · JavaScript

New projects on this topic in Telegram

Telegram →
Inference — 95 open-source GitHub projects | REDDYX AI