R_REDDYX.XYZ

Cuda: open-source projects on GitHub

48 repositories on Cuda, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Inference 95Qwen 106LLM Inference 76Moe 23Gguf 47GPU 48Pytorch 98Deepseek 229Llama 106Vllm 69Local LLM 107Speculative Decoding 23Llama Cpp 56
debpalash/VoiceStudio

LOCAL AI VOICE STUDIO WITH CLONING AND DUBBING

Open-source cross-platform tool for voice cloning, video dubbing, and audiobook creation in 646 languages. Runs fully locally without cloud APIs or registration.

★ 110,614 · Python
vllm-project/vllm

High-performance LLM engine

A high-performance and memory-efficient engine for inference and serving large language models.

★ 79,783 · Python
sgl-project/sglang

SGLang: LLM Framework

SGLang is a high-performance framework for serving large language and multimodal models.

★ 27,854 · Python
LMCache/LMCache

LMCache: LLM Acceleration

The fastest KV cache layer for large language models, boosting performance and reducing latency.

★ 8,292 · Python
gpustack/gpustack

GPU cluster manager

Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.

★ 5,041 · Python
NVIDIA/skills

NVIDIA AGENT SKILLS

A set of skills for coding agents that enables running physical AI, robotics, simulations, CUDA, and RAG in Claude Code and Codex.

★ 4,002 · Python
BBuf/how-to-optim-algorithm-in-cuda

OPTIMIZING ALGORITHMS IN CUDA

Guide on how to optimize algorithms on CUDA for maximum performance. ⭐️ 3091 stars

★ 3,353 · Cuda
mw00/project-maya

PROJECT MAYA: LOCAL LLM ENGINE

Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,

★ 2,898 · C++
containers/ramalama

RAMALAMA EASY AI LAUNCH

RamaLama is an open Python tool that lets you quickly run any AI models in containers and use them for inference in production.

★ 2,872 · Python
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
withcatai/node-llama-cpp

RUN LLM LOCALLY IN NODE.JS

Run AI models directly on your computer via Node.js. Enforced JSON schema validation right at the generation level.

★ 2,205 · TypeScript
Luce-Org/lucebox-hub

Lucebox Hub

High-speed server for running LLM locally on consumer hardware. Written in C++ for maximum performance.

★ 2,167 · C++
datawhalechina/zero-to-sglang

ZERO-TO-SGLANG: LLM INFERENCE TRAINING

The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.

★ 2,136 · Python
sgl-project/sglang-omni

SGLANG-OMNI: OMNI-SERVICE FOR AI

SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.

★ 1,791 · Python
beam-cloud/beta9

BETA9 - Lightning GPU Output

Ultrafast serverless GPU inference with sandboxes and background jobs, written in Go for instant scaling.

★ 1,664 · Go
uccl-project/uccl

UCCL: GPU Communication at Max

A fast library for efficient data exchange between GPUs: collective operations, P2P, and inference acceleration.

★ 1,586 · C++
psalias2006/gpu-hot

🔥 GPU REAL‑TIME DASHBOARD

GPU-hot — a web dashboard for monitoring NVIDIA GPU showing temperature, load, and memory in real time. Already over 1,500 stars on GitHub! 🌟

★ 1,543 · JavaScript
tenstorrent/tt-metal

tt-metal - Metallic Operations Library

TT-NN operator library and low-level programming model for TT-Metalium kernels to accelerate AI workloads on proprietary hardware.

★ 1,505 · C++
jmaczan/tiny-vllm

TINY VLLM: BUILD YOUR LLM

A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.

★ 1,194 · C++
wici-ai/inferssd

RUN MODELS LARGER THAN VRAM

inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.

★ 1,108 · C++
NVIDIA/raft

RAFT - Accelerate ML on CUDA

The RAFT library provides CUDA‑accelerated ML and IR algorithms, simplifying the creation of high‑performance applications.

★ 1,054 · Cuda
simplexai-labs/PolyStrata

RUN LARGE LLMS LOCALLY

PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.

★ 1,037 · C++
SemiAnalysisAI/InferenceX

InferenceX – Inference Benchmark

Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.

★ 1,016 · Python
NVIDIA/cuvs

CUVS - Accelerated Vector Search

cuvs is a library for fast vector search and clustering on GPU, using CUDA for maximum performance.

★ 897 · Cuda
TrenTorch/TrenTorch

ML from First Principles

Educational resource teaching machine learning concepts from first principles, covering inference, CUDA, deep learning, and reinforcement learning with Python implementat

★ 842 · Python
invergent-ai/surogate

SURROGATE — POWERFUL FINE-TUNING IN C++

Training and fine-tuning models at the speed of light. 796 stars on GitHub.

★ 796 · C++
pegainfer-project/pegainfer

PEGAINFER: Rust + CUDA Infer

Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.

★ 742 · Rust
kanaworksai-hue/audio-studio-mcp

LOCAL AI AUDIO GENERATION

audio-studio-mcp runs Stable Audio 3 on Apple Silicon via MLX and on NVIDIA GPU via PyTorch. Generates background music and sound effects for Claude Code and any MCP clie

★ 669 · Python
inducer/loopy

Loopy: CPU & GPU Generator

Loopy automatically generates bulk code for CPUs and GPUs in Python. 631 stars on GitHub.

★ 640 · Python
BobMcDear/attorch

ATORCH: Pythonic PyTorch

The attorch library offers a subset of PyTorch neural network modules implemented in Python using OpenAI's Triton for accelerated computation.

★ 607 · Python
Code-Amadeus/Amadeus

AMADEUS: AI AGENT FOR DESKTOP

Amadeus is a real‑time multimedia desktop agent built on Python, using CUDA, Electron and GPT‑SoVITS. The project is evolving toward a persistent AI OS interface and alre

★ 554 · Python
Indras-Mirror/Xolotl

RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC

Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent

★ 534 · C++
OmniJev/Flide

FLIDE: INSTANT DECISION INFERENCE

Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s

★ 501 · Python
hybridgroup/yzma

YZMA: Local LLM in Go

Integration of llama.cpp into Go applications for local inference with hardware acceleration. 496 stars on GitHub.

★ 496 · Go
ekzhang/resin

EXPERIMENTAL INFERENCE LIBRARY

resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25

★ 432 · Python
Entrpi/ds4-on-spark

DS4 on Spark: 3X Speedup

One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.

★ 415 · Shell
JakeATX/llamAmpere

LLAMAMPERE: LLM ACCELERATION ON RTX 3090

llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.

★ 380 · C++
ModelEngine-Group/unified-cache-management

ACCELERATE LLM WITH CACHE

The unified-cache-management project enables storing and reusing KV Cache to accelerate large language models.

★ 356 · Python
raketenkater/ggrun

Not a Llama, no stance

Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.

★ 284 · Go
Avarok-Cybersecurity/atlas

Inference Engine for Cybersecurity

High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.

★ 240 · Rust
giannisanni/pulsar

PULSAR — ULTRAFAST MOE INFERENCE

Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.

★ 238 · Rust
RustingStudio/RustingStrata

PURE-RUST ENGINE FOR QWEN GGUF

RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.

★ 231 · Rust
divyaprakash0426/autoshorts

AUTOSHORTS AI CLIPS GAMEPLAY

Automatically creates vertical short clips from long gameplay recordings, analyzing scenes with AI. GPU‑accelerated rendering and optional AI voiceovers make virus‑ready

★ 207 · Python
surya-koritala/Glyd

GLYD: LOSSLESS AI COMPRESSION

Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma

★ 200 · Rust
kingcharleslzy-ai/DSH-Spicy

NSFW Content Creation Plugin

Purge-compatible DeepSeek Harness plugin for creating adult images, voiced videos, and multi-shot films with self-hosted models, reference editing, LoRA control, and agen

★ 133 · TypeScript
DevDock-AI/Speedy

125B MoE Inference on RTX

Run Qwen3.8-Flash-Next (125B MoE) with 1M context on one or two RTX 3080 10GB cards. Tuned, benchmarked fork of Strata.

★ 71 · C++
yassa9/dvlt.cu

DVLT on CUDA without dependencies

The project ports NVIDIA DVLT to pure CUDA/C++, allowing images to be fed in and a 3D point cloud together with camera poses to be obtained. Everything is packed into a s

★ 31 · Cuda
avifenesh/memra

MEMRA - AI Accelerator for RTX

Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.

★ 6 · Rust

New projects on this topic in Telegram

Telegram →
Cuda — 48 open-source GitHub projects | REDDYX AI