Cuda: open-source projects on GitHub
48 repositories on Cuda, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
LOCAL AI VOICE STUDIO WITH CLONING AND DUBBING
Open-source cross-platform tool for voice cloning, video dubbing, and audiobook creation in 646 languages. Runs fully locally without cloud APIs or registration.
★ 110,614 · PythonHigh-performance LLM engine
A high-performance and memory-efficient engine for inference and serving large language models.
★ 79,783 · PythonSGLang: LLM Framework
SGLang is a high-performance framework for serving large language and multimodal models.
★ 27,854 · PythonLMCache: LLM Acceleration
The fastest KV cache layer for large language models, boosting performance and reducing latency.
★ 8,292 · PythonGPU cluster manager
Management of GPU clusters with automatic configuration and orchestration of inference engines (vLLM, SGLang) for high-performance deployment of AI models.
★ 5,041 · PythonNVIDIA AGENT SKILLS
A set of skills for coding agents that enables running physical AI, robotics, simulations, CUDA, and RAG in Claude Code and Codex.
★ 4,002 · PythonOPTIMIZING ALGORITHMS IN CUDA
Guide on how to optimize algorithms on CUDA for maximum performance. ⭐️ 3091 stars
★ 3,353 · CudaPROJECT MAYA: LOCAL LLM ENGINE
Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,
★ 2,898 · C++RAMALAMA EASY AI LAUNCH
RamaLama is an open Python tool that lets you quickly run any AI models in containers and use them for inference in production.
★ 2,872 · PythonHyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonRUN LLM LOCALLY IN NODE.JS
Run AI models directly on your computer via Node.js. Enforced JSON schema validation right at the generation level.
★ 2,205 · TypeScriptLucebox Hub
High-speed server for running LLM locally on consumer hardware. Written in C++ for maximum performance.
★ 2,167 · C++ZERO-TO-SGLANG: LLM INFERENCE TRAINING
The Datawhale and SGLang course teaches LLM inference from scratch: build a mini‑SGLang, study the source code, and land your first PR. Available in English and Chinese.
★ 2,136 · PythonSGLANG-OMNI: OMNI-SERVICE FOR AI
SGLang-Omni provides high-performance execution of TTS, ASR, and speech omni-models on Python.
★ 1,791 · PythonBETA9 - Lightning GPU Output
Ultrafast serverless GPU inference with sandboxes and background jobs, written in Go for instant scaling.
★ 1,664 · GoUCCL: GPU Communication at Max
A fast library for efficient data exchange between GPUs: collective operations, P2P, and inference acceleration.
★ 1,586 · C++🔥 GPU REAL‑TIME DASHBOARD
GPU-hot — a web dashboard for monitoring NVIDIA GPU showing temperature, load, and memory in real time. Already over 1,500 stars on GitHub! 🌟
★ 1,543 · JavaScripttt-metal - Metallic Operations Library
TT-NN operator library and low-level programming model for TT-Metalium kernels to accelerate AI workloads on proprietary hardware.
★ 1,505 · C++TINY VLLM: BUILD YOUR LLM
A mini version of vLLM in C++ and CUDA for fast inference of large language models. Enables building your own high-performance inference engine.
★ 1,194 · C++RUN MODELS LARGER THAN VRAM
inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.
★ 1,108 · C++RAFT - Accelerate ML on CUDA
The RAFT library provides CUDA‑accelerated ML and IR algorithms, simplifying the creation of high‑performance applications.
★ 1,054 · CudaRUN LARGE LLMS LOCALLY
PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.
★ 1,037 · C++InferenceX – Inference Benchmark
Open-source continuous inference benchmark for Qwen3.5, DeepSeek, GPTOSS and new accelerators.
★ 1,016 · PythonCUVS - Accelerated Vector Search
cuvs is a library for fast vector search and clustering on GPU, using CUDA for maximum performance.
★ 897 · CudaML from First Principles
Educational resource teaching machine learning concepts from first principles, covering inference, CUDA, deep learning, and reinforcement learning with Python implementat
★ 842 · PythonSURROGATE — POWERFUL FINE-TUNING IN C++
Training and fine-tuning models at the speed of light. 796 stars on GitHub.
★ 796 · C++PEGAINFER: Rust + CUDA Infer
Pure Rust LLM engine with CUDA acceleration, OpenAI API compatible, runs Qwen3 and Kimi‑K2 without PyTorch.
★ 742 · RustLOCAL AI AUDIO GENERATION
audio-studio-mcp runs Stable Audio 3 on Apple Silicon via MLX and on NVIDIA GPU via PyTorch. Generates background music and sound effects for Claude Code and any MCP clie
★ 669 · PythonLoopy: CPU & GPU Generator
Loopy automatically generates bulk code for CPUs and GPUs in Python. 631 stars on GitHub.
★ 640 · PythonATORCH: Pythonic PyTorch
The attorch library offers a subset of PyTorch neural network modules implemented in Python using OpenAI's Triton for accelerated computation.
★ 607 · PythonAMADEUS: AI AGENT FOR DESKTOP
Amadeus is a real‑time multimedia desktop agent built on Python, using CUDA, Electron and GPT‑SoVITS. The project is evolving toward a persistent AI OS interface and alre
★ 554 · PythonRUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC
Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent
★ 534 · C++FLIDE: INSTANT DECISION INFERENCE
Flide is a decision model inference engine with CUDA support. Provides deterministic inference for Gemma and other models in microseconds. Ideal for AI agents requiring s
★ 501 · PythonYZMA: Local LLM in Go
Integration of llama.cpp into Go applications for local inference with hardware acceleration. 496 stars on GitHub.
★ 496 · GoEXPERIMENTAL INFERENCE LIBRARY
resin — experimental Python library for building LLM inference engines. Supports CUDA, GPU acceleration, and models like DeepSeek and GPT-OSS. Early-stage project with 25
★ 432 · PythonDS4 on Spark: 3X Speedup
One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.
★ 415 · ShellLLAMAMPERE: LLM ACCELERATION ON RTX 3090
llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.
★ 380 · C++ACCELERATE LLM WITH CACHE
The unified-cache-management project enables storing and reusing KV Cache to accelerate large language models.
★ 356 · PythonNot a Llama, no stance
Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.
★ 284 · GoInference Engine for Cybersecurity
High-performance inference engine in Rust with safe memory handling for cybersecurity tasks.
★ 240 · RustPULSAR — ULTRAFAST MOE INFERENCE
Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.
★ 238 · RustPURE-RUST ENGINE FOR QWEN GGUF
RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.
★ 231 · RustAUTOSHORTS AI CLIPS GAMEPLAY
Automatically creates vertical short clips from long gameplay recordings, analyzing scenes with AI. GPU‑accelerated rendering and optional AI voiceovers make virus‑ready
★ 207 · PythonGLYD: LOSSLESS AI COMPRESSION
Glyd reduces GPU memory by 33% without losing data, ensuring bit‑for‑bit identity. The project is written in Rust and supports CUDA, C‑ABI, and CLI for convenient benchma
★ 200 · RustNSFW Content Creation Plugin
Purge-compatible DeepSeek Harness plugin for creating adult images, voiced videos, and multi-shot films with self-hosted models, reference editing, LoRA control, and agen
★ 133 · TypeScript125B MoE Inference on RTX
Run Qwen3.8-Flash-Next (125B MoE) with 1M context on one or two RTX 3080 10GB cards. Tuned, benchmarked fork of Strata.
★ 71 · C++DVLT on CUDA without dependencies
The project ports NVIDIA DVLT to pure CUDA/C++, allowing images to be fed in and a 3D point cloud together with camera poses to be obtained. Everything is packed into a s
★ 31 · CudaMEMRA - AI Accelerator for RTX
Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.
★ 6 · RustNew projects on this topic in Telegram
Telegram →