Gguf: open-source projects on GitHub
47 repositories on Gguf, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
Soup: Train LLM Easily
Soup fine-tunes an LLM from a single YAML using a streaming layer — enabling an 8B model to run on a laptop with 4 GB VRAM.
★ 10,552 · PythonATOMIC AGENT: LOCAL AI
The local AI agent atomic-agent runs entirely on your device, supports long context and precise tool invocation.
★ 3,682 · TypeScriptOGAM Offline AI Swiss Knife
OGAM — chat, vision, speech, and image generation offline on your device without keys or network data.
★ 3,380 · TypeScriptMAID: MANAGE AI MODELS
Free tool for working with local and remote LLMs: llama.cpp, Anthropic, DeepSeek, etc.
★ 2,776 · TypeScriptNATIVE LLM SERVER FOR MAC
LLM server compatible with OpenAI+Anthropic, no Python required. Includes MLX Core macOS: chat, agent, and tool calling.
★ 2,472 · ZigLLM Agent Framework in ComfyUI
Framework unites LLM agents, Omost, GPT‑Sovits, ChatTTS, and OCR in ComfyUI with support for Feishu, Discord, and all popular LLMs.
★ 2,390 · PythonRUN LLM LOCALLY IN NODE.JS
Run AI models directly on your computer via Node.js. Enforced JSON schema validation right at the generation level.
★ 2,205 · TypeScriptLOCAL OPEN-SOURCE AI AGENT
Atomic-Chat is a private AI engine for running LLM locally without internet, fully offline on your computer. ⭐ 967 stars on GitHub.
★ 2,008 · TypeScriptGOLLAMA: MANAGE OLLAMA MODELS
gollama — Go utility for managing Ollama models via simple CLI. Lets you quickly list, pull, remove, and update models without extra commands. Supports Linux, macOS and w
★ 1,847 · GoSTARTLUX‑DECISION: AI DECISIONS
Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference
★ 1,745 · PythonRUN 284B LLM ON CPU WITHOUT GPU
llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto
★ 1,234 · PythonRUN MODELS LARGER THAN VRAM
inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.
★ 1,108 · C++RUN LARGE LLMS LOCALLY
PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.
★ 1,037 · C++JEVOS: BINARY DECISIONS ON LAPTOP
Jevos is an open-source Jev alternative for yes/no decisions. Runs fully locally on CPU via FastAPI. Data never leaves your device.
★ 963 · PythonWhich local LLM to choose?
Find a local model with the best performance on your hardware based on real benchmarks, not parameter count. One command — instant launch.
★ 947 · PythonLOCAL AI DRAFT HUMANIZER
The project rephrases AI-generated drafts so they read like human-written text while preserving all facts. It runs fully locally, with no data sent to the cloud, and supp
★ 935 · PythonMULTIMODAL DECISION MODEL
Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.
★ 918 · PythonLLM.SWIFT - LOCAL ACCESS TO LLM
Simple library for working with large language models locally on all Apple platforms.
★ 884 · C++BIGMOEONEDGE - SUPER MOE ON PHONE
The project enables running 284 B Mixture‑of‑Experts models on a device with 12 GB of RAM, using only CPU and llama.cpp without loss of accuracy.
★ 855 · C++JANAS: FAST LLM INFERENCE!!
janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU
★ 712 · CTENSORSHARP - POWERFUL .NET LLM
A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.
★ 679 · C#DISLLM INFERENCE ON MOJO
Runs diffusion LLMs (LLaDA, Dream) locally on CPU without GPU. Written in Mojo with GGUF format support. 289 stars on GitHub.
★ 540 · MojoRUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC
Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent
★ 534 · C++Audar-ASR-V1: Generative ASR
The project presents an Arabic-first generative speech recognition system with Flash and Turbo models, ranking first on the Open Universal Arabic ASR leaderboard. Include
★ 503 · PythonYZMA: Local LLM in Go
Integration of llama.cpp into Go applications for local inference with hardware acceleration. 496 stars on GitHub.
★ 496 · GoGERBIL: Local LLMs on Your PC
Desktop app for running large language models right on your computer. No cloud, no subscriptions — full control over AI.
★ 482 · TypeScriptDS4 on Spark: 3X Speedup
One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.
★ 415 · ShellLLAMAMPERE: LLM ACCELERATION ON RTX 3090
llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.
★ 380 · C++STRIX HALO LLM Setup
Guide for LLM on Ryzen AI MAX+ 395 and Radeon 8060S via Ollama, llama.cpp and ROCm. Achieving 100 tokens/s 30B Qwen and 140 tokens/s CHADROCK MTP.
★ 373 · PythonPMETAL: THE ACCEL LLM ON APPLE SILICON
PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele
★ 333 · RustLOCALMIND: OFFLINE LLM CHAT
Localmind — a mobile Dart/Flutter application for running local LLM models and AI servers without data leakage. Provides a fast chat, support for Gemma models, and compa
★ 326 · DartLLAMASTASH TERMINAL LLM MANAGER
llamastash — terminal-native manager for local LLMs with zero overhead. Launches, supervises, and routes GGUF models via a single OpenAI‑compatible endpoint.
★ 299 · RustTURBO LLM: OFFLINE WITH AI
Run any local LLM with automatic GPU tuning, a web UI, and an OpenAI/Anthropic‑compatible API.
★ 296 · TypeScriptJITLLM: HIGH PERFORMANCE LLM ON JVM
JITLLM — Java framework for high‑performance LLM inference and serving on the JVM. Supports GPU acceleration, GGUF and quantized model formats, and integration with DeepS
★ 295 · JavaNot a Llama, no stance
Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.
★ 284 · GoLLAMA_ROS ROS 2 LLM
The project combines llama.cpp and llava.cpp with ROS 2, allowing the use of GGUF models for real-time robotics.
★ 268 · C++LLMKUBE: LLM IN KUBERNETES
LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi
★ 266 · GoModel-Agnostic Quantization Framework
Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.
★ 254 · C++PULSAR — ULTRAFAST MOE INFERENCE
Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.
★ 238 · RustJANGQ Quantization for Apple Silicon
JANG — adaptive mixed-precision quantization of GGUF models for MLX on Apple Silicon. Requires using the JANG_Q runtime for maximum performance.
★ 233 · PythonPURE-RUST ENGINE FOR QWEN GGUF
RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.
★ 231 · RustCAMELID: LOCAL AI ON RUST
Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful
★ 221 · RustETOS-LLM-STUDIO:AI ASSISTANT ON iOS
Native LLM client iOS & Apple Watch:run local GGUF via llama.cpp or OpenAI/Claude/Gemini Local RAG, MCP tools, Siri Shortcuts, sync Built with Swift.
★ 219 · SwiftLocal-First AI Desktop Assistant
Capri is a Windows AI companion running local or API models with Hugging Face support, voice, memory, and permission-based desktop automation tools.
★ 194 · JavaScriptDeepSeek Local Setup Guide
A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi
★ 192 · ShellSelf-hosted LLM gateway
A single Go binary turns your Mac and Linux machines into a private inference cluster featuring multi‑machine routing, sharding via llama.cpp‑RPC, and support for the Ope
★ 38 · GoMEMRA - AI Accelerator for RTX
Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.
★ 6 · RustNew projects on this topic in Telegram
Telegram →