R_REDDYX.XYZ

Gguf: open-source projects on GitHub

47 repositories on Gguf, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Llama Cpp 56Local LLM 107Inference 95Ollama 249Cuda 48Self Hosted 359Quantization 34Apple Silicon 56Metal 32LLM Inference 76Mixture Of Experts 20Moe 23Local AI 82
MakazhanAlpamys/Soup

Soup: Train LLM Easily

Soup fine-tunes an LLM from a single YAML using a streaming layer — enabling an 8B model to run on a laptop with 4 GB VRAM.

★ 10,552 · Python
AtomicBot-ai/atomic-agent

ATOMIC AGENT: LOCAL AI

The local AI agent atomic-agent runs entirely on your device, supports long context and precise tool invocation.

★ 3,682 · TypeScript
off-grid-ai/OGAM

OGAM Offline AI Swiss Knife

OGAM — chat, vision, speech, and image generation offline on your device without keys or network data.

★ 3,380 · TypeScript
Mobile-Artificial-Intelligence/maid

MAID: MANAGE AI MODELS

Free tool for working with local and remote LLMs: llama.cpp, Anthropic, DeepSeek, etc.

★ 2,776 · TypeScript
ddalcu/mlx-serve

NATIVE LLM SERVER FOR MAC

LLM server compatible with OpenAI+Anthropic, no Python required. Includes MLX Core macOS: chat, agent, and tool calling.

★ 2,472 · Zig
heshengtao/comfyui_LLM_party

LLM Agent Framework in ComfyUI

Framework unites LLM agents, Omost, GPT‑Sovits, ChatTTS, and OCR in ComfyUI with support for Feishu, Discord, and all popular LLMs.

★ 2,390 · Python
withcatai/node-llama-cpp

RUN LLM LOCALLY IN NODE.JS

Run AI models directly on your computer via Node.js. Enforced JSON schema validation right at the generation level.

★ 2,205 · TypeScript
AtomicBot-ai/Atomic-Chat

LOCAL OPEN-SOURCE AI AGENT

Atomic-Chat is a private AI engine for running LLM locally without internet, fully offline on your computer. ⭐ 967 stars on GitHub.

★ 2,008 · TypeScript
sammcj/gollama

GOLLAMA: MANAGE OLLAMA MODELS

gollama — Go utility for managing Ollama models via simple CLI. Lets you quickly list, pull, remove, and update models without extra commands. Supports Linux, macOS and w

★ 1,847 · Go
StartLuxLabs/StartLux-Decision

STARTLUX‑DECISION: AI DECISIONS

Open decision models from 0.8B to 27B give calibrated probability for each option in one pass and a short thought only when unsure. They serve as the available inference

★ 1,745 · Python
synchopate/llama-modchip

RUN 284B LLM ON CPU WITHOUT GPU

llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto

★ 1,234 · Python
wici-ai/inferssd

RUN MODELS LARGER THAN VRAM

inferssd runs MoE models exceeding RAM and VRAM by streaming experts from SSD. Built on llama.cpp with GGUF, CUDA and Apple Silicon support. 288 stars on GitHub.

★ 1,108 · C++
simplexai-labs/PolyStrata

RUN LARGE LLMS LOCALLY

PolyStrata runs Qwen3.8-Flash, GLM 5.3 Flash, DeepSeek V4 and more on your PC. One-click install for Windows and Linux with OpenAI/Anthropic API on localhost.

★ 1,037 · C++
feder-cr/jev

JEVOS: BINARY DECISIONS ON LAPTOP

Jevos is an open-source Jev alternative for yes/no decisions. Runs fully locally on CPU via FastAPI. Data never leaves your device.

★ 963 · Python
Andyyyy64/whichllm

Which local LLM to choose?

Find a local model with the best performance on your hardware based on real benchmarks, not parameter count. One command — instant launch.

★ 947 · Python
sgaofen/humanizer-local-model

LOCAL AI DRAFT HUMANIZER

The project rephrases AI-generated drafts so they read like human-written text while preserving all facts. It runs fully locally, with no data sent to the cloud, and supp

★ 935 · Python
Valen-Team/Valen

MULTIMODAL DECISION MODEL

Valen-Decision is a multimodal System One decision model. It takes text, images, and video, returning probabilities over supplied candidates and bounding box coordinates.

★ 918 · Python
eastriverlee/LLM.swift

LLM.SWIFT - LOCAL ACCESS TO LLM

Simple library for working with large language models locally on all Apple platforms.

★ 884 · C++
Helldez/BigMoeOnEdge

BIGMOEONEDGE - SUPER MOE ON PHONE

The project enables running 284 B Mixture‑of‑Experts models on a device with 12 GB of RAM, using only CPU and llama.cpp without loss of accuracy.

★ 855 · C++
prabanta-dev/janas

JANAS: FAST LLM INFERENCE!!

janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU

★ 712 · C
zhongkaifu/TensorSharp

TENSORSHARP - POWERFUL .NET LLM

A library for quickly launching GGUF models on .NET with GPU support, console app, web chat, and an API compatible with Ollama and OpenAI.

★ 679 · C#
vladimirrott/mojo-dllm

DISLLM INFERENCE ON MOJO

Runs diffusion LLMs (LLaDA, Dream) locally on CPU without GPU. Written in Mojo with GGUF format support. 289 stars on GitHub.

★ 540 · Mojo
Indras-Mirror/Xolotl

RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC

Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent

★ 534 · C++
AudarAI/Audar-ASR-V1

Audar-ASR-V1: Generative ASR

The project presents an Arabic-first generative speech recognition system with Flash and Turbo models, ranking first on the Open Universal Arabic ASR leaderboard. Include

★ 503 · Python
hybridgroup/yzma

YZMA: Local LLM in Go

Integration of llama.cpp into Go applications for local inference with hardware acceleration. 496 stars on GitHub.

★ 496 · Go
lone-cloud/gerbil

GERBIL: Local LLMs on Your PC

Desktop app for running large language models right on your computer. No cloud, no subscriptions — full control over AI.

★ 482 · TypeScript
Entrpi/ds4-on-spark

DS4 on Spark: 3X Speedup

One‑command installation of the ds4 fork on NVIDIA DGX Spark with ~3× prefill speedup and ~1.5× decode speedup.

★ 415 · Shell
JakeATX/llamAmpere

LLAMAMPERE: LLM ACCELERATION ON RTX 3090

llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.

★ 380 · C++
hogeheer499-commits/strix-halo-guide

STRIX HALO LLM Setup

Guide for LLM on Ryzen AI MAX+ 395 and Radeon 8060S via Ollama, llama.cpp and ROCm. Achieving 100 tokens/s 30B Qwen and 140 tokens/s CHADROCK MTP.

★ 373 · Python
Epistates/pmetal

PMETAL: THE ACCEL LLM ON APPLE SILICON

PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele

★ 333 · Rust
abdulmominsakib/localmind

LOCALMIND: OFFLINE LLM CHAT

Localmind — a mobile Dart/Flutter application for running local LLM models and AI servers without data leakage. Provides a fast chat, support for Gemma models, and compa

★ 326 · Dart
llamastash/llamastash

LLAMASTASH TERMINAL LLM MANAGER

llamastash — terminal-native manager for local LLMs with zero overhead. Launches, supervises, and routes GGUF models via a single OpenAI‑compatible endpoint.

★ 299 · Rust
mohitsoni48/TurboLLM

TURBO LLM: OFFLINE WITH AI

Run any local LLM with automatic GPU tuning, a web UI, and an OpenAI/Anthropic‑compatible API.

★ 296 · TypeScript
beehive-lab/jitllm

JITLLM: HIGH PERFORMANCE LLM ON JVM

JITLLM — Java framework for high‑performance LLM inference and serving on the JVM. Supports GPU acceleration, GGUF and quantized model formats, and integration with DeepS

★ 295 · Java
raketenkater/ggrun

Not a Llama, no stance

Auto-tuned launcher for GGUF models on llama.cpp — server with multi‑GPU support, hardware‑specific tuning, and crash recovery. Stars: 226.

★ 284 · Go
mgonzs13/llama_ros

LLAMA_ROS ROS 2 LLM

The project combines llama.cpp and llava.cpp with ROS 2, allowing the use of GGUF models for real-time robotics.

★ 268 · C++
defilantech/LLMKube

LLMKUBE: LLM IN KUBERNETES

LLMKube - Kubernetes operator for LLM inference on heterogeneous GPU fleet: NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal. Supports llama.cpp, vLLm, TGI, mlx-server, multi

★ 266 · Go
huawei-csl/MACaQ

Model-Agnostic Quantization Framework

Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.

★ 254 · C++
giannisanni/pulsar

PULSAR — ULTRAFAST MOE INFERENCE

Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.

★ 238 · Rust
jjang-ai/jangq

JANGQ Quantization for Apple Silicon

JANG — adaptive mixed-precision quantization of GGUF models for MLX on Apple Silicon. Requires using the JANG_Q runtime for maximum performance.

★ 233 · Python
RustingStudio/RustingStrata

PURE-RUST ENGINE FOR QWEN GGUF

RustingStrata is a pure-Rust inference engine for Qwen GGUF (Qwen3–3.8, MoE up to 770B+). Supports CUDA/Vulkan, hybrid CPU+GPU experts, MTP speculative decoding. Up to 1.

★ 231 · Rust
timtoole02/Camelid

CAMELID: LOCAL AI ON RUST

Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful

★ 221 · Rust
Eric-Terminal/ETOS-LLM-Studio

ETOS-LLM-STUDIO:AI ASSISTANT ON iOS

Native LLM client iOS & Apple Watch:run local GGUF via llama.cpp or OpenAI/Claude/Gemini Local RAG, MCP tools, Siri Shortcuts, sync Built with Swift.

★ 219 · Swift
CapriChooms/CapriCore

Local-First AI Desktop Assistant

Capri is a Windows AI companion running local or API models with Hugging Face support, voice, memory, and permission-based desktop automation tools.

★ 194 · JavaScript
justxor/deepseek-local-guide-ru

DeepSeek Local Setup Guide

A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi

★ 192 · Shell
hadihonarvar/flock

Self-hosted LLM gateway

A single Go binary turns your Mac and Linux machines into a private inference cluster featuring multi‑machine routing, sharding via llama.cpp‑RPC, and support for the Ope

★ 38 · Go
avifenesh/memra

MEMRA - AI Accelerator for RTX

Memra is a from‑scratch LLM inference implementation optimized for RTX 5090 (sm_120a) and H100 (sm_90a) architectures in Rust.

★ 6 · Rust

New projects on this topic in Telegram

Telegram →
Gguf — 47 open-source GitHub projects | REDDYX AI