R_REDDYX.XYZ

Quantization: open-source projects on GitHub

34 repositories on Quantization, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.

// RELATED TOPICS

LLM 2094Gguf 47LLM Inference 76Local LLM 107Fine Tuning 50Moe 23Qwen 106Deep Learning 105Vllm 69Mixture Of Experts 20Speculative Decoding 23Deepseek 229Lora 67Transformers 50
hiyouga/LlamaFactory

LlamaFactory: Unified LLM Tuning

LlamaFactory offers a unified library for efficient fine‑tuning of 100+ LLMs and VLMs, with support for LoRA, QLoRA and distributed multi‑GPU training.

★ 71,186 · Python
SYSTRAN/faster-whisper

FAST SPEECH TRANSCRIPTION

Faster Whisper — an accelerated version of the Whisper model for speech recognition, built on CTranslate2. It supports quantized models and provides real‑time transcripti

★ 26,758 · Python
FareedKhan-dev/kimi-k3-in-c

KIMI K3: 2.78 Trillion Parameters

Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.

★ 12,109 · C
bitsandbytes-foundation/bitsandbytes

BITSANDBYTES: LLM COMPRESSION

Library for k-bit quantization of large language models, enabling LLM execution on regular hardware via PyTorch.

★ 8,246 · Python
mw00/project-maya

PROJECT MAYA: LOCAL LLM ENGINE

Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,

★ 2,898 · C++
syv-ai/HyperQwen

HyperQwen: Qwen on Consumer GPUs

The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.

★ 2,713 · Python
stochasticai/xTuring

XTRUING: PERSONALIZE YOUR LLM

xTuring simplifies creating and personalizing your own LLMs: from data preparation to fine‑tuning. The tool supports adapters, ready‑made pipelines and model export. Join

★ 2,681 · Python
RightNow-AI/picolm

TinyML LLM for cheap boards

Running a language model with 1B parameters on a $10 device with 256 MB RAM in C.

★ 1,614 · C
amitshekhariitbhu/ai-engineering-interview-questions

AI Interview Cheat Sheet

A collection of questions and answers for preparing for an AI engineer interview. Covers topics in machine learning, deep learning, and system design.

★ 1,523 · Markdown
Tencent/AngelSlim

ANGELSLIM: COMPRESSING MODELS WITHOUT LOSS

A tool for model compression focusing on convenience and efficiency. 1257 stars on GitHub.

★ 1,257 · Python
synchopate/llama-modchip

RUN 284B LLM ON CPU WITHOUT GPU

llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto

★ 1,234 · Python
openvinotoolkit/nncf

NNCFF - ACCELERATING MODEL INFERENCE

NNCFF is a neural network compression framework for accelerated inference in OpenVINO™. It reduces model size while preserving accuracy.

★ 1,221 · Python
whitecircle/halo

HALO: FRAMEWORK FOR LLM TRAINING

Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an

★ 862 · Python
prabanta-dev/janas

JANAS: FAST LLM INFERENCE!!

janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU

★ 712 · C
hyperionaidev/smolmaxxing

SMOLMAXXING: LLM ON MICROCONTROLLERS

C library for running LLMs on microcontrollers and weak hardware. Alternative to llama.cpp focused on minimal memory usage. Supports quantization and various model archit

★ 691 · C
RyanCodrai/turbovec

TurboVec: Vector Index on Rust

A high-performance vector index based on TurboQuant with Python bindings for fast nearest-neighbor search.

★ 599 · Rust
Indras-Mirror/Xolotl

RUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC

Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent

★ 534 · C++
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash

QWEN3.6-27B ULTIMATE VERSION

Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.

★ 495 · Python
david-xinyuwei/david-share

AI AGENTS ON AZURE AND GPU

Repository with code, benchmarks, and guides for AI agents and Deep Learning. Deployment on Microsoft Foundry and Azure, fine-tuning, inference optimization, and GPU test

★ 451 · Jupyter Notebook
FujitsuResearch/OneCompression

ONECOMPRESSION: Lossless LLM Compression

The Python package OneCompression simplifies compression of large language models while preserving accuracy and speeding up inference.

★ 445 · Python
JakeATX/llamAmpere

LLAMAMPERE: LLM ACCELERATION ON RTX 3090

llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.

★ 380 · C++
Epistates/pmetal

PMETAL: THE ACCEL LLM ON APPLE SILICON

PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele

★ 333 · Rust
Picovoice/picollm

PICOLLM - OFFLINE LLM ON DEVICE

picollm enables running LLM on-device through extreme k‑bit quantization, delivering fast and private inference without the cloud.

★ 320 · Python
qdrant/skills

QDRANT AGENT: VECTOR SEARCH SKILLS

A set of skills for optimizing search in Qdrant: scaling, quality, monitoring, and SDK for Python, TypeScript, Rust, and more.

★ 270 · Python
huawei-csl/MACaQ

Model-Agnostic Quantization Framework

Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.

★ 254 · C++
giannisanni/pulsar

PULSAR — ULTRAFAST MOE INFERENCE

Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.

★ 238 · Rust
Yamz-Labs/kyojin

AMD Strix Halo Inference Engine

Kyojin runs ExLlamaV3 on AMD Strix Halo (ROCm gfx1151) to execute 300B‑parameter Mixture‑of‑Experts models on a single 128 GB mini PC.

★ 238 · Python
jjang-ai/jangq

JANGQ Quantization for Apple Silicon

JANG — adaptive mixed-precision quantization of GGUF models for MLX on Apple Silicon. Requires using the JANG_Q runtime for maximum performance.

★ 233 · Python
timtoole02/Camelid

CAMELID: LOCAL AI ON RUST

Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful

★ 221 · Rust
justxor/deepseek-local-guide-ru

DeepSeek Local Setup Guide

A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi

★ 192 · Shell
huawei-csl/KVarN

KVarN: KV-Cache Quantization

KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act

★ 159 · Python
cookiespiggy/agentic-rl

Agentic RL Tutorial & Benchmark

33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.

★ 146 · Python
ronitgupta138/vectra-core

Fast In-Memory Vector Search Engine

A Java 21 Loom-based engine delivering sub‑millisecond KNN search with HNSW graphs and 8‑bit quantization.

★ 65 · Java
Greninja9257/LabLLM

LabLLM on macOS

LabLLM is a native macOS app where you can build the architecture of a tiny language model, train its weights, and see the model emerge from scratch on Apple Silicon usin

★ 39 · Swift

New projects on this topic in Telegram

Telegram →
Quantization — 34 open-source GitHub projects | REDDYX AI