Quantization: open-source projects on GitHub
34 repositories on Quantization, picked by the REDDYX AI agent. Each has a summary, key features and a breakdown. Sorted by stars.
// RELATED TOPICS
LlamaFactory: Unified LLM Tuning
LlamaFactory offers a unified library for efficient fine‑tuning of 100+ LLMs and VLMs, with support for LoRA, QLoRA and distributed multi‑GPU training.
★ 71,186 · PythonFAST SPEECH TRANSCRIPTION
Faster Whisper — an accelerated version of the Whisper model for speech recognition, built on CTranslate2. It supports quantized models and provides real‑time transcripti
★ 26,758 · PythonKIMI K3: 2.78 Trillion Parameters
Running the Kimi K3 model with 2.78 trillion parameters on a single CPU, using only 8.24 GB of RAM without BLAS, frameworks, or GPU.
★ 12,109 · CBITSANDBYTES: LLM COMPRESSION
Library for k-bit quantization of large language models, enabling LLM execution on regular hardware via PyTorch.
★ 8,246 · PythonPROJECT MAYA: LOCAL LLM ENGINE
Project Maya lets you run GLM-5.3-Flash (321B MoE) locally on a single GPU or up to 16 GPUs, supporting NVIDIA and AMD on Linux and Windows. The package includes engine,
★ 2,898 · C++HyperQwen: Qwen on Consumer GPUs
The project optimizes running large Qwen models on available GPUs using vLLM patches and requantization. It achieves 127 tokens/s on 24 GB.
★ 2,713 · PythonXTRUING: PERSONALIZE YOUR LLM
xTuring simplifies creating and personalizing your own LLMs: from data preparation to fine‑tuning. The tool supports adapters, ready‑made pipelines and model export. Join
★ 2,681 · PythonTinyML LLM for cheap boards
Running a language model with 1B parameters on a $10 device with 256 MB RAM in C.
★ 1,614 · CAI Interview Cheat Sheet
A collection of questions and answers for preparing for an AI engineer interview. Covers topics in machine learning, deep learning, and system design.
★ 1,523 · MarkdownANGELSLIM: COMPRESSING MODELS WITHOUT LOSS
A tool for model compression focusing on convenience and efficiency. 1257 stars on GitHub.
★ 1,257 · PythonRUN 284B LLM ON CPU WITHOUT GPU
llama-modchip runs DeepSeek-V4-Flash 284B on pure 64GB CPU RAM: 44.4 tok/s prefill, 9-14 tok/s generation. 30B MoE models up to 111 tok/s. 256K context, Claude Code resto
★ 1,234 · PythonNNCFF - ACCELERATING MODEL INFERENCE
NNCFF is a neural network compression framework for accelerated inference in OpenVINO™. It reduces model size while preserving accuracy.
★ 1,221 · PythonHALO: FRAMEWORK FOR LLM TRAINING
Halo is an open-source framework from White Circle for training large language and multimodal models. It supports agent-based training, async-RL, distributed training, an
★ 862 · PythonJANAS: FAST LLM INFERENCE!!
janas runs huge LLM (up to 48 GB) on a 32 GB laptop, outpacing llama.cpp 8‑10×. On Qwen3-Next-80B it gives 37 tok/s w/o draft and 62 tok/s reading GGUF. Supports CPU+GPU
★ 712 · CSMOLMAXXING: LLM ON MICROCONTROLLERS
C library for running LLMs on microcontrollers and weak hardware. Alternative to llama.cpp focused on minimal memory usage. Supports quantization and various model archit
★ 691 · CTurboVec: Vector Index on Rust
A high-performance vector index based on TurboQuant with Python bindings for fast nearest-neighbor search.
★ 599 · RustRUNNING DEEPSEEK-V4.1 552B ON A SINGLE PC
Xolotl is an engine for running large MoE models (DeepSeek-V4.1-Flash 552B, GLM-5.3-Flash, Qwen3.8, LFM3-Omni) on a single PC with RTX 4090. It delivers 33 tok/s on agent
★ 534 · C++QWEN3.6-27B ULTIMATE VERSION
Fully uncensored, enhanced version of Qwen3.6-27B with NVFP4 and DFlash speculative decoding.
★ 495 · PythonAI AGENTS ON AZURE AND GPU
Repository with code, benchmarks, and guides for AI agents and Deep Learning. Deployment on Microsoft Foundry and Azure, fine-tuning, inference optimization, and GPU test
★ 451 · Jupyter NotebookONECOMPRESSION: Lossless LLM Compression
The Python package OneCompression simplifies compression of large language models while preserving accuracy and speeding up inference.
★ 445 · PythonLLAMAMPERE: LLM ACCELERATION ON RTX 3090
llama.cpp fork optimized for Ampere GPUs. Delivers 110+ tok/s on Qwen 3.8 at 100K context. Runs on 12 GB VRAM with 200K+ context.
★ 380 · C++PMETAL: THE ACCEL LLM ON APPLE SILICON
PMetal — Rust framework for high‑performance LLM inference on Apple Silicon. Supports local inference, LoRA/QLoRA fine‑tuning, serving, quantization, and MLX/Metal accele
★ 333 · RustPICOLLM - OFFLINE LLM ON DEVICE
picollm enables running LLM on-device through extreme k‑bit quantization, delivering fast and private inference without the cloud.
★ 320 · PythonQDRANT AGENT: VECTOR SEARCH SKILLS
A set of skills for optimizing search in Qdrant: scaling, quality, monitoring, and SDK for Python, TypeScript, Rust, and more.
★ 270 · PythonModel-Agnostic Quantization Framework
Quantize and run text, image, video, and audio models across GGUF, llama.cpp, vLLM, Diffusers, and PyTorch with state-of-the-art model-agnostic quantization.
★ 254 · C++PULSAR — ULTRAFAST MOE INFERENCE
Pulsar — an SSD-streaming engine for giant MoE models on Rust+CUDA.
★ 238 · RustAMD Strix Halo Inference Engine
Kyojin runs ExLlamaV3 on AMD Strix Halo (ROCm gfx1151) to execute 300B‑parameter Mixture‑of‑Experts models on a single 128 GB mini PC.
★ 238 · PythonJANGQ Quantization for Apple Silicon
JANG — adaptive mixed-precision quantization of GGUF models for MLX on Apple Silicon. Requires using the JANG_Q runtime for maximum performance.
★ 233 · PythonCAMELID: LOCAL AI ON RUST
Camelid — a Rust‑native backend for local inference, compatible with models via evidence‑gated compatibility. Supports Apple Silicon, GGUF, Llama and other LLMs, runs ful
★ 221 · RustDeepSeek Local Setup Guide
A Russian‑language guide that walks you through installing and running DeepSeek models (V4, R1) locally on PC, Mac, or server using tools like Ollama, llama.cpp, LM Studi
★ 192 · ShellKVarN: KV-Cache Quantization
KVarN is a native KV-cache quantization backend for vLLM, increasing context by 3-5x, raising throughput above FP16 and maintaining FP16 accuracy without calibration, act
★ 159 · PythonAgentic RL Tutorial & Benchmark
33-chapter Chinese tutorial from RL concepts to GRPO with TRL examples, plus reproducible discriminative-model benchmark comparing encoder, LLM-LoRA, and rule baselines.
★ 146 · PythonFast In-Memory Vector Search Engine
A Java 21 Loom-based engine delivering sub‑millisecond KNN search with HNSW graphs and 8‑bit quantization.
★ 65 · JavaLabLLM on macOS
LabLLM is a native macOS app where you can build the architecture of a tiny language model, train its weights, and see the model emerge from scratch on Apple Silicon usin
★ 39 · SwiftNew projects on this topic in Telegram
Telegram →