R_REDDYX.XYZ
Models2026-06-09

LLM Quantization in 2026: GGUF, AWQ, GPTQ Explained Simply

LLM Quantization in 2026: GGUF, AWQ, GPTQ Explained Simply
TL;DR: Quantization reduces model weight precision from 16 bits to 8/4/3 so that a 70B model fits on a single GPU instead of three. GGUF — for CPU/mixed hardware and llama.cpp, AWQ and GPTQ — for pure GPU inference. In 2026, the default is: GGUF Q4_K_M for home builds, AWQ for production on vLLM.

WHY QUANTIZATION AT ALL

LLM weights are stored by default in FP16 or BF16 — two bytes per parameter. Simple arithmetic: a 70-billion-parameter model in FP16 requires about 140 GB just for weights, plus KV-cache for context. That's three to four A100s with 40 GB each. Expensive, noisy, inaccessible from home.

Quantization is model compression by storing weights in lower precision: 8, 4, and sometimes 3 or 2 bits. The same 70B in 4-bit format takes about 35–40 GB and runs on a single RTX 4090 (24 GB) with partial offloading to RAM or on a pair of consumer cards. The key idea: neural network weights are redundant, and moving from 16 bits to 4 bits loses noticeably less quality than one might expect — according to community measurements, degradation on benchmarks such as MMLU is usually within 1–3% for 4-bit schemes.

Tracking fresh quantized releases is convenient via the catalog REDDYX — new GGUF builds of popular models appear there within hours of release.

HOW IT WORKS UNDER THE HOOD

The essence of any quantization is to map the range of float values to a small set of integers. A group of weights is taken, a scale is found (scale) and sometimes a zero-point (zero-point), and each weight is replaced by an integer plus a reconstruction formula. During inference, weights are dequantized on the fly back to float for matrix multiplications.

Format differences lie in three aspects:

  • Group size — how many weights share one scale (usually 32, 64, or 128). Smaller group — more precise, but more overhead data.
  • Activation awareness — “naive” quantization ignores which weights are actually important. Advanced methods (AWQ, GPTQ) look at calibration data and protect critical channels.
  • Mixed precision — important layers (e.g., attention) retain higher precision, while the rest is compressed more aggressively.

GGUF — FORMAT FOR llama.cpp

GGUF (successor of the outdated GGML) is not so much a quantization method as a container format of the llama.cpp and Ollama ecosystem. Inside lie the weights, metadata, tokenizer, and a dictionary of quantization schemes.

The main advantage of GGUF is flexible layer offloading between GPU and CPU. You can run a model that doesn’t fit entirely in VRAM: some layers go to VRAM, the rest is computed on the CPU and in regular RAM. That’s why GGUF is the king of home builds and Mac (Apple Silicon with unified memory).

Decoding the Suffixes

Schemes like Q4_K_M are read as: Q4 — 4 bits, K — “k-quant” with block structure and mixed precision, the size suffix — S (small), M (medium), L (large). In practice:

  • Q8_0 — almost lossless, but heavy; for checking the baseline.
  • Q5_K_M — the sweet spot if memory suffices.
  • Q4_K_M — 2026 default, best quality/size balance.
  • Q3_K_S / IQ3 — when memory is tight; noticeable degradation.
  • IQ schemes (imatrix-quant) — use the importance matrix from calibration, giving better quality at the same bit width at the cost of slightly slower generation.

GPTQ — CLASSIC GPU INFERENCE

GPTQ (2022, but still going strong) is a layer-wise quantization method that minimizes the output error of each layer, using an approximation via the inverse Hessian on a calibration set. Put simply: it doesn’t just round weights, it compensates for rounding error in the remaining weights so that the layer’s output hardly changes.

GPTQ resides entirely on GPU and requires calibration data (usually a few hundred samples from C4 or similar). Fast inference on the graphics card, broad support in exllama/exllamav2. Downside — it cannot offload to CPU; everything must fit in VRAM.

AWQ — EMPHASIS ON IMPORTANT WEIGHTS

AWQ (Activation-aware Weight Quantization) starts from the observation: not all weights are equal. About 1% of channels contribute disproportionately to quality. AWQ identifies these important channels via activation statistics and scales them so that quantization affects them less — without storing them in full precision.

In practice, AWQ in 2026 is the preferred format for production on vLLM and TGI: stable quality at 4 bits, excellent throughput under batching, good compatibility with the serving stack. GPTQ and AWQ are close in quality, but AWQ is usually a bit more robust on instruction-tuned models and easier to serve.

FORMAT COMPARISON

CriterionGGUFGPTQAWQ
Main enginellama.cpp, Ollamaexllamav2, AutoGPTQvLLM, TGI, AutoAWQ
HardwareCPU + GPU + MacOnly GPU (NVIDIA)Only GPU (NVIDIA)
Offload to RAMYes, layer-wiseNoNo
Requires calibrationNo (IQ — yes)YesYes
Throughput (batch)MediumHighHigh
Typical bits2-8 (flexible)3-4-84 (mainly)
Best use caseLocally, at home, MacSingle queries on GPUProduction-serving

It's worth remembering the new schemes for 2025-2026: FP8 (native on Hopper/Ada and newer — almost no loss, default for serving fresh cards) and experimental 2-bit quantizations, which are currently suitable mainly for giant MoE models, where even 2 bits retain reasonable quality.

PRACTICE: QUANTIZING OURSELVES

The most reproducible path — GGUF via llama.cpp. Below is a real pipeline: conversion from HF format to FP16-GGUF and subsequent compression to Q4_K_M.

# 1. Ставим llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt
cmake -B build && cmake --build build --config Release -j

# 2. Скачиваем модель с Hugging Face (пример)
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
    --local-dir ./models/llama3-8b

# 3. Конвертируем HF -> GGUF (FP16, промежуточный эталон)
python convert_hf_to_gguf.py ./models/llama3-8b \
    --outfile ./models/llama3-8b-f16.gguf \
    --outtype f16

# 4. Квантизуем FP16 -> Q4_K_M (дефолт 2026)
./build/bin/llama-quantize \
    ./models/llama3-8b-f16.gguf \
    ./models/llama3-8b-Q4_K_M.gguf \
    Q4_K_M

# 5. Проверяем: запускаем инференс
./build/bin/llama-cli \
    -m ./models/llama3-8b-Q4_K_M.gguf \
    -p "Объясни квантизацию одним предложением:" \
    -n 128 -ngl 99   # -ngl = слоёв на GPU

For AWQ, the Python pipeline is different — calibration on real data:

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "./llama3-8b-awq"

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# 4-битная квантизация, group_size=128 — стандарт
quant_config = {"w_bit": 4, "q_group_size": 128,
                "zero_point": True, "version": "GEMM"}

model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
# Дальше грузится напрямую в vLLM: --quantization awq

HOW TO CHOOSE THE FORMAT

  1. Running locally / on Mac / without a top-tier GPU → GGUF, Q4_K_M. Not enough memory — go down to IQ3, enough with spare — take Q5_K_M.
  2. One NVIDIA card, speed of single answers important → GPTQ 4-bit on exllamav2 or the same GGUF with full offload to VRAM.
  3. Production, many parallel requests, vLLM/TGI → AWQ 4-bit, or FP8 if the card is Hopper/Ada+.
  4. Need a quality reference for comparison → keep FP16 or Q8_0 as reference and measure the degradation of your quantizations on your own prompts, not just on public benchmarks.

The main rule of 2026: don't chase minimal bits for the sake of the number. The difference between Q4_K_M and Q3 in memory is small, but in quality — noticeable. Ready-made quantizations for specific hardware are often already posted — you should look in the REDDYX catalog before quantizing an 8B model for half a day yourself.

Frequently Asked Questions

Which is better — GGUF or AWQ?

Depends on hardware. GGUF is chosen for local execution on CPU, mixed GPU+RAM, and Mac thanks to layer-wise offloading. AWQ — for pure GPU inference in production on vLLM, where throughput matters with batching. In terms of quality at 4 bits, both are close.

How much quality is lost with 4-bit quantization?

According to community measurements, degradation on 4-bit schemes such as Q4_K_M or AWQ is usually around 1-3% on benchmarks like MMLU. For most tasks, this is imperceptible in live use. Significant drop starts at 3 bits and especially 2 bits.

How do GPTQ and AWQ differ?

GPTQ minimizes rounding error layer-wise via Hessian approximation. AWQ instead finds roughly 1% critical channels via activation statistics and protects them via scaling. AWQ is usually slightly more robust on instruction-tuned models and more convenient for serving.

Is a graphics card needed to run GGUF?

No. GGUF via llama.cpp works even on bare CPU, just slower. A graphics card accelerates generation: the -ngl parameter sets how many layers to offload to VRAM. On Apple Silicon, unified memory is used without a separate GPU.

Quantization formats change faster than the models themselves are released: today the default is Q4_K_M and AWQ, tomorrow FP8 and 2-bit MoE. To not miss fresh quants and tools for local runs, visit the Telegram channel REDDYX AI — new repositories every 30-60 minutes.

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram

← All articles

LLM Quantization in 2026: GGUF, AWQ, GPTQ Explained Simply | REDDYX AI