WHY QUANTIZATION AT ALL
LLM weights are stored by default in FP16 or BF16 — two bytes per parameter. Simple arithmetic: a 70-billion-parameter model in FP16 requires about 140 GB just for weights, plus KV-cache for context. That's three to four A100s with 40 GB each. Expensive, noisy, inaccessible from home.
Quantization is model compression by storing weights in lower precision: 8, 4, and sometimes 3 or 2 bits. The same 70B in 4-bit format takes about 35–40 GB and runs on a single RTX 4090 (24 GB) with partial offloading to RAM or on a pair of consumer cards. The key idea: neural network weights are redundant, and moving from 16 bits to 4 bits loses noticeably less quality than one might expect — according to community measurements, degradation on benchmarks such as MMLU is usually within 1–3% for 4-bit schemes.
Tracking fresh quantized releases is convenient via the catalog REDDYX — new GGUF builds of popular models appear there within hours of release.
HOW IT WORKS UNDER THE HOOD
The essence of any quantization is to map the range of float values to a small set of integers. A group of weights is taken, a scale is found (scale) and sometimes a zero-point (zero-point), and each weight is replaced by an integer plus a reconstruction formula. During inference, weights are dequantized on the fly back to float for matrix multiplications.
Format differences lie in three aspects:
- Group size — how many weights share one scale (usually 32, 64, or 128). Smaller group — more precise, but more overhead data.
- Activation awareness — “naive” quantization ignores which weights are actually important. Advanced methods (AWQ, GPTQ) look at calibration data and protect critical channels.
- Mixed precision — important layers (e.g., attention) retain higher precision, while the rest is compressed more aggressively.
GGUF — FORMAT FOR llama.cpp
GGUF (successor of the outdated GGML) is not so much a quantization method as a container format of the llama.cpp and Ollama ecosystem. Inside lie the weights, metadata, tokenizer, and a dictionary of quantization schemes.
The main advantage of GGUF is flexible layer offloading between GPU and CPU. You can run a model that doesn’t fit entirely in VRAM: some layers go to VRAM, the rest is computed on the CPU and in regular RAM. That’s why GGUF is the king of home builds and Mac (Apple Silicon with unified memory).
Decoding the Suffixes
Schemes like Q4_K_M are read as: Q4 — 4 bits, K — “k-quant” with block structure and mixed precision, the size suffix — S (small), M (medium), L (large). In practice:
- Q8_0 — almost lossless, but heavy; for checking the baseline.
- Q5_K_M — the sweet spot if memory suffices.
- Q4_K_M — 2026 default, best quality/size balance.
- Q3_K_S / IQ3 — when memory is tight; noticeable degradation.
- IQ schemes (imatrix-quant) — use the importance matrix from calibration, giving better quality at the same bit width at the cost of slightly slower generation.
GPTQ — CLASSIC GPU INFERENCE
GPTQ (2022, but still going strong) is a layer-wise quantization method that minimizes the output error of each layer, using an approximation via the inverse Hessian on a calibration set. Put simply: it doesn’t just round weights, it compensates for rounding error in the remaining weights so that the layer’s output hardly changes.
GPTQ resides entirely on GPU and requires calibration data (usually a few hundred samples from C4 or similar). Fast inference on the graphics card, broad support in exllama/exllamav2. Downside — it cannot offload to CPU; everything must fit in VRAM.
AWQ — EMPHASIS ON IMPORTANT WEIGHTS
AWQ (Activation-aware Weight Quantization) starts from the observation: not all weights are equal. About 1% of channels contribute disproportionately to quality. AWQ identifies these important channels via activation statistics and scales them so that quantization affects them less — without storing them in full precision.
In practice, AWQ in 2026 is the preferred format for production on vLLM and TGI: stable quality at 4 bits, excellent throughput under batching, good compatibility with the serving stack. GPTQ and AWQ are close in quality, but AWQ is usually a bit more robust on instruction-tuned models and easier to serve.
FORMAT COMPARISON
| Criterion | GGUF | GPTQ | AWQ |
|---|---|---|---|
| Main engine | llama.cpp, Ollama | exllamav2, AutoGPTQ | vLLM, TGI, AutoAWQ |
| Hardware | CPU + GPU + Mac | Only GPU (NVIDIA) | Only GPU (NVIDIA) |
| Offload to RAM | Yes, layer-wise | No | No |
| Requires calibration | No (IQ — yes) | Yes | Yes |
| Throughput (batch) | Medium | High | High |
| Typical bits | 2-8 (flexible) | 3-4-8 | 4 (mainly) |
| Best use case | Locally, at home, Mac | Single queries on GPU | Production-serving |
It's worth remembering the new schemes for 2025-2026: FP8 (native on Hopper/Ada and newer — almost no loss, default for serving fresh cards) and experimental 2-bit quantizations, which are currently suitable mainly for giant MoE models, where even 2 bits retain reasonable quality.
PRACTICE: QUANTIZING OURSELVES
The most reproducible path — GGUF via llama.cpp. Below is a real pipeline: conversion from HF format to FP16-GGUF and subsequent compression to Q4_K_M.
# 1. Ставим llama.cpp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt
cmake -B build && cmake --build build --config Release -j
# 2. Скачиваем модель с Hugging Face (пример)
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct \
--local-dir ./models/llama3-8b
# 3. Конвертируем HF -> GGUF (FP16, промежуточный эталон)
python convert_hf_to_gguf.py ./models/llama3-8b \
--outfile ./models/llama3-8b-f16.gguf \
--outtype f16
# 4. Квантизуем FP16 -> Q4_K_M (дефолт 2026)
./build/bin/llama-quantize \
./models/llama3-8b-f16.gguf \
./models/llama3-8b-Q4_K_M.gguf \
Q4_K_M
# 5. Проверяем: запускаем инференс
./build/bin/llama-cli \
-m ./models/llama3-8b-Q4_K_M.gguf \
-p "Объясни квантизацию одним предложением:" \
-n 128 -ngl 99 # -ngl = слоёв на GPU
For AWQ, the Python pipeline is different — calibration on real data:
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Llama-3.1-8B-Instruct"
quant_path = "./llama3-8b-awq"
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
# 4-битная квантизация, group_size=128 — стандарт
quant_config = {"w_bit": 4, "q_group_size": 128,
"zero_point": True, "version": "GEMM"}
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
# Дальше грузится напрямую в vLLM: --quantization awq
HOW TO CHOOSE THE FORMAT
- Running locally / on Mac / without a top-tier GPU → GGUF, Q4_K_M. Not enough memory — go down to IQ3, enough with spare — take Q5_K_M.
- One NVIDIA card, speed of single answers important → GPTQ 4-bit on exllamav2 or the same GGUF with full offload to VRAM.
- Production, many parallel requests, vLLM/TGI → AWQ 4-bit, or FP8 if the card is Hopper/Ada+.
- Need a quality reference for comparison → keep FP16 or Q8_0 as reference and measure the degradation of your quantizations on your own prompts, not just on public benchmarks.
The main rule of 2026: don't chase minimal bits for the sake of the number. The difference between Q4_K_M and Q3 in memory is small, but in quality — noticeable. Ready-made quantizations for specific hardware are often already posted — you should look in the REDDYX catalog before quantizing an 8B model for half a day yourself.
Frequently Asked Questions
Which is better — GGUF or AWQ?
Depends on hardware. GGUF is chosen for local execution on CPU, mixed GPU+RAM, and Mac thanks to layer-wise offloading. AWQ — for pure GPU inference in production on vLLM, where throughput matters with batching. In terms of quality at 4 bits, both are close.
How much quality is lost with 4-bit quantization?
According to community measurements, degradation on 4-bit schemes such as Q4_K_M or AWQ is usually around 1-3% on benchmarks like MMLU. For most tasks, this is imperceptible in live use. Significant drop starts at 3 bits and especially 2 bits.
How do GPTQ and AWQ differ?
GPTQ minimizes rounding error layer-wise via Hessian approximation. AWQ instead finds roughly 1% critical channels via activation statistics and protects them via scaling. AWQ is usually slightly more robust on instruction-tuned models and more convenient for serving.
Is a graphics card needed to run GGUF?
No. GGUF via llama.cpp works even on bare CPU, just slower. A graphics card accelerates generation: the -ngl parameter sets how many layers to offload to VRAM. On Apple Silicon, unified memory is used without a separate GPU.