HOW MEMORY IS COMPOSED
Let's break it down by parts, because ignoring any of them leads to the model not fitting, even though it seemed to fit on paper.
1. Model weights
Main chunk. It's calculated directly: number of parameters × bytes per parameter. At FP16 each parameter weighs 2 bytes, at Q8 — about 1 byte, at Q4 — about 0.55 bytes accounting for quantization service data.
2. KV-cache
Invisible consumer. The model stores intermediate states for each context token. It grows linearly: double the context — double the cache. On large windows the cache can occupy more than the model itself.
3. Overhead
Framework buffers, activations, fragmentation. Allocate 10-15% extra. Without a margin the app will crash with OOM at the most interesting moment.
TABLE: WHAT WILL FIT ON YOUR HARDWARE
| VRAM | Comfortably | With strain | Typical card |
|---|---|---|---|
| 8 GB | 7-8B in Q4 | 14B in Q3 | RTX 3060 Ti, 4060 |
| 12 GB | 8B in Q8 / 14B in Q4 | 27B in Q3 | RTX 3060 12G, 4070 |
| 16 GB | 14B in Q4-Q5 | 32B in Q3 | RTX 4060 Ti 16G, A4000 |
| 24 GB | 32B in Q4 | 70B in Q2 | RTX 3090, 4090 |
| 48 GB | 70B in Q4 | 110B in Q3 | 2×3090, A6000 |
| 64+ GB (unified) | 70B in Q5-Q8 | over 100B | Mac Studio M-series |
Note Apple: unified memory means that 64 GB Mac Studio behaves differently than 64 GB regular RAM — GPU accesses it directly without copying.
QUANTIZATION LEVELS: WHAT WE LOSE
Quantization is weight compression by reducing numerical precision. The key question: where is the threshold after which the model noticeably dulls.
| Format | Bytes/parameter | Quality loss | When to take |
|---|---|---|---|
| FP16 | 2.0 | reference | when there is excess memory and maximum accuracy is needed |
| Q8_0 | ~1.1 | practically unnoticeable | sweet spot when memory is sufficient |
| Q6_K | ~0.8 | barely distinguishable | good compromise |
| Q5_K_M | ~0.7 | small | practical choice under shortage |
| Q4_K_M | ~0.55 | noticeable on complex tasks | de facto standard, best balance |
| Q3_K_M | ~0.45 | noticeable | only if there is no alternative |
| Q2_K | ~0.35 | strong degradation | experiments, not for work |
Practical conclusion: Q4_K_M — the point below which you shouldn't go without extreme need. A larger model in Q4 is almost always better than a smaller model in Q8 at the same memory volume.
CALCULATING KV-CACHE
Approximate formula, but sufficient for planning:
KV_кэш (ГБ) ≈ 2 × слоёв × размер_скрытого × контекст × байт / 1e9
# для 8B-модели (32 слоя, hidden 4096) при FP16:
# 2 × 32 × 4096 × 32768 × 2 / 1e9 ≈ 17 ГБ (!)
The number is frightening, and this is the reality of large contexts: the cache easily outweighs the model. Solution — quantize the cache itself, which is supported by main runtimes:
# llama.cpp — кэш в 8 бит вместо 16
./llama-server -m model.gguf -c 32768 \
--cache-type-k q8_0 --cache-type-v q8_0
# в Ollama через переменную
OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
Gives 3-3.5 times more context with almost imperceptible quality loss. One of the most underestimated techniques.
WHEN MEMORY IS STILL NOT ENOUGH
Four techniques in descending order of usefulness:
- Partial offload to CPU. Some layers on GPU, part in regular RAM. It works, but each offloaded layer hurts speed — keep as much as possible on GPU.
- KV-cache quantization. Discussed above, best technique in terms of benefit/loss ratio.
- Trim the context. Honestly ask yourself if you really need 128k. Most tasks fit with 8-16k, and memory savings are multiplicative.
- MoE models. Mixture of Experts activates only a subset of parameters per token — quality-wise like a large model, computation-wise like a small one. But the weights still need to be stored somewhere.
Fresh builds of quantized models and dynamic quants are easy to find in the REDDYX catalog — there you can see which projects around quantization are gaining traction.
CHECK BEFORE DOWNLOADING
A quick script that estimates the requirement before you spend traffic:
def vram_gb(params_b, quant_bytes=0.55, ctx=8192, layers=32, hidden=4096, kv_bytes=2):
weights = params_b * quant_bytes
kv = 2 * layers * hidden * ctx * kv_bytes / 1e9
return round((weights + kv) * 1.12, 1) # +12% overhead
print(vram_gb(14, ctx=8192)) # 14B, context 8k
print(vram_gb(14, ctx=32768)) # same 14B, context 32k
The difference between the two lines is the reason why 'it works for the neighbor, but I get OOM'.
FAQ
Which is better: 14B in Q4 or 8B in Q8 with equal memory?
Almost always 14B in Q4. More parameters matter more than the precision of their representation — this is a stable observation on most benchmarks.
Does quantization affect speed?
Yes, positively. Less data is moved across the memory bus — higher generation speed. Q4 is usually faster than FP16 on the same card.
Can two video cards be combined?
Yes, llama.cpp and vLLM can distribute layers across GPUs. Two 3090s with 24 GB each give usable 48 GB, but there is overhead for inter-card communication.
Why did the model take more memory than I calculated?
Three common reasons: forgot KV-cache, overlooked framework overhead, or the runtime reserved a buffer for max context even if you don't use it.
