R_REDDYX.XYZ
Infrastructure2026-07-19

How Much VRAM Do You Need for LLMs in 2026: Calculate Before Downloading a Model

How Much VRAM Do You Need for LLMs in 2026: Calculate Before Downloading a Model
TL;DR: Memory for LLM consists of three parts: model weights, KV-cache for context, and service overhead. Base formula for Q4: GB ≈ parameters(B) × 0.55 + context. A 14B model in Q4_K_M will take about 9 GB, plus 2-3 GB with a 32k context. Calculate before downloading — typical error is discovered after 40 GB of traffic.

HOW MEMORY IS COMPOSED

Let's break it down by parts, because ignoring any of them leads to the model not fitting, even though it seemed to fit on paper.

1. Model weights

Main chunk. It's calculated directly: number of parameters × bytes per parameter. At FP16 each parameter weighs 2 bytes, at Q8 — about 1 byte, at Q4 — about 0.55 bytes accounting for quantization service data.

2. KV-cache

Invisible consumer. The model stores intermediate states for each context token. It grows linearly: double the context — double the cache. On large windows the cache can occupy more than the model itself.

3. Overhead

Framework buffers, activations, fragmentation. Allocate 10-15% extra. Without a margin the app will crash with OOM at the most interesting moment.

TABLE: WHAT WILL FIT ON YOUR HARDWARE

VRAMComfortablyWith strainTypical card
8 GB7-8B in Q414B in Q3RTX 3060 Ti, 4060
12 GB8B in Q8 / 14B in Q427B in Q3RTX 3060 12G, 4070
16 GB14B in Q4-Q532B in Q3RTX 4060 Ti 16G, A4000
24 GB32B in Q470B in Q2RTX 3090, 4090
48 GB70B in Q4110B in Q32×3090, A6000
64+ GB (unified)70B in Q5-Q8over 100BMac Studio M-series

Note Apple: unified memory means that 64 GB Mac Studio behaves differently than 64 GB regular RAM — GPU accesses it directly without copying.

QUANTIZATION LEVELS: WHAT WE LOSE

Quantization is weight compression by reducing numerical precision. The key question: where is the threshold after which the model noticeably dulls.

FormatBytes/parameterQuality lossWhen to take
FP162.0referencewhen there is excess memory and maximum accuracy is needed
Q8_0~1.1practically unnoticeablesweet spot when memory is sufficient
Q6_K~0.8barely distinguishablegood compromise
Q5_K_M~0.7smallpractical choice under shortage
Q4_K_M~0.55noticeable on complex tasksde facto standard, best balance
Q3_K_M~0.45noticeableonly if there is no alternative
Q2_K~0.35strong degradationexperiments, not for work

Practical conclusion: Q4_K_M — the point below which you shouldn't go without extreme need. A larger model in Q4 is almost always better than a smaller model in Q8 at the same memory volume.

CALCULATING KV-CACHE

Approximate formula, but sufficient for planning:

KV_кэш (ГБ) ≈ 2 × слоёв × размер_скрытого × контекст × байт / 1e9

# для 8B-модели (32 слоя, hidden 4096) при FP16:
# 2 × 32 × 4096 × 32768 × 2 / 1e9 ≈ 17 ГБ (!)

The number is frightening, and this is the reality of large contexts: the cache easily outweighs the model. Solution — quantize the cache itself, which is supported by main runtimes:

# llama.cpp — кэш в 8 бит вместо 16
./llama-server -m model.gguf -c 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0

# в Ollama через переменную
OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Gives 3-3.5 times more context with almost imperceptible quality loss. One of the most underestimated techniques.

WHEN MEMORY IS STILL NOT ENOUGH

Four techniques in descending order of usefulness:

  • Partial offload to CPU. Some layers on GPU, part in regular RAM. It works, but each offloaded layer hurts speed — keep as much as possible on GPU.
  • KV-cache quantization. Discussed above, best technique in terms of benefit/loss ratio.
  • Trim the context. Honestly ask yourself if you really need 128k. Most tasks fit with 8-16k, and memory savings are multiplicative.
  • MoE models. Mixture of Experts activates only a subset of parameters per token — quality-wise like a large model, computation-wise like a small one. But the weights still need to be stored somewhere.

Fresh builds of quantized models and dynamic quants are easy to find in the REDDYX catalog — there you can see which projects around quantization are gaining traction.

CHECK BEFORE DOWNLOADING

A quick script that estimates the requirement before you spend traffic:

def vram_gb(params_b, quant_bytes=0.55, ctx=8192, layers=32, hidden=4096, kv_bytes=2):
    weights = params_b * quant_bytes
    kv = 2 * layers * hidden * ctx * kv_bytes / 1e9
    return round((weights + kv) * 1.12, 1)  # +12% overhead

print(vram_gb(14, ctx=8192))    # 14B, context 8k
print(vram_gb(14, ctx=32768))   # same 14B, context 32k

The difference between the two lines is the reason why 'it works for the neighbor, but I get OOM'.

FAQ

Which is better: 14B in Q4 or 8B in Q8 with equal memory?

Almost always 14B in Q4. More parameters matter more than the precision of their representation — this is a stable observation on most benchmarks.

Does quantization affect speed?

Yes, positively. Less data is moved across the memory bus — higher generation speed. Q4 is usually faster than FP16 on the same card.

Can two video cards be combined?

Yes, llama.cpp and vLLM can distribute layers across GPUs. Two 3090s with 24 GB each give usable 48 GB, but there is overhead for inter-card communication.

Why did the model take more memory than I calculated?

Three common reasons: forgot KV-cache, overlooked framework overhead, or the runtime reserved a buffer for max context even if you don't use it.

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram

← All articles

How Much VRAM Do You Need for LLMs in 2026: Calculate Before Downloading a Model | REDDYX AI