Why this matters
GPU cost is the main expense item for ML teams in production. A 2x throughput difference equals a 2x difference in infrastructure cost. At 10M requests per day, this amounts to hundreds of thousands of dollars per year.
Test configuration
| Parameter | Value |
|---|---|
| GPU | NVIDIA A100 80GB SXM |
| Model | Llama 3.1 8B, fp16 |
| Input length | 512 tokens |
| Output length | 128 tokens |
| Concurrent requests | 1, 8, 32, 128 |
Throughput results
| Concurrency | vLLM req/s | TGI req/s | Difference |
|---|---|---|---|
| 1 | 12.4 | 11.8 | +5% |
| 8 | 48.2 | 38.1 | +26% |
| 32 | 156.3 | 98.4 | +59% |
| 128 | 312.7 | 136.2 | +130% |
Why vLLM is faster
PagedAttention — the key difference of vLLM. Instead of static allocation of KV-cache for maximum context, vLLM allocates in pages of 16 tokens. Fragmentation drops from 60-80% to less than 4%. Continuous batching adds new requests to the batch without waiting.
Running vLLM
pip install vllm
# Простой запуск OpenAI-совместимого сервера
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--max-model-len 4096 \
--tensor-parallel-size 1 \
--port 8000
Running TGI
docker run --gpus all -p 8080:80 \
-v /data:/data \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id meta-llama/Llama-3.1-8B-Instruct \
--max-input-length 4096
FAQ
Does vLLM work without GPU?
Yes, via --device cpu, but performance drops by an order of magnitude. For production without GPU use llama.cpp or Ollama.
What quantization is sufficient for production?
AWQ 4-bit results in less than 1% quality loss on most benchmarks with 2x memory reduction. GPTQ is slightly worse in quality but quantizes faster.