R_REDDYX.XYZ
Infrastructure2026-06-19

Accelerating Inference in 2026: ONNX, TensorRT, and vLLM in Practice

Accelerating Inference in 2026: ONNX, TensorRT, and vLLM in Practice
TL;DR: In 2026, the choice of stack for inference acceleration decides everything: ONNX Runtime — universal glue and top pick for classics and small models on CPU/edge, TensorRT — maximum latency optimization on NVIDIA GPU, vLLM — throughput king for LLMs thanks to PagedAttention and continuous batching. There is no single "best" — the right combo depends on what you're running: a BERT classifier, diffusion, or a 70B chat.

WHY INFERENCE SUDDENLY BECAME THE MAIN COST ITEM

Training is a one-time event. Inference runs 24/7 and pays for itself with every query. By 2026, costs for most teams have shifted specifically to serving: the model is already trained or taken off-the-shelf, and the question becomes — how many tokens per second can you squeeze out of a rented H100 at $2–3 per hour.

The difference between naive model.generate() in pure PyTorch and a properly built pipeline on TensorRT or vLLM is not 10–20%. It's multiples. By community benchmarks, continuous batching in vLLM gives a multiple increase in throughput against static batching, and TensorRT on encoder transformers regularly cuts latency by 2–5x relative to eager mode. Ignoring optimization means overpaying for hardware by multiples. Fresh tool and engine releases are tracked in the REDDYX catalog.

ONNX: THE UNIVERSAL ADAPTER

ONNX (Open Neural Network Exchange) is a format-contract. You trained a model in PyTorch, but run it in a C# backend, on mobile, or on a CPU-only server — ONNX breaks the coupling "training framework = runtime framework".

Where ONNX Runtime Actually Wins

  • CPU and edge. For classic models, embedders, small encoders, ORT on CPU is often faster and lighter than dragging all of PyTorch.
  • Execution providers. The same .onnx runs via CPUExecutionProvider, CUDA, TensorRT, OpenVINO, DirectML — code barely changes.
  • Cross-language. Python, C++, C#, Java, JS/WASM in the browser.

Basic export and run look like this:

import torch, onnxruntime as ort
import numpy as np

# экспорт из PyTorch
dummy = torch.randn(1, 3, 224, 224)
torch.onnx.export(
    model, dummy, "model.onnx",
    input_names=["input"], output_names=["logits"],
    dynamic_axes={"input": {0: "batch"}, "logits": {0: "batch"}},
    opset_version=18,
)

# инференс через ORT с приоритетом TensorRT -> CUDA -> CPU
sess = ort.InferenceSession(
    "model.onnx",
    providers=["TensorrtExecutionProvider", "CUDAExecutionProvider", "CPUExecutionProvider"],
)
out = sess.run(None, {"input": np.random.randn(8, 3, 224, 224).astype(np.float32)})
print(out[0].shape)

The key moment is dynamic_axes. Without it, the graph will fix batch=1, and you'll hit a wall when batching. Specify dynamics for batch and sequence right away.

TENSORRT: WHEN YOU NEED MINIMAL LATENCY ON GPU

TensorRT is an NVIDIA engine that takes a graph and recompiles it for a specific GPU architecture: fuses layers (layer fusion), selects optimal kernels for your dimensions (kernel auto-tuning), throws out unnecessary transposes. The result is a "flat" optimized engine tailored to one card.

Strengths and Weaknesses

  • Pro: best latency on single and small batches, excellent FP16/INT8/FP8 support on Ada and Hopper.
  • Pro: for generative LLMs there is TensorRT-LLM with in-flight batching and paged KV-cache.
  • Con: engine is tied to GPU architecture and often to version — built for H100 won't move to A100 without rebuild.
  • Con: build is slow (minutes), and INT8 requires calibration on representative data.

The practical path in 2026 — don't write TensorRT by hand, but build engine from ONNX via trtexec:

# сборка FP16-движка из ONNX с динамическим батчем
trtexec \
  --onnx=model.onnx \
  --saveEngine=model_fp16.engine \
  --fp16 \
  --minShapes=input:1x3x224x224 \
  --optShapes=input:8x3x224x224 \
  --maxShapes=input:32x3x224x224 \
  --builderOptimizationLevel=5

Specify three shape profiles — min/opt/max. The engine optimizes specifically for optShapes, so put your most frequent working batch there.

vLLM: THROUGHPUT MACHINE FOR LLM

If the task is serving a large language model to many users, TensorRT-LLM and vLLM solve one pain point: KV-cache and batching. vLLM did it first and remains the default choice thanks to two ideas.

PagedAttention

Classic KV-cache requires a contiguous block of memory for each request and reserves it up to the maximum length — hence huge fragmentation and idle memory. PagedAttention stores KV in pages, like virtual memory in an OS. Result — fragmentation drops to a few percent, and in the same VRAM volume noticeably more concurrent requests fit.

Continuous batching

Instead of waiting for the entire batch to finish generation, vLLM feeds new requests at each decoding step. GPU does not idle between varying response lengths — hence a multiplicative growth in tokens per second under load.

Launching an OpenAI-compatible server — one command:

# запуск сервера vLLM с квантизацией и тензорным параллелизмом
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

# запрос как к OpenAI
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
       "messages":[{"role":"user","content":"Объясни PagedAttention в двух предложениях"}]}'

Turn up --gpu-memory-utilization until you start seeing OOM — the more memory allocated to KV-cache, the longer the queue of concurrent requests.

QUANTIZATION: FREE ACCELERATION (ALMOST)

Reducing weight precision is the cheapest way to speed up and fit into a smaller GPU. By 2026 the set has settled:

  • FP16/BF16 — default, almost no quality loss, base for everything else.
  • INT8 / SmoothQuant — good balance, requires calibration; noticeable speedup on matrix cores.
  • INT4 (AWQ, GPTQ) — model weight several times smaller, 70B fits into one GPU; for chat loads degradation usually within noise.
  • FP8 — on Hopper/Ada gives INT8 speed with quality closer to FP16; becoming default on new hardware.

Practitioner's rule: start with FP16, if memory runs out switch to AWQ INT4 and necessarily run your eval set. Benchmarks «on average without loss» say nothing about your specific task.

COMPARISON: WHAT TO TAKE WHEN

CriterionONNX RuntimeTensorRT / TRT-LLMvLLM
Main scenarioClassic, embedders, edge, CPUMinimum latency on NVIDIA GPUServing LLM with high throughput
HardwareCPU, GPU, mobile devices, browserOnly NVIDIA GPUPrimarily GPU NVIDIA (+ support for others)
StrengthPortability, cross-languageLatency, layer fusion, FP8/INT8PagedAttention, continuous batching
WeaknessNot top in peak LLM speedGPU-bound, long buildTailored for LLM, not for classic
BatchingStaticIn-flight (in TRT-LLM)Continuous (dynamic)
Entry barrierLowHighLow (one command)

Common workflow: ONNX as exchange format, TensorRT as execution provider under it for CV models, and vLLM separately for everything related to text generation.

TYPICAL PITFALLS 2026

  1. Forgot dynamic_axes. Model works at batch=1 and fails in production under load.
  2. Measuring latency without warmup. First requests include kernel compilation and loading — always do warmup and measure p50/p95, not just a «single number».
  3. Moved TensorRT-engine to another card. Not portable — rebuild for target GPU.
  4. Quantized and didn't check quality. INT4 can drop exactly your narrow domain, even if overall benchmark holds.
  5. Pushed max-model-len to the limit. Each extra context token consumes VRAM for KV-cache and cuts the number of parallel requests.

HOW TO APPROACH STACK SELECTION

Briefly, by load type:

  • CV / classification / embeddings, need portability → ONNX Runtime, with GPU — using TensorRT provider.
  • One model, latency critical on NVIDIA → TensorRT (or TRT-LLM for generative).
  • LLM chat/API for many users → vLLM, AWQ quantization, continuous batching.
  • Mixed hardware park, part without GPU → ONNX as common denominator.

Measure on your data and your hardware — there is no universal leader, there is the right tool for the specific query profile.

Frequently Asked Questions

What is faster for inference — ONNX, TensorRT, or vLLM?

It depends on the task. For minimal latency of a single model on NVIDIA GPU, TensorRT is faster. For maximum throughput when serving LLM, vLLM wins thanks to continuous batching. ONNX Runtime is the best universal choice for CPU, edge, and classic models.

Can ONNX and TensorRT be used together?

Yes, this is standard practice. The model is exported to ONNX, then run via TensorrtExecutionProvider in ONNX Runtime or a TensorRT engine is built from ONNX via trtexec. ONNX serves as an intermediate format.

Does INT4 quantization significantly degrade LLM quality?

For most chat workloads, degradation from AWQ/GPTQ INT4 is usually within noise, but on narrow domains it can be noticeable. Always run your own eval set after quantization — generic benchmarks do not guarantee results for your task.

What is PagedAttention in vLLM in simple terms?

This is a way to store KV-cache in pages, like virtual memory in an operating system, instead of a single contiguous block per request. This sharply reduces VRAM fragmentation and allows serving more requests simultaneously on the same GPU.

Serving tools change literally every week — new quantizations, engines, and forks appear constantly. To not miss a fresh release that could save you half of your GPU budget, check out the Telegram channel REDDYX AI — new repositories every 30-60 minutes.

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram

← All articles

Accelerating Inference in 2026: ONNX, TensorRT, and vLLM in Practice | REDDYX AI