GPU inference throughput optimization

If your GPU utilization is low during inference, NVIDIA Triton Inference Server is my top recommendation for general-purpose model serving, while vLLM is the better starting point if you're serving large language models (LLMs).

The key technique you're looking for is dynamic batching: collecting individual inference requests into larger batches so the GPU spends less time underutilized and processes more requests per second. NVIDIA identifies dynamic batching as one of the most effective ways to improve inference throughput and GPU utilization.docs.nvidia.com+1

Best tools for your use case

1. vLLM — Best for LLMs

(Top pick for LLM inference)

Automatically uses continuous batching, PagedAttention, and efficient GPU memory management to serve concurrent requests. Particularly useful for chatbots, text generation, and other autoregressive LLM workloads.docs.vllm.cc+1

Website:vLLM documentation

2. NVIDIA Triton Inference Server — Best for general inference

Its dynamic batcher groups incoming requests, while concurrent model instances can increase GPU utilization. A strong fit for computer vision, classification, embeddings, and models using TensorRT, ONNX Runtime, or other supported backends.docs.nvidia.com+1

Website: Triton repository ·Batching guide

3. NVIDIA TensorRT-LLM — Best for NVIDIA-specific optimization

Worth evaluating when you need to squeeze additional performance out of NVIDIA GPUs through optimized LLM execution and advanced serving techniques. It is a more optimization-focused route than simply adding a batching layer.

What I would do first

For Triton, enable dynamic batching and benchmark a few settings:

max_batch_size: 16

dynamic_batching {
  max_queue_delay_microseconds: 100
}

This is an illustrative starting point, not a universal optimum. Verify that the model supports batch size 16, then tune batch size and queue delay against your latency budget. NVIDIA recommends measuring throughput and latency with its Performance Analyzer rather than assuming larger batches are always better.docs.nvidia.com+2

For vLLM, start with continuous batching enabled through its normal serving engine, then tune concurrent sequences, batched-token limits, and GPU memory allocation.

Also measure requests or tokens per second, p95 latency, GPU memory use, and GPU utilization. Low utilization can indicate insufficient request concurrency, CPU preprocessing bottlenecks, or memory-bandwidth limitations—not just poor batching.

One question to narrow it down

What kind of model are you serving?

Model type

LLM (Llama, Qwen, etc.)

Embedding or reranking model

Computer vision / image model

Other model or custom PyTorch model

GPU hardware

NVIDIA GPU (A100, H100, L4, etc.)

AMD GPU

Mixed hardware / other accelerator

Workload

Real-time API requests

Offline bulk inference

[Recommend my stack]