High-volume inference batching

For processing a huge volume of data through a model, the biggest inference-throughput gains usually come from maximizing hardware utilization, batching requests efficiently, and minimizing wasted computation. The best approach depends on whether you're running an LLM, an embedding model, an image model, or a conventional neural network.

The key is to optimize throughput (items or tokens per second) without exceeding GPU memory or introducing unacceptable latency.

1. The batching techniques that matter most

• 

1. Dynamic batching

Collect incoming requests for a short time window and process them together in one model invocation. Requests arriving at slightly different times can share the same GPU execution.

• Best for: online inference with continuous traffic.
• Tune: maximum batch size and maximum batching delay.
• Trade-off: higher throughput, potentially higher per-request latency.

---

• 

2. Large static batches

Assemble a fixed batch of inputs and run inference over the batch before moving to the next one.

• Best for: offline processing of millions of records.
• Tune: batch size until GPU utilization is high and memory headroom remains safe.
• Trade-off: excellent throughput potential, but more memory use and longer time before a batch completes.

---

• 

3. Continuous batching

Particularly useful for autoregressive LLMs. As soon as a sequence finishes, a new request can enter the active batch instead of waiting for every other sequence to finish.

• Best for: text generation with variable output lengths.
• Tune: concurrent sequences, token budget, and KV-cache memory.
• Trade-off: better GPU occupancy, but scheduling and memory management become more complex.

---

• 

4. Length bucketing and token-budget batching

Group inputs with similar sequence lengths, or create batches subject to a maximum total token count. This prevents short inputs from wasting compute on padding to match much longer inputs.

• Best for: LLM prompts, NLP, variable-resolution images, and variable-length sequences.
• Tune: bucket boundaries, maximum tokens per batch, and padding strategy.
• Trade-off: more complicated data scheduling, usually less wasted computation.

---

• 

5. Multi-GPU parallel inference

Distribute independent batches across GPUs, typically using data parallelism. Each GPU processes different inputs while sharing the same model weights.

• Best for: large offline workloads and high aggregate request volume.
• Tune: per-GPU batch size, work distribution, CPU input pipeline, and GPU utilization.
• Trade-off: near-linear scaling is possible in favorable conditions, but input bottlenecks and communication overhead can limit gains.

2. How to tune batch size for maximum throughput

Don't assume that the largest possible batch is the fastest. Throughput typically improves as you increase batch size, then plateaus or drops when memory pressure, padding, or compute limits become significant.

A practical tuning procedure:

1. Start with batch sizes of 1, 2, 4, 8, 16, 32, 64, and continue upward if memory permits.
2. Run enough concurrent batches to keep the accelerator busy.
3. Measure throughput, GPU utilization, peak memory, and latency for each configuration.
4. Keep increasing batch size until throughput stops improving or the configuration becomes unstable.
5. Test the winning batch size under sustained load, not just a short benchmark.

For example, if a batch of 32 takes 40 ms and a batch of 64 takes 65 ms:

The batch of 64 delivers about 23% more throughput, even though its execution time is longer. These numbers are illustrative, not benchmark results.

For online serving, NVIDIA Triton's documentation recommends starting with dynamic batching and increasing batch size or queue delay only while latency remains within your budget.docs.nvidia.com+1

3. Additional optimizations that compound

Mixed precision

Use BF16 or FP16 where supported and validated. This can increase accelerator throughput and reduce memory use, allowing larger batches.

---

Parallelize the input pipeline

Pre-tokenize, decode, normalize, or preprocess data in parallel. Use worker processes, prefetching, and pinned memory where appropriate so the GPU doesn't wait for the CPU or storage.

---

Optimize model execution

Use optimized runtimes such as TensorRT or ONNX Runtime where supported. For PyTorch, benchmark inference mode, compilation, and efficient attention kernels where applicable.

---

Stream data instead of loading everything into memory

Read chunks, batch them, execute inference, and write results incrementally. This makes the pipeline scalable and avoids unnecessary memory pressure.

---

Measure bottlenecks separately

Track items/sec or tokens/sec, accelerator utilization, memory bandwidth, queue time, preprocessing time, and output-writing time. More batching won't help if storage or postprocessing is already the bottleneck.

4. If your model is an LLM, optimize tokens—not just requests

For LLM inference, the number of requests per batch can be misleading because requests vary in prompt length and generated output length.

The highest-impact techniques are often:

• Continuous batching: replace completed sequences immediately with new ones.
• Token-budget batching: limit the total tokens processed in a scheduling step rather than using only a fixed request count.
• Chunked prefill: break long prompts into chunks so they don't monopolize execution while other requests are generating tokens.
• Prefix caching: reuse computed prompt states when requests share the same prefix.
• Paged KV cache: manage attention-cache memory efficiently so more sequences can remain active concurrently.

These techniques are supported in modern LLM inference engines such as vLLM. Prefix caching is most beneficial when prompts actually share prefixes, while chunked prefill helps balance prompt processing against token generation.docs.vllm.ai+1

5. A practical starting configuration

Use this as your initial plan, then benchmark and adjust based on your hardware.

For PyTorch, use torch.inference_mode() for inference-only execution and evaluate mixed precision when supported by your model and hardware.docs.pytorch.org+1

6. Let's narrow this down to your workload

The optimal strategy changes substantially depending on the model architecture and GPU. Tell me a little about your setup, and I can suggest concrete batch sizes, concurrency settings, and an implementation approach.

Model type

Large language model (generation or chat)

Embedding model

Classification or regression

Image or vision model

Other neural network

---

Hardware

NVIDIA GPU

AMD GPU

Apple Silicon

CPU only

Multi-GPU server or cluster

Cloud inference API

---

Processing mode

Offline bulk processing (data already available)

Online serving (live requests)

---

Framework or serving engine

PyTorch

vLLM

TensorRT / Triton

ONNX Runtime

Other / not decided

---

Approximate data volume

Thousands

Millions

Billions

Continuous high-volume stream

Primary objective

Maximum total throughput

High throughput with a latency limit

Lowest possible latency

[Recommend my configuration]