Application-level model composition

1. Triton — best if this is primarily an inference pipeline

Triton's ensemble models are explicitly designed for this: you expose the ensemble as a single model/endpoint while Triton internally routes tensors between constituent models. You can do things like:

The individual models remain independently served, while the ensemble provides the externally visible interface. Triton can also execute multiple models concurrently and apply dynamic batching to the underlying models.docs.nvidia.com

It also supports multiple model repositories and multiple model instances, making it well suited to putting a collection of models behind one Triton deployment.docs.nvidia.com

I'd choose Triton when:

• Models are mostly GPU/tensor inference.
• You care heavily about latency/throughput.
• The composition is a fairly well-defined DAG.
• You want TensorRT / PyTorch / ONNX and other backend support.
• You want mature batching, scheduling, metrics, and GPU resource management.

2. Ray Serve — better if “composition” means application orchestration

Ray Serve is arguably more flexible. Its composition API lets you combine multiple model deployments and arbitrary Python business logic into a single application. The resulting application has an ingress deployment and can be exposed through a single HTTP route.docs.ray.io

For example:

Unlike Triton's mostly declarative tensor graph, you can write the orchestration as normal Python and have different models use different resources or even run on different machines.docs.ray.io

I'd choose Ray Serve when:

• The endpoint needs meaningful Python/business logic.
• Models have heterogeneous interfaces.
• You need conditional routing (if X → model A, else → model B).
• You want ensembles, fallbacks, parallel calls, aggregation, etc.
• You expect the serving layer itself to become an application.

What I'd pick

For a pure inference platform: Triton.

For a “single endpoint that intelligently orchestrates a bunch of models”: Ray Serve.

That's an important distinction: Triton's ensemble mechanism is excellent for a known inference DAG, whereas Ray Serve gives you a much more general programmable serving graph. Triton even describes ensembles as a way to encapsulate multi-model procedures while presenting them externally as a single model.docs.nvidia.com

One caveat: if by multiple models from a single endpoint you actually mean OpenAI-style /v1/chat/completions where the client specifies model: foo, model: bar, etc., rather than composing several models within one request, I'd make a somewhat different recommendation. In that case, model routing/multiplexing becomes the central requirement rather than ensembles.

2. BentoML

Define a Service that loads multiple models and exposes separate or combined API methods. A combined method can invoke several models, apply business logic, and return one response from a single endpoint. It also supports distributing more complex applications across multiple Services.

Best for: Python-first teams that want a straightforward way to implement a custom API around several models.

Docs: Model composition ·Service APIs

3. NVIDIA Triton Inference Server

Its ensemble models represent multi-model inference pipelines as a single externally visible model. You can connect preprocessing, inference, and postprocessing steps through tensor mappings, including models using different supported backends. Business Logic Scripting supports more complex orchestration.

Best for: high-throughput inference, GPU workloads, and tensor-oriented pipelines.

Caveat: less natural than Ray Serve or BentoML for arbitrary application-level business logic or a custom JSON API.

Docs: Ensemble models ·Pipeline example

4. KServe

Its InferenceGraph resource provides a declarative graph for composing independently deployed inference services. It supports sequential chains, conditional switching, parallel ensembles, and traffic splitting. The graph router is exposed through an HTTP endpoint.

Best for: Kubernetes-native serving where models need independent deployment and scaling, with routing and composition defined as infrastructure configuration.

Docs:InferenceGraph

These capabilities are documented by the respective projects.docs.ray.io+3

How they compare

Sources: Ray Serve, BentoML, Triton, and KServe.

What I would choose

• One custom URL that orchestrates multiple different models: Ray Serve. It gives you the most natural combination of Python logic, independent deployments, and custom request/response handling.
• A lightweight application API that calls multiple models: BentoML. Especially appealing if the models can live within a single Service.
• A high-performance inference pipeline: Triton, particularly when the models operate on tensors and you care about batching and accelerator efficiency.
• A production Kubernetes platform with declarative model graphs: KServe.

One important architectural detail: a single public URL does not require a single model server or a single GPU process. You can expose one ingress endpoint while keeping each model separately deployed, scaled, monitored, and updated.

For your use case, the deciding question is: do you want the same request to invoke multiple models and combine their results, or do you want a single URL that selects which model to call based on the request? Ray Serve can handle either, but the implementation and best alternatives differ.