I need an orchestrator to deploy machine learning models to Kubernetes clusters automatically.

If by “orchestrator” you mean “I push a model artifact, and the system automatically deploys, scales, updates, and monitors it across Kubernetes”, I’d look at these:

Option	What it gives you	Best fit
KServe	Kubernetes-native model serving, autoscaling, scale-to-zero, GPU autoscaling, canaries	Best general-purpose choice
Seldon Core 2	Model deployment + routing/pipelines + scaling + production ML features	Large model fleets / complex inference
Kubeflow	End-to-end ML platform: training pipelines, experiments, model serving via KServe	Full MLOps platform
Ray Serve + KubeRay	Programmable model/application serving, autoscaling, model composition	Python-heavy ML/LLM applications
BentoML	Package model → container → Kubernetes deployment workflow	Simplicity and developer experience

My recommendation: KServe + GitOps

KServe is probably closest to what you're describing. It uses Kubernetes custom resources to represent model deployments and handles things like autoscaling, networking, health checks, GPU autoscaling, scale-to-zero, and canary rollouts.www.kubeflow.org

A typical architecture would be:

                  Git repository
                       │
                       ▼
              CI/CD / GitOps
             (Argo CD / Flux)
                       │
                       ▼
             ┌─────────────────┐
             │ Kubernetes      │
             │                 │
             │    KServe       │
             │       │         │
             │       ▼         │
             │  Model Server   │
             │       │         │
             │       ▼         │
             │  Model artifact │
             │   S3/GCS/etc.   │
             └─────────────────┘
                       │
                       ▼
                Inference API

For example, your deployment pipeline could effectively be:

model trained
     ↓
model registered/versioned
     ↓
artifact uploaded to S3/GCS
     ↓
CI updates model version in Git
     ↓
Argo CD syncs Kubernetes
     ↓
KServe creates/updates endpoint
     ↓
health check
     ↓
canary 10%
     ↓
promote to 100%

KServe is particularly attractive if you want the Kubernetes cluster itself to be the control plane rather than introducing another proprietary deployment platform.

When I'd choose Seldon instead

Seldon Core is worth serious consideration if you expect hundreds/thousands of models or complicated inference graphs. Seldon Core 2 models are Kubernetes resources, can pull artifacts from object storage, and can be matched to appropriate inference servers.docs.seldon.io

It also has explicit support for model pipelines and more elaborate inference architectures.docs.seldon.io

When Ray makes more sense

Ray Serve is different: it's more of a programmable distributed serving layer than a Kubernetes-native model orchestrator. KubeRay lets you package a Ray cluster and Serve application into a Kubernetes RayService resource, which Kubernetes can then manage.docs.ray.io

I'd lean toward Ray if your models are part of complex Python applications—for example:

API
 │
 ├── LLM
 ├── embedding model
 ├── reranker
 ├── feature service
 └── business logic

rather than simply:

model artifact → production endpoint

A practical stack I'd build

For a new platform today, I'd consider:

Kubernetes → KServe → Argo CD → MLflow/model registry → S3/GCS → Prometheus + Grafana → OpenTelemetry

Then give developers a tiny interface such as:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-model
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/fraud/42/
      resources:
        requests:
          cpu: "1"
          memory: 2Gi

The important architectural distinction is that KServe/Seldon/Ray handle serving/orchestration, while something like Argo CD handles the automatic deployment mechanism and MLflow handles model lifecycle/registry. You generally don't want to make one tool responsible for all three jobs.

If you tell me whether you're deploying traditional ML, PyTorch/TF models, or LLMs, and whether you have one Kubernetes cluster or multiple clusters, I can propose a concrete production architecture and repository layout.

If you mean “I want a control plane/orchestrator that automatically takes a trained model and deploys it onto Kubernetes, with scaling and rollout management,” there are a few strong options.

Option	What it gives you	Best fit
KServe	Kubernetes-native InferenceService CRDs, autoscaling, scale-to-zero, GPU support, canary rollouts, multiple model frameworks	Best general-purpose choice
Seldon Core 2	Model deployment, model pipelines, scaling, experiments, monitoring, synchronous/asynchronous inference	More sophisticated inference workflows
Kubeflow + KServe	Model registry → deployment automation → serving → broader ML platform	Full MLOps platform
Ray Serve	Python-native distributed serving and autoscaling	Complex Python/LLM inference workloads

My recommendation: KServe + GitOps

KServe is particularly close to what you're describing. It adds Kubernetes CRDs specifically for ML inference and handles things like autoscaling, networking, health checks, GPU autoscaling, scale-to-zero, and canary deployment.www.kubeflow.org+1

A practical architecture would be:

             ┌─────────────────────┐
             │ Model Registry      │
             │ MLflow / Kubeflow   │
             └──────────┬──────────┘
                        │
                  new model version
                        │
                        ▼
             ┌─────────────────────┐
             │ Git / Model Event   │
             └──────────┬──────────┘
                        │
                        ▼
             ┌─────────────────────┐
             │ GitOps Controller    │
             │ Argo CD / Flux       │
             └──────────┬──────────┘
                        │
             ┌──────────┴──────────┐
             ▼                     ▼
       ┌───────────┐         ┌───────────┐
       │ K8s       │         │ K8s       │
       │ Cluster A │         │ Cluster B │
       │ KServe    │         │ KServe    │
       └─────┬─────┘         └─────┬─────┘
             │                     │
          model v17              model v17
             │                     │
       autoscaling             autoscaling
       canary                  canary
       monitoring              monitoring

The important distinction is that KServe is primarily the model-serving/control layer, while something like Argo CD/Flux provides multi-cluster GitOps orchestration. If you want one system to decide which cluster gets a model, manage cluster capacity, and promote models automatically, I'd put a small ML deployment controller above KServe rather than trying to make KServe do everything.

Kubeflow's Model Registry can also feed KServe directly: its documentation explicitly describes using registered model/version metadata to automatically create KServe inference endpoints.www.kubeflow.org

If you need more sophisticated orchestration

Seldon Core 2 is worth considering. It models ML artifacts as Kubernetes resources and supports model pipelines, experiments, monitoring-related components, and deployment at large scale.docs.seldon.io

For example, you can represent a model with a Kubernetes resource pointing at an object-storage artifact:

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: fraud-model
spec:
  storageUri: s3://models/fraud/v17
  requirements:
    - sklearn

Seldon then matches the model requirements to an appropriate server.docs.seldon.io

What I'd build

If you're building this yourself, I'd use:

Model Registry → Event → Deployment Orchestrator → Argo CD → KServe → Kubernetes

Your orchestrator would expose something like:

model:
  name: fraud-model
  version: 17
  artifact: s3://ml-models/fraud/17

deployment:
  environment: production
  clusters:
    - us-east
    - us-west

resources:
  cpu: "2"
  memory: "8Gi"
  gpu: 1

rollout:
  strategy: canary
  initialTraffic: 5
  promotionMetric:
    latencyP95: "< 200ms"
    errorRate: "< 1%"

It could then automatically:

1. Discover a new model version.
2. Validate the model/artifact.
3. Determine eligible Kubernetes clusters.
4. Generate the KServe deployment.
5. Deploy through GitOps.
6. Wait for readiness.
7. Send 5% traffic to the new version.
8. Evaluate metrics.
9. Promote or roll back.
10. Scale replicas/GPU capacity according to traffic.

KServe already supports canary traffic percentages and autoscaling, so you wouldn't need to implement those primitives yourself.www.kubeflow.org+1

If you tell me whether you want to use AWS/GKE/Azure/on-prem and whether your models are mainly sklearn/PyTorch/LLMs, I can sketch the actual orchestrator architecture and Kubernetes CRDs I'd use.

If your goal is “give an orchestrator a model artifact + deployment spec and have it automatically deploy, scale, update, and roll back models across Kubernetes”, I’d look at these:

Platform	Best fit	Kubernetes-native	Autoscaling	Canary / rollout	Multi-model	LLM support
KServe	General-purpose ML serving platform	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	✅	✅	⭐⭐⭐⭐⭐
Seldon Core 2	Large model fleets + MLOps	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	✅	✅	⭐⭐⭐⭐
Ray Serve + KubeRay	Complex Python/LLM inference applications	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	Via your orchestration	—	⭐⭐⭐⭐⭐
BentoML	Developer-friendly model packaging/deployment	⭐⭐⭐⭐	⭐⭐⭐⭐	Via Kubernetes/GitOps	—	⭐⭐⭐⭐⭐

1. KServe — probably the closest match

KServe documentation

KServe extends Kubernetes with an InferenceService CRD. You declare the model, runtime, resources, and deployment configuration; KServe handles much of the serving lifecycle, including networking, health checks, autoscaling and revisions. It supports TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, Hugging Face and custom runtimes.kserve.github.io+1

It also supports canary deployments, e.g. sending 10% of traffic to a new model version and rolling back by changing the traffic allocation.kserve.github.io

For LLMs, current KServe supports vLLM/Hugging Face-style deployments, multi-node inference, GPU-aware autoscaling and metrics-based scaling.kserve.github.io

A simplified deployment looks like:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-model
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://models/fraud/v42
      resources:
        requests:
          cpu: "1"
          memory: 2Gi
        limits:
          cpu: "2"
          memory: 4Gi

Then:

kubectl apply -f fraud-model.yaml

KServe creates/manages the Kubernetes resources required to serve it.

2. Seldon Core 2 — particularly interesting for model fleets

Seldon Core 2 documentation

Seldon Core 2 is explicitly designed around deploying and managing potentially thousands of models on Kubernetes. It separates model artifacts from inference servers, so models can be assigned to compatible serving runtimes.docs.seldon.io

That architecture is attractive if your orchestrator needs to say something like:

“Deploy model X to the cluster, find a compatible inference runtime, allocate resources, expose the endpoint, and manage the model lifecycle.”

Seldon also has components for pipelines, experiments, monitoring and inference routing.github.com

3. Ray Serve + KubeRay — if models are really applications

Ray Serve documentation

I'd consider Ray if your inference system involves Python business logic + multiple models + distributed computation, rather than simply serving an individual model.

KubeRay's RayService CR lets you package a Ray cluster and Serve application into a Kubernetes resource. The controller handles creating the Ray cluster and deploying the Serve application.docs.seldon.io+2

Ray Serve also has request-based autoscaling; replicas can automatically increase/decrease based on ongoing request load.docs.ray.io

What I'd build

If you're actually building an ML deployment orchestrator, rather than just looking for an inference server, I'd use:

                    ┌─────────────────────┐
                    │   ML Control Plane  │
                    │                     │
Git / Model Registry ──► Deployment API   │
                    │       │             │
                    │       ▼             │
                    │  Scheduler/Policy   │
                    └───────┬─────────────┘
                            │
                 Kubernetes API / CRDs
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                 ▼
      Cluster A          Cluster B         Cluster C
       KServe             KServe             KServe
          │                 │                 │
       GPU/CPU           GPU/CPU           GPU/CPU
       models             models             models

KServe would be my first component to evaluate. It already gives you the Kubernetes-native model lifecycle primitives, while your orchestrator can sit above KServe and decide which cluster, which version, which resources, and when to deploy.

For example:

model:
  name: fraud-model
  version: "42"

deployment:
  strategy: canary
  canaryPercent: 10

resources:
  gpu: 1
  memory: 8Gi

scaling:
  minReplicas: 1
  maxReplicas: 10

placement:
  clusters:
    - gpu-prod-east
    - gpu-prod-west

Your orchestrator translates that higher-level specification into KServe/Kubernetes resources.

One important distinction: KServe/Seldon/Ray are primarily serving/deployment infrastructure. If by “orchestrator” you mean a system that also handles model registry → approval → deployment → evaluation → canary → monitoring → automatic rollback → multi-cluster placement, you'll want a control plane around one of these rather than expecting the serving framework itself to provide the entire workflow.

If you tell me whether you're deploying traditional ML models, LLMs, or both, I can narrow this down to a concrete architecture and open-source stack.

If your goal is “a model gets registered → automatically selected/configured → deployed to the appropriate Kubernetes cluster → health-checked → exposed → optionally rolled back”, I’d build the orchestrator around KServe rather than writing a Kubernetes deployment controller from scratch.

KServe provides Kubernetes-native InferenceService resources and handles model serving concerns such as autoscaling, networking, health checks, canary deployments, and multiple ML frameworks.KServe+1

Recommended architecture

                 ┌─────────────────────┐
                 │ Model Registry       │
                 │ MLflow / Kubeflow    │
                 │ Model Registry       │
                 └──────────┬──────────┘
                            │
                       model approved
                            │
                            ▼
                 ┌─────────────────────┐
                 │ ML Deployment       │
                 │ Orchestrator        │
                 │                     │
                 │ • policy engine     │
                 │ • cluster selector  │
                 │ • resource planner  │
                 │ • deployment engine │
                 │ • rollout manager   │
                 └──────────┬──────────┘
                            │
                 ┌──────────┴──────────┐
                 │                     │
                 ▼                     ▼
          ┌──────────────┐      ┌──────────────┐
          │ Kubernetes A │      │ Kubernetes B │
          │              │      │              │
          │ KServe       │      │ KServe       │
          │ GPU nodes    │      │ CPU nodes    │
          └──────┬───────┘      └──────┬───────┘
                 │                     │
                 ▼                     ▼
             inference              inference
              endpoint               endpoint

Kubeflow Model Registry is particularly useful if you want the orchestrator to consume standardized model/version/artifact metadata rather than having deployment configuration scattered across systems. Kubeflow explicitly documents using Model Registry metadata to automate creation of KServe inference endpoints.Kubeflow

What I'd make the orchestrator responsible for

1. Model event

Something like:

{
  "model": "fraud-detector",
  "version": "42",
  "artifact": "s3://ml-models/fraud-detector/42",
  "framework": "xgboost",
  "environment": "production"
}

The orchestrator receives this from your registry, CI/CD system, or message bus.

2. Policy evaluation

For example:

model: fraud-detector

deployment:
  environment: production
  min_replicas: 2
  max_replicas: 20

resources:
  cpu: "2"
  memory: "4Gi"

routing:
  region: us-east
  cluster: auto

rollout:
  strategy: canary
  initial_percent: 10
  promotion_threshold:
    error_rate: 0.01
    latency_p95_ms: 250

The important part is separating desired deployment policy from Kubernetes implementation details.

3. Cluster selection

Give every cluster metadata:

cluster: prod-us-east-gpu
labels:
  region: us-east
  environment: production
  gpu: nvidia
  capacity: high

Then the orchestrator can determine:

model requirements
       +
deployment policy
       +
cluster capabilities
       ↓
selected cluster

For multiple clusters, I'd strongly consider keeping a small control plane outside the individual clusters, with KServe acting as the serving control plane inside each cluster.

4. Generate a KServe resource

For example:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
spec:
  predictor:
    model:
      modelFormat:
        name: xgboost
      storageUri: s3://ml-models/fraud-detector/42
      resources:
        requests:
          cpu: "2"
          memory: 4Gi

KServe's current documentation supports predictive models including scikit-learn, XGBoost, TensorFlow, PyTorch, ONNX and others, as well as generative models and custom runtimes.KServe+1

The orchestrator then applies that resource to the selected cluster.

5. Deployment state machine

I'd make deployments explicit state machines:

PENDING
   │
   ▼
VALIDATING
   │
   ▼
SCHEDULING
   │
   ▼
DEPLOYING
   │
   ▼
STARTING
   │
   ▼
HEALTH_CHECK
   │
   ├──────── failure ───────► ROLLBACK
   │
   ▼
CANARY
   │
   ├──────── failure ───────► ROLLBACK
   │
   ▼
PROMOTING
   │
   ▼
ACTIVE

This makes retries, crash recovery, and observability much easier than having one giant deployment function.

6. Reconciliation rather than scripts

The orchestrator should behave like a Kubernetes controller:

Desired state:
    fraud-detector v42
    10 replicas
    cluster = prod-us-east
    traffic = 100%

             ↓ reconcile

Actual state:
    fraud-detector v41
    2 replicas
    cluster = prod-us-east
    traffic = 100%

             ↓

Create/update resources
             ↓
wait
             ↓
observe
             ↓
reconcile again

That means a temporary API failure doesn't leave your deployment system permanently confused.

KServe itself follows this Kubernetes-native model through its custom resources and control/data-plane architecture.KServe

Suggested technology stack

Component	Choice
Orchestrator	Go
Kubernetes integration	client-go / controller-runtime
Model serving	KServe
Model registry	MLflow or Kubeflow Model Registry
Workflow orchestration	, , or Kubernetes-native reconciliation

KServe also supports both Knative/serverless mode and standard Kubernetes deployment mode, so you can choose whether workloads should use serverless scaling or ordinary Kubernetes deployments.KServe

The key design decision

I would not make your orchestrator responsible for actually running model servers.

Instead:

Your orchestrator
      │
      │ decides WHAT / WHERE / WHEN
      ▼
KServe
      │
      │ manages HOW the model runs
      ▼
Kubernetes
      │
      ▼
Pods / GPUs / Services

That gives you a relatively small orchestration layer while delegating model-serving complexity to a system designed specifically for it.

If you want, I can next sketch the actual Go orchestrator, including the CRDs, controller architecture, multi-cluster selection, deployment state machine, and a working KServe deployment example.