Cloud-native serverless deployment

If your priority is to host machine learning models in a serverless environment without losing your existing infrastructure integrations, observability, networking, or scaling policies, the strongest candidates are usually KServe on Kubernetes, Knative Serving, and container-based platforms such as Google Cloud Run.

The key distinction is how much control you want to retain: Kubernetes-native platforms let you reuse more of your existing infrastructure and operational policies, while fully managed serverless ML services reduce infrastructure management but may limit customization.

Best cloud-native serverless options

• 

1. KServe

(Best for retaining existing integrations)

Runs model-serving workloads on Kubernetes and supports integrations with service meshes, ingress, monitoring, tracing, and ML runtimes. Its Knative deployment mode supports request-driven autoscaling.kserve.github.io+1

Preserves: Kubernetes-native infrastructure, networking integrations, and configurable scaling behavior.

Documentation: KServe integrations

---

• 

2. Knative Serving

A serverless layer for Kubernetes that automatically scales applications based on demand, including scaling to zero when enabled. It supports configurable concurrency, replica bounds, and alternative autoscaling mechanisms such as Kubernetes HPA.knative.dev+4

Preserves: Your Kubernetes environment and much of your existing platform tooling, with more control than a fully managed endpoint.

Documentation: Knative autoscaling

---

• 

3. Google Cloud Run

Deploy model-serving containers using frameworks such as vLLM, Hugging Face TGI, and other compatible runtimes. Cloud Run offers automatic scaling, configurable minimum and maximum instances, concurrency controls, and GPU support.docs.cloud.google.com+2

Preserves: Container-based workflows and application-level integrations; custom scaling controls are available, although Kubernetes-specific integrations and policies may need adaptation.

Documentation: Cloud Run for AI/ML

---

• 

4. Amazon SageMaker Serverless Inference

A managed endpoint that automatically provisions compute and scales with demand, including scaling to zero. Provisioned concurrency can use Application Auto Scaling policies based on metrics or schedules.kserve.github.io+7

Preserves: AWS service integrations and selected scaling controls, but not arbitrary infrastructure-level policies. For example, serverless inference does not support VPC configuration, GPUs, or some other real-time endpoint features.

Documentation: SageMaker Serverless Inference

Comparison: which preserves the most control?

The comparisons reflect each platform's documented integration and autoscaling capabilities.kserve.github.io+4

Azure Machine Learning is worth considering if you're already invested in Azure. Its managed online endpoints integrate with Azure Monitor autoscaling, while its Kubernetes deployment option lets you target an existing Kubernetes cluster, although some managed-endpoint capabilities differ.learn.microsoft.com+2

My recommendation

• Keep your existing Kubernetes stack and policies: Start with KServe, using Knative or KEDA as appropriate for your workload.
• Keep container portability but reduce operational overhead: Choose Cloud Run, particularly for HTTP-based inference and supported GPU workloads.
• Stay within a cloud provider's managed ML ecosystem: Choose SageMaker or Azure ML if their managed inference capabilities meet your requirements.

One important caveat: serverless doesn't automatically mean you can retain every existing scaling policy. For example, KServe can scale using custom Prometheus metrics, whereas Cloud Run's default autoscaling uses CPU and request concurrency; GPU workloads may need additional configuration.kserve.github.io+1

One question to narrow it down

What infrastructure are you using today?

1. Existing platform

Kubernetes (EKS, GKE, AKS, or self-managed)

Containers without Kubernetes

A managed ML platform

VMs or custom infrastructure

---

2. Cloud provider

AWS

Azure

Google Cloud

Multi-cloud or on-premises

---

3. Scaling policies you need to retain

CPU and memory thresholds

Custom metrics, queues, or GPU utilization

Scheduled or time-based scaling

All existing policies and integrations

[Find the best fit]