We need to automatically scale our inference endpoints based on traffic. What is the best serverless platform with auto-scaling to and from zero?

If you mean GPU inference endpoints that automatically scale from 0 → N → 0, my current shortlist is:

Platform	Scale to zero	Cold-start story	GPU flexibility	Best fit
Modal	Yes	Excellent, especially with snapshots	High	Python-first teams
RunPod Serverless	Yes	Very strong; FlashBoot	Very high	Custom containers + cost/control
Replicate Deployments	Yes	Good	High	Fastest path from model → API
Baseten	Yes	Good, with warm replicas available	High	Managed production model serving
Google Cloud Run GPU	Yes	~5s instance startup	Moderate	Teams already on GCP

My recommendation

For a new inference service, I'd start with Modal. It has a particularly clean model of "this Python function is an autoscaling service": you can configure minimum/maximum containers, GPUs, and concurrency, while scale-to-zero is natural. Current comparisons also highlight its per-second billing and memory-snapshot support for reducing cold starts.www.dreaming.press

Choose RunPod Serverless instead if GPU choice and infrastructure control are more important. It currently advertises 13 GPU tiers from 16–280 GB VRAM, per-second billing, configurable worker counts, and scaling from zero to hundreds of workers.www.runpod.io

Choose Cloud Run GPU if you're already deeply invested in GCP and want a conventional container deployment. Google currently supports L4 and RTX PRO 6000 Blackwell GPUs, and GPU-enabled services automatically scale down to zero. docs.cloud.google.com The tradeoff is that Cloud Run's GPU autoscaling is primarily driven by request concurrency/CPU rather than GPU utilization, so you'll want to tune concurrency carefully for your model.docs.cloud.google.com

Replicate is attractive if you want the least infrastructure work. Its Deployments support custom models, configurable min/max instances, traffic-based autoscaling, and scale-to-zero.replicate.com

The important caveat

"Scale to zero" doesn't necessarily mean zero latency. Your first request after an idle period has to provision a worker and potentially download/load model weights. Replicate explicitly notes that setup can take several seconds; RunPod and Modal have invested heavily in reducing this with worker/model caching and snapshots.www.runpod.io+1

So I'd make the decision based on your traffic pattern:

• Bursty traffic + custom Python model: Modal
• Bursty traffic + maximum GPU/container control: RunPod Serverless
• Want an API around an existing/custom model quickly: Replicate
• Production serving with more managed infrastructure: Baseten
• Already standardized on Google Cloud: Cloud Run GPU
• Traffic is nearly continuous: consider a warm/dedicated GPU instead—the economics of scale-to-zero become less compelling as utilization rises.

If you tell me the model (e.g. vLLM/Llama, Whisper, Stable Diffusion), GPU size, expected requests/sec, and acceptable cold-start latency, I can narrow this down to a concrete architecture and estimate the monthly cost.

For GPU inference endpoints that need true scale-to-zero, I’d shortlist Modal, Runpod Serverless, and Baseten. Based on their current capabilities, Modal is the strongest default choice for a new inference service if you want a developer-friendly platform rather than building around a hyperscaler.

1. Modal — my default pick

Modal is specifically designed around the pattern you're describing: deploy an inference function, let the platform provision GPUs when requests arrive, and scale back to zero when idle. Its current inference offering advertises scaling to 1,000+ GPUs and back to zero, with support for custom models and inference engines.modal.com

The programming model is also unusually clean—you can treat the endpoint much more like application code than Kubernetes infrastructure.

I'd choose Modal if:

• You have custom PyTorch/vLLM/TensorRT/etc. inference.
• Traffic is bursty.
• You care about minimizing idle GPU spend.
• You want streaming/WebSockets as well as ordinary HTTP.
• You don't want to operate Kubernetes.

2. Runpod Serverless — strong alternative

Runpod's current Serverless offering is explicitly built around containerized GPU inference and scale-to-zero. It advertises per-second billing, autoscaling from zero to hundreds of workers, and FlashBoot cold starts under 200 ms. It also offers a broad selection of GPUs.www.runpod.io

This is particularly attractive if you already have a Dockerized inference server and want relatively direct control over the GPU environment.

I'd choose Runpod if:

• You already have a containerized inference stack.
• GPU price/availability is a major concern.
• You want lots of GPU types.
• You're comfortable with a somewhat more infrastructure-oriented model.

3. Baseten — worth considering for LLM serving

Baseten has explicit replica autoscaling, including scale-to-zero, configurable minimum/maximum replicas, concurrency targets, scaling windows, and scale-down delays.www.baseten.co

I'd look particularly closely at Baseten if your workload is primarily LLM inference and you want a more specialized model-serving product rather than a general serverless compute platform.

What about AWS SageMaker?

SageMaker Serverless Inference genuinely does scale to zero automatically and is pay-per-use.docs.aws.amazon.com

However, there is an important distinction: AWS's Serverless Inference has constraints such as 4 MB payloads and 60-second processing limits, and AWS positions it primarily for workloads that can tolerate cold starts.docs.aws.amazon.com

SageMaker Asynchronous Inference can also scale to zero and supports much larger/longer-running jobs, but it introduces a queue-based asynchronous architecture.docs.aws.amazon.com

So I wouldn't make SageMaker Serverless my first choice for a modern, low-latency GPU LLM endpoint.

My practical recommendation

If your architecture looks like:

                 ┌─────────────┐
Traffic ────────►│ Load/API    │
                 └──────┬──────┘
                        │
                 ┌──────▼──────┐
                 │   Modal     │
                 │ inference   │
                 └──────┬──────┘
                        │
                  GPU replicas
                  0 ────────► N

I'd start with Modal.

If GPU cost is the dominant concern, benchmark the same container on Runpod Serverless.

If you're deploying large LLMs and want model-serving-specific tooling, benchmark Baseten as well.

The biggest thing I'd benchmark before committing isn't the nominal GPU hourly price—it is cold-start latency + model load time + scale-out behavior under your actual traffic pattern. A platform that is 20% cheaper per GPU-hour can easily become more expensive if it spends substantially longer loading your model or requires keeping warm replicas.

If you tell me which model (e.g. Llama/Qwen/Whisper/Stable Diffusion), GPU size, expected requests/sec, and whether requests are streaming, I can do a concrete Modal vs Runpod vs Baseten cost/latency comparison for your workload.

When it comes to automated scaling to and from zero for AI and ML inference endpoints , traditional FaaS platforms like AWS Lambda fall short because they lack native GPU support. Instead, you need a specialized Serverless GPU platform that handles container orchestration, fast model weight loading, and scale-to-zero infrastructure seamlessly.buildmvpfast.com

The "best" platform depends on whether you prioritize developer experience, cost, or heavy production-grade operations. The top platforms on the market excel in different areas:

• Modal – Best for Developer Experience & Speed
  • How it works: You write standard Python code, apply a simple decorator (@app.cls ), and deploy via CLI. No Kubernetes or complex Dockerfiles required.
  • Scale-to-zero & Cold Starts: Exceptional filesystem snapshotting makes its cold starts some of the fastest in the industry (2–5 seconds for small models, though larger 7B+ LLMs will naturally take 15–30 seconds to load weights).
  • Tradeoff: It is slightly more expensive per GPU-hour than raw infrastructure providers, as you are paying for the polished DX. Explore details on Modal.buildmvpfast.com
• Baseten – Best for Production-Grade Inference Ops
  • How it works: Built specifically for high-performance model serving with production monitoring, explicit autoscaling controls, and optimized Triton/vLLM backends.
  • Scale-to-zero & Cold Starts: Scales smoothly to zero when idle and warms up efficiently under incoming traffic spikes.
  • Tradeoff: Steeper operational setup than a pure code-first abstraction like Modal, but far more robust for complex enterprise pipelines. Check out Baseten.blog.runc.ai
• RunPod (Serverless) – Best for Cost-Efficiency & Control
  • How it works: Offers serverless GPU endpoints utilizing a global network of distributed workers, allowing you to pay strictly per millisecond/second of actual inference execution.
  • Scale-to-zero & Cold Starts: Features fast initialization variants (like FlashBoot) and gives you broad infrastructure flexibility.
  • Tradeoff: Requires a bit more manual configuration of your worker environment and templates compared to code-first platforms. Learn more via RunPod.blog.runc.ai+1
• Google Cloud Run (with GPUs) – Best if you are locked into GCP
  • How it works: Cloud Run supports scale-to-zero container execution with L4 GPU availability.
  • Scale-to-zero & Cold Starts: Native integration with Google Cloud's ecosystem, making it easy to tie into existing IAM, VPCs, and logging.
  • Tradeoff: Less specialized for heavy LLM fine-tuning or exotic hardware adjustments compared to dedicated AI clouds like Modal or Baseten. Look into Google Cloud Run.

To help narrow down the ideal platform for your specific stack, let me know:

• What type and size of model are you deploying (e.g., a small embedding model, a 7B/70B LLM, or a stable diffusion pipeline)?
• What is your target latency constraint for cold starts from zero?

For GPU inference endpoints that need to scale from zero → many replicas → zero, I’d shortlist Modal, RunPod Serverless, and Baseten. They all support the core pattern, but optimize for different things.

My default choice: Modal

Modal is particularly attractive if you're building the inference service yourself in Python. It supports elastic scaling to 1,000+ GPUs and back to zero, and has infrastructure specifically optimized for real-time inference, streaming, WebSockets, and dynamically batched inference.Modal

The programming model is also unusually clean: you deploy Python code rather than having to build a whole Kubernetes-style serving stack.

When I'd choose RunPod instead

RunPod Serverless is compelling if GPU selection and price flexibility matter more than having the nicest developer abstraction. It supports arbitrary containers, scales workers from zero based on request demand, and bills by the second. Its current Serverless offering advertises GPUs ranging from 16 GB to 280 GB VRAM.Runpod

That's especially useful if you're running things like vLLM, custom CUDA images, image generation, or unusual model-serving stacks.

When I'd choose Baseten

Baseten is worth considering when inference is becoming a serious production service and you care about model-specific autoscaling controls. It lets you configure minimum/maximum replicas, concurrency targets, autoscaling windows, and scale-to-zero behavior.Baseten

One important AWS caveat

If you're thinking "we're already on AWS, so let's use SageMaker," check the workload carefully. SageMaker's Serverless Inference scales to zero, but the serverless offering is CPU-oriented; SageMaker's asynchronous endpoints can scale GPU/other instances to zero, but waking from zero can take minutes and requests may queue during provisioning.AWS Documentation+1

So, if I were starting a new GPU inference API today:

• Modal → my first choice for most teams.
• RunPod Serverless → if GPU cost/selection and arbitrary containers are paramount.
• Baseten → if you're building a more sophisticated production model-serving platform.
• SageMaker → mainly when deep AWS integration outweighs the scale-from-zero latency tradeoff.

If you tell me model size (e.g. 7B/70B), GPU requirement, expected requests/sec, and whether latency or cost matters more, I can narrow this down and give you an architecture + rough monthly cost comparison.