I need a distributed training framework that works across "spot" instances to save money.

If your main goal is distributed ML training on cheap, interruptible Spot instances, I’d look closely at Ray Train.

Recommended architecture

Ray Train + PyTorch + Kubernetes/EKS + Spot GPU workers + persistent object storage

Ray Train is specifically designed to recover distributed training after worker/node failures. When a Spot node disappears, it can restart the worker group on replacement nodes and resume from the latest checkpoint.

A typical setup would be:

                 ┌─────────────────────┐
                 │  Stable driver      │
                 │  CPU / On-Demand    │
                 └──────────┬──────────┘
                            │
                     Ray Train job
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                 ▼
     Spot GPU #1       Spot GPU #2       Spot GPU #3
          │                 │                 │
          └─────────────────┼─────────────────┘
                            │
                     checkpoints
                            ▼
                    S3 / object storage

The important part is not treating the Spot workers as durable machines. AWS explicitly warns that Spot capacity can disappear and recommends Spot for fault-tolerant workloads.

Why Ray Train fits

• PyTorch integration: TorchTrainer handles the distributed worker setup.
• Node failure recovery: Ray can replace failed workers/nodes and restart training from a checkpoint.
• Spot/preemption support: Ray's documentation explicitly describes using preemptible instances and recovering interrupted training.
• Persistent checkpoints: Put checkpoints in S3 or equivalent persistent storage rather than the Spot disks.
• Elasticity: Ray can be configured to work with changing worker capacity rather than assuming every machine is permanently available.

A minimal configuration looks roughly like:

from ray.train import RunConfig, ScalingConfig, FailureConfig
from ray.train.torch import TorchTrainer

trainer = TorchTrainer(
    train_loop_per_worker=train_fn,

    scaling_config=ScalingConfig(
        num_workers=8,
        use_gpu=True,
    ),

    run_config=RunConfig(
        storage_path="s3://my-bucket/checkpoints",
        failure_config=FailureConfig(
            max_failures=-1,
        ),
    ),
)

trainer.fit()

Your train_fn needs to save and restore the model/optimizer/training state. Ray then supplies the latest checkpoint to restarted workers.

One important caveat

For tightly synchronized large-model training, losing a worker generally means the whole worker group may be restarted, rather than the remaining GPUs simply continuing indefinitely. That's still useful with Spot, but checkpoint frequency becomes important.

I'd also keep the Ray driver/head node on On-Demand capacity rather than Spot. Ray's own guidance recommends putting the driver on a safer node because losing the driver can take down the training workers.

AWS Spot itself can provide substantial savings—AWS advertises discounts of up to 90% versus On-Demand, although actual savings and availability vary.

If you tell me AWS/GCP/Azure, PyTorch vs. JAX, and whether you're training an LLM or something smaller, I can lay out a concrete architecture (including Kubernetes/VM setup, checkpointing, autoscaling, and how to handle Spot interruptions).

                 ┌─────────────────────┐
                 │  Stable driver      │
                 │  CPU / On-Demand    │
                 └──────────┬──────────┘
                            │
                     Ray Train job
                            │
          ┌─────────────────┼─────────────────┐
          ▼                 ▼                 ▼
     Spot GPU #1       Spot GPU #2       Spot GPU #3
          │                 │                 │
          └─────────────────┼─────────────────┘
                            │
                     checkpoints
                            ▼
                    S3 / object storage

The important part is not treating the Spot workers as durable machines. AWS explicitly warns that Spot capacity can disappear and recommends Spot for fault-tolerant workloads.

Why Ray Train fits

Running distributed training on spot/preemptible instances (which save up to 90% on compute costs)requires a framework that handles node preemption (sudden termination) gracefully . Because cloud providers can pull the rug out from under your GPUs at any moment, standard multi-node scripts will crash.

The top frameworks and infrastructure orchestrators designed specifically to solve this problem approach it from slightly different layers of the stack:

• Item SkyPilot : An infrastructure orchestrator (not a deep learning framework itself, but runs with PyTorch/Ray) built explicitly for spot instances. It handles multi-cloud spot allocation, automatic fallback to on-demand if spot capacity runs out, and managed spot recovery (restarting and resuming your training job from the last checkpoint when a spot node gets preempted). You write a simple YAML config, and it spins up the cluster across AWS, GCP, or Lambda Labs.
• Item PyTorch Elastic (TorchElastic / torchrun) : If you are doing pure PyTorch distributed training, native torchrun supports elastic training . It allows the cluster size to shrink or grow dynamically when a spot instance is reclaimed or added back. Paired with frequent checkpointing, the job detects the node drop, re-configures the remaining ranks, and continues training without a total pipeline crash.
• Item Ray (Ray Train / Ray Core) : Excellent if you want a Python-native distributed computing layer. Ray has built-in actor fault tolerance and dynamic cluster scaling. Ray Train integrates tightly with PyTorch, Hugging Face, and XGBoost, making it smooth to handle worker loss and automatic cluster recovery on volatile spot node pools.
• Item Mosaic Platform / Composer : Built heavily with efficient, multi-node resilient training in mind, specifically targeted at cutting LLM/foundation model training costs via streaming datasets and robust asynchronous checkpoint-recovery loops.

To help narrow down the best fit, tell me:

• What model architecture or library are you using (e.g., PyTorch, Hugging Face, JAX)?
• Are you training on a single cloud (like AWS/GCP) or multi-provider/neo-clouds?
• Does your codebase already implement frequent checkpointing?

PyTorch Elastic (TorchElastic) , integrated natively into PyTorch via torchrun , is the leading distributed training framework for handling spot instance interruptions cost-effectively.

When a cloud provider reclaims a spot instance (giving a brief 2-minute warning), standard distributed jobs crash. PyTorch Elastic detects membership changes dynamically, pauses or shrinks the worker pool, and resumes training from the latest checkpoint without requiring a manual restart.

Top Distributed Frameworks for Spot Instances

Key Architectural Requirements for Spot Instances

To successfully run distributed training on spot capacity, frameworks alone are not enough. You must implement the following patterns:

1. Frequent Asynchronous Checkpointing: Write checkpoints to a fast, shared object store (like Amazon S3 or Google Cloud Storage) every few hundred steps so that a sudden preemption only discards minutes of compute instead of hours.
2. Elastic Agent Configuration: Use torchrun with --max-restarts parameters configured so that the agent layer waits for a replacement spot instance to spin up and dynamically re-integrates it into the rank list.
3. Orchestration Tooling: Combine your framework with orchestrators like Kubernetes using the Kubeflow Training Operator or SkyPilot to automatically re-request spot nodes upon termination signals.

If you'd like to narrow this down, let me know:

• Which cloud provider (AWS, GCP, Azure) are you using?
• Are you training via native PyTorch, Hugging Face, or Ray?

If you want distributed training on spot/preemptible instances, you need more than just a distributed trainer—you need elasticity + fault tolerance + checkpointing. Good options:

1. PyTorch Elastic + Kubernetes (most flexible)

Best default choice for custom PyTorch workloads.

Stack:

• PyTorch Elastic / torchrun
• Kubernetes
• GPU spot node pools
• Persistent checkpoint storage (S3/GCS/Azure Blob)

Why it works:

• Workers can disappear and rejoin.
• Training can restart from checkpoints instead of losing days of work.
• Kubernetes can replace reclaimed spot nodes.

Typical architecture:

                Object Storage
              (checkpoints/data)
                    |
                    v
Spot GPU nodes --> PyTorch Elastic --> Trainer
      |                 |
      |                 v
  node eviction     rendezvous
      |
      v
Kubernetes replaces node

Good for:

• LLM fine-tuning
• CV models
• large PyTorch training jobs
• mixed spot/on-demand clusters

---

2. Ray Train + spot autoscaling

Best if you want easier orchestration.

Anyscale builds around Ray, which supports distributed training workflows and elastic cloud scaling.

Advantages:

• Simple APIs
• Handles cluster provisioning
• Works well with hyperparameter tuning
• Easier than managing raw Kubernetes for many teams

Tradeoff:

• More abstraction than PyTorch Elastic.

---

3. Determined AI

Best if you want a training platform instead of building infrastructure.

Determined AI provides experiment management, distributed training, checkpoint handling, and spot-instance workflows.

Good when you want:

• researchers running many experiments
• automatic checkpointing
• dashboards
• less cluster engineering

---

4. Cloud-native managed options

AWS

• Amazon SageMaker HyperPod supports Spot Instances for fault-tolerant AI workloads and can scale capacity as spot availability changes.

Google Cloud

• Spot VM training works well for checkpointable workloads; Google recommends checkpointing or elastic approaches for preemptible capacity.

---

My recommended architecture for lowest cost

For most teams:

Kubernetes
 ├── On-demand small control plane
 ├── Spot GPU worker pool
 │     ├── A100/H100/L40/etc.
 │     └── multiple instance types
 ├── PyTorch Elastic
 ├── NCCL
 └── Object storage checkpoints

Key design choices:

• Checkpoint often

• Save model weights

• optimizer state

• scheduler state

• RNG state

• dataloader position if possible

• Use multiple spot instance types

• Avoid depending on one scarce GPU SKU.

• Keep a small on-demand anchor

• Rendezvous/control services are cheap compared with losing a large training job.

• Prefer elastic data parallel training

• Rigid multi-node jobs are much harder to run on spot because one lost GPU can kill the whole job.

A framework like PyTorch Elastic on Kubernetes is usually the best balance of cost, control, and reliability for a production training system.

If you tell me your stack (PyTorch/TensorFlow/JAX), cloud (AWS/GCP/Azure), GPU type, and whether this is LLM pretraining vs fine-tuning, I can suggest a more specific architecture.

Running distributed training on spot (preemptible) instances can slash your GPU costs by 60–90%, but it requires a setup that handles node interruptions gracefully. Because hyperscalers like AWS and GCP give little warning (2 minutes and 30 seconds respectively) before yanking an instance, your framework choice needs native fault tolerance, elastic resizing, or robust auto-recovery.

The best frameworks and orchestration tools designed to tackle spot instance volatility include:

1. SkyPilot (Best Multi-Cloud Orchestrator for Spot)

• What it is: An open-source framework by UC Berkeley designed to run ML and large-scale training jobs across multiple clouds (AWS, GCP, Azure, and specialized GPU clouds).
• Why it works on spot: SkyPilot has a built-in Managed Jobs API that handles spot instance interruptions natively. It automatically finds the cheapest region/cloud, provisions the spot cluster, monitors for preemptions, and—crucially—auto-recovers and resumes training from your last checkpoint when a spot node is reclaimed and restarted.
• Learn more: Check out the official SkyPilot Documentation.

2. PyTorch Elastic (torchrun / TorchElastic)

• What it is: The native elasticity engine built directly into PyTorch for distributed data-parallel (DDP) and fully-sharded data-parallel (FSDP) training.
• Why it works on spot: Standard distributed training crashes entirely if a single worker node drops. TorchElastic (torchrun ) allows training to dynamically change the number of active workers (scale down when a spot instance is preempted, scale back up when a replacement is provisioned) without tearing down the entire job, provided your code periodically saves states to a shared checkpoint.
• Learn more: Read the PyTorch Distributed Overview.

3. Ray Train (with Ray Cluster Fault Tolerance)

• What it is: The distributed compute framework from Anyscale tailored for scaling AI and Python workloads.
• Why it works on spot: Ray can orchestrate multi-node worker pools where head nodes and worker nodes run on spot instances. Ray Train integrates asynchronous and distributed checkpointing (streaming shards to remote storage in the background without blocking training). If a spot worker dies, Ray detaches the failed actor, spins up a replacement instance, and resumes from the latest checkpoint.
• Learn more: Explore the Ray Train Documentation.

Key Architecture Strategy for Spot Training

Whichever framework you pick, spot-instance training always relies on a two-part safety net:

1. Aggressive Checkpointing: Save model states, optimizer states, and dataloader states frequently (e.g., every 5–10 minutes) to a fast persistent bucket (like AWS S3 or Google Cloud Storage).
2. Orchestration Layer: Use a tool like SkyPilot to manage the lifecycle and auto-restart loop so you don't have to manually script instance re-creations when AWS or GCP reclaims your GPUs.

If you'd like to narrow this down, tell me:

• Which cloud provider (AWS, GCP, or multi-cloud) you prefer to use
• What training library you are using (PyTorch, Hugging Face, DeepSpeed, etc.)
• The model size or number of GPUs you plan to spin up

I can help outline a sample configuration file or workflow for your setup.

Yes. If your goal is distributed ML training on interruptible/Spot instances, I’d prioritize frameworks that can survive node loss rather than plain torchrun/DDP.

My top choices

• Ray Train — probably the best general-purpose choice. It supports multi-node/multi-GPU training, worker/node fault tolerance, checkpointing, and can continue with fewer resources after Spot nodes disappear. It can run PyTorch DDP, FSDP, DeepSpeed, etc.
• Amazon SageMaker Managed Spot Training — best if you're already on AWS and want the infrastructure managed for you. SageMaker handles Spot interruptions and can restore checkpoints from S3; AWS says savings can reach up to 90% versus On-Demand in suitable workloads.
• PyTorch Distributed + your own orchestration — good if you need maximum control, but vanilla DDP isn't itself a Spot-tolerance framework. You need an external mechanism to replace nodes and robustly restore distributed checkpoints.

What I'd build

For an AWS-based setup, I'd lean toward:

                    ┌───────────────┐
                    │  S3 / durable │
                    │  checkpoints  │
                    └───────┬───────┘
                            │
                     checkpoint/load
                            │
        ┌───────────────────┼───────────────────┐
        │                   │                   │
   Spot GPU #1         Spot GPU #2         Spot GPU #3
        │                   │                   │
        └──────────── Ray Train / PyTorch ──────┘
                            │
                    Spot node disappears
                            │
                     worker replaced
                            │
                    resume from checkpoint

The important part is checkpointing to durable storage frequently enough that losing a Spot node doesn't cost hours of work. Ray specifically supports distributed checkpointing, including sharded checkpoints for FSDP/DeepSpeed/Megatron-style training.

If you're on AWS, SageMaker makes this even simpler: its managed Spot training automatically handles the Spot lifecycle and restores checkpoints when the job resumes.

Which I'd choose

One subtle point: ordinary synchronous data-parallel training doesn't naturally tolerate losing a worker—the other workers generally have to stop waiting for it. That's why Ray's worker/node fault tolerance or SageMaker's managed restart/checkpoint mechanism is valuable for Spot workloads.

If you tell me AWS/GCP/Azure, GPU type, PyTorch/Hugging Face/etc., and whether this is LLM pretraining or fine-tuning, I can recommend a concrete architecture and estimated cost-saving strategy.

Running distributed training on spot (or preemptible) instances is one of the best ways to slash cloud compute bills by up to 70%-90%, but it requires handling sudden node terminations gracefully.

Rather than choosing just one isolated framework, the modern approach splits the stack into two layers: an infrastructure/orchestration layer that manages spot instance interruptions, and a training framework layer that actually coordinates the GPU compute.

Phase 1: Infrastructure & Spot Management (The Orchestrator)

These tools handle spot instance provisioning, auto-recovery, and failover when AWS, GCP, or Azure reclaims a spot node.

• SkyPilot : Check out the SkyPilot Documentation for a framework designed specifically to run AI and batch workloads on any cloud using spot instances. It automatically handles spot recovery, multi-node queuing, and fallback to on-demand instances if spot capacity is unavailable.
• Ray / Ray Train : Explore Ray Documentation for a distributed computing framework that natively supports fault-tolerant actor management. Ray Train integrates smoothly with spot nodes by checkpointing state and restarting failed workers.
• Torch Distributed Elastic (TDE) : Read the AWS EKS Distributed Training Guide to see how Kubernetes and PyTorch Elastic handle dynamic membership—meaning if a spot node drops, the cluster resizes and resumes training without crashing the entire job.

Phase 2: Distributed Training Frameworks (The Computations)

Once your infrastructure can survive a spot interruption, you run your actual training logic using these frameworks (which all support fault-tolerant state-saving):

1. PyTorch FSDP / DDP + PyTorch Elastic : Best for native PyTorch workflows. It dynamically re-ranks remaining nodes when a spot interruption occurs.
2. DeepSpeed : Excellent for large language models (LLMs) and multi-node optimization. Must be paired with robust checkpointing (saving to S3/GCS every few hundred steps) so you can restart quickly after a preemption.
3. Ray Train : Provides a clean abstraction layer over PyTorch or Hugging Face, making multi-node scaling on volatile infrastructure much less painful.

To help narrow down the best setup for your project, let me know:

• Which cloud provider (AWS, GCP, Azure) are you targeting?
• What model architecture or size (e.g., Llama-3-scale, a custom ResNet, etc.) are you training?
• Are you deploying on raw VMs/instances or using Kubernetes?

Yes. If you're on AWS, I’d strongly consider PyTorch + TorchElastic on Kubernetes/EKS with EC2 Spot instances. That combination is designed around the exact problem you’re describing: workers can disappear and be replaced without treating the whole training run as permanently failed. AWS specifically documents TorchElastic for Spot-based distributed training and dynamic worker scaling.

My shortlist

• EKS + Karpenter + PyTorch TorchElastic — best if you want maximum control and low cost. Karpenter can provision Spot capacity from multiple instance families, while TorchElastic handles worker membership/replacement. AWS recommends checkpointing to S3/FSx for Spot training.
• Amazon SageMaker Managed Spot Training — best if you want less infrastructure to manage. SageMaker handles Spot interruptions and can resume from checkpoints; AWS says savings can be up to 90% versus On-Demand. It supports distributed training as well as single-instance training.
• Plain PyTorch DDP + Spot — good for simpler jobs, but I'd only use it if you're prepared to restart/reform the distributed job when a worker disappears. Ordinary DDP isn't inherently interruption-tolerant.

Architecture I'd use

                    S3
             checkpoints / data
                     │
                     ▼
              ┌─────────────┐
              │   EKS       │
              │             │
              │ TorchElastic│
              └──────┬──────┘
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
       Spot GPU    Spot GPU    Spot GPU
        worker      worker      worker
          │          │          │
          └──────────┼──────────┘
                     │
              distributed
              PyTorch/FSDP

The critical piece is checkpointing. Save model + optimizer + scheduler + scaler + training position frequently to durable storage. When Spot reclaims a node, the replacement worker(s) can recover from the latest checkpoint rather than throwing away hours of GPU time. SageMaker's implementation, for example, automatically synchronizes checkpoints through S3.

One important caveat: don't build a tightly coupled distributed system that assumes every Spot node will remain alive. AWS explicitly notes that Spot capacity isn't guaranteed and recommends Spot for fault-tolerant workloads; distributed training therefore needs interruption/recovery mechanisms.

If you tell me (1) PyTorch/TensorFlow, (2) model size, (3) number/type of GPUs, and (4) AWS vs GCP/Azure, I can recommend a concrete framework/architecture and give you the cluster + training configuration.

For distributed training across spot/preemptible instances, you want an elastic, fault-tolerant training framework rather than a traditional fixed-size distributed launcher. Spot nodes can disappear, so the system needs checkpointing, worker replacement, and ideally the ability to change worker counts during a run.

Good options:

1. PyTorch Elastic + Kubernetes (strong default)

PyTorch Elastic is designed for this use case. It adds elastic worker management on top of PyTorch distributed training: workers can leave and rejoin, and jobs can recover after node loss.

Typical stack:

• Kubernetes cluster
• Spot GPU node pool
• PyTorch DistributedDataParallel + TorchElastic
• Kubernetes operator/controller for job lifecycle
• Persistent checkpoint storage (S3/GCS/Azure Blob/etc.)

Architecture:

                Control plane
                    |
             TorchElastic/K8s
                    |
       +------------+------------+
       |            |            |
    Spot GPU    Spot GPU    Spot GPU
      worker      worker      worker

              checkpoint store
                    |
              resume after loss

Advantages:

• Native PyTorch support
• Handles spot interruptions well
• Works with autoscaling
• Good fit for large GPU training jobs

Example scenarios using Kubernetes spot pools with TorchElastic have been demonstrated for fault-tolerant training.

2. Ray Train

Ray Train is another good choice if you want a broader distributed compute platform.

Pros:

• Easy cluster management
• Handles distributed training, tuning, and serving
• Works well with heterogeneous resources
• Integrates with PyTorch, TensorFlow, and others

Tradeoff:

• More infrastructure than PyTorch Elastic if you only need training.

3. Horovod Elastic

Horovod supports elastic training, allowing workers to be added or removed without restarting the entire job.

Good when:

• You already have Horovod-based training code
• You need multi-framework support

4. DeepSpeed + checkpointing

DeepSpeed is popular for large language models.

Usually paired with:

• ZeRO optimization
• frequent checkpoints
• Kubernetes/cloud scheduler handling replacement

It is excellent for reducing memory requirements, but the spot-resilience layer usually comes from the orchestration system around it.

---

Recommended production setup

For most teams today:

Kubernetes
  |
  +-- On-demand small control nodes
  |
  +-- Spot GPU workers
          |
          +-- PyTorch Elastic
          |
          +-- DistributedDataParallel
          |
          +-- Checkpoints -> object storage

Important practices:

• Save checkpoints frequently (minutes, not hours)
• Use multiple spot instance types to improve availability
• Keep a small on-demand control plane
• Detect termination notices and checkpoint before eviction
• Make training resumable from any worker count

Cloud providers specifically recommend checkpointing and resilient distributed training for spot workloads because interruptions are expected.

If you are starting fresh, I would pick:

• PyTorch + TorchElastic + Kubernetes for deep learning training
• Ray Train if you need a broader ML platform
• DeepSpeed + TorchElastic/Kubernetes for large LLM training.

Important practices:

• Save checkpoints frequently (minutes, not hours)
• Use multiple spot instance types to improve availability
• Keep a small on-demand control plane
• Detect termination notices and checkpoint before eviction
• Make training resumable from any worker count

Cloud providers specifically recommend checkpointing and resilient distributed training for spot workloads because interruptions are expected.

If you are starting fresh, I would pick:

Running distributed training on spot instances (preemptible VMs) can slash your compute bills by 50% to 70%, but it requires a framework that handles node preemption (eviction) gracefully without crashing the entire job.

No single framework magically solves spot interruptions on its own; rather, the best approach is a combination of an elastic training framework and an orchestration layer that manages instance replacement and fast state-recovery.

1. The Core Training Frameworks (Elastic & Fault-Tolerant)

• PyTorch FSDP / DDP with torchrun (TorchElastic):
  • Why it works: PyTorch’s native elastic launcher (torchrun ) supports node failures and dynamic membership. If a spot instance is reclaimed, torchrun can pause the job, wait for the cluster autoscaler to spin up a replacement node, re-mesh the remaining and new nodes, reload the latest checkpoint, and resume training seamlessly.
  • Best for: Standard native PyTorch workflows, Fully Shared Data Parallel (FSDP), and multi-node GPU scaling.
• Ray Train / Ray Actor Fault Tolerance:
  • Why it works: Ray treats actors (workers) as resilient entities. Ray Train integrates natively with state-store checkpointing and can automatically restart failed workers on new spot instances, re-attaching them to the active distributed cluster.
  • Best for: Reinforcement learning, hyperparameter tuning, and distributed python-heavy ML workloads.
• DeepSpeed (with ZeRO-3 & persistent checkpointing):
  • Why it works: DeepSpeed handles massive models via ZeRO partitioning, but to survive spot interruptions, you must pair it with frequent, asynchronous checkpointing to a shared object store (like S3 or GCS) so recovery takes minutes instead of hours.

2. The Orchestration Layer (Crucial for Spot)

Frameworks alone don't request new spot instances when old ones die. You need an orchestrator that watches for the cloud provider's preemption warning (AWS gives a 2-minute notice; GCP/Azure give 30 seconds) and auto-provisions replacements:

• SkyPilot:
  • An open-source framework designed to run LLMs and batch training across multiple clouds and spot instances seamlessly. It automatically handles spot recovery, multi-node queueing, and failover to on-demand instances if spot capacity vanishes . Check out the SkyPilot Model Training Guide.
• Kubernetes + Karpenter or Volcano:
  • Running your training container pods via Kubernetes using an intelligent autoscaler like Karpenter allows rapid fallback to alternative spot pools or on-demand nodes the second an eviction notice is caught.

Recommended Architecture Stack

To help narrow down the exact stack, tell me:

• Which cloud provider (AWS, GCP, Azure, or multi-cloud) are you targeting?
• What is the model size / type (e.g., a 7B LLM, CNN, or tabular model) and which training paradigm (DDP vs. FSDP/ZeRO)?

I can provide a concrete configuration or architecture blueprint.