LLM provider routing

Since Sep 24Updated Nov 20, 2026Based on 20 AI answersReviewed by

How we measure

Brands AI recommends

Mentioned in, on ChatGPT and Google AI Mode
  • Portkey AIChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 90%
  • LiteLLMChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 90%
  • OpenRouterChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 60%
  • Cloudflare AI GatewayChatGPTChatGPT 60%Google AI ModeGoogle AI Mode 40%
  • Kong AI GatewayChatGPTChatGPT 80%Google AI ModeGoogle AI Mode 20%

Questions and answers

  • ChatGPTNov 18, 2026

    1. 1AWS
    2. 2LiteLLM
    3. 3Redis
    4. 4Kong AI Gateway
    5. 5OpenRouter
    6. 6Gemini
    7. 7PostgreSQL
    8. 8Portkey AI
    9. 9Google Cloud Gemini Enterprise Agent Platform

    Architecture options for a multi-provider LLM gateway

    What you're describing is generally called an LLM Gateway or LLM Router.

    It sits between your application and multiple LLM providers, providing a unified API, request/response normalization, provider selection, load balancing, retries, and automatic failover.

    Rather than integrating OpenAI, Anthropic, Google, AWS Bedrock, Azure, and other providers independently into your application, you integrate with one abstraction layer.

    1. Recommended high-level architecture

    Client applications

    Web apps · Backend services · Agents · APIs

    LLM Gateway API

    Unified OpenAI-compatible interface

    Normalization

    Request mapping

    Response mapping

    Error translation

    Routing engine

    Provider selection

    Load balancing

    Policies

    Reliability

    Retries

    Circuit breakers

    Failover

    Governance

    API keys

    Rate limits

    Cost tracking

    Provider adapters

    External LLM services and self-hosted models

    The important architectural decision is to separate three concerns:

    1. Normalization: Your application shouldn't need to understand individual provider APIs.
    2. Routing: The gateway decides which provider/model receives the request.
    3. Resilience: Provider failures shouldn't automatically become application failures.

    2. Available technology options

    These are the main candidates worth evaluating.

    • LiteLLM

    1. LiteLLM

    (Best overall self-hosted option)

    • OpenAI-compatible API across 100+ providers.
    • Request and response normalization.
    • Retries, fallbacks, cooldowns and load balancing.
    • Redis-backed shared routing state.
    • Spend tracking, virtual keys and budgets.

    Particularly attractive if you want to own the gateway infrastructure.

    Documentation: Architecture ·


    2. Portkey

    (Best for enterprise governance)

    • Unified provider API.
    • Configurable fallback chains.
    • Weighted load balancing.
    • Retries and conditional routing.
    • Guardrails, logging and observability.

    Good when reliability policies and centralized governance are important.

    Documentation: Gateway repository ·


    3. Kong AI Gateway

    • Provider-agnostic API.
    • Request format conversion.
    • Credential injection.
    • Load balancing and priority-based failover.
    • Enterprise API management and observability.

    Strong choice if your organization already uses Kong or needs broader API governance.

    Documentation: Architecture ·


    • OpenRouter Logo PNG & SVG Vector

    4. OpenRouter

    • Single API for hundreds of models.
    • Provider-level automatic failover.
    • Model-level fallback chains.
    • Routing based on price, latency and reliability.
    • Minimal infrastructure to operate.

    Best when you want to outsource provider aggregation and routing rather than operate your own gateway.

    Documentation: Provider selection ·

    3. Feature comparison

    CapabilityLiteLLMPortkeyKongOpenRouter
    Unified APIYesYesYesYes
    Provider normalizationYesYesYesYes
    Automatic retriesYesYesYesYes
    Provider failoverYesYesYesYes
    Model fallbackYesYesConfigurableYes
    Load balancingYesYesYesYes
    Self-hostingYesYesYesNo
    Centralized governanceYesYesYesLimited

    Sources:

    4. How I would implement failover

    I recommend implementing failover at two separate levels.

    Level 1: Provider failover

    Keep the same logical model, but change the provider.

    Example:

    Logical model: Claude Sonnet
    
        Anthropic API
             |
             | 429 / 500 / timeout
             v
        AWS Bedrock
             |
             | failure
             v
        Google Vertex AI

    This preserves model identity as much as possible.

    Level 2: Model failover

    If all providers for the primary model fail, switch to a compatible alternative.

    Primary: Claude Sonnet
             |
             v
       All providers fail
             |
             v
    Fallback: GPT model
             |
             v
    Fallback: Gemini model

    The second level can change response characteristics, so it should be explicitly configured.

    LiteLLM documents both deployment-level retries and cross-model fallback mechanisms.

    5. Important implementation considerations

    Regardless of which product you choose, I would build around these principles.

    ComponentRecommended behavior
    Provider registryStore provider endpoints, models, capabilities and credentials
    NormalizationConvert requests into a canonical internal schema
    Provider adaptersTranslate canonical requests into provider-specific APIs
    Response normalizationReturn consistent responses, usage and finish reasons
    Error classificationDistinguish 429, 5xx, timeout, authentication and invalid requests
    Circuit breakerTemporarily remove unhealthy providers from routing
    Retry policyExponential backoff with jitter
    Failover policyConfigurable provider and model fallback chains
    ObservabilityTrack latency, tokens, cost, provider and failure rates

    A few particularly important details:

    • Don't retry non-retryable errors such as invalid request parameters.
    • Respect provider Retry-After headers.
    • Avoid retrying a streaming request after partial output has already reached the client unless you explicitly support restarting the stream.
    • Check model capabilities before routing, particularly tool calling, structured output, vision and context-window requirements.
    • Make provider selection and fallback decisions observable.
    • Use a shared state store such as Redis when running multiple gateway instances.

    6. My recommended deployment architecture

    For a production system, I'd start with:

    Production reference stack

    Application layer

    Your existing applications and services

    LiteLLM Gateway

    Normalization, routing, authentication, retries and fallbacks

    Redis

    Cooldowns, routing state and caching

    PostgreSQL

    Configuration, keys and usage records

    OpenAI

    Anthropic

    Bedrock

    Additional adapters can support Azure, Google Vertex AI, Gemini, Mistral and self-hosted models.

    I'd deploy multiple gateway instances behind a load balancer, use Redis for shared routing health state, PostgreSQL for persistent configuration, and centralized telemetry for operational monitoring.

    My recommendation: Start with LiteLLM unless you have a compelling enterprise governance requirement for Portkey or Kong. It gives you a relatively complete foundation without requiring you to build the routing and provider-normalization layers yourself.


    7. A few architectural questions to narrow down the right solution

    The best architecture depends on whether you're building a shared enterprise platform or a gateway for one application.

    1. Where do you want to run the gateway?

    Self-hosted (Kubernetes, Docker, VMs)

    Managed cloud service

    Hybrid (self-hosted with managed fallback)


    2. How many providers do you expect to support?

    2–3 providers

    5–10 providers

    10+ providers, including custom endpoints


    3. What API normalization do you want?

    OpenAI-compatible API as the standard interface

    Custom canonical schema independent of providers

    Both, with support for provider-specific extensions


    4. Expected traffic volume?

    Under 100 requests/second

    100–1,000 requests/second

    Over 1,000 requests/second


    5. What's the primary objective?

    High availability and automatic failover

    Cost optimization and intelligent routing

    Security, governance and centralized management

    Complete control over the architecture and implementation

    [Design my architecture]

    With these answers, I can put together a more concrete architecture including component selection, deployment topology, provider adapter interfaces, routing algorithms, failover state machines and an implementation roadmap.

    Google AI ModeNov 18, 2026

    1. 1OpenAI
    2. 2Anthropic
    3. 3vLLM
    4. 4LiteLLM
    5. 5Langfuse
    6. 6OpenRouter
    7. 7Vercel AI Gateway
    8. 8Kong
    9. 9Cloudflare Workers
    10. 10FastAPI
    11. 11Microsoft Azure
    12. 12Portkey AI

    Managing connections across multiple LLM providers requires an LLM Gateway / Model Router architecture . This pattern places a single proxy layer between your application and various providers (OpenAI, Anthropic, Gemini, Azure, local vLLM, etc.), handling request normalization, retries, fallbacks, and load balancing.

    Core Architectural Pattern

    [ Your App / Agents ]
             │ (Unified OpenAI-compatible API)
             ▼
     ┌─────────────────────────────────────────────────────────┐
     │                   LLM Gateway / Router                  │
     │  ┌─────────────────┐ ┌───────────────┐ ┌─────────────┐  │
     │  │  Normalization  │ │  Load Balanc. │ │  Failover   │  │
     │  └─────────────────┘ └───────────────┘ └─────────────┘  │
     └───────────────────────────┬─────────────────────────────┘
             ┌───────────────────┼───────────────────┐
             ▼                   ▼                   ▼
        [ OpenAI ]        [ Anthropic ]         [ Local / vLLM ]
    • Request Normalization: Translates a standard input format (typically OpenAI’s /v1/chat/completions schema) into provider-specific payloads (e.g., Anthropic Messages API, Vertex AI format) and normalizes streaming/non-responses back.
    • Failover & Cooldowns: Automatically catches rate limits (429 ), server errors (5xx ), or context/content policy triggers and routes the retry down an ordered fallback list.
    • Observability & Budgeting: Tracks token usage, costs per key/team, and latency across all underlying providers.

    Top Architecture & Gateway Options

    1. LiteLLM Proxy (Open Source / Self-Hosted)

    • Best for: The standard self-hosted default for production teams needing high control.
    • Normalization: Exposes a universal OpenAI-compatible REST API for 100+ providers.
    • Failover: Supports ordered fallbacks, per-model/per-provider cooldowns, and distinct strategies for context-window versus content-policy errors.
    • Ecosystem: Easily pairs with observability tools like Langfuse for tracing and cost attribution.

    2. Portkey (Open Source Core / Managed & Self-Hosted)

    • Best for: Enterprise governance, compliance, and fine-grained request routing.
    • Normalization: Unified API with built-in semantic caching and format translation.
    • Failover: Configurable ordered lists triggered by any non-2xx status code or specific HTTP errors, with ultra-low added proxy overhead (<1 ms).

    3. OpenRouter (Managed Service / API Router)

    • Best for: Rapid prototyping or multi-provider routing without managing infrastructure.
    • Normalization: Single API endpoint mapped across hundreds of frontier and open-weight models.
    • Failover: Handles dynamic fallbacks based on context length, moderation flags, rate limits, or provider downtime, billing dynamically for the model that actually services the request.

    4. Vercel AI Gateway (Managed Edge Router)

    • Best for: Next.js and edge-first application architectures looking for ultra-low latency.
    • Normalization: Seamless integration with the AI SDK and standard provider handshakes.
    • Failover: Fast edge-native routing and caching.

    5. Custom Gateway (Kong / Cloudflare Workers / FastAPI + Custom Proxy)

    • Best for: Organizations with heavy existing API management infrastructure.
    • Kong / Cloudflare: Good for edge caching and rate-limiting, though custom translation logic for complex tool-use/streaming schemas across providers must be written or maintained.

    Key Production Trade-offs to Keep in Mind

    1. Mid-Conversation State vs. Gateway Failover:
      No gateway magically migrates a half-finished multi-turn conversation or server-side prompt cache from Provider A to Provider B mid-stream. Your application layer still needs to own and pass the clean transcript context if a failover triggers.
    2. Streaming & Tool-Call Normalization:
      Function/tool calling schemas differ subtly between OpenAI and Anthropic. A good gateway abstracts these differences, but complex streaming tool-use should always be integration-tested against all failover targets.
    3. Latency Overhead:
      Self-hosted proxies (like LiteLLM or Portkey) typically add a p95 overhead of roughly 1 ms to 8 ms, which is negligible compared to TTFT (Time to First Token) from the upstream LLM.

    Are you planning to self-host the gateway (e.g., Kubernetes / Docker) or looking for a fully managed SaaS , and which primary providers (e.g., OpenAI, Anthropic, local vLLM) are you trying to unify?

  • ChatGPTNov 9, 2026

    1. 1Anthropic
    2. 2Google
    3. 3LiteLLM
    4. 4Kong AI Gateway
    5. 5Cloudflare AI Gateway
    6. 6OpenRouter
    7. 7Gemini
    8. 8Claude
    9. 9Redis
    10. 10PostgreSQL
    11. 11Prometheus
    12. 12Grafana
    13. 13OpenTelemetry
    14. 14Portkey AI

    What you're describing is generally called an LLM Gateway, AI Gateway, or LLM Router.

    It sits between your application and multiple model providers, exposing a unified API while handling provider selection, retries, failover, request/response translation, and observability.

    For your requirements, I'd focus on solutions that provide three capabilities:

    1. Provider abstraction: One API/interface for OpenAI, Anthropic, Google, AWS Bedrock, Azure, self-hosted models, etc.
    2. Intelligent routing and failover: Retry, load balancing, health-aware routing, fallback to another provider, and potentially cost/latency-based selection.
    3. Format normalization: Convert requests and responses between provider-specific formats, ideally without changing your application code.

    1. Tools worth evaluating

    These are the strongest candidates to investigate.

    • LiteLLM

    1. LiteLLM

    (Best starting point for self-hosting)

    • OpenAI-compatible API across many providers.
    • Provider and model fallback chains.
    • Retries, cooldowns, load balancing and routing strategies.
    • Centralized credentials, budgets and usage tracking.
    • Python SDK or standalone proxy service.

    Particularly interesting: it distinguishes retrying another deployment of the same model from falling back to a different model group.

    Documentation:


    • Best AI Gateway Solutions

    2. Portkey

    (Best managed gateway candidate)

    • Universal API for multiple LLM providers.
    • Request and response normalization.
    • Configurable fallback and retry strategies.
    • Conditional routing and load balancing.
    • Logging, tracing, analytics and governance.

    Worth evaluating if you want gateway capabilities without building as much infrastructure yourself.

    Documentation:


    • Announcing General Availability of Kong Gateway 3.0  | Kong Inc.

    3. Kong AI Gateway

    (Best for enterprise API infrastructure)

    • OpenAI-compatible standardized interface.
    • Translates provider-specific request and response formats.
    • Multiple upstream targets per model.
    • Priority-based failover, round-robin and latency-based routing.
    • Authentication, policies, rate limiting and centralized governance.

    Especially compelling if you already use Kong for conventional API management.

    Documentation:


    • Free High-Quality Cloudflare Logo Svg for Creative Design

    4. Cloudflare AI Gateway

    • Provider fallbacks and request timeouts.
    • Unified provider routing.
    • Automatic model selection through Auto Router.
    • Integration with Cloudflare's infrastructure.

    Good if your infrastructure already runs on Cloudflare. Its Auto Router also considers request capabilities and provider health.

    Documentation:


    • OpenRouter Logo PNG & SVG Vector

    5. OpenRouter

    • One API across a large model catalog.
    • Provider selection and provider-level failover.
    • Model-level fallback chains.
    • Provider ordering and routing preferences.

    Best when you want to consume a hosted routing service rather than operate your own gateway. Less suitable if you need complete control over gateway infrastructure.

    Documentation:

    2. Architecture I'd recommend

    For a system that needs to support multiple providers while keeping the application independent of them, I'd use a layered gateway architecture.

    • What is an AI Gateway? Architecture, Features, and Why It Matters

    The conceptual architecture:

                 Applications
                       |
                       v
              Unified API Interface
                       |
                       v
              Request Normalization
                       |
                       v
               Routing Engine
              /       |       \
          Policy    Health    Load
          Engine    Monitor   Balancer
              \       |       /
                       |
                       v
               Provider Adapters
              /       |        \
           OpenAI  Anthropic   Gemini
              |       |         |
              v       v         v
           Provider-specific APIs
                       |
                       v
              Response Normalization
                       |
                       v
                 Applications

    I'd divide the system into these components:

    ComponentResponsibility
    API interfaceStable application-facing contract
    NormalizerTranslate request/response formats
    RouterSelect provider and model
    Health monitorTrack availability and failure rates
    Failover managerRetry and switch providers
    Provider adaptersHandle authentication and API differences
    ObservabilityCapture latency, cost, errors and usage

    3. Important design decisions

    There are a few architectural decisions that will determine whether this works reliably in production.

    A. Choose a canonical API format

    I'd start with an OpenAI-compatible interface as the canonical format.

    For example, your application sends:

    json
    {
      "model": "fast-general",
      "messages": [
        {
          "role": "user",
          "content": "Summarize this document"
        }
      ],
      "temperature": 0.2
    }

    The gateway translates this into the appropriate provider-specific format.

    Internally:

    • OpenAI adapter → OpenAI API format.
    • Anthropic adapter → Anthropic Messages format.
    • Gemini adapter → Gemini API format.
    • Bedrock adapter → Bedrock-specific format.

    Responses are translated back into your canonical format.

    Important: Normalization shouldn't mean pretending every provider supports identical capabilities. Tool calling, structured output, multimodal inputs, reasoning metadata and streaming need explicit capability mapping.

    B. Implement intelligent failover

    I'd use a policy-based fallback system rather than simply retrying every error.

    For example:

    Example routing policy

    Primary (Priority 1)

    OpenAI GPT model

    Secondary (Priority 2)

    Anthropic Claude model

    Tertiary

    Google Gemini model

    Suggested policies:

    FailureRecommended behavior
    HTTP 429Retry with backoff or switch provider
    HTTP 500/502/503Retry, then fail over
    TimeoutRetry within request deadline
    Authentication errorAlert; don't blindly retry
    Invalid requestReturn error
    Context window exceededSwitch to a compatible larger-context model
    Unsupported featureRoute to a provider supporting that feature

    Also implement circuit breakers so unhealthy providers are temporarily removed from routing.

    Avoid multiple independent retry loops in your SDK, application and gateway, as they can multiply request volume.

    C. Preserve provider-specific capabilities

    I would not discard provider-specific information during normalization.

    A good internal response object should preserve:

    • Normalized response content.
    • Original provider response metadata.
    • Provider and model identifiers.
    • Token usage.
    • Finish reason.
    • Tool calls.
    • Reasoning metadata where supported.
    • Streaming events.

    This gives you a stable application API without losing important provider functionality.

    4. Build versus buy

    Here's how I'd narrow the choices:

    RequirementMy choice
    Self-hosted, customizable routingLiteLLM
    Managed gateway and observabilityPortkey
    Enterprise API managementKong
    Cloudflare-native infrastructureCloudflare AI Gateway
    Simplest hosted multi-provider accessOpenRouter

    My first two candidates would be LiteLLM and Portkey.

    I'd prototype the same set of requests against both, particularly cross-provider failover and streaming response normalization.

    5. A more advanced architecture to consider

    If you're building a gateway that will become a core infrastructure component, I'd consider separating the routing engine from the provider adapters.

    For example:

                        LLM Gateway
                             |
                     Canonical Request
                             |
                     Routing Policies
                             |
                  Provider Selection Engine
                             |
                  +----------+----------+
                  |          |          |
               Adapter A  Adapter B  Adapter C
                  |          |          |
               OpenAI    Anthropic   Gemini
                  |          |          |
                  +----------+----------+
                             |
                    Canonical Response

    This makes it possible to add new providers without rewriting the routing logic.

    I'd also consider:

    • Redis for distributed health and circuit-breaker state.
    • PostgreSQL for provider configurations and routing policies.
    • Prometheus/Grafana for latency and availability metrics.
    • OpenTelemetry for distributed tracing.
    • A configurable policy engine for routing decisions.

    6. A few questions to narrow down the best solution

    The right recommendation depends considerably on whether you're looking for an existing product or building your own abstraction layer.

    1. How do you want to deploy it?

    Self-hosted / open source

    Managed SaaS

    Hybrid (self-hosted gateway + managed services)

    Open to any option


    2. What are you looking to build?

    Use an existing LLM gateway

    Build a custom routing and normalization layer

    A complete multi-provider LLM platform


    3. Which providers do you need?

    Select all that apply.


    4. What's your primary priority?

    Reliability and automatic failover

    Cost optimization

    Lowest latency

    Consistent API and format normalization

    All of the above

    [Get tailored architecture recommendations]

    With these answers, I can narrow this down to a concrete architecture and recommend whether you should adopt LiteLLM, Portkey, Kong, or build a dedicated routing layer.

    Google AI ModeNov 9, 2026

    1. 1OpenAI
    2. 2LiteLLM
    3. 3Portkey AI
    4. 4OpenRouter
    5. 5Cloudflare AI Gateway
    6. 6Vercel AI SDK
    7. 7Ollama
    8. 8GPT-4o
    9. 9vLLM
    10. 10Llama 3.3 70B
    11. 11Deepseek R1

    To route requests across multiple LLM providers with normalized formats and automatic failover , you are looking for an LLM Gateway or Proxy architecture . Instead of writing custom retry logic and provider-specific adapters in your application code, an abstraction layer standardizes the interface (typically mimicking the OpenAI API spec) and handles operational resilience.

    Top Tools to Consider

    1. Self-Hosted Gateways (Best for Data Privacy & Custom Control)

    • LiteLLMLiteLLM (Proxy): The most popular open-source, Python-heavy standard. It exposes an OpenAI-compatible REST API for 100+ LLM providers, supports YAML-defined fallback and retry chains, virtual keys, team-based budgets, and prompt/response logging. Learn more on the LiteLLM DocumentationLiteLLM Documentation.
    • PortkeyPortkey : A lightweight, high-performance gateway (Node.js/Rust core) offering MIT-licensed self-hosting or a managed cloud tier. It features granular JSON-based routing rules, conditional fallbacks based on specific HTTP status codes or latency thresholds, and built-in guardrails/observability. Explore via Portkey AIPortkey AI.

    2. Managed / Hosted Gateways (Best for Zero-Infra Setup)

    • OpenRouterOpenRouter : Acts as a universal API endpoint routing across hundreds of commercial and open-source models with automatic fallback options and unified billing. Best if you don’t mind third-party routing infrastructure. Check out OpenRouter.
    • Cloudflare AI GatewayCloudflare AI Gateway : Sits in front of your LLM calls to provide caching, rate-limiting, analytics, and basic fallback/retry mechanics with minimal latency overhead. Details on Cloudflare AI Gateway.
    • Vercel AI SDKVercel AI SDK / AI Gateway: Great if your application stack is built heavily around TypeScript/Next.js and you want a unified streaming and provider-abstraction layer. See Vercel AI SDK.

    Core Architectural Patterns

    • The Proxy / Sidecar Pattern: Your application sends every LLM request to a local or centralized gateway instance using a standard format (e.g., /v1/chat/completions ). The gateway translates the payload into Anthropic, Gemini, Bedrock, or Ollama formats on the fly.
    • Tiered Fallback Chains: Configure sequential fallback rules. For example:
      1. Primary: claude-3-5-sonnet (via Anthropic)
      2. Fallback 1: gpt-4o (via OpenAI) if Anthropic returns a 5xx or rate-limit (429) error.
      3. Fallback 2: deepseek-r1 or a local self-hosted model via vLLM if both major APIs experience a wider outage.
    • Semantic or Rule-Based Routing: Use conditional rules where simple or high-volume queries route to cheaper/faster models (e.g., Llama 3.3 70B), while complex reasoning tasks automatically get bumped to frontier models.

    To help narrow down the best fit, tell me:

    • Do you prefer to self-host (for data residency/compliance) or use a managed cloud service?
    • What is your primary application stack (Python, TypeScript, Go, etc.)?
    • Are you prioritizing cost optimization, lowest latency , or maximum uptime/failover?
  • ChatGPTOct 29, 2026

    1. 1LiteLLM
    2. 2OpenRouter
    3. 3Vercel AI Gateway
    4. 4Vercel AI SDK
    5. 5Cloudflare AI Gateway
    6. 6Portkey AI
    7. 7Microsoft Azure

    If you want to route requests across different LLM providers while handling failover and normalizing inputs, the most useful architecture is an LLM gateway with a provider adapter layer, policy-based routing, and centralized retry/fallback logic.

    This lets your application use one internal API while routing requests to OpenAI, Anthropic, Google, self-hosted models, or other providers. It also helps you standardize message formats, tool calls, model parameters, errors, and response handling.

    1. Architecture options

    A. Unified LLM gateway

    A single API sits between your application and all providers. It normalizes requests and responses, applies routing rules, and handles failover.

    Best for: most production applications, multiple providers, centralized governance.


    B. Application-level adapter pattern

    Your application defines a common interface, with one adapter per provider. Routing and retries live in your own code.

    Best for: smaller systems, strict customization, minimal infrastructure.


    C. Model router with intelligent selection

    A router chooses models using task type, latency, cost, quality, context length, or live provider health.

    Best for: high-volume systems, cost optimization, workloads with different quality requirements.


    D. Resilient multi-provider control plane

    A gateway combines routing with circuit breakers, timeouts, concurrency limits, quotas, observability, and provider-specific policies.

    Best for: enterprise systems and applications where availability is critical.

    These approaches are complementary: for example, you can use a unified gateway with an intelligent router and a resilience layer.

    2. Tools worth evaluating

    These are the strongest options to shortlist, depending on whether you want self-hosting, a managed service, or routing inside your application.

    • LiteLLM

    LiteLLM — self-hosted gateway and routing

    Provides an OpenAI-compatible API across many providers, plus retries, fallbacks, load balancing, virtual keys, budgets, and observability integrations. Its routing layer supports strategies such as latency-based and rate-limit-aware routing.

    Choose it when you want infrastructure control and a configurable, provider-neutral API.

    Documentation


    • Remote Admin Jobs

    OpenRouter — managed multi-provider routing

    Offers a unified API, automatic provider-level failover, provider ordering, and optional model-level fallbacks. This is useful when you want multiple inference providers without operating the routing infrastructure yourself.

    Choose it for quick integration and broad model access.

    Provider routing documentation


    • AI Gateway

    Vercel AI Gateway — managed routing with AI SDK integration

    Supports provider preference and filtering, provider timeouts, and ordered model fallbacks. A good fit for TypeScript applications using the Vercel AI SDK.

    Choose it when you're already using the AI SDK or deploying on Vercel.

    Gateway documentation


    • We open sourced our AI gateway written in TS - DEV Community

    Portkey — policy-oriented gateway

    Offers conditional routing, automatic retries, fallbacks, load balancing, timeouts, and request tracing. Useful for teams that want routing policies and operational controls managed through a gateway.

    Choose it for centralized policies, observability, and managed operations.

    AI Gateway features


    • LLM Gateway | Cloudflare

    Cloudflare AI Gateway — configurable dynamic routing

    Its dynamic routing feature lets you build conditional routing flows with quotas, model selection, and fallbacks, including versioned route configurations.

    Choose it when you want routing policies that can evolve without application-code changes.

    Dynamic routing documentation

    3. How to normalize requests across providers

    The key is to normalize into your own canonical request format, rather than assuming that a shared API schema makes every provider feature equivalent.

    A practical internal contract could look like this:

    typescript
    type LLMRequest = {
      task: "chat" | "json" | "embedding";
      messages: Array<{
        role: "system" | "user" | "assistant" | "tool";
        content: string | Array<ContentPart>;
      }>;
      modelTier?: "fast" | "balanced" | "best";
      temperature?: number;
      maxOutputTokens?: number;
      tools?: ToolDefinition[];
      responseFormat?: "text" | "json";
      timeoutMs?: number;
      metadata?: Record<string, string>;
    };

    Then implement the following pipeline:

    1. Application request

    Your canonical schema

    2. Validation and normalization

    Validate roles, content, tokens, tools, and capabilities

    3. Policy router

    Select eligible models using health, cost, latency, and task

    4. Provider adapter

    Map to native API fields and provider-specific capabilities

    5. Response normalizer

    Standardize output, usage, finish reasons, and errors

    Important normalization rules:

    • Messages: Convert roles and multimodal content into each provider's supported format.
    • Parameters: Define canonical meanings for token limits, temperature, stop sequences, and structured output; don't assume parameter equivalence.
    • Tools: Normalize tool definitions and results, while handling differences in tool-call IDs and argument formats.
    • Capabilities: Track whether each model supports vision, JSON schema, tool calling, streaming, and the context length you need.
    • Responses: Return a stable envelope with content, tool calls, usage, provider/model metadata, and normalized error categories.
    • Unsupported features: Reject them clearly or use an explicit downgrade policy. Never silently discard a critical parameter.

    For a ready-made OpenAI-compatible interface, LiteLLM is a particularly relevant starting point. If you need a custom contract, keep the canonical schema in your application and use the gateway as the transport and routing layer.

    4. Failover design: what to implement

    Use two distinct fallback levels. They solve different problems.

    Level 1: Provider failover

    Keep the same model but switch provider or deployment.

    Example: OpenAI-hosted deployment → Azure-hosted deployment of the same model family.


    Level 2: Model fallback

    Switch to a different compatible model when the primary model is unavailable or cannot handle the request.

    Example: primary reasoning model → smaller general-purpose model for a supported task.

    Gateways such as LiteLLM, OpenRouter, and Vercel AI Gateway support variations of these mechanisms.

    Recommended production behavior:

    FailureRecommended handling
    HTTP 429 rate limitRetry with backoff or use another eligible deployment
    HTTP 5xxFail over to a healthy provider
    Connection timeoutApply a deadline, then try a backup
    Invalid request / unsupported featureFix the request or route to a compatible model; don't repeat unchanged
    Context limit exceededShorten input or select a model with sufficient context
    Authentication / permission errorMark the affected credential or route unhealthy; alert operators
    Failure after streaming beginsUsually terminate the stream; restarting risks duplicate or contradictory output

    Use bounded retries, exponential backoff with jitter, per-provider circuit breakers, and an overall request deadline. Be careful retrying operations after a timeout if the provider might already have processed the request.

    5. A practical implementation plan

    I'd build this in stages:

    1. Start with a unified API. Use LiteLLM if you want self-hosting, or a managed gateway if minimizing operations is more important.
    2. Define a canonical schema. Normalize messages, content parts, tool definitions, model requirements, and response metadata.
    3. Create a capability registry. Record which models support required features and maintain limits for context size, output tokens, and modalities.
    4. Add deterministic routing. Begin with a primary model and an ordered list of compatible fallbacks.
    5. Add resilience controls. Configure timeouts, retry budgets, circuit breakers, rate limits, and concurrency caps.
    6. Instrument every attempt. Record the selected provider, model, latency, retry reason, token usage, and cost.
    7. Test semantic compatibility. Run the same representative requests through each provider, especially tool calls, JSON output, and streaming.

    One additional standard worth investigating is Open Responses. It defines a shared schema and interoperability approach for model requests, responses, streaming, and tool workflows.

    My recommendation: for a typical production service, start with a gateway plus a canonical application-level request contract. Keep routing policies and provider adapters separate so you can replace the gateway later without rewriting your business logic. Add dynamic cost- or latency-based routing only after you have enough telemetry to make those decisions reliably.

    Google AI ModeOct 29, 2026

    1. 1LiteLLM
    2. 2Anthropic
    3. 3Portkey AI
    4. 4Apache APISIX
    5. 5Kong AI Gateway
    6. 6GPT-4o
    7. 7Bifrost by Maxim AI
    8. 8DeepSeek Chat

    Routing requests across multiple LLM providers while managing input normalization (usually translating diverse provider payloads into an OpenAI-compatible schema ) and automatic failover is best achieved using specialized AI Gateways or Reverse Proxy architectures.

    Recommended Tools & Open-Source Gateways

    Instead of building custom retry loops and mapping layers from scratch, production setups typically leverage dedicated proxy servers or SDK-based routers:

    1. LiteLLM Proxy / SDK
      • What it does: Translates standard OpenAI format requests to over 100+ providers (Anthropic, Vertex, Bedrock, Mistral, Cohere). Supports built-in load balancing, fallback/failover chains, budget tracking, and rate limiting.
      • Form Factor: Can be deployed as a standalone reverse proxy container or used as a Python library.
    2. Portkey AI Gateway
      • What it does: An enterprise-grade, open-source AI gateway featuring robust fallback routing, semantic caching, request/response logging, and standard OpenAI-compatible API normalization.
      • Form Factor: Cloud-hosted or self-hosted proxy with extensive observability integration (e.g., Langfuse, Datadog).
    3. Bifrost
      • What it does: A high-performance, standalone Go-based gateway optimized for multi-provider routing, hedging, and low-latency fallbacks across 25+ model providers.
      • Form Factor: Lightweight binary/container well-suited for microservice meshes.
    4. Apache APISIX (with AI Plugins) / Kong AI Gateway
      • What it does: Traditional API gateways extended with native AI plugins (ai-proxy and ai-proxy-multi ). They handle upstream token tracking, load balancing, and fallback behavior natively at the infrastructure edge.
      • Form Factor: Enterprise API Gateway integrated with Kubernetes or bare-metal infrastructure.

    Core Architectural Patterns

    When designing your routing topology, choose one of these standard structural implementations:

    • The Centralized Reverse-Proxy Pattern (Recommended)
      • Flow: App Client -> AI Gateway Proxy (LiteLLM/Portkey) -> Provider API (OpenAI/Anthropic/Bedrock)
      • Why use it: Your application code only ever talks to a single, static OpenAI-compatible endpoint. The proxy intercepts the payload, translates it on the fly to the target downstream provider, tracks quotas, and executes instant fallbacks if a 429 (Rate Limit) or 5xx error returns.
    • The Sidecar / Service Mesh Pattern
      • Flow: App container communicates with a local host-level gateway instance (like an Envoy-managed or lightweight local proxy container) which coordinates cluster-wide failovers.
      • Why use it: Minimizes external network hops for strict latency requirements and decouples credential management entirely from application runtimes.

    How Normalization and Failover Work Under the Hood

    • Input Normalization: Gateways accept a standard request body (/v1/chat/completions ). If the target fallback switches from OpenAI to Anthropic Claude, the proxy maps JSON parameters dynamically (e.g., translating max_tokens , system roles, or tool definitions into Anthropic's distinct schema format) before dispatching the HTTP call.
    • Failover Logic: You define a priority list (e.g., primary: gpt-4o , fallback 1: claude-3-5-sonnet , fallback 2: deepseek-chat ). If the primary upstream times out or returns a service error, the router catches the exception internally, rewrites the payload for the backup provider, and completes the request seamlessly within the same client connection window.

    If you'd like to narrow this down, let me know:

    • Are you looking for a self-hosted open-source proxy or a managed cloud service?
    • Do you need integration with specific observability or cost-tracking tools?

    I can provide a reference configuration setup for your preferred architecture.

For agents: this page as Markdown, parse.gl/topics/llm-provider-routing.md.

This page

Reuse: Figures and tables on this page are free to republish with attribution: a link to https://parse.gl/topics/llm-provider-routing beside them. A standard followed link (dofollow) is preferred.

Read and search

Every brand's share on this topic, with 95% ranges: GET agent.parse.gl/v1/tools/query_metrics?args=%7B%22measures%22%3A%5B%22share_named%22%5D%2C%22by%22%3A%5B%22brand%22%5D%2C%22filter%22%3A%7B%22topic%22%3A%5B%22top_cf5be6d8-85a2-4f3f-b0bd-670db754c457%22%5D%7D%7D
Search the index: GET parse.gl/api/public/search?q=<words>

Connect

Connect an assistant: agent.parse.gl/mcp

An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces

What it costs

No account: Public figures and up to 5 examples per list, about 120 brands a day per address.
Free account: $5 in credit, your brand's first check (20 buyer questions) and a free weekly check.
After your first top-up: Exports and deeper search results. The first top-up starts at $5.
Live checks: $0.01 per question per engine.
Full public answers and pages: 25 free per day, then $0.005 each.
Public answers older than 90 days: $0.001 each the first time your workspace counts or lists one, then yours.