LLM provider routing
Since Sep 24Updated Nov 20, 2026Based on 20 AI answers
Reviewed by Dimitry Apollonsky
Brands AI recommends
Also mentioned rarely: Gemini, GPT-4o, Bifrost by Maxim AI, Vercel AI Gateway, FastAPI, PostgreSQL, vLLM, Claude · anthropic.com, Apache APISIX, Claude · claude.ai, RouteLLM, Redis, OpenAI, Vercel AI SDK, AWS, Agent Router, Axonhub, Azure OpenAI Service, DeepSeek Chat, ClickHouse, Google Cloud Gemini Enterprise Agent Platform, Deepseek R1, Maxim AI, Go, Google, Grafana, Helicone, Kong, LangChain, LangSmith, LangChain LangGraph, Llama 3.3 70B, Meta AI, Ollama, OpenTelemetry, Prometheus, PyBreaker, Requesty, Cloudflare Workers, Amazon Bedrock, Microsoft Azure, Cohere, Google Gemini, GPT-5, Langfuse
- Portkey AIChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 90%
- LiteLLMChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 90%
- OpenRouterChatGPTChatGPT 100%Google AI ModeGoogle AI Mode 60%
- Cloudflare AI GatewayChatGPTChatGPT 60%Google AI ModeGoogle AI Mode 40%
- Kong AI GatewayChatGPTChatGPT 80%Google AI ModeGoogle AI Mode 20%
Questions and answers
ChatGPTNov 18, 2026
Architecture options for a multi-provider LLM gateway
What you're describing is generally called an LLM Gateway or LLM Router.
It sits between your application and multiple LLM providers, providing a unified API, request/response normalization, provider selection, load balancing, retries, and automatic failover.
Rather than integrating OpenAI, Anthropic, Google, AWS Bedrock, Azure, and other providers independently into your application, you integrate with one abstraction layer.
1. Recommended high-level architecture
Client applications
Web apps · Backend services · Agents · APIs
LLM Gateway API
Unified OpenAI-compatible interface
Normalization
Request mapping
Response mapping
Error translation
Routing engine
Provider selection
Load balancing
Policies
Reliability
Retries
Circuit breakers
Failover
Governance
API keys
Rate limits
Cost tracking
Provider adapters
External LLM services and self-hosted models
The important architectural decision is to separate three concerns:
- Normalization: Your application shouldn't need to understand individual provider APIs.
- Routing: The gateway decides which provider/model receives the request.
- Resilience: Provider failures shouldn't automatically become application failures.
2. Available technology options
These are the main candidates worth evaluating.
1. LiteLLM
(Best overall self-hosted option)
- OpenAI-compatible API across 100+ providers.
- Request and response normalization.
- Retries, fallbacks, cooldowns and load balancing.
- Redis-backed shared routing state.
- Spend tracking, virtual keys and budgets.
Particularly attractive if you want to own the gateway infrastructure.
Documentation: Architecture ·
2. Portkey
(Best for enterprise governance)
- Unified provider API.
- Configurable fallback chains.
- Weighted load balancing.
- Retries and conditional routing.
- Guardrails, logging and observability.
Good when reliability policies and centralized governance are important.
Documentation: Gateway repository ·
3. Kong AI Gateway
- Provider-agnostic API.
- Request format conversion.
- Credential injection.
- Load balancing and priority-based failover.
- Enterprise API management and observability.
Strong choice if your organization already uses Kong or needs broader API governance.
Documentation: Architecture ·
4. OpenRouter
- Single API for hundreds of models.
- Provider-level automatic failover.
- Model-level fallback chains.
- Routing based on price, latency and reliability.
- Minimal infrastructure to operate.
Best when you want to outsource provider aggregation and routing rather than operate your own gateway.
Documentation: Provider selection ·
3. Feature comparison
Capability LiteLLM Portkey Kong OpenRouter Unified API Yes Yes Yes Yes Provider normalization Yes Yes Yes Yes Automatic retries Yes Yes Yes Yes Provider failover Yes Yes Yes Yes Model fallback Yes Yes Configurable Yes Load balancing Yes Yes Yes Yes Self-hosting Yes Yes Yes No Centralized governance Yes Yes Yes Limited Sources:
4. How I would implement failover
I recommend implementing failover at two separate levels.
Level 1: Provider failover
Keep the same logical model, but change the provider.
Example:
Logical model: Claude Sonnet Anthropic API | | 429 / 500 / timeout v AWS Bedrock | | failure v Google Vertex AIThis preserves model identity as much as possible.
Level 2: Model failover
If all providers for the primary model fail, switch to a compatible alternative.
Primary: Claude Sonnet | v All providers fail | v Fallback: GPT model | v Fallback: Gemini modelThe second level can change response characteristics, so it should be explicitly configured.
LiteLLM documents both deployment-level retries and cross-model fallback mechanisms.
5. Important implementation considerations
Regardless of which product you choose, I would build around these principles.
Component Recommended behavior Provider registry Store provider endpoints, models, capabilities and credentials Normalization Convert requests into a canonical internal schema Provider adapters Translate canonical requests into provider-specific APIs Response normalization Return consistent responses, usage and finish reasons Error classification Distinguish 429, 5xx, timeout, authentication and invalid requests Circuit breaker Temporarily remove unhealthy providers from routing Retry policy Exponential backoff with jitter Failover policy Configurable provider and model fallback chains Observability Track latency, tokens, cost, provider and failure rates A few particularly important details:
- Don't retry non-retryable errors such as invalid request parameters.
- Respect provider
Retry-Afterheaders. - Avoid retrying a streaming request after partial output has already reached the client unless you explicitly support restarting the stream.
- Check model capabilities before routing, particularly tool calling, structured output, vision and context-window requirements.
- Make provider selection and fallback decisions observable.
- Use a shared state store such as Redis when running multiple gateway instances.
6. My recommended deployment architecture
For a production system, I'd start with:
Production reference stack
Application layer
Your existing applications and services
LiteLLM Gateway
Normalization, routing, authentication, retries and fallbacks
Redis
Cooldowns, routing state and caching
PostgreSQL
Configuration, keys and usage records
OpenAI
Anthropic
Bedrock
Additional adapters can support Azure, Google Vertex AI, Gemini, Mistral and self-hosted models.
I'd deploy multiple gateway instances behind a load balancer, use Redis for shared routing health state, PostgreSQL for persistent configuration, and centralized telemetry for operational monitoring.
My recommendation: Start with LiteLLM unless you have a compelling enterprise governance requirement for Portkey or Kong. It gives you a relatively complete foundation without requiring you to build the routing and provider-normalization layers yourself.
7. A few architectural questions to narrow down the right solution
The best architecture depends on whether you're building a shared enterprise platform or a gateway for one application.
1. Where do you want to run the gateway?
Self-hosted (Kubernetes, Docker, VMs)
Managed cloud service
Hybrid (self-hosted with managed fallback)
2. How many providers do you expect to support?
2–3 providers
5–10 providers
10+ providers, including custom endpoints
3. What API normalization do you want?
OpenAI-compatible API as the standard interface
Custom canonical schema independent of providers
Both, with support for provider-specific extensions
4. Expected traffic volume?
Under 100 requests/second
100–1,000 requests/second
Over 1,000 requests/second
5. What's the primary objective?
High availability and automatic failover
Cost optimization and intelligent routing
Security, governance and centralized management
Complete control over the architecture and implementation
[Design my architecture]
With these answers, I can put together a more concrete architecture including component selection, deployment topology, provider adapter interfaces, routing algorithms, failover state machines and an implementation roadmap.
Pages cited in this answer 9
- docs.litellm.aiLife of a Request | liteLLM
- docs.litellm.aiRetries, fallbacks and cooldowns | liteLLM
- github.comgateway/README.md at main · Portkey-AI/gateway · GitHub
- portkey.aiFailover routing strategies for LLMs in production
- developer.konghq.comAI Gateway architecture - AI Gateway | Kong Docs
- developer.konghq.comAI Gateway providers | Kong Docs
- github.comdocs/guides/routing/provider-selection.mdx at main · OpenRouterTeam/docs · GitHub
- openrouter.aiOpenRouter Failover and Fallbacks Explained
- docs.litellm.aiRouter - Load Balancing | liteLLM
Google AI ModeNov 18, 2026
Managing connections across multiple LLM providers requires an LLM Gateway / Model Router architecture . This pattern places a single proxy layer between your application and various providers (OpenAI, Anthropic, Gemini, Azure, local vLLM, etc.), handling request normalization, retries, fallbacks, and load balancing.
Core Architectural Pattern
[ Your App / Agents ] │ (Unified OpenAI-compatible API) ▼ ┌─────────────────────────────────────────────────────────┐ │ LLM Gateway / Router │ │ ┌─────────────────┐ ┌───────────────┐ ┌─────────────┐ │ │ │ Normalization │ │ Load Balanc. │ │ Failover │ │ │ └─────────────────┘ └───────────────┘ └─────────────┘ │ └───────────────────────────┬─────────────────────────────┘ ┌───────────────────┼───────────────────┐ ▼ ▼ ▼ [ OpenAI ] [ Anthropic ] [ Local / vLLM ]- Request Normalization: Translates a standard input format (typically OpenAI’s
/v1/chat/completionsschema) into provider-specific payloads (e.g., Anthropic Messages API, Vertex AI format) and normalizes streaming/non-responses back. - Failover & Cooldowns: Automatically catches rate limits (
429), server errors (5xx), or context/content policy triggers and routes the retry down an ordered fallback list. - Observability & Budgeting: Tracks token usage, costs per key/team, and latency across all underlying providers.
Top Architecture & Gateway Options
1. LiteLLM Proxy (Open Source / Self-Hosted)
- Best for: The standard self-hosted default for production teams needing high control.
- Normalization: Exposes a universal OpenAI-compatible REST API for 100+ providers.
- Failover: Supports ordered fallbacks, per-model/per-provider cooldowns, and distinct strategies for context-window versus content-policy errors.
- Ecosystem: Easily pairs with observability tools like Langfuse for tracing and cost attribution.
2. Portkey (Open Source Core / Managed & Self-Hosted)
- Best for: Enterprise governance, compliance, and fine-grained request routing.
- Normalization: Unified API with built-in semantic caching and format translation.
- Failover: Configurable ordered lists triggered by any non-2xx status code or specific HTTP errors, with ultra-low added proxy overhead (<1 ms).
3. OpenRouter (Managed Service / API Router)
- Best for: Rapid prototyping or multi-provider routing without managing infrastructure.
- Normalization: Single API endpoint mapped across hundreds of frontier and open-weight models.
- Failover: Handles dynamic fallbacks based on context length, moderation flags, rate limits, or provider downtime, billing dynamically for the model that actually services the request.
4. Vercel AI Gateway (Managed Edge Router)
- Best for: Next.js and edge-first application architectures looking for ultra-low latency.
- Normalization: Seamless integration with the AI SDK and standard provider handshakes.
- Failover: Fast edge-native routing and caching.
5. Custom Gateway (Kong / Cloudflare Workers / FastAPI + Custom Proxy)
- Best for: Organizations with heavy existing API management infrastructure.
- Kong / Cloudflare: Good for edge caching and rate-limiting, though custom translation logic for complex tool-use/streaming schemas across providers must be written or maintained.
Key Production Trade-offs to Keep in Mind
- Mid-Conversation State vs. Gateway Failover:
No gateway magically migrates a half-finished multi-turn conversation or server-side prompt cache from Provider A to Provider B mid-stream. Your application layer still needs to own and pass the clean transcript context if a failover triggers. - Streaming & Tool-Call Normalization:
Function/tool calling schemas differ subtly between OpenAI and Anthropic. A good gateway abstracts these differences, but complex streaming tool-use should always be integration-tested against all failover targets. - Latency Overhead:
Self-hosted proxies (like LiteLLM or Portkey) typically add a p95 overhead of roughly 1 ms to 8 ms, which is negligible compared to TTFT (Time to First Token) from the upstream LLM.
Are you planning to self-host the gateway (e.g., Kubernetes / Docker) or looking for a fully managed SaaS , and which primary providers (e.g., OpenAI, Anthropic, local vLLM) are you trying to unify?
ChatGPTNov 9, 2026
What you're describing is generally called an LLM Gateway, AI Gateway, or LLM Router.
It sits between your application and multiple model providers, exposing a unified API while handling provider selection, retries, failover, request/response translation, and observability.
For your requirements, I'd focus on solutions that provide three capabilities:
- Provider abstraction: One API/interface for OpenAI, Anthropic, Google, AWS Bedrock, Azure, self-hosted models, etc.
- Intelligent routing and failover: Retry, load balancing, health-aware routing, fallback to another provider, and potentially cost/latency-based selection.
- Format normalization: Convert requests and responses between provider-specific formats, ideally without changing your application code.
1. Tools worth evaluating
These are the strongest candidates to investigate.
1. LiteLLM
(Best starting point for self-hosting)
- OpenAI-compatible API across many providers.
- Provider and model fallback chains.
- Retries, cooldowns, load balancing and routing strategies.
- Centralized credentials, budgets and usage tracking.
- Python SDK or standalone proxy service.
Particularly interesting: it distinguishes retrying another deployment of the same model from falling back to a different model group.
Documentation:
2. Portkey
(Best managed gateway candidate)
- Universal API for multiple LLM providers.
- Request and response normalization.
- Configurable fallback and retry strategies.
- Conditional routing and load balancing.
- Logging, tracing, analytics and governance.
Worth evaluating if you want gateway capabilities without building as much infrastructure yourself.
Documentation:
3. Kong AI Gateway
(Best for enterprise API infrastructure)
- OpenAI-compatible standardized interface.
- Translates provider-specific request and response formats.
- Multiple upstream targets per model.
- Priority-based failover, round-robin and latency-based routing.
- Authentication, policies, rate limiting and centralized governance.
Especially compelling if you already use Kong for conventional API management.
Documentation:
4. Cloudflare AI Gateway
- Provider fallbacks and request timeouts.
- Unified provider routing.
- Automatic model selection through Auto Router.
- Integration with Cloudflare's infrastructure.
Good if your infrastructure already runs on Cloudflare. Its Auto Router also considers request capabilities and provider health.
Documentation:
5. OpenRouter
- One API across a large model catalog.
- Provider selection and provider-level failover.
- Model-level fallback chains.
- Provider ordering and routing preferences.
Best when you want to consume a hosted routing service rather than operate your own gateway. Less suitable if you need complete control over gateway infrastructure.
Documentation:
2. Architecture I'd recommend
For a system that needs to support multiple providers while keeping the application independent of them, I'd use a layered gateway architecture.
The conceptual architecture:
Applications | v Unified API Interface | v Request Normalization | v Routing Engine / | \ Policy Health Load Engine Monitor Balancer \ | / | v Provider Adapters / | \ OpenAI Anthropic Gemini | | | v v v Provider-specific APIs | v Response Normalization | v ApplicationsI'd divide the system into these components:
Component Responsibility API interface Stable application-facing contract Normalizer Translate request/response formats Router Select provider and model Health monitor Track availability and failure rates Failover manager Retry and switch providers Provider adapters Handle authentication and API differences Observability Capture latency, cost, errors and usage 3. Important design decisions
There are a few architectural decisions that will determine whether this works reliably in production.
A. Choose a canonical API format
I'd start with an OpenAI-compatible interface as the canonical format.
For example, your application sends:
json{ "model": "fast-general", "messages": [ { "role": "user", "content": "Summarize this document" } ], "temperature": 0.2 }The gateway translates this into the appropriate provider-specific format.
Internally:
- OpenAI adapter → OpenAI API format.
- Anthropic adapter → Anthropic Messages format.
- Gemini adapter → Gemini API format.
- Bedrock adapter → Bedrock-specific format.
Responses are translated back into your canonical format.
Important: Normalization shouldn't mean pretending every provider supports identical capabilities. Tool calling, structured output, multimodal inputs, reasoning metadata and streaming need explicit capability mapping.
B. Implement intelligent failover
I'd use a policy-based fallback system rather than simply retrying every error.
For example:
Example routing policy
Primary (Priority 1)
OpenAI GPT model
Secondary (Priority 2)
Anthropic Claude model
Tertiary
Google Gemini model
Suggested policies:
Failure Recommended behavior HTTP 429 Retry with backoff or switch provider HTTP 500/502/503 Retry, then fail over Timeout Retry within request deadline Authentication error Alert; don't blindly retry Invalid request Return error Context window exceeded Switch to a compatible larger-context model Unsupported feature Route to a provider supporting that feature Also implement circuit breakers so unhealthy providers are temporarily removed from routing.
Avoid multiple independent retry loops in your SDK, application and gateway, as they can multiply request volume.
C. Preserve provider-specific capabilities
I would not discard provider-specific information during normalization.
A good internal response object should preserve:
- Normalized response content.
- Original provider response metadata.
- Provider and model identifiers.
- Token usage.
- Finish reason.
- Tool calls.
- Reasoning metadata where supported.
- Streaming events.
This gives you a stable application API without losing important provider functionality.
4. Build versus buy
Here's how I'd narrow the choices:
Requirement My choice Self-hosted, customizable routing LiteLLM Managed gateway and observability Portkey Enterprise API management Kong Cloudflare-native infrastructure Cloudflare AI Gateway Simplest hosted multi-provider access OpenRouter My first two candidates would be LiteLLM and Portkey.
I'd prototype the same set of requests against both, particularly cross-provider failover and streaming response normalization.
5. A more advanced architecture to consider
If you're building a gateway that will become a core infrastructure component, I'd consider separating the routing engine from the provider adapters.
For example:
LLM Gateway | Canonical Request | Routing Policies | Provider Selection Engine | +----------+----------+ | | | Adapter A Adapter B Adapter C | | | OpenAI Anthropic Gemini | | | +----------+----------+ | Canonical ResponseThis makes it possible to add new providers without rewriting the routing logic.
I'd also consider:
- Redis for distributed health and circuit-breaker state.
- PostgreSQL for provider configurations and routing policies.
- Prometheus/Grafana for latency and availability metrics.
- OpenTelemetry for distributed tracing.
- A configurable policy engine for routing decisions.
6. A few questions to narrow down the best solution
The right recommendation depends considerably on whether you're looking for an existing product or building your own abstraction layer.
1. How do you want to deploy it?
Self-hosted / open source
Managed SaaS
Hybrid (self-hosted gateway + managed services)
Open to any option
2. What are you looking to build?
Use an existing LLM gateway
Build a custom routing and normalization layer
A complete multi-provider LLM platform
3. Which providers do you need?
Select all that apply.
4. What's your primary priority?
Reliability and automatic failover
Cost optimization
Lowest latency
Consistent API and format normalization
All of the above
[Get tailored architecture recommendations]
With these answers, I can narrow this down to a concrete architecture and recommend whether you should adopt LiteLLM, Portkey, Kong, or build a dedicated routing layer.
Pages cited in this answer 16
- docs.litellm.aiRetries, fallbacks and cooldowns | liteLLM
- docs.litellm.aiRouter concepts | liteLLM
- docs.litellm.aiLiteLLM documentation | liteLLM
- portkey.aiThe AI Control Plane for the Enterprise - Palo Alto Networks
- docs1.portkey.aiUniversal API
- portkey.aiWhat is Portkey? - Portkey (Prisma AIRS AI Gateway) Docs
- developer.konghq.comAI Gateway architecture - AI Gateway | Kong Docs
- developer.konghq.comAI Models - AI Gateway | Kong Docs
- developer.konghq.comLoad balancing with Kong AI Gateway - AI Gateway | Kong Docs
- developer.konghq.comKong AI Gateway | Kong Docs
- developers.cloudflare.comFallbacks · Cloudflare AI Gateway docs
- developers.cloudflare.comAuto Router · Cloudflare AI Gateway docs
- developers.cloudflare.comOverview · Cloudflare AI Gateway docs
- openrouter.aiHow OpenRouter Model Routing Works
- openrouter.aiOpenRouter Quickstart Guide
- developers.openai.comDesigning failover across OpenAI model providers
Google AI ModeNov 9, 2026
To route requests across multiple LLM providers with normalized formats and automatic failover , you are looking for an LLM Gateway or Proxy architecture . Instead of writing custom retry logic and provider-specific adapters in your application code, an abstraction layer standardizes the interface (typically mimicking the OpenAI API spec) and handles operational resilience.
Top Tools to Consider
1. Self-Hosted Gateways (Best for Data Privacy & Custom Control)
- LiteLLMLiteLLM (Proxy): The most popular open-source, Python-heavy standard. It exposes an OpenAI-compatible REST API for 100+ LLM providers, supports YAML-defined fallback and retry chains, virtual keys, team-based budgets, and prompt/response logging. Learn more on the LiteLLM DocumentationLiteLLM Documentation.
- PortkeyPortkey : A lightweight, high-performance gateway (Node.js/Rust core) offering MIT-licensed self-hosting or a managed cloud tier. It features granular JSON-based routing rules, conditional fallbacks based on specific HTTP status codes or latency thresholds, and built-in guardrails/observability. Explore via Portkey AIPortkey AI.
2. Managed / Hosted Gateways (Best for Zero-Infra Setup)
- OpenRouterOpenRouter : Acts as a universal API endpoint routing across hundreds of commercial and open-source models with automatic fallback options and unified billing. Best if you don’t mind third-party routing infrastructure. Check out OpenRouter.
- Cloudflare AI GatewayCloudflare AI Gateway : Sits in front of your LLM calls to provide caching, rate-limiting, analytics, and basic fallback/retry mechanics with minimal latency overhead. Details on Cloudflare AI Gateway.
- Vercel AI SDKVercel AI SDK / AI Gateway: Great if your application stack is built heavily around TypeScript/Next.js and you want a unified streaming and provider-abstraction layer. See Vercel AI SDK.
Core Architectural Patterns
- The Proxy / Sidecar Pattern: Your application sends every LLM request to a local or centralized gateway instance using a standard format (e.g.,
/v1/chat/completions). The gateway translates the payload into Anthropic, Gemini, Bedrock, or Ollama formats on the fly. - Tiered Fallback Chains: Configure sequential fallback rules. For example:
- Primary:
claude-3-5-sonnet(via Anthropic) - Fallback 1:
gpt-4o(via OpenAI) if Anthropic returns a 5xx or rate-limit (429) error. - Fallback 2:
deepseek-r1or a local self-hosted model via vLLM if both major APIs experience a wider outage.
- Primary:
- Semantic or Rule-Based Routing: Use conditional rules where simple or high-volume queries route to cheaper/faster models (e.g., Llama 3.3 70B), while complex reasoning tasks automatically get bumped to frontier models.
To help narrow down the best fit, tell me:
- Do you prefer to self-host (for data residency/compliance) or use a managed cloud service?
- What is your primary application stack (Python, TypeScript, Go, etc.)?
- Are you prioritizing cost optimization, lowest latency , or maximum uptime/failover?
ChatGPTOct 29, 2026
If you want to route requests across different LLM providers while handling failover and normalizing inputs, the most useful architecture is an LLM gateway with a provider adapter layer, policy-based routing, and centralized retry/fallback logic.
This lets your application use one internal API while routing requests to OpenAI, Anthropic, Google, self-hosted models, or other providers. It also helps you standardize message formats, tool calls, model parameters, errors, and response handling.
1. Architecture options
A. Unified LLM gateway
A single API sits between your application and all providers. It normalizes requests and responses, applies routing rules, and handles failover.
Best for: most production applications, multiple providers, centralized governance.
B. Application-level adapter pattern
Your application defines a common interface, with one adapter per provider. Routing and retries live in your own code.
Best for: smaller systems, strict customization, minimal infrastructure.
C. Model router with intelligent selection
A router chooses models using task type, latency, cost, quality, context length, or live provider health.
Best for: high-volume systems, cost optimization, workloads with different quality requirements.
D. Resilient multi-provider control plane
A gateway combines routing with circuit breakers, timeouts, concurrency limits, quotas, observability, and provider-specific policies.
Best for: enterprise systems and applications where availability is critical.
These approaches are complementary: for example, you can use a unified gateway with an intelligent router and a resilience layer.
2. Tools worth evaluating
These are the strongest options to shortlist, depending on whether you want self-hosting, a managed service, or routing inside your application.
LiteLLM — self-hosted gateway and routing
Provides an OpenAI-compatible API across many providers, plus retries, fallbacks, load balancing, virtual keys, budgets, and observability integrations. Its routing layer supports strategies such as latency-based and rate-limit-aware routing.
Choose it when you want infrastructure control and a configurable, provider-neutral API.
OpenRouter — managed multi-provider routing
Offers a unified API, automatic provider-level failover, provider ordering, and optional model-level fallbacks. This is useful when you want multiple inference providers without operating the routing infrastructure yourself.
Choose it for quick integration and broad model access.
Provider routing documentation
Vercel AI Gateway — managed routing with AI SDK integration
Supports provider preference and filtering, provider timeouts, and ordered model fallbacks. A good fit for TypeScript applications using the Vercel AI SDK.
Choose it when you're already using the AI SDK or deploying on Vercel.
Portkey — policy-oriented gateway
Offers conditional routing, automatic retries, fallbacks, load balancing, timeouts, and request tracing. Useful for teams that want routing policies and operational controls managed through a gateway.
Choose it for centralized policies, observability, and managed operations.
Cloudflare AI Gateway — configurable dynamic routing
Its dynamic routing feature lets you build conditional routing flows with quotas, model selection, and fallbacks, including versioned route configurations.
Choose it when you want routing policies that can evolve without application-code changes.
3. How to normalize requests across providers
The key is to normalize into your own canonical request format, rather than assuming that a shared API schema makes every provider feature equivalent.
A practical internal contract could look like this:
typescripttype LLMRequest = { task: "chat" | "json" | "embedding"; messages: Array<{ role: "system" | "user" | "assistant" | "tool"; content: string | Array<ContentPart>; }>; modelTier?: "fast" | "balanced" | "best"; temperature?: number; maxOutputTokens?: number; tools?: ToolDefinition[]; responseFormat?: "text" | "json"; timeoutMs?: number; metadata?: Record<string, string>; };Then implement the following pipeline:
1. Application request
Your canonical schema
2. Validation and normalization
Validate roles, content, tokens, tools, and capabilities
3. Policy router
Select eligible models using health, cost, latency, and task
4. Provider adapter
Map to native API fields and provider-specific capabilities
5. Response normalizer
Standardize output, usage, finish reasons, and errors
Important normalization rules:
- Messages: Convert roles and multimodal content into each provider's supported format.
- Parameters: Define canonical meanings for token limits, temperature, stop sequences, and structured output; don't assume parameter equivalence.
- Tools: Normalize tool definitions and results, while handling differences in tool-call IDs and argument formats.
- Capabilities: Track whether each model supports vision, JSON schema, tool calling, streaming, and the context length you need.
- Responses: Return a stable envelope with content, tool calls, usage, provider/model metadata, and normalized error categories.
- Unsupported features: Reject them clearly or use an explicit downgrade policy. Never silently discard a critical parameter.
For a ready-made OpenAI-compatible interface, LiteLLM is a particularly relevant starting point. If you need a custom contract, keep the canonical schema in your application and use the gateway as the transport and routing layer.
4. Failover design: what to implement
Use two distinct fallback levels. They solve different problems.
Level 1: Provider failover
Keep the same model but switch provider or deployment.
Example: OpenAI-hosted deployment → Azure-hosted deployment of the same model family.
Level 2: Model fallback
Switch to a different compatible model when the primary model is unavailable or cannot handle the request.
Example: primary reasoning model → smaller general-purpose model for a supported task.
Gateways such as LiteLLM, OpenRouter, and Vercel AI Gateway support variations of these mechanisms.
Recommended production behavior:
Failure Recommended handling HTTP 429 rate limit Retry with backoff or use another eligible deployment HTTP 5xx Fail over to a healthy provider Connection timeout Apply a deadline, then try a backup Invalid request / unsupported feature Fix the request or route to a compatible model; don't repeat unchanged Context limit exceeded Shorten input or select a model with sufficient context Authentication / permission error Mark the affected credential or route unhealthy; alert operators Failure after streaming begins Usually terminate the stream; restarting risks duplicate or contradictory output Use bounded retries, exponential backoff with jitter, per-provider circuit breakers, and an overall request deadline. Be careful retrying operations after a timeout if the provider might already have processed the request.
5. A practical implementation plan
I'd build this in stages:
- Start with a unified API. Use LiteLLM if you want self-hosting, or a managed gateway if minimizing operations is more important.
- Define a canonical schema. Normalize messages, content parts, tool definitions, model requirements, and response metadata.
- Create a capability registry. Record which models support required features and maintain limits for context size, output tokens, and modalities.
- Add deterministic routing. Begin with a primary model and an ordered list of compatible fallbacks.
- Add resilience controls. Configure timeouts, retry budgets, circuit breakers, rate limits, and concurrency caps.
- Instrument every attempt. Record the selected provider, model, latency, retry reason, token usage, and cost.
- Test semantic compatibility. Run the same representative requests through each provider, especially tool calls, JSON output, and streaming.
One additional standard worth investigating is Open Responses. It defines a shared schema and interoperability approach for model requests, responses, streaming, and tool workflows.
My recommendation: for a typical production service, start with a gateway plus a canonical application-level request contract. Keep routing policies and provider adapters separate so you can replace the gateway later without rewriting your business logic. Add dynamic cost- or latency-based routing only after you have enough telemetry to make those decisions reliably.
Pages cited in this answer 15
- docs.litellm.aiLiteLLM AI Gateway | liteLLM
- docs.litellm.aiRouter - Load Balancing | liteLLM
- openrouter.aiOpenRouter Failover and Fallbacks Explained
- vercel.comAI Gateway Model Fallbacks
- docs.litellm.aiGetting Started | liteLLM
- github.comdocs/guides/routing/provider-selection.mdx at main · OpenRouterTeam/docs · GitHub
- openrouter.aiProvider Routing - Smart Multi-Provider Request Management
- vercel.comAI Gateway Provider Routing and Fallbacks
- vercel.comAI Gateway Provider Timeouts
- vercel.comVercel AI Gateway: Models, Routing, and Observability
- portkey.aiThe AI Control Plane for the Enterprise - Palo Alto Networks
- docs1.portkey.aiFallback configuration
- developers.cloudflare.comDynamic routes · Cloudflare AI Gateway docs
- openresponses.orgOpen Responses
- openresponses.orgSpecification
Google AI ModeOct 29, 2026
Routing requests across multiple LLM providers while managing input normalization (usually translating diverse provider payloads into an OpenAI-compatible schema ) and automatic failover is best achieved using specialized AI Gateways or Reverse Proxy architectures.
Recommended Tools & Open-Source Gateways
Instead of building custom retry loops and mapping layers from scratch, production setups typically leverage dedicated proxy servers or SDK-based routers:
- LiteLLM Proxy / SDK
- What it does: Translates standard OpenAI format requests to over 100+ providers (Anthropic, Vertex, Bedrock, Mistral, Cohere). Supports built-in load balancing, fallback/failover chains, budget tracking, and rate limiting.
- Form Factor: Can be deployed as a standalone reverse proxy container or used as a Python library.
- Portkey AI Gateway
- What it does: An enterprise-grade, open-source AI gateway featuring robust fallback routing, semantic caching, request/response logging, and standard OpenAI-compatible API normalization.
- Form Factor: Cloud-hosted or self-hosted proxy with extensive observability integration (e.g., Langfuse, Datadog).
- Bifrost
- What it does: A high-performance, standalone Go-based gateway optimized for multi-provider routing, hedging, and low-latency fallbacks across 25+ model providers.
- Form Factor: Lightweight binary/container well-suited for microservice meshes.
- Apache APISIX (with AI Plugins) / Kong AI Gateway
- What it does: Traditional API gateways extended with native AI plugins (
ai-proxyandai-proxy-multi). They handle upstream token tracking, load balancing, and fallback behavior natively at the infrastructure edge. - Form Factor: Enterprise API Gateway integrated with Kubernetes or bare-metal infrastructure.
- What it does: Traditional API gateways extended with native AI plugins (
Core Architectural Patterns
When designing your routing topology, choose one of these standard structural implementations:
- The Centralized Reverse-Proxy Pattern (Recommended)
- Flow:
App Client -> AI Gateway Proxy (LiteLLM/Portkey) -> Provider API (OpenAI/Anthropic/Bedrock) - Why use it: Your application code only ever talks to a single, static OpenAI-compatible endpoint. The proxy intercepts the payload, translates it on the fly to the target downstream provider, tracks quotas, and executes instant fallbacks if a 429 (Rate Limit) or 5xx error returns.
- Flow:
- The Sidecar / Service Mesh Pattern
- Flow: App container communicates with a local host-level gateway instance (like an Envoy-managed or lightweight local proxy container) which coordinates cluster-wide failovers.
- Why use it: Minimizes external network hops for strict latency requirements and decouples credential management entirely from application runtimes.
How Normalization and Failover Work Under the Hood
- Input Normalization: Gateways accept a standard request body (
/v1/chat/completions). If the target fallback switches from OpenAI to Anthropic Claude, the proxy maps JSON parameters dynamically (e.g., translatingmax_tokens, system roles, or tool definitions into Anthropic's distinct schema format) before dispatching the HTTP call. - Failover Logic: You define a priority list (e.g., primary:
gpt-4o, fallback 1:claude-3-5-sonnet, fallback 2:deepseek-chat). If the primary upstream times out or returns a service error, the router catches the exception internally, rewrites the payload for the backup provider, and completes the request seamlessly within the same client connection window.
If you'd like to narrow this down, let me know:
- Are you looking for a self-hosted open-source proxy or a managed cloud service?
- Do you need integration with specific observability or cost-tracking tools?
I can provide a reference configuration setup for your preferred architecture.
Pages cited in this answer 2
For agents: this page as Markdown, https://parse.gl/topics/llm-provider-routing.md. JSON, MCP · free for the last 90 days, $0.01 a check
This page
Read and search
Connect
An agent can save work before signup: POST https://agent.parse.gl/v1/workspaces
What it costs
Prices: https://parse.gl/pricing.md
Map of the site: https://parse.gl/llms.txt