We want to implement rate limiting and caching for our internal LLM API. What's the best open-source API gateway for this?

For an internal LLM API, I’d lean toward Apache APISIX if you want the gateway itself to understand LLM-specific traffic.

Apache APISIX has several relevant pieces built in:

• Token-aware rate limiting: its ai-rate-limiting plugin can throttle LLM usage based on token consumption, rather than merely counting HTTP requests.Apache APISIX+1
• LLM proxying and routing: ai-proxy and ai-proxy-multi support model/provider routing, load balancing, retries, and fallbacks.Apache APISIX
• LLM response caching: APISIX has an ai-cache plugin specifically for LLM responses, with exact-prompt caching and optional semantic caching backed by Redis.GitHub
• Traditional API caching: its proxy-cache plugin can cache GET/POST/HEAD responses with memory or disk storage.Apache APISIX
• It's fully open-source and built around NGINX + etcd, with dynamically loaded plugins.Apache APISIX

How I'd choose

Gateway	LLM-specific features	Rate limiting	Caching	My take
APISIX	Excellent	Request + token	HTTP + LLM/semantic	Strong fit for an LLM gateway
Kong Gateway	Good

Kong is the main alternative I'd consider. Its open-source Gateway has request rate limiting with local, cluster, and Redis strategies, plus a proxy-cache plugin.Kong Docs+1

For your use case, I'd start with APISIX + Redis:

Internal clients
       │
       ▼
┌─────────────────────────┐
│      Apache APISIX      │
│                         │
│ auth / API keys         │
│ request rate limits     │
│ token rate limits       │
│ LLM response cache      │
│ retries / fallbacks     │
│ metrics / tracing       │
└───────────┬─────────────┘
            │
       ┌────┴─────┐
       ▼          ▼
   Redis       LLM API(s)
  cache/RL     OpenAI/etc.

One important caveat: don't blindly cache arbitrary LLM POST requests. Your cache key should incorporate the model, effective system/developer prompt, user prompt, relevant generation parameters, and any other input that affects the response. For personalized or nondeterministic responses, caching can be inappropriate.

If you tell me whether you're running Kubernetes or VMs/bare metal, and roughly requests/sec + number of internal users, I can suggest a concrete APISIX architecture and configuration.

When implementing rate limiting and caching for an internal LLM API, the "best" open-source gateway depends on whether your priority is LLM-native management and token budgeting or integrating into an existing enterprise API infrastructure.orq.ai

The top open-source options tailored for LLM workloads stand out across distinct use cases:

• LiteLLM Proxy
  • Best for: Native LLM management, virtual API key budgeting, and multi-provider routing.
  • Rate Limiting: Highly granular. It tracks Requests Per Minute (RPM) and Tokens Per Minute (TPM) across global server limits, teams, virtual keys, and individual users. It also supports budget-hard limits (e.g., maximum spend per user per month).
  • Caching: Supports Redis-backed caching for LLM responses to reduce latency and save costs.
  • Trade-offs: It is purpose-built as an LLM proxy rather than a general-purpose web gateway, which makes LLM-specific configurations trivial, but lacks traditional reverse-proxy features of heavy-duty edge infrastructure.docs.litellm.ai+1
• Kong AI Gateway
  • Best for: Organizations already running Kong or wanting to fold AI traffic into a mature, enterprise-grade API gateway ecosystem.
  • Rate Limiting: Offers robust AI Rate Limiting plugins that can calculate limits based not just on HTTP requests, but on actual token consumption returned by the LLM provider. (Note: Some advanced AI rate-limiting features require Kong Enterprise/Plus plugins).
  • Caching: Features dedicated proxy-caching and AI-focused plugins to handle response caching cleanly.
  • Trade-offs: Can be operationally heavy to set up and manage compared to lightweight Python/Go-based LLM proxies if you only need an AI-specific layer.developer.konghq.com+1
• Apache APISIX (with AI Gateway Plugins)
  • Best for: High-performance, cloud-native environments needing deep extensibility.
  • Rate Limiting: Uses the ai-rate-limiting plugin to track provider-reported token usage and throttle requests once quotas are hit.
  • Caching: Offers an ai-cache plugin paired with Redis for exact-match response caching.
  • Trade-offs: Smaller community mindshare specifically for LLM workflows compared to LiteLLM, though its architectural performance under load is stellar.apisix.apache.org

Summary Recommendation

If your primary goal is tracking token spend, enforcing user/team budgets, and spinning up an LLM-specific gateway quickly , go with LiteLLM Proxy Documentation.docs.litellm.ai+1

If you already use Kong for production microservices and need strict, unified governance across traditional and AI APIs alike, extend it with Kong AI Gateway Plugins.orq.ai+1

To help narrow down the exact fit, tell me:

• Are you looking for exact-match caching or do you need semantic caching (vector-based similarity)?
• How are you managing user authentication and team budgets right now?

For an internal LLM API, the “best” open-source API gateway depends on whether you mainly need classic API controls (rate limits, auth, caching, routing) or LLM-specific controls (token budgets, model routing, prompt policies, semantic caching). For most teams, I would shortlist these:

Gateway	Best fit	Rate limiting	Caching	LLM-specific features	Operational complexity
Apache APISIX	General-purpose gateway with AI traffic	Excellent (plugins)	Good (plugins/external cache)	Some AI gateway features	Medium
Kong	Teams already using Kong ecosystem	Excellent	Good	AI gateway extensions	Medium
LiteLLM	LLM proxy rather than generic API gateway	Excellent for LLM quotas	Strong LLM caching options	Strongest LLM-native features	Low–medium
Envoy	Kubernetes/service mesh environments	Excellent	Usually external	Requires more assembly	High

Apache APISIX Apache APISIX## My usual recommendation

If you want a gateway in front of your internal LLM service:

Start with Apache APISIX or Kong.

They give you the normal gateway features you will probably need:

• Per-user/team/API-key rate limits
• Request authentication
• Routing between model backends
• Retries and circuit breaking
• Observability hooks
• Kubernetes integration
• TLS termination

APISIX is attractive for greenfield deployments because it is a full API gateway with dynamic configuration and a large plugin ecosystem. Kong is a strong choice if your organization already operates Kong or wants its ecosystem.Apache APISIX+1

If your main problem is LLM cost and usage governance:

Consider LiteLLM as an LLM gateway, possibly behind a normal API gateway.

It is designed around LLM workloads:

• OpenAI-compatible API surface
• Model routing/fallbacks
• Per-user/team spend tracking
• Token-based limits
• Provider abstraction
• LLM-specific caching patterns

A common architecture is:

Clients
   |
   v
API Gateway (APISIX/Kong/Envoy)
   |
   v
LiteLLM Proxy
   |
   +--> OpenAI
   +--> Anthropic
   +--> vLLM
   +--> Local models

Caching considerations

For LLM APIs, “cache” can mean different things:

1. Exact response cache

• Same prompt + parameters → return stored response
• Easy to implement with Redis
• Useful for repeated internal queries

2. Semantic cache

• Similar prompts → reuse previous answers
• Requires embeddings/vector search
• More useful for knowledge assistants

3. Prompt/result caching

• Cache expensive retrieval steps or intermediate outputs

A normal API gateway often handles (1), while LLM gateways or application layers are usually better positioned for (2) and (3).

A practical production stack

For many internal deployments:

                 ┌─────────────┐
Users/services ->│ APISIX/Kong │
                 └──────┬──────┘
                        |
                 ┌──────▼──────┐
                 │  LiteLLM    │
                 └──────┬──────┘
                        |
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
     vLLM            OpenAI        Anthropic

Add:

• Redis for rate-limit counters and exact caching
• PostgreSQL for usage records
• Prometheus/OpenTelemetry for metrics

What I would pick by scenario

• Small internal team, one/few models: LiteLLM alone may be enough.
• Platform team exposing LLM APIs to many internal teams: APISIX/Kong + LiteLLM.
• Already on Kubernetes with Istio/Envoy: Evaluate Envoy-based solutions, but expect more assembly work.Apache APISIX
• Need one gateway for both normal APIs and LLM APIs: APISIX or Kong.

If you share your environment (Kubernetes vs VMs, OpenAI-compatible endpoint vs custom, number of users, and whether you need token-level quotas), I can narrow this to a specific architecture.

1. Exact response cache

• Same prompt + parameters → return stored response
• Easy to implement with Redis
• Useful for repeated internal queries

2. Semantic cache

• Similar prompts → reuse previous answers
• Requires embeddings/vector search
• More useful for knowledge assistants

3. Prompt/result caching

• Cache expensive retrieval steps or intermediate outputs

A normal API gateway often handles (1), while LLM gateways or application layers are usually better positioned for (2) and (3).

A practical production stack

For many internal deployments:

                 ┌─────────────┐
Users/services ->│ APISIX/Kong │
                 └──────┬──────┘
                        |
                 ┌──────▼──────┐
                 │  LiteLLM    │
                 └──────┬──────┘
                        |
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
     vLLM            OpenAI        Anthropic

Add:

• Redis for rate-limit counters and exact caching
• PostgreSQL for usage records
• Prometheus/OpenTelemetry for metrics

What I would pick by scenario

For an internal LLM API, choosing the "best" open-source API gateway depends heavily on your existing infrastructure and how deeply you need to manage tokens versus standard HTTP traffic.getmaxim.ai

The top open-source options for handling LLM-specific rate limiting (Requests Per Minute/Tokens Per Minute) and caching (exact match or semantic) break down as follows:konghq.com+1

1. LiteLLM (Proxy) — Best for pure LLM-native management and fast setup
   • Rate Limiting: Highly granular. Supports RPM (Requests Per Minute) and TPM (Tokens Per Minute) limits, real-time budget enforcement, and per-key/per-team/per-user cost tracking.
   • Caching: Supports both in-memory and robust Redis/Valkey integration for exact-match caching, plus semantic caching via Redis or Qdrant.
   • Why choose it: It acts as an drop-in OpenAI-compatible proxy supporting 100+ LLM providers out of the box. If your goal is specifically managing LLM spend, multi-tenant keys, and fallbacks without managing raw infrastructure proxies, LiteLLM Proxy Docs is the community favorite.docs.litellm.ai+3
2. Kong AI Gateway — Best if you already use Kong for traditional API management
   • Rate Limiting: Offers token-aware rate-limiting plugins natively that track token consumption instead of just counting HTTP requests.
   • Caching: Provides standard and semantic caching plugins (though some advanced semantic caching capabilities are locked behind Kong Enterprise).
   • Why choose it: If your engineering organization already routes traffic through Kong AI Gateway , adding their AI gateway plugins allows you to govern LLM traffic alongside your standard microservices. Note that the architecture adds a small Lua/Nginx-based latency overhead.dev.to+1
3. Envoy AI Gateway — Best for Kubernetes-native and service-mesh environments
   • Rate Limiting: Features native, low-latency, token-based rate limiting extending the Kubernetes Gateway API.
   • Caching: Supports unified provider-agnostic prompt caching (translating cache controls across Anthropic, Bedrock, and Vertex AI).
   • Why choose it: If your team already runs Istio or Envoy at scale in Kubernetes, plugging in Envoy AI Gateway keeps your infrastructure unified rather than introducing an entirely separate proxy stack.aigateway.envoyproxy.io+1
4. Bifrost — Best for ultra-low latency and lightweight setups
   • Rate Limiting: Built for high-throughput multi-tenant environments with token-aware limits and automatic 429 failovers.
   • Caching: Includes built-in semantic caching and Prometheus observability.
   • Why choose it: A newer open-source contender designed specifically for microsecond-level proxy overhead and quick NPX/container deployments.reddit.com+1

To help narrow down the ideal choice, tell me:

• Are you currently running Kubernetes / a service mesh , or do you prefer a standalone proxy?
• Do you need semantic caching (matching similar prompts via vector embeddings) or just exact-match caching?
• What is your primary LLM infrastructure (self-hosted vLLM/Ollama, or commercial APIs like OpenAI/Anthropic)?

For an internal LLM API, I would shortlist these:

Gateway	Best fit	Rate limiting	LLM-specific features	Caching
Apache APISIX	Best overall for new LLM gateways	Excellent (request, consumer, token-style controls)	Strong AI gateway direction	Built-in cache plugins + Redis patterns
Kong Gateway	Teams wanting a mature ecosystem	Excellent	Good via plugins	Good, often paired with Redis
Envoy Proxy	Service mesh / high-scale infra	Excellent	Requires more assembly	Usually external
Traefik	Simple Kubernetes ingress use cases	Good	Limited	Usually external

Apache APISIX Apache APISIX## My recommendation: Apache APISIX

For an internal LLM API, I would start with Apache APISIX. It gives you the pieces you typically need in front of an LLM service:

• Per-user / per-team rate limits Example: - Team A: 10k requests/hour
• Team B: 100k tokens/day
• Production workloads: higher burst allowance
• Request routing - /v1/chat/completions → primary model
• fallback model on failure
• canary new models
• Authentication - API keys
• JWT/OIDC
• mTLS for internal services
• Observability - latency
• token usage
• errors
• model/provider metrics

APISIX has built-in traffic-control plugins such as request-count, request-rate, and connection limiting, and supports Redis-backed shared counters for multi-node deployments.Apache APISIX+1

A typical architecture:

                 Internal Clients
                       |
                       v
              +----------------+
              |  APISIX Gateway |
              +----------------+
                |      |       |
                |      |       |
          rate limit  cache   auth
                |
                v
        +----------------+
        | LLM Backend(s) |
        +----------------+
          |      |      |
        GPT   Llama   Claude

LLM caching considerations

LLM caching is different from normal API caching.

Good candidates:

• identical embedding requests
• repeated classification prompts
• deterministic completions (temperature=0)
• expensive retrieval/tool calls

Be careful caching:

• user-specific conversations
• prompts containing secrets
• responses with time-sensitive data

A common setup:

APISIX
  |
  +-- Redis
       |
       +-- response cache
       +-- token quotas
       +-- request counters

APISIX supports Redis integrations for use cases such as AI response caching and distributed quota counters.Apache APISIX

When I would pick Kong instead

Choose Kong Gateway if:

• your organization already runs Kong,
• you want its ecosystem and operational familiarity,
• you expect to buy enterprise support later.

Kong and APISIX are probably the two closest comparisons; both are mature NGINX-based gateways with plugin ecosystems and strong traffic-management features.Apache APISIX

When I would not use a gateway cache

For many LLM workloads, I would not rely only on gateway-level response caching. I would combine:

• Gateway:

• authentication

• quotas

• throttling

• routing

• Application layer:

• semantic cache (embedding similarity)

• prompt normalization

• user/session awareness

Example:

Client
  |
APISIX
  |
LLM Service
  |
Semantic Cache (Redis / vector DB)
  |
Model Provider

Practical stack I would deploy

For a Kubernetes-based internal LLM platform:

• Gateway: Apache APISIX
• Cache + quotas: Redis
• Metrics: Prometheus + Grafana
• Tracing: OpenTelemetry
• Secrets: Vault/Kubernetes Secrets
• Models: vLLM, TGI, Ollama, or external APIs

For a small internal deployment, APISIX + Redis is probably the simplest strong choice. For a very large platform with an existing service mesh, Envoy-based architectures become more attractive.apisix.incubator.apache.org

For a Kubernetes-based internal LLM platform:

Choosing the best open-source API gateway for an internal LLM implementation depends on whether you view your gateway through an AI-native/application lens or a traditional infrastructure/proxy lens.

LLM traffic differs from standard REST traffic because you need to track tokens (not just requests), manage budgets , handle streaming responses (Server-Sent Events), and utilize semantic caching rather than strict URL-hash caching.

Top Recommendations

1. LiteLLM Proxy (Best for AI-First Teams & Multi-Provider Routing)

While technically an application-layer proxy rather than a traditional low-level API gateway, LiteLLM Proxy has become the de facto open-source standard for internal LLM setups. It translates standard OpenAI-formatted requests to over 100+ LLM providers (Anthropic, Azure, Vertex, local vLLM, etc.).getmaxim.ai+2

• Rate Limiting: Supports robust virtual-key generation, user-level rate limits, and budget tracking (e.g., max dollar spend or token limits per user/team/duration) backed by Redis.dev.to+1
• Caching: Supports Redis-backed exact/prompt response caching out-of-the-box, meaning identical prompts don't hit the expensive LLM backend twice.dev.to+1
• Why choose it: If your main priority is fast iteration, granular user-level token budgeting, and seamlessly swapping underlying LLM providers without changing app code.getmaxim.ai+2

2. Apache APISIX (Best for Cloud-Native Infrastructure & Token-Rate Limiting)

If your organization already relies on a cloud-native microservices architecture or needs a high-performance reverse proxy, Apache APISIX features native AI plugins (ai-proxy and ai-rate-limiting ) designed directly for LLM workloads.apisix.apache.org+2

• Rate Limiting: Features specialized token-aware rate limiting (ai-rate-limiting ) allowing you to cap traffic based on total tokens consumed (per minute/day) rather than just standard HTTP requests.apisix.apache.org+2
• Caching: Highly performant, though advanced semantic caching features may require external configuration or enterprise additions depending on exact matching needs.
• Why choose it: If you want a heavy-duty, scalable edge/API gateway that sits natively in your infrastructure stack and handles traditional APIs alongside AI traffic.

3. Kong AI Gateway (Best for Enterprises Already on Kong)

Kong AI Gateway extends Kong's mature API management framework with plugins explicitly built for AI routing.dev.to+2

• Rate Limiting & Caching: Inherits Kong’s industrial-strength rate-limiting and Redis caching infrastructure, mapping them cleanly to LLM endpoints.
• Trade-off: While the core proxy is open-source, several advanced AI governance, security guardrails, and advanced semantic caching options are locked behind Kong Enterprise.
• Why choose it: If your company is already standardized on Kong for API gateways and you want unified management under one roof.dev.to+1

Comparison at a Glance

To help narrow down the ideal choice, tell me:

• Are you self-hosting your models (e.g., via vLLM / Ollama ) or calling external APIs (OpenAI, Anthropic)?
• Do you need semantic caching (catching similar/paraphrased prompts) or just exact match caching?

When implementing rate limiting and caching for an internal LLM API, the "best" choice depends heavily on your traffic scale, infrastructure stack (Kubernetes vs. standalone containers), and whether you need semantic caching (caching similar prompts via vector embeddings) or simple exact-match caching.respan.ai+1

The top open-source options tailored for LLM workloads stand out across different scenarios:

1. LiteLLM Proxy (Best Overall for Feature Depth & Ease of Use)

• What it is: An open-source, Python-based proxy designed specifically to translate 100+ LLM provider APIs into a unified OpenAI-compliant format.reddit.com+1
• Caching: Supports Redis, DynamoDB, and in-memory caches for exact-match caching, plus advanced semantic caching through backends like Redis and Qdrant.getmaxim.ai+1
• Rate Limiting: Granular, built-in rate limiting per user, per virtual key, per team, or per dollar budget.respan.ai+1
• Trade-offs: In high-throughput production environments exceeding 1,000+ requests per second (RPS), Python-based performance can occasionally become a bottleneck compared to compiled proxy architectures.reddit.com+1
• Get started: Check out the repository on the LiteLLM GitHub.

2. Kong AI Gateway (Best for Enterprise Infrastructure & Scale)

• What it is: Built on top of the ultra-fast, Lua/Nginx-based Kong Gateway, augmented with native AI plugins.
• Caching: Features an AI Semantic Cache plugin that automatically generates embeddings for incoming prompts, checks a vector store, and returns cached outputs to slash latency.
• Rate Limiting: Enterprise-grade rate limiting, request size limiting, and dynamic traffic shaping plugins.
• Trade-offs: Configuration complexity is significantly higher than purpose-built Python tools, and advanced AI features might require enterprise tiers or heavy configuration.
• Explore docs: Read more via the Kong AI Gateway Overview.getmaxim.ai+1

3. Envoy AI Gateway (Best for Kubernetes & Service Mesh)

• What it is: A relatively new, high-performance cloud-native AI gateway built directly on top of Envoy Proxy.
• Caching: Currently focused heavily on routing, token-based rate limiting, and cost tracking; semantic caching support is minimal or absent in early versions.
• Rate Limiting: Extremely low-latency, native token-based and request-based rate limiting with minimal overhead (1–3ms).
• Trade-offs: Early stage; provider support is more limited than LiteLLM, and it lacks semantic caching out-of-the-box. Ideal if you already run Istio or Envoy on Kubernetes.
• Check the project: Track development on the Envoy AI Gateway GitHub.dev.to

Quick Comparison Matrix

To help me narrow down the best fit for your team, could you share:

• Your estimated request volume (RPS) or daily token usage?
• Whether you prefer exact-match caching or need semantic caching for similar user prompts?
• Your current infrastructure environment (e.g., Kubernetes, AWS ECS, or bare-metal VMs)?

For an internal LLM API, I’d pick LiteLLM Proxy if the gateway’s primary job is LLM-specific routing, quotas, caching, and provider abstraction. If you want a more traditional, general-purpose API gateway, Kong is the stronger choice.

My recommendation

LiteLLM Proxy	Kong Gateway	Envoy AI Gateway
LLM-native	Excellent	Excellent
Request rate limiting	✅	Excellent
Token/cost-based limiting	Excellent	Excellent
Semantic/LLM caching	Excellent	Excellent
Multi-provider routing	Excellent	Excellent
General API gateway	Good	Excellent
Operational complexity	Lower	Medium
Kubernetes-native	Good	Good

1. Best overall for an LLM API: LiteLLM

LiteLLM is purpose-built around the LLM gateway use case. Its proxy supports authentication, logging, cost tracking, and rate limiting, and gives applications a unified OpenAI-compatible API.LiteLLM

I'd use it if your architecture looks roughly like:

Applications
     |
     v
  LiteLLM
     |
     +----> OpenAI
     +----> Anthropic
     +----> Azure OpenAI
     +----> vLLM
     +----> Ollama
     +----> other internal models

It's particularly attractive if you want limits such as:

• 100 requests/minute per application
• 1M tokens/day per team
• $50/month budget per project
• model-specific quotas
• different access policies for different internal users

LiteLLM also has multi-instance rate-limiting support backed by Redis.GitHub

2. Best traditional gateway: Kong

Kong is my choice if you already have a broader API-gateway infrastructure.

Kong has conventional request rate limiting, including distributed Redis-backed limits, sliding windows, consumer-level limits, etc.Kong Docs+1

More importantly for your use case, Kong's AI Gateway can rate-limit based on LLM token usage/cost rather than merely request count. Kong Docs Kong Docs It also has an AI semantic-cache capability.Kong Docs

So if you already have:

                    +--> REST APIs
                    |
Clients --> Kong ---+--> LLM API
                    |
                    +--> Internal services

I'd strongly favor Kong rather than adding a second gateway.

3. Envoy AI Gateway

Envoy AI Gateway is compelling if you're heavily invested in Kubernetes/Envoy.

It provides a two-tier gateway architecture, with authentication, routing, and global rate limiting at the first tier and more fine-grained model-serving controls at the second. GitHub Envoy Gateway also supports both global and local rate limiting.Envoy Gateway

I'd choose it primarily for an infrastructure/platform team already standardized on Envoy, rather than introducing it solely to get LLM caching.

What I'd deploy

For most internal LLM platforms, I'd start with:

                 ┌─────────────────┐
Applications --->│    LiteLLM      │
                 │                 │
                 │ Auth            │
                 │ Rate limits     │
                 │ Token quotas    │
                 │ Routing         │
                 │ Caching         │
                 │ Cost tracking   │
                 └───────┬─────────┘
                         |
                    ┌────▼────┐
                    │  Redis  │
                    └─────────┘
                         |
             ┌───────────┼───────────┐
             v           v           v
          vLLM       Azure OA     OpenAI

Redis-backed rate limiting + semantic/request caching is the combination I'd focus on. For LLMs, don't make request count your only quota: a 10-token request and a 100,000-token request shouldn't necessarily consume the same budget.

Bottom line:

• Starting fresh and primarily serving LLMs → LiteLLM
• Already using an API gateway / need broad API-management features → Kong
• Kubernetes + Envoy-centric platform → Envoy AI Gateway

If you tell me whether your LLMs are self-hosted (vLLM/Ollama), cloud APIs, or a mix, I can sketch the recommended architecture and Redis/cache strategy.

Kong has conventional request rate limiting, including distributed Redis-backed limits, sliding windows, consumer-level limits, etc.Kong Docs+1

More importantly for your use case, Kong's AI Gateway can rate-limit based on LLM token usage/cost rather than merely request count. Kong Docs Kong Docs It also has an AI semantic-cache capability.Kong Docs

So if you already have:

                    +--> REST APIs
                    |
Clients --> Kong ---+--> LLM API
                    |
                    +--> Internal services

I'd strongly favor Kong rather than adding a second gateway.

3. Envoy AI Gateway

Envoy AI Gateway is compelling if you're heavily invested in Kubernetes/Envoy.

It provides a two-tier gateway architecture, with authentication, routing, and global rate limiting at the first tier and more fine-grained model-serving controls at the second. GitHub Envoy Gateway also supports both global and local rate limiting.Envoy Gateway

I'd choose it primarily for an infrastructure/platform team already standardized on Envoy, rather than introducing it solely to get LLM caching.

What I'd deploy

For most internal LLM platforms, I'd start with:

                 ┌─────────────────┐
Applications --->│    LiteLLM      │
                 │                 │
                 │ Auth            │
                 │ Rate limits     │
                 │ Token quotas    │
                 │ Routing         │
                 │ Caching         │
                 │ Cost tracking   │
                 └───────┬─────────┘
                         |
                    ┌────▼────┐
                    │  Redis  │
                    └─────────┘
                         |
             ┌───────────┼───────────┐
             v           v           v
          vLLM       Azure OA     OpenAI

Redis-backed rate limiting + semantic/request caching is the combination I'd focus on. For LLMs, don't make request count your only quota: a 10-token request and a 100,000-token request shouldn't necessarily consume the same budget.

Bottom line:

• Starting fresh and primarily serving LLMs → LiteLLM
• Already using an API gateway / need broad API-management features → Kong
• Kubernetes + Envoy-centric platform → Envoy AI Gateway

If you tell me whether your LLMs are self-hosted (vLLM/Ollama), cloud APIs, or a mix, I can sketch the recommended architecture and Redis/cache strategy.