Open-source API gateway for internal LLM rate limiting and caching

For an internal LLM API, I’d start by evaluating Apache APISIX, Kong Gateway, and Envoy-based gateways. The best choice depends on whether you need ordinary API rate limiting and response caching, or LLM-specific features such as token-aware quotas, semantic caching, and per-model budgets.

One important distinction: caching LLM responses is not the same as caching ordinary HTTP responses. You need to account for model parameters, prompts, tenant isolation, streaming responses, and potentially sensitive data. A gateway that handles HTTP caching well may not provide semantic or token-aware LLM caching out of the box.

My shortlist

• 

1. Apache APISIX

(Best overall for LLM APIs)

A strong fit if you want traditional API controls and LLM-specific features in one open-source gateway. Its AI plugins include token rate limiting and Redis-backed exact-match or optional semantic response caching.apisix.apache.org+3

License: Apache 2.0

Official AI Gateway documentation

---

• 

2. Kong Gateway

A good choice if your team already uses Kong or wants a conventional API gateway with consumer-level rate limits and mature routing and authentication. Check the exact caching and advanced rate-limiting features against your required edition before committing.docs.konghq.com

License: verify the specific edition and plugins

Rate limiting documentation

---

• 

3. Envoy Gateway

Best if you're Kubernetes-first and want policy-driven traffic management. It supports local and global rate limits, with shared limits using an external rate-limit service and typically Redis. HTTP caching is available in the Envoy ecosystem, but LLM-aware caching may require additional components.gateway.envoyproxy.io+1

License: Apache 2.0 for Envoy

Rate limiting documentation

How I'd implement it

For a new internal LLM API, I'd use APISIX as the gateway and Redis for shared counters and response-cache storage where appropriate.

Internal applications

Service identity · API key · Tenant

Apache APISIX

Authentication · Request limits · Token quotas

Cache lookup · Routing · Metrics

Redis

Shared quotas and cache

LLM backends

Internal model servers or providers

APISIX documents both conventional caching via proxy-cache and LLM-aware caching via ai-cache. Its standard limit-count plugin supports request-count limits, while its AI rate-limiting plugin can enforce token quotas.apisix.apache.org+2

A few implementation details matter:

• Rate-limit at two levels: requests per second/minute for overload protection, plus per-tenant token budgets for cost control.
• Build safe cache keys: include the model, normalized request/prompt, relevant generation parameters, and tenant or authorization scope.
• Handle streaming separately: don't assume streamed responses can be cached like complete responses.
• Protect sensitive prompts: use short TTLs or disable caching for requests involving confidential or user-specific data.
• Measure cache effectiveness: track hit rate, latency saved, tokens avoided, and cache storage costs.

One question that would narrow the recommendation

What does your internal LLM API look like?

1. Deployment environment

Kubernetes

VMs / Docker Compose

Already using Envoy or a service mesh

---

2. What kind of caching?

Exact prompt/request match

Semantic caching (similar prompts reuse answers)

Both

Only ordinary HTTP response caching

---

3. What limits do you need?

Requests per second/minute

Tokens or spend per tenant

Both request and token limits

[Compare for my setup]