ChatGPT SearchOct 12, 2026
For an internal LLM API, I’d start by evaluating Apache APISIX, Kong Gateway, and Envoy-based gateways. The best choice depends on whether you need ordinary API rate limiting and response caching, or LLM-specific features such as token-aware quotas, semantic caching, and per-model budgets.
One important distinction: caching LLM responses is not the same as caching ordinary HTTP responses. You need to account for model parameters, prompts, tenant isolation, streaming responses, and potentially sensitive data. A gateway that handles HTTP caching well may not provide semantic or token-aware LLM caching out of the box.
My shortlist
1. Apache APISIX
(Best overall for LLM APIs)
A strong fit if you want traditional API controls and LLM-specific features in one open-source gateway. Its AI plugins include token rate limiting and Redis-backed exact-match or optional semantic response caching.
