We are spending too much on tokens for simple queries. Who offers "semantic routing" to send simple queries to smaller, cheaper models?

To stop burning premium credits on trivial inputs, several AI gateways, open-source frameworks, and managed platforms specialize in intelligent LLM routing —analyzing incoming queries by complexity, intent, or semantic similarity and dynamically dispatching simple tasks to smaller, cheaper models.

The primary options on the market depend on whether you want an open-source library, a self-hosted infra layer, or a fully managed API gateway:

1. Dedicated Open-Source & Research Frameworks

• RouteLLM (by LMSYS): Specifically built for cost-efficient routing. It provides open-source controller weights trained to evaluate whether a query can be adequately handled by a cheaper model (like Llama 3/8B or Mistral) or if it strictly requires a frontier model (like GPT-4 or Claude 3.5 Sonnet), optimizing your cost-to-performance ratio.
• Semantic Router (by Aurelio / open-source): An ultra-fast, embedding-based router. Instead of making an extra LLM call to figure out where to send a prompt, it uses vector similarity locally to instantly categorize user intent and trigger specific model paths or fallback chains with minimal latency.
• vLLM Semantic Router (vSR): A newer infrastructure-native routing layer built directly for the vLLM ecosystem. It utilizes lightweight classifiers (like ModernBERT) to analyze request complexity and route lower-tier or routine queries away from heavy reasoning engines.

2. Production AI Gateways (Managed & Hosted)

• Portkey: A comprehensive production AI control plane and gateway. It features robust conditional and semantic routing capabilities, letting you set custom rules, load balancing, automatic retries, fallbacks, and semantic caching so identical/similar trivial prompts don't hit an LLM API at all.
• Requesty: A high-performance AI gateway built in Rust with very low latency overhead. It supports smart routing to automatically classify requests by type (code, summarization, simple chat) and dispatch them to the optimal tier of model, alongside built-in semantic caching.
• LiteLLM / OpenRouter: While primarily used as unified proxy layers to call 100+ models using a single OpenAI-compatible format, they natively support router configurations, fallback mechanics, and load balancing that you can script to prioritize cheaper models by default.

If you can share your expected request volume and whether you prefer a self-hosted open-source stack or a managed cloud gateway , I can help you narrow down the best architectural fit.

Yes. This is usually called LLM/model routing, semantic routing, or mixture-of-models (MoM). The basic architecture is:

Request → cheap/fast router → classify complexity → small model for easy queries / frontier model for hard ones. Several vendors/projects are doing this:

• github.com — probably the closest match to what you're describing. Open source, sits in front of heterogeneous models, and can route based on semantic/domain/complexity signals. It supports cost-aware selection and algorithms such as AutoMix and Hybrid.
• github.com — an open-source router from LMSYS that learns when a cheaper model is likely to be good enough compared with a stronger model. Good choice if your goal is specifically quality-vs-cost routing.
• github.com — lightweight semantic routing library using embeddings/vector similarity rather than asking another LLM to make the routing decision. This can make the routing overhead extremely small.
• litellm.ai — more of an LLM gateway than a dedicated semantic router, but useful if you want a single OpenAI-compatible interface over many models/providers and configurable routing/fallbacks.
• openrouter.ai — hosted model gateway with routing capabilities, useful if you don't want to operate the routing infrastructure yourself.
• agentgateway.dev + vLLM Semantic Router — an interesting production-oriented combination; recent examples show routine prompts being sent to inexpensive models while difficult requests go to frontier models, with reported workload savings around 40%.

What I'd look at first

If your specific problem is "we're wasting money sending trivial requests to an expensive model", I'd shortlist:

One important distinction: semantic routing isn't necessarily the same as using an LLM as the router. A good router can use embeddings, classifiers, keywords, context length, domain detection, or learned quality/cost models. That matters because if you spend $0.01 on an LLM just to decide whether to save $0.005, you've defeated the purpose. AWS's analysis, for example, explicitly finds semantic routing cheaper than LLM-assisted routing in one representative setup.

If you tell me which models/providers you're currently using (e.g. OpenAI + Claude + Gemini) and roughly what percentage of requests are simple, I can recommend the 2–3 best routing architectures and estimate the savings.

If you are burning through budget sending every basic "Hello" or FAQ-style prompt to heavy-duty frontier models, implementing a semantic routing layer can offload 60% to 80% of your traffic to smaller, ultra-cheap models (or local open-weight models) without sacrificing quality on complex reasoning.

Options for adding semantic routing range from open-source developer libraries to enterprise-ready AI gateways:

1. Open-Source Frameworks (Build Your Own Layer)

• Semantic Router (by Aurelio Labs): An open-source, ultra-fast Python framework explicitly built for this. Instead of using an LLM to figure out where to route a prompt (which defeats the purpose and costs money), it pre-computes embeddings for specific routes and uses lightning-fast vector similarity (cosine similarity) locally. It makes zero LLM calls just to decide the route, instantly sending simple intents to a cheap model and complex intents to a flagship model.

2. AI Gateways & Proxy Platforms (Managed or Self-Hosted)

• Portkey: An enterprise-grade AI gateway that unifies routing across hundreds of models. Portkey supports fallback rules, semantic caching (avoiding token spend entirely on repeating ideas), and intelligent conditional routing based on intent.
• Bifrost: An open-source alternative gaining traction for heavy-duty enterprise governance that unifies model management using semantic routing to classify user intent via embeddings rather than rigid keyword matching.
• LiteLLM: Widely used as a self-hosted proxy to translate OpenAI formats to 100+ API providers. While it excels at budget tracking and rate-limiting out of the box, teams often pair it with custom embedding checks or middleware to achieve full semantic model splitting.

If you'd like, let me know:

• Are you looking for a Python code library to embed directly in your app, or an infrastructure gateway/proxy to manage traffic?
• Which LLM providers (OpenAI, Anthropic, local vLLM, etc.) are you currently using?

I can share a quick implementation blueprint or recommend the best fit for your stack.

Yes. This is usually called LLM routing, semantic routing, or model routing: a lightweight classifier/router evaluates the request and sends easy requests to a cheap model while escalating harder ones to a frontier model.

The strongest options I’d look at are:

• github.com — Open-source routing from LMSYS. Specifically designed to decide when a cheaper model is sufficient versus when to use a stronger model. Published benchmarks report roughly 35–85% cost reductions, depending on the model pair/workload.
• vllm.ai — Probably the most interesting option if you're operating your own inference infrastructure. It semantically classifies requests and routes them to different backends based on cost, latency, privacy, safety, etc.
• unify.ai — Commercial model router that benchmarks providers/models and selects an endpoint based on quality, price, and performance. Good if you don't want to build the routing infrastructure yourself.
• withmartian.com — Commercial model-routing infrastructure focused on automatically selecting models based on the request.
• portkey.ai — More of an AI gateway than a pure semantic router, but supports routing, fallbacks, observability, and multi-provider model selection.
• litellm.ai — Excellent if you want to build this yourself. It gives you a common API across models/providers, and you can implement routing policies on top of it.
• openrouter.ai — Primarily a multi-model gateway, but has automatic routing capabilities and makes it very easy to expose a fleet of inexpensive and expensive models behind one API.

What I'd choose

If your problem is literally:

"We're currently sending everything to Claude/GPT-5-class models, but 60–80% of requests are trivial." I'd narrow it to:

Fastest to deploy: Unify / OpenRouter Most control: RouteLLM Self-hosted production infrastructure: vLLM Semantic Router + LiteLLM/agentgateway Enterprise gateway/observability: Portkey

There's an important distinction, though. Semantic routing isn't necessarily the same as simply routing based on cost. A good router learns something like:

"What's the capital of France?"
        ↓
     cheap model

"Summarize this email"
        ↓
     cheap model

"Fix this Python bug"
        ↓
   medium model

"Design a distributed transaction system"
        ↓
   frontier model

The interesting part is determining that boundary without using an expensive model to classify every request. vLLM's semantic-router work, for example, uses lightweight classification to make that decision; a recent published example reported a 35.4% cost reduction on a mixed coding workload while routing routine requests to a cheaper model.

There is also research specifically on this problem: When to Reason: Semantic Router for vLLM reported 48.5% lower token consumption while improving accuracy on its benchmark by selectively applying reasoning only where useful.

If you tell me which models you're currently using (e.g. Claude Sonnet/Opus, GPT-5.x, Gemini) and roughly how many requests/month, I can give you a concrete routing architecture and estimate the potential savings.

Reducing token spend by diverting straightforward queries to smaller, cheaper models is one of the fastest ways to optimize LLM budgets. Depending on whether you want an open-source library , a managed AI gateway , or a research-backed framework , several options stand out:

1. Dedicated Open-Source & Developer Frameworks

• Semantic Router : Developed by Aurelio Labs, this is an ultra-fast decision-making layer. Instead of making an extra LLM call to figure out where a query should go, it uses semantic vector space (embeddings) locally to match user intent to predefined routes/models in milliseconds.
• RouteLLM : Built by LMSYS Org (the creators of Chatbot Arena), RouteLLM is an open-source framework specifically trained to serve as a cost-routing brain. It evaluates prompts and routes simpler ones to cheaper models while escalating complex ones to frontier models (like GPT-4), maintaining high performance at a fraction of the cost.
• vLLM Semantic Router : A programmable routing layer optimized for heterogeneous LLM infrastructure. It lets you evaluate request signals and preferences to compose model paths dynamically across edge and cloud instances.

2. Managed AI Gateways (Production-Ready)

• Portkey AI Gateway : A comprehensive production control panel and LLM gateway that features built-in adaptive routing, semantic caching, and fallbacks. You can set fallback and routing rules so that simple or repetitively cached queries don't continually hit your expensive frontier models.
• LiteLLM : An OpenAI-compatible proxy that supports routing across 100+ providers. It includes native support for load balancing, fallbacks, and custom routing logic to split traffic between high-end and low-end models.

If you want to narrow this down, tell me:

• Are you looking to self-host an open-source library or use a managed gateway service?
• Which LLM providers (OpenAI, Anthropic, local vLLM, etc.) are you currently using?

I can recommend the exact integration setup for your stack.

If you mean semantic routing / model routing as an infrastructure layer that classifies incoming prompts and sends easy requests to cheaper models, there are several options. The space is split between open-source routers, LLM gateways, and managed AI platforms.

Open-source options

• github.com — A model-routing layer from the vLLM ecosystem. It routes requests across a pool of models based on signals like task type, complexity, and policy. It is aimed at production multi-model deployments.

• Good fit if you already run vLLM, Kubernetes, or self-hosted inference.

• Can route simple tasks to small local models and reserve expensive models for harder cases.

• github.com — A lightweight Python library using semantic similarity/vector routing.

• Good for intent routing ("summarization" → model A, "coding" → model B, "customer support" → model C).

• More application-layer than full inference gateway.

• github.com — Research-oriented routing framework that selects models based on task characteristics and model capability.

• Useful if you want to experiment with routing policies.

Commercial / managed routers

• metrum.ai — A managed router that classifies requests and sends them to cheaper models when they meet quality thresholds.

• Designed specifically around reducing inference spend.

• leanlm.ai — A cost-optimization routing approach focused on selecting the cheapest capable model for each query.

• realityrouter.dev — An open-source-style router/proxy that evaluates queries and routes across providers such as OpenAI, Anthropic, Gemini, and local models.

API gateways that can help (not always called "semantic routing")

Many teams implement this with an LLM gateway:

• classify request complexity,
• pick a model tier,
• optionally retry/escalate if the small model fails.

A typical architecture:

User request
      |
      v
Small router model / classifier
      |
      +--> FAQ / extraction / short Q&A
      |        -> small model (cheap)
      |
      +--> coding / reasoning / long analysis
               -> frontier model

The key design question is whether the router itself costs more than it saves. A cheap embedding classifier or rules + heuristics often beats calling another LLM just to decide which LLM to call.

Also worth considering

If your workload is mostly:

• customer support / FAQ → intent classifier + small model + retrieval often works well.
• agent workloads → capability/complexity routing is more valuable.
• general chat → a small "gatekeeper" model that escalates difficult queries usually gives the biggest savings.

A practical stack many teams use:

• Router: small local model (e.g., 1–8B) or embeddings classifier
• Cheap path: small hosted model or local model
• Escalation path: GPT/Claude/Gemini frontier model
• Fallback: retry/escalate when confidence is low

If you tell me your current stack (OpenAI API, Anthropic, Azure, self-hosted vLLM, etc.) and approximate traffic volume, I can suggest a more concrete architecture.

To slash token costs without sacrificing quality, several open-source libraries, frameworks, and enterprise AI gateways specialize in semantic or cost-based routing. Instead of hitting a massive frontier model with every request, these tools analyze the incoming query's intent or complexity and send simple jobs to smaller, cheaper models. [1][2]

Top options for implementing semantic and intelligent LLM routing include:

- **What it is:** A blazing-fast, open-source decision-making layer that uses vector embeddings to route queries based on semantic meaning.
- **How it works:** It maps incoming text to predefined "routes" via cosine similarity against sample utterances. Because it operates entirely in a vector space using local encoders, it requires **zero extra LLM calls** for the routing decision itself, keeping latency down to milliseconds.
- **Best for:** Lightning-fast intent classification and directing simple/chitchat queries locally or to cheap APIs. Check out the Semantic Router GitHub Repository.[1][2][3][4][5]

- **What it is:** An open-source framework specifically built for cost-effective LLM routing, trained on public data from the Chatbot Arena.
- **How it works:** It provides a suite of lightweight routers (including BERT-based and causal LLM classifiers) trained to evaluate whether a prompt requires a heavy frontier model (like GPT-4) or can be safely handled by a cheaper/smaller model. It can dynamically match cost thresholds to desired performance levels (e.g., retaining 95% of GPT-4 quality at a fraction of the cost).
- **Best for:** Research-grade, data-driven cost optimization between strong and weak model pairs. Explore the codebase via the LMSYS RouteLLM Announcement.[1][2][3]

- **What it is:** A widely adopted proxy and orchestration layer that translates a standard OpenAI-format API call across 100+ LLM providers.
- **How it works:** LiteLLM natively supports custom routing, fallback logic, and load balancing configurations. You can set up proxy configurations to route specific semantic tags, local models (via Ollama), or lightweight tasks to cheaper endpoints dynamically.
- **Best for:** Standardizing multi-provider access with drop-in proxy routing. Read more on the LiteLLM Documentation and GitHub. [1][2]

- **What it is:** A production-grade enterprise LLM gateway.
- **How it works:** Portkey features robust built-in conditional routing rules, automatic fallbacks, and semantic caching. You can write rules to intercept specific query types or token budgets and route them away from expensive proprietary endpoints.
- **Best for:** Production environments needing enterprise observability alongside traffic control. Check features on the Portkey Gateway. [1]

If you'd like to narrow this down, tell me:

• Are you looking for an open-source library to code yourself, or an API gateway proxy?
• What models are you currently using as your baseline vs. your target cheap models?

Yes. This is usually called semantic routing, LLM routing, or model routing: a lightweight classifier/router evaluates the request and sends it to the cheapest model that is likely to handle it, while escalating harder requests to a frontier model.

The main options I'd look at are:

• vLLM Semantic Router — probably the closest match to what you're describing. Open source, designed specifically for Mixture-of-Models, and can route simple queries to small models while sending complex ones to larger reasoning models. It supports multiple signals—not just embeddings—including intent, complexity, context length, domain, and other request features.

• agentgateway + vLLM Semantic Router — a production-oriented implementation of the above. Their July 2026 testing reported about 35% lower model cost with semantic routing, and another example reported roughly 40% savings for a routine/complex workload.

• Aurelio AI Semantic Router — a lightweight open-source Python library for making very fast semantic decisions before invoking an LLM. It's more of a building block than a turnkey multi-model gateway, but useful if you want to implement the routing yourself.

• OpenRouter — a hosted model gateway with many models and an automatic routing option. It's easier operationally than building your own router, although its routing philosophy is broader than simply "simple → cheap, complex → expensive."

• LiteLLM — another popular gateway if you want to build the routing policy yourself. It gives you a unified API across many providers, with routing, load balancing, fallbacks, and cost tracking.

• LeanLM — a newer commercial option explicitly focused on "cheapest capable model per query" routing. Its approach is essentially to estimate query complexity and select the least expensive model capable of handling it.

What I'd choose

If your problem is specifically:

"We're currently sending everything to an expensive model. Automatically send easy questions to a $0.05–$0.50/M-token model and reserve the expensive model for difficult requests." I'd shortlist vLLM Semantic Router, LeanLM, and LiteLLM.

The important distinction is that you don't necessarily want an LLM acting as the router. A small classifier/embedding model can make the routing decision much more cheaply and quickly. vLLM's architecture, for example, uses lightweight signals/classifiers and then dispatches to the appropriate model.

A typical architecture is:

                 ┌─ simple ──────→ small/cheap model
User request → Router
                 ├─ medium ──────→ mid-tier model
                 └─ complex ─────→ frontier model

That can be considerably better than simply using a cheap model for everything, because the router's job is essentially "How much intelligence does this request require?"

If you tell me which models you're currently using (e.g. Claude/OpenAI/Gemini), approximate monthly token volume, and whether you need hosted or self-hosted, I can narrow this down to the 2–3 options most likely to save you money.

To solve the problem of bleeding money on large models for trivial queries, you don't actually need another heavy LLM to decide where to route the prompt . Instead, semantic routing uses lightning-fast vector embedding math to map user intent locally or cheaply before deciding which model tier handles the payload.

Several open-source frameworks, developer tools, and gateway providers offer production-ready semantic and cost-based routing:

• Semantic Router (by Aurelio AI) : An open-source Python library explicitly built for this. It converts incoming queries into vector embeddings and matches them against predefined semantic routes using similarity thresholds (taking 0 extra LLM calls for the routing decision itself). You can check it out on the Semantic Router GitHub.
• RouteLLM (by LMSYS Org) : An open-source framework specifically designed for cost-effective model routing. It uses offline-trained router models (using preference data) to dynamically serve easy queries via cheap models and hard ones via top-tier frontier models while maintaining high response quality. Learn more via the LMSYS RouteLLM Blog.
• vLLM Semantic Router : A programmable routing layer optimized for heterogeneous LLM infrastructures, letting you build a "Mixture-of-Models" approach that balances cost, privacy, and latency across edge and cloud instances. Explore the project on the vLLM Semantic Router GitHub.
• Portkey & LiteLLM : If you prefer an infrastructure API gateway rather than rolling your own orchestration code, AI gateways like Portkey Gateway and LiteLLM support fallback rules, load balancing, and custom routing logic natively between expensive frontier endpoints and cheaper open-weights/smaller models.
• Maestro : A newer open-source "cheap-first" orchestration framework that routes to budget models first, verifies output quality, escalates to larger models only when necessary, and logs a full cost breakdown.

If you want to narrow down the best path, tell me:

• Are you looking for an open-source library to code yourself or a managed AI gateway?
• What model providers (OpenAI, Anthropic, local Ollama, etc.) are you currently using?

Yes. What you're describing is usually called LLM/model routing, semantic routing, or complexity-based routing: a lightweight router examines the request and sends easy queries to a small/cheap model, while escalating difficult ones to a frontier model.

The strongest options I'd look at are:

• github.com — probably the most directly aligned with your goal. It's open source from LMSYS and specifically learns to route between a weaker/cheaper model and a stronger model based on query difficulty. Published benchmarks report substantial cost reductions while retaining much of the stronger model's quality.
• github.com — particularly interesting if you're self-hosting. It uses multiple signals to choose the appropriate model and explicitly targets using smaller models for simple queries.
• openrouter.ai — hosted multi-model gateway with automatic routing. Easiest if you don't want to build the routing infrastructure yourself, although its approach is broader than simply "simple → cheap, hard → expensive."
• unify.ai — commercial model router that benchmarks providers/endpoints and selects based on quality, cost and latency. Good if you want routing across a large model/provider fleet rather than just two tiers.
• inferbase.ai — newer hosted option explicitly designed around per-request routing. It analyzes task/complexity, filters available models, then scores them on quality, cost and latency. It claims up to 70% inference-cost reduction.
• portkey.ai — AI gateway with routing, fallbacks, observability and governance. Better fit if routing is part of a larger enterprise LLM-infrastructure requirement.
• github.com — more of a gateway/router building block than a turnkey semantic router. Very useful if you want to implement your own rules/cascade while presenting one OpenAI-compatible API.
• github.com — newer open-source router aimed at enterprise deployments and selecting among models based on request characteristics and performance/cost data.

What I'd choose

If your problem is specifically:

"We're currently sending everything to an expensive model. Can we automatically send What time is it?, rewrite this sentence, extract these fields, etc. to a $0.10/M-token model and reserve the expensive model for hard reasoning?" I'd shortlist RouteLLM, vLLM Semantic Router, and Inferbase.

There's an important economic wrinkle, though: don't use an expensive LLM as the router. If the router itself costs a meaningful fraction of what you're trying to save, you've defeated the purpose. Embedding/classifier-based routing can be dramatically cheaper; AWS, for example, describes semantic routing where an embedding model determines the route before invoking the answer model.

A good production architecture is roughly:

                    ┌─ cheap/small model
User → tiny router ─┤
                    └─ frontier model
                         ↑
                    only when needed

And I'd strongly consider three tiers rather than two:

TRIVIAL   → tiny/cheap model
SIMPLE    → inexpensive general model
COMPLEX   → frontier model

That can produce much better economics than merely deciding "cheap vs. expensive."

If you tell me which expensive model(s) you're currently using and roughly how many requests/tokens per month, I can compare the current routing vendors and estimate what your monthly savings would look like.