LLM Routing: Strategies, Tools and How to Test Them
Written By:
Matt Tanner
Published on

Alt text (deployment only): A router selects a small, mid-tier, or frontier model, with dashed escalation paths. Listed strategies include rules, classifiers, semantic routing, LLM routing, and cascades.
A support assistant sends every ticket to a frontier model. Most tickets are refund lookups a small model handles fine, so the team adds a router: small model first, frontier model if the small one's answer fails a check. The bill drops, but long, messy tickets now pay for two calls and wait for both, and nobody knows how many of the small model's "passing" answers were wrong.
That's LLM routing in practice: routing rules and thresholds whose right values depend on your traffic. This guide covers the main routing strategies, how an LLM router works, the tools, when routing pays off, and how to test a router on your own queries.
What is LLM routing?
LLM routing is the practice of sending each request to one of several language models based on what that request needs: its task, its difficulty, the data it contains, or its cost and latency budget. The component that makes the call is an LLM router, and the per-request decision is often called model routing.
That's a different job from LLM model selection, which picks one model for a task ahead of time. LLM routing makes that choice on every request. In a support assistant, the router might send a refund lookup to a small model, a contract dispute to a frontier model, and anything containing card details to the one model your security team approved.
Router vs. gateway, in one line: the LLM router decides which model gets a request. An AI gateway is the proxy in front of your providers that handles keys, rate limits, retries, and logs, and it often ships a router as one feature.
Why route requests at all
Teams adopt LLM routing for one of five reasons:
Lower cost. Small and frontier models are priced far apart, and much production traffic is routine. Count savings the way our LLM cost optimization guide does, as cost per successful task: a cheap answer that needs a retry saves nothing.
Latency. A smaller model can return the first token sooner, but the router's own work sits on the request path before that first token. Time to first token already includes network travel, queueing, and prompt processing, so any latency optimization work should measure the router's share too.
Output quality. Some requests need your strongest or a specialized model. Model routing gives them that model without paying frontier prices for everything else.
Rate limits. Spreading traffic across providers keeps one provider's rate limits from capping throughput.
Failover. When a provider returns 429s or 5xx errors, or times out, a fallback model keeps the request alive.
The first three are quality and price decisions. The last two are about availability, and a different mechanism usually handles them. A failover fires when a call errors out; an escalation fires when a call succeeds but the answer isn't good enough. Gateway fallbacks in OpenRouter, LiteLLM and Portkey trigger on errors, so a wrong answer that comes back with HTTP 200 sails through them. We show that in a test further down.
Routing strategies
Five families cover most production LLM routers, and real systems often chain two: a rule layer for hard constraints, then something smarter for the rest. Static and rule-based routing follow conditions you write. The other four are dynamic routing, where a model or a check makes the call per request at runtime.
Static and rule-based routing
Static routing gives each task its own endpoint or model. Rule-based routing reads request attributes (tenant, plan, language, prompt length, a PII flag) and applies routing rules written as plain conditions. Portkey's conditional routing is typical: conditions match request metadata or parameters, and "if no conditions pass, then the default target name is called".
Rules add no model calls and are easy to audit, so they're the right layer for compliance and approved-model constraints. They're blind to difficulty, though: prompt length says little about whether a request is hard.
Classifier-based routing
A trained model scores each request, and the score picks the route. RouteLLM is an open-source example. Its routers choose between "a stronger, more expensive model and a cheaper but weaker model". Each one estimates the strong model's win rate for the prompt, and the request goes to the strong model when that win rate is above your cost threshold. The RouteLLM paper trains these routers on human preference data.
Classifiers are fast once trained, but they need labeled data and drift as your traffic changes.
Semantic (embedding-based) routing
A semantic router embeds the request, compares it with reference examples for each route, and picks the closest route above a similarity threshold, or a default. Semantic routing scales to many intents, and you update it by adding examples. AWS's reference implementation warns that "adequate coverage for all possible task categories in your reference prompt set is crucial for accurate routing". That's the failure to watch, since a request type with no examples still gets sent, confidently, to whichever route is nearest.
LLM-as-router
A small LLM reads the request and returns a label or model name. It handles nuance, and adding a category means editing a prompt. The cost is an extra model call per request, which AWS notes "introduces additional costs and latency".
Cascade routing (try cheap first, escalate)
Cascade routing calls a cheaper model first, checks the answer, and escalates only when the check fails. FrugalGPT framed this as an LLM cascade that "learns which combinations of LLMs to use for different queries". It suits checkable outputs: our guide to small vs. large models describes a text-to-SQL service that falls back to an LLM "whenever the SLM's JSON fails validation".
Two costs are easy to miss. Every escalated request pays for both calls and waits for both: Vercel's AI Gateway docs say a triggered decision fallback "executes and bills both the primary and fallback stages" and that the request "includes the latency of both model calls". And the validator is the real router: if it passes a wrong answer, the cascade looks cheap and is simply wrong.
Strategy | Decides using | Added work per request | Main failure mode | Good fit |
|---|---|---|---|---|
Static/rule-based | Endpoint, metadata, request attributes | None | Blind to difficulty | Compliance, tenant tiers, known task types |
Classifier | Trained score vs. threshold | A small model inference, sometimes an embedding call | Drift; threshold tuned on other traffic | High volume, labeled history |
Semantic | Similarity to reference examples | One embedding call | Uncovered intents sent to the nearest wrong route | Many distinct intents |
LLM-as-router | LLM-generated label | One extra LLM call | Latency and cost on every request | Lower volume, nuanced categories |
Cascade | Check on the cheap model's answer | A second call for every escalation | Weak validator accepts bad answers | Checkable outputs: JSON, SQL, extraction |
Anatomy of an LLM router
Whatever the strategy, the routing layer runs six steps per request:
Request. The prompt arrives with metadata such as tenant, task and plan.
Signal extraction. The router computes its inputs: rule flags, an embedding, a classifier score.
Decision. It builds the candidate list, hard constraints first, then ordered by cost or predicted quality.
Model call. It calls the first candidate.
Fallback. An error moves the request to the next candidate (failover); a rejected answer escalates it (cascade).
Logging. It records every attempt (model, reason, cost, latency) so evaluation can replay the routing decisions.

A router selects a candidate model, records the result, and uses separate paths for errors and rejected answers.
Alt text (deployment only): Routing separates API errors and timeouts, which fail over to the next allowed candidate, from rejected answers, which escalate to a stronger candidate. Each attempt is logged; offline evaluation informs reviewed, versioned routing-policy updates.
Here's that loop in plain Python: one hard rule, then a three-tier cascade, with failover and escalation logged separately. The model call is a stub so it runs offline; prices are placeholders. The check validates JSON shape, not whether the answer is correct. The stub treats timeouts as free and does not measure latency; production logs must capture elapsed time and actual usage, including any billed work before a timeout.
Running it with Python 3 prints:
The second request paid for a rejected small-model call and sat through a timeout before reaching the frontier model, so it cost more than going there directly and took longer. Those per-attempt logs are what you'll aggregate when you test the router.
LLM router vs. AI gateway vs. mixture of experts
Three different things share the word "routing":
| Where it runs | What it chooses | When |
|---|---|---|---|
LLM router | Your app or a gateway | A whole model or provider | Once per request (more with fallbacks) |
AI gateway | A proxy between your app and providers | Provider and model, plus keys, limits, retries and logs | Per request; routing is one feature among several |
Mixture of experts | Inside a single model | Expert sub-networks | For each input, inside the forward pass |
Mixture of experts is a model architecture. In the original sparsely gated MoE paper, "a trainable gating network determines a sparse combination of these experts to use for each example". You don't configure it from your application, and an MoE model can sit behind your LLM router like any other.
Tools and frameworks
LLM routing tools fall into three groups: gateways with routing features, open source routers you train or calibrate, and managed routers that decide for you.
Tool | Type | Routing it offers | Worth knowing |
|---|---|---|---|
Hosted API |
| Billed at the model that answered; no extra fee for Auto Router | |
Open-source SDK and proxy | Load balancing ( | Fallbacks work in the Python Router or the proxy config | |
Gateway | Conditional routing on metadata and parameters with a default target; fallbacks on any non-2xx by default, adjustable with | Conditions run in order | |
Open-source framework | Trained strong/weak routers ( | Apache-2.0; calibrate on your own traffic | |
Vercel AI Gateway | Hosted gateway | Ordered model fallbacks on failure; decision fallbacks that rerun uncertain decision answers on another model | Escalations bill both stages |
Managed | Exactly two models from the same family; a fallback model as anchor; switches on predicted response quality difference | Optimized for English prompts | |
Router API | Predicts the best model per input; custom routers trained on your evaluation data; cost or latency tradeoff | Pre-trained router or one trained on your data |
The gateways handle failover similarly and differ on quality routing, where each product decides with its own trained model, heuristic, or anchor. Bedrock's docs say its router "can't adjust routing decisions or responses based on application-specific performance data". RouteLLM's README says the share of calls going to each model "will differ based on the actual queries received" unless you calibrate on data like yours. To train your own, LLMRouter is an MIT-licensed library with over 16 routing models.
When routing is worth it (and when it is not)
LLM routing earns its keep when:
Your traffic is mixed. A real share of requests is easy, and you can spot them before the call (rules, classifier) or right after it (validator).
The quality gap is uneven. On your hard requests, the frontier model wins clearly; on the easy ones, small and large models tie.
Volume pays for upkeep. Labeled sets, retraining, and threshold reviews cost engineering time that per-request savings must cover.
You have hard constraints. Data residency, approved-model lists and PII handling justify rule-based routing even when cost doesn't.
Skip it, or keep it to rules, when traffic is uniform; pick the right LLM for the task instead. The same goes when the small model rarely passes your check, or when a tight latency budget can't absorb an extra classifier call or a second sequential one.
For a cascade, the break-even math is short. Going direct costs one frontier call per request; a cascade costs the cheap call, the check, and a frontier call for each escalation:
Cascade is cheaper only if: cost(small call) + cost(check) < (1 - escalation rate) ร cost(frontier call)
The formula hides quality. The escalation rate only counts answers your check rejected, and wrong answers it accepted never show up in cost, only in support tickets.
How to test a router before shipping
The right thresholds depend on your queries. RouteLLM calibrates on public Chatbot Arena data by default and still recommends "calibrating on a dataset that closely resembles the types of queries you receive". Testing turns that advice into a procedure:
Build a labeled query set from real traffic. Sample production requests across categories, languages, and difficulty, and label each with an expected answer or acceptance criteria. Hold out a slice for the final check.
Measure quality, cost, and latency per route. Run every request through the router and record each attempt: model, accepted or rejected, cost, and end-to-end latency.
Compare routing configurations and thresholds side by side. Treat each threshold, classifier choice, and model pair as a configuration, add always-small and always-frontier baselines, and run them all on the same set.
These are the numbers to compare. Our guide to evaluation metrics covers how to score the answers themselves.
Metric | Why it matters for routing |
|---|---|
Acceptance rate, per slice | Averages hide the slice a router hurts |
Cost per accepted answer | Counts every attempt, including rejected calls |
Escalation rate | Drives cascade cost and tail latency |
Misroute rate | Easy-route requests the cheap model got wrong |
p95 end-to-end latency | Escalations land in the tail, not the mean |
Our open-source RapidFire engine runs this kind of comparison. Put each router configuration behind an OpenAI-compatible endpoint, such as a RouteLLM server where router-mf-<threshold> sets the threshold or a LiteLLM proxy, and point an RFAPIModelConfig at it with provider "openai" and a custom api_base_url. A List() of those configs under api_config gives one run per routing configuration on the same eval set.
The engine runs configs in parallel, one data shard at a time, so with typed metrics and an accumulate_metrics_fn you get running estimates with confidence intervals.
You can then stop a losing threshold after a few shards, or clone a promising run with a new threshold, using IC Ops; in the open-source engine you trigger them yourself. Converge automates those calls, logs the reason for each, and stays within the time and spend you set.
Two details matter for routers. If your labeled set is sorted by difficulty or category, shuffle it first, or early estimates will be biased. RapidFire's built-in spend metric uses LiteLLM's model pricing and is omitted when a price isn't known, while its latency metric is a per-query average. For router endpoints, compute cost per accepted answer and p95 latency from the router's log of which model answered.
Common mistakes
Over-routing to cheap models. Tune thresholds for spend alone and borderline requests drift to the small model. RouteLLM's README is direct about it: "a higher cost threshold leads to lower cost but may lead to lower-quality responses". Set a quality floor per slice first, then minimize cost above it.
No fallback, or one you've never triggered. Install the tested version with pip install litellm==1.109.1. Here's a minimal LiteLLM Router with one error fallback and two test calls; mock_response keeps the backup call offline too:
We ran it on LiteLLM 1.109.1 with mocked provider responses, so no real model was called:
The forced error reached the backup. The wrong answer stayed on the primary because LiteLLM falls back to another model group only "if a call fails after num_retries". Quality escalation needs its own check, like the validator in the cascade example. The LiteLLM proxy strips mock_testing_* flags since v1.85.0, so test proxy fallbacks trigger real provider errors outside production.
No drift monitoring. Classifier and semantic routing learn last quarter's traffic. Log routing decisions and acceptance per route, and rerun the labeled set when you launch a product, add a language, or a provider updates a model.
Evaluating only on averages. An overall acceptance rate can look fine while one slice collapses: non-English tickets on a router optimized for English prompts, for example, which is a stated limit of Bedrock's router. Report every metric per slice, and watch p95 latency, since escalations live in the tail.
Start small: one rule layer, one cascade, every attempt logged, and three or four thresholds compared against the two baselines. When you're ready to run that comparison, the RapidFire walkthrough gets the open-source engine installed. It needs Python 3.12+, and an OpenAI-only setup needs an API key but no GPU.
FAQ
What is LLM routing?
LLM routing sends each request to one of several language models based on what it needs. The router can use fixed rules, a trained classifier, embedding similarity, an LLM that labels the request, or a cascade that escalates when a cheap model's answer fails a check.
Is RouteLLM free?
RouteLLM's code is open source under the Apache-2.0 license, so there's no license fee. You still pay for the model calls it routes, and its README notes that the mf and sw_ranking routers need an OPENAI_API_KEY to generate embeddings.
What is LLM an acronym for?
LLM stands for large language model. In LLM routing, the LLMs are the candidate models, and the router's job is picking the right LLM for each request.
What are the three types of routing?
In networking, the three types of routing are usually listed as static, default, and dynamic. LLM routing maps onto them loosely. Static routing sends a task to a fixed model, and a default route catches requests when no rule matches (Portkey's default target, for example). Dynamic routing picks a model per request with a classifier, embeddings, an LLM, or a cascade.