What is model routing?

Model Routing Basics

Most AI products begin with one model. That is usually the right decision. One endpoint is easy to debug and easy to budget. When something goes wrong, the failure path is short.

The arrangement starts to look wasteful once the workload splits into obviously different kinds of work. Imagine a support product handling password resets, billing disputes, log analysis, and open-ended troubleshooting. Sending all four to the strongest model buys quality the password-reset flow probably does not need. Sending all four to the cheapest model creates a different problem: the difficult cases fail first.

A router sits between the application and a pool of models. For each request, it chooses a model. It may also choose a provider, tool chain, or sequence of models using information available at decision time. That information might be as simple as a task label. It might include context length, user tier, data policy, recent provider health, or a prediction of how each model will perform.

The point is not to find a universally best model. It is to beat the result of using one model indiscriminately across a mixed workload. RouteLLM, for example, studies learned selection between a stronger and weaker model using preference data. FrugalGPT studies cascades that call additional models only when needed. Both are more involved than most teams need on day one.

A useful trigger is usually mundane: a model bill that keeps climbing or a latency target one model can't meet. A repeatable task may also fail on the cheaper default. Start with the failure you can measure. If you can't name the group of requests that should move and the metric that should improve, a router will add machinery without giving you a better system.

if (request.contains_sensitive_data) return PRIVATE_MODEL
if (request.task === 'extract_fields') return FAST_MODEL
if (request.requires_deep_reasoning) return STRONG_MODEL
return DEFAULT_MODEL

What's required to use a Model Router?

You do not need a research team. You do need enough plumbing to make the routing decision observable and reversible. A router that cannot explain where a request went is just another source of production ambiguity.

You can build this yourself, but most teams will be better served by a third-party router or AI gateway. There are many good options. The routine work includes provider adapters, retries, credentials, logging, and policy enforcement. It adds up quickly. Build the selection logic yourself only when it is central to the product or unusually specific to the workload.

Response normalization is the part most diagrams skip. Two providers can accept similar chat requests and still disagree on tool-call shapes, token accounting, error codes, or streaming events. Test those seams before you send live traffic through OpenRouter, Vercel AI Gateway, Not Diamond, or a homegrown proxy. A successful hello-world request proves very little.

• Start with at least two useful destinations. They should differ in a way that matters so you can compare results. Some common ones are model strength, price, speed, context size, privacy posture, or availability.
• Manage a consistent request contract. Messages, tools, structured outputs, errors, and usage data need to survive a change of model.
• Define a policy that governs your routing. Start with an explicit answer to: what signal causes a request to take a different route?
• Hard constraints should run before price or speed decisions. Data residency, zero-retention requirements, required modalities, and context limits should remove ineligible routes.
• Make sure there's a fallback. Decide what happens on a timeout, rate limit, invalid response, or provider outage before one occurs.
• Track traces for every decision so you can tune: candidate routes, selected route, policy version, latency, token use, estimated cost, and outcome.

When does it make sense to introduce routing?

As a starting point, you should assess if you have enough volume to make this exercise worth it. Suppose a team spends $10,000 a month with one premium model. If 60% of those requests can move to a model that costs one quarter as much, the gross saving is $4,500 a month. That is a worked example, not a forecast. Retries, longer outputs, gateway fees, evaluation, and engineering time all come out of that number.

Your requests should have some variability. If you are dealing with approximately the same question every time, you're better off finding an optimal model and routing all traffic to that option. Routing becomes useful when prompts vary by task, payload, user, latency budget, privacy requirement, or expected value. Those differences predict which model will do well.

Task complexity also needs to spread out. A workflow that mixes extraction, summarization, code execution, and difficult planning is a better routing candidate than a batch job that performs the same classification all day.

There is no magic request threshold. A low-volume medical workflow might justify routing for privacy and model-strength reasons. A high-volume but perfectly uniform workload might not justify it at all. The test is whether you can name the decision, measure the outcome, and afford the extra failure modes.

Use your invoice for the first pass. Mark the traffic that has an obvious cheaper destination, multiply it by the price difference, then discount the result for errors and operating cost. If the remaining figure would not change a budget decision, keep the single-model setup.

• The traffic is large enough for small unit differences to matter.
• The requests fall into recognizably different groups.
• Those groups need different models or operating constraints.
• A fixed-model baseline is leaving measurable cost, speed, quality, or resilience on the table.
• Someone owns evaluation and rollback after launch.

What's the difference between a model router and an AI gateway? Do I need both?

The clean distinction is this: a router chooses; a gateway controls. A model router asks which destination should handle this request. An AI gateway handles the shared path to those destinations. It covers authentication, provider adapters, rate limits, logs, retries, caching, data controls, and cost accounting.

Product pages blur the distinction because the products themselves overlap. Gateways add routing. Routers add observability and policy controls. OpenRouter’s provider-routing documentation, for example, exposes sorting by price or throughput alongside fallbacks and data-policy filters. Latency can drive provider selection too. That puts gateway behavior and routing behavior in one request object.

If your application already knows the model it wants, start with a gateway. If the difficult question is which model should answer each prompt, evaluate the quality of the routing decision. If you need both, first see whether one platform covers the requirements. Two separate products create two policy surfaces and two places to investigate a bad route.

A combined product gives the on-call engineer one trace to follow from request to provider response. Separate layers offer more freedom to replace the routing logic or gateway later. The cost appears during an incident: a timeout may pass through two retry policies and two logs. It may also put two support teams on the same ticket before anyone finds the cause.

Choose from the failure backward. Write down who owns retries, who can block a provider, where sensitive fields are removed, and which log explains the final route. If those answers span two products, the extra control may be worth it. If nobody owns the seam, use one platform.

What are some routing strategies?

Most routing taxonomies make the topic look more settled than it is. In practice, teams combine a few plain mechanisms and add sophistication only where the mistakes are expensive.

Rules are the natural starting point. They are fast and deterministic. Overrides are easy to reason about. Eligibility filters are rules too: eliminate models that cannot accept the context length, tool schema, modality, region, or retention policy. Provider routing keeps the model fixed and chooses a host based on price, latency, throughput, or availability.

Classifiers and semantic routers infer a task or match the request with known examples. Cascades reverse the order of commitment: try the cheaper model first, validate its answer, and escalate when it fails. Learned routers predict which model is likely to win for the request. Pareto and utility-based policies weigh predicted quality against cost and latency.

Session routing changes the unit of the decision. Instead of rerouting every prompt, it keeps a conversation or workflow on one model until a defined event breaks affinity. That avoids erratic behavior between turns but may preserve a poor initial choice.

For a first production policy, use hard eligibility filters and three to five readable task rules. Add one tested fallback. Replay at least 200 representative requests against that policy. Look at the wrong routes before you look at aggregate savings; ten expensive mistakes can matter more than a tidy average.

Whichever strategy you choose, evaluate the decision, not just the final answer. Compare it with fixed strong and cheap models, then compare it with simple rules. An oracle that selects the best known outcome provides a useful upper bound. RouteLLM and FrugalGPT are useful research references, but neither result substitutes for replaying your own traffic.