Intelligent LLM Routing: Spending Compute Where It Matters

This article was rewritten in October 2026 to describe how route-switch actually works and to withdraw results that had not been published.

Most applications that call a language model call the same model for everything. That is the simplest thing to build and almost never the cheapest thing to run. Model prices span a wide range, and the quality difference between a small model and a large one is real but not uniformly distributed. For some requests — simple classification, format conversion, straightforward extraction — a smaller model is often good enough. For others — multi-step reasoning, nuanced generation, ambiguous instructions — the larger model earns its price.

The research question behind route-switch is how to make that trade-off empirically rather than by intuition: route traffic across models and providers, record what happened, and use the record to improve both the routing and the prompts.

The Cost-Quality Frontier

Every LLM provider offers a spectrum of models at different price points, and the relationship between cost and quality is not linear. It follows a frontier:

Quality
  ^
  |          * Largest frontier models
  |        *
  |      *   Mid-tier models
  |    *
  |  *       Small hosted models
  | *
  |*         Small open models
  +---------------------------------> Cost

Moving right buys quality, with diminishing returns, and the returns depend on the task. A more expensive model can be clearly better on hard requests and indistinguishable on easy ones.

The interesting strategy is not to pick one point on the curve but to pick different points for different traffic. The system-level cost is a weighted average; the system-level quality is determined by whether each request reaches a model — and a prompt — that handles it adequately. The difficulty is knowing where each piece of your traffic actually falls, which is not something you can read off a leaderboard.

Why Static Rules Fail

The first approach most teams try is a rule-based router: if the request contains certain keywords, use the big model; if it is under some length, use the small one; if it involves code, use the code model.

This works for a while and then breaks in predictable ways.

Coverage gaps. Rules encode the author’s intuition about what makes a request hard. That intuition is always incomplete, and novel request patterns fall through.

Maintenance burden. As the application evolves and the traffic shifts, the rules need constant updating. The rule set grows until nobody can reason about it as a whole.

No quality signal. Rules are written from input features alone. Nothing in them records whether the routing decision was right — whether the chosen model produced an adequate result.

The third failure is the important one. You cannot improve a routing decision you never evaluate, and you cannot evaluate one you never recorded.

What route-switch Does

route-switch is a self-hosted gateway written in Go. It exposes an OpenAI-compatible POST /v1/chat/completions endpoint, so an existing client can point at it without a rewrite, and it reaches OpenAI, Anthropic, Google, Ollama, Cohere, Mistral and others through the gollm library.

Routing happens over combinations: a prompt template paired with a model and a provider. The gateway selects among enabled combinations using a configured strategy:

Strategy	Behaviour
round_robin	Rotate through combinations evenly
random	Uniform random selection
weighted_round_robin	Weighted random selection by each combination’s weight
performance_based	Prefer combinations with a higher success rate and lower latency
least_connections	Send to the combination with the fewest in-flight requests

A configuration with a primary and a cheaper fallback combination looks like this:

gateway:
  strategy: "performance_based"
  combinations:
    - id: "default"
      name: "default"
      prompt: "Default onboarding prompt: {topic}"
      model: "gpt-4"
      provider: "openai"
      is_primary: true
      weight: 10
    - id: "fallback"
      name: "fallback"
      prompt: "Default onboarding prompt: {topic}"
      model: "gpt-3.5-turbo"
      provider: "openai"
      weight: 5

It is worth being precise about what this is not. route-switch does not run a classifier, or a small “router model”, that reads each request and predicts its difficulty. The performance_based strategy uses each combination’s observed success rate and latency, not the content of the incoming request. Per-request difficulty routing is a real research direction, and an open one; it is not what this gateway does today.

Capture First

What distinguishes route-switch from a plain gateway is what happens to the traffic after the response is returned. Every invocation is written to a per-prompt SQLite dataset — the rendered input, the output, the variables, the model and provider, whether the call succeeded, its estimated cost and a timestamp — trimmed to a configured maximum number of records. Request summaries, latency, error rates and cost go to a DuckDB analytics store, which backs the /status, /v1/prompts/{id}/stats and /v1/system/analytics endpoints.

This is the unglamorous part, and the part that makes everything else possible. Once each routing decision is logged alongside its outcome, “which combination is better for this prompt?” stops being a matter of opinion and becomes a query.

Where MIPROv2 Comes In

MIPROv2 (Multi-prompt Instruction Proposal Optimiser v2) is a prompt-optimisation algorithm from the DSPy project. route-switch uses it for what it was designed for: optimising prompts. The captured traces become the evaluation set, so a prompt is tuned against the requests you actually receive rather than a benchmark somebody else assembled.

The loop, as implemented:

1. Bootstrap. Load recent records from the prompt’s dataset as calibration data.
2. Generate. Propose instruction candidates through provider calls.
3. Search. Run Bayesian optimisation across combinations of instructions and demonstrations.
4. Evaluate. Replay dataset rows against each candidate and score the outputs with the configured evaluation strategy.
5. Deploy. Write the optimised prompt to the gateway’s registry.

Three evaluation strategies are built in: Similarity (the default, based on token overlap and length heuristics, with a configurable threshold), ExactMatch for deterministic answers, and Keyword for outputs where particular concepts must appear. A custom strategy can implement the EvaluationStrategy interface to plug in a domain metric.

Optimisation can be run on demand from the CLI:

./route-switch --config config.yaml \
  --prompt "Explain {topic} to a 10 year old" \
  --model gpt-4 \
  --provider gollm \
  --optimize-prompt

or periodically by the gateway itself:

gateway:
  optimization:
    enabled: true
    interval_seconds: 1800

There is also a --find-best-model mode that evaluates one prompt across all configured models, which is the closest the tool comes to answering “which model should handle this?” — per prompt, offline, against your own data.

Results: What We Would Measure

route-switch publishes no cost-saving, latency or quality benchmark, and this article quotes none. The honest position is that the trade-off depends on the traffic, so the useful thing a tool can do is make it measurable on yours. If you are evaluating it, the comparison we would run is:

• Baseline. All traffic on the single model you use today, with your current prompt. Record success rate, cost and latency per prompt for a representative period.
• Strategy change alone. Add a cheaper combination and switch to weighted_round_robin or performance_based. Compare the same three numbers, per prompt rather than in aggregate, because an average can hide a prompt that got much worse.
• Prompt optimisation alone. Keep the original model, run MIPROv2 on the captured traces, and compare again. This separates the gain from better wording from the gain from a cheaper model.
• Both together, with the quality measure checked against a human-reviewed sample, because an automated evaluator’s verdict is only as good as the evaluator.

Limitations and Open Questions

Quality measurement is the bottleneck. The optimisation loop needs a quality signal, and the built-in evaluators are heuristics. Similarity scoring rewards outputs that look like previous outputs; an LLM judge introduces its own biases and costs; human review is slow. The quality of the optimisation is bounded by the quality of the measurement, which is a somewhat uncomfortable recursion.

It needs traffic. Optimisation against captured traces is worth nothing until there are traces. The project’s own guidance is to accumulate at least a hundred requests before optimising, which makes route-switch a poor fit for a prototype.

Drift. A prompt optimised against last month’s distribution is not necessarily optimal against this month’s. Background optimisation on a timer is a blunt instrument, and we do not yet have a principled trigger for re-optimisation that is better than a schedule.

Per-request routing is unsolved here. Choosing the model per request from its content — the problem most people mean by “intelligent routing” — would need a predictor whose own cost and error rate are smaller than the savings it produces. That is an open question, and one the captured traces could help answer.

Where This Goes

Routing is a necessary component of any system that uses multiple LLMs. The question is whether it is done implicitly (model names hard-coded per endpoint), explicitly but statically (rule-based routers), or empirically (decisions logged, evaluated and revised).

We think the empirical approach is the right one for most non-trivial applications, and that its first requirement is not a clever router but a record of what happened. The main barrier is evaluation infrastructure: you need a way to measure quality before you can optimise for it.

route-switch is written in Go, released under the MIT licence, and available on GitHub. We welcome contributions, particularly around evaluation strategies and principled triggers for re-optimisation.