route-switch

Overview

route-switch investigates the cost-quality frontier in LLM serving empirically rather than by rule. The premise is uncontroversial: a small model answers most requests acceptably and fails on a minority, and the failures are hard to predict from the surface features of the request. The usual response is a hand-written routing heuristic based on prompt length or a keyword list, which encodes an assumption nobody has tested.

The service is an OpenAI-compatible gateway, so it drops into an existing client without a rewrite, and it routes across OpenAI, Anthropic, Google, Ollama, Cohere and Mistral. What distinguishes it from a gateway is what happens to the traffic afterwards. Per-prompt analytics are written to DuckDB, and MIPROv2 optimisation is rerun against those captured traces to tune per-model prompts on the real request distribution rather than on a benchmark set someone else assembled.

That is the argument in one sentence: a gateway gives you provider portability, and a router that learns gives you a prompt tuned to the traffic you actually receive. The two are separable, and most deployments that believe they need the second are currently buying the first.

Go was chosen for the proxy path because the workload is concurrency-bound rather than compute-bound and the operational story matters. This is a component that sits in front of production traffic, and a single static binary with predictable memory behaviour is easier to justify to whoever carries the pager.

Where it loses: LiteLLM has broader provider coverage and a larger community, Portkey and OpenRouter offer managed operation with no infrastructure to run, and for a team that wants routing solved without a research question attached those are better answers. route-switch also needs traffic before it is useful — optimisation against captured traces is worth nothing until there are traces — which makes it a poor fit for a prototype.

The open question is drift. A prompt optimised against last month's distribution is not necessarily optimal against this month's, and we do not yet have a principled trigger for re-optimisation that is better than a schedule.

In portfolio terms route-switch sits next to promptel and blogus without overlapping them. Those two concern how a prompt is written and governed before it ships; route-switch concerns what happens to it afterwards, when live traffic reveals that the wording which tested well is not the wording that performs. The three can be used together and none of them depends on the others.

Two practical notes for anyone evaluating it. DuckDB as the analytics store is a deliberate choice to keep the whole thing self-contained: trace analysis is an embarrassingly analytical workload, and requiring a separate warehouse to answer questions about your own gateway is a poor trade. And routing decisions are logged with the request, so a surprising answer can be traced back to the model that produced it rather than inferred from a latency graph.