Route Changelogs to DeepSeek, Keep Claude on Code Review

Hello builders,

Done properly, llm model routing sends the changelog to DeepSeek on every single request without you remembering to decide, keeps code review on Claude, and when somebody asks six months from now why the review still costs what it costs, you hand them a row of numbers instead of an opinion. Done the usual way it looks like this, which is one line out of a real trace log:

{"ok": true, "latency_ms": 1840, "cost_usd": 0.0009, "model": "ds-flash",
 "task_type": "code-review", "output": "No issues found."}

Successful call, under two seconds, at nine hundredths of a cent against the seven cents the frontier would have charged.

The diff it reviewed had an off-by-one in a retry loop that shipped to production and paged somebody at 4am.

Notice what the instrumentation says about that event. ok: true. Cost per task went down. Nothing anywhere went red. A router is a machine for spending less money, and pointed at a task nobody measured it becomes a machine for buying worse answers at high speed, quietly, until something expensive breaks.

No number, no route

So here is the rule, stated before a single line of routing code. A task may only be routed to a model that has a measured pass rate on that task type. No number, no route.

That’s the whole difference between this and the thing half the internet has built and abandoned. Ask how people decide which calls go where and you get a parade of heuristics: route by prompt length, route by keyword, route by whether it looks hard. Every one of those is a guess about quality wearing an engineering costume.

So which of your task types has a number attached to it right now? If the honest answer is none, you are not ready to route yet, and that is the right place to start rather than a reason to reach for a heuristic. Most people picture a router as something clever that inspects each request and reasons about which model deserves it. Ours is going to be a dictionary. That’s not a teaching simplification, it’s the correct design, because all the thinking already happened when we scored the candidates. A routing table is those measurements sorted by cost and filtered to the ones that cleared the bar.

{
  "changelog": {
    "chosen": "ds-flash",
    "why": "rel_pass 0.97 of baseline at 0.6% of cost, n=40",
    "fallback": "claude-opus-4-8"
  },
  "code-review": {
    "chosen": "claude-opus-4-8",
    "why": "no candidate cleared the bar; best was 0.71 of baseline",
    "fallback": null
  }
}

Read the why fields. Every routing decision now carries the sentence that justifies it, and that sentence contains a sample size. We generate this file from our scorecards, never by hand, and put our risk appetite in two constants at the top of the generator: a minimum relative pass rate (0.95 is a defensible bar) and whether you require the confidence intervals to still overlap. Those two lines are the entire policy of the system, sitting where a colleague can argue with them.

Then our dispatcher is eleven lines of logic:

"""router.py - read the table, pick the model, log the call."""
import json, os
from providers import complete
from tracelog import traced

TABLE = json.load(open(os.environ.get("ROUTES", "routes.json")))

def route(task):
    if os.environ.get("ROUTER_OFF") == "1":    # the kill switch
        return os.environ["DEFAULT_MODEL"]
    rule = TABLE.get(task)
    if rule is None:                    # unknown task = most capable model
        return os.environ["DEFAULT_MODEL"]
    return rule["chosen"]

def run(task, messages, **kw):
    model = route(task)
    return traced(complete, model=model, task=task,
                  prompt_text=messages[-1]["content"],
                  alias=model, messages=messages, **kw)

Your call site now names a job, not a model: run("code-review", messages).

Two lines there are worth arguing about. An unrecognised task goes to the most capable model, never the cheapest, because new work is unmeasured work and unmeasured work hasn’t earned a discount. And ROUTER_OFF is one environment variable that sends everything back to the frontier with no redeploy, so whoever is awake at an inconvenient hour can undo your entire cost project in ten seconds without understanding any of it.

Why not just a gateway

Because a gateway routes on plumbing. LiteLLM ships routing as configuration, and the strategies it accepts are exactly these:

• simple-shuffle, the default and the one the docs recommend for production
• least-busy
• usage-based-routing-v2
• latency-based-routing
• cost-based-routing

(LiteLLM routing docs.) Every one of those is a real, useful thing to route on, and not one of them is can this model do my job. Look at cost-based-routing: it sends work to the cheapest model in a group, with no idea whether that model can finish the task, because nothing in a gateway has ever seen your success criteria. It’s our own routing table with the measurement removed.

The same page says that on failure “the router moves on to the next entry in fallbacks (a different model group).” Read that carefully, because it’s provider failover, not quality failover. It fires on an error, a timeout, a rate limit. It doesn’t fire on a fast, well-formed, confident, completely wrong answer. Go back to the trace at the top: no gateway on earth reroutes that one.

So here is my recommendation, since I said I’d give you opinions rather than options: own the routing decision, rent the plumbing. Keep the table in your repo where it can be tested and where every rule carries its evidence, and let a gateway handle keys, retries and spend caps.

Turn it on safely

Don’t flip the switch. We run the router in shadow mode first: serve the frontier answer to your user exactly as today, send the same request to the model the table chose, log it under a separate shadow: task tag so it never contaminates your real cost numbers, and throw the answer away. Score the pair offline. If the cheap candidate runs at 0.6% of the frontier’s cost per task, a week of shadowing adds 0.6% to your bill, which is the cheapest insurance you will buy this year.

Then we promote by percentage, one task type at a time. Five percent, then 25, then all of it, and never two task types on the same day, because when something goes wrong you want one suspect. Watch downstream retries rather than the cost graph. The cost graph will improve immediately, which is exactly why it tells you nothing.

Now go build something this weekend!

John Cook