How DeepSeek Can Fail 19 Times in 20 and Still Beat Claude on Cost

Hello builders,

The question of when not to use a cheaper llm model has an exact answer, and at a wide enough price gap that answer is 19 failures out of 20. Most people shop the way you would buy insurance: hedge toward quality, pick the cheapest model that is still clearly good, and keep everything borderline on Claude. The arithmetic says pick the cheapest model that is merely acceptable and let a guardrail absorb the misses, because the gap is what pays for the failures.

Here is the whole thing. Routing with a guardrail costs the cheap call every time, plus the frontier call on the fraction that fail. Never routing costs the frontier call every time. So routing wins while:

escalation_rate < 1 - (cheap / frontier)

That’s it. So how much failure can your cheap model actually afford? Let’s run it on real numbers, because the answer runs against almost everybody’s instinct.

The bigger the gap, the more you can miss

Take our cheap model at $0.0031 per task against a frontier model at $0.0684.

break-even escalation rate = 1 - (0.0031 / 0.0684) = 95.5%

Ninety-five and a half percent. With a gap that wide the cheap model could fail nineteen tasks out of twenty and routing would still be cheaper than not routing. Watch what that does across the range:

at 10% escalation:  $0.0031 + 0.10 x $0.0684 = $0.0099   (-85% vs frontier-only)
at 50% escalation:  $0.0031 + 0.50 x $0.0684 = $0.0373   (-45%)
at 90% escalation:  $0.0031 + 0.90 x $0.0684 = $0.0647   (-5%)

Now we narrow the gap. A cheap model at $0.048 against the same $0.0684:

break-even escalation rate = 1 - (0.048 / 0.0684) = 29.8%

Thirty percent. Three failures in ten and your clever routing setup is costing you money.

Same machinery, opposite verdict, and the only variable that moved is the gap. So which of those two are you? Whichever it is, the rule is the same: the bigger the price gap, the more failure you can afford. Which inverts how most people shop. A model at 70% of frontier price has to be nearly perfect to be worth the machinery. A model at 5% of frontier price can be mediocre and still win.

Two things lower the real break-even, and we have to account for both. Latency, because an escalated task takes roughly twice the wall-clock time: you ran the cheap model, waited, judged it, then ran the expensive one. For batch work that’s free; for anything a human waits on, a 20% escalation rate means one request in five feels broken, and users notice p95 latency long before they notice a bill. And your predicate’s false-pass rate, because every escalation is a failure you caught, and the ones your check misses ship.

Nobody can hand you your easy fraction

The one number that decides all of this is the share of your work that is genuinely easy, and that is a property of your business, not of any model. The clearest public evidence is a routing paper whose own results range over a factor of two and a half on the same routers:

• MT Bench: over 85% cost reduction
• MMLU: 45%
• GSM8K: 35%

All three while retaining 95% of GPT-4’s performance (RouteLLM, LMSYS). Same method, three benchmarks, and the savings swing by 2.5x, because MT Bench has a lot of easy conversational turns and MMLU is harder, so the router has to call the strong model more often and saves less.

Which is the argument for measuring our own escalation rate, made by the literature itself. You will also see “the strong model is only needed 14% of the time” quoted from this work. Careful: which benchmark? That is MT Bench with a particular judge augmentation. On Arena data alone the same work reports 26%, and on MMLU it is 54%. Anyone quoting 14% without the benchmark attached is quoting a number they don’t understand.

What a gateway will not catch

Our escalation rate only exists if something is measuring quality, and here is where two products will let you believe you already have that.

Model-group fallbacks in a gateway are provider failover, not quality failover. They fire when a provider returns an error, times out, or rate-limits you. They are excellent at that. What they cannot do is notice that a model returned a fast, well-formed, confident, completely wrong answer. So who catches that one? Nobody but you, because nothing in a gateway has ever seen your definition of correct:

# guard.py - wraps the router you already have.
def guarded(task, messages, **kw):
    """Route, check, escalate on a failed check. Both attempts share one job id."""
    job, cheap = uuid.uuid4().hex[:12], route(task)
    check = TASKS[task]["check"]
    if cheap != FRONTIER:
        text = text_of(traced(complete, model=cheap, task=task, meta={"job": job},
                              alias=cheap, messages=messages, **kw))
        try:
            if check(text, dict(kw, input=messages[-1]["content"], id=job)):
                return text, {"job": job, "escalated": False}
        except Exception:
            pass                      # a predicate that throws is a failure
    text = text_of(traced(complete, model=FRONTIER, task=f"{task}:escalated",
                          meta={"job": job}, alias=FRONTIER, messages=messages, **kw))
    return text, {"job": job, "escalated": cheap != FRONTIER}

It returns (text, {"job": ..., "escalated": True|False}), and that boolean is the whole metric: escalations over routed attempts, per task type, is the rate you compare against the break-even.

Two choices matter more than the code. Escalate on a failed check, never on a low self-reported confidence score, because models are badly calibrated about their own uncertainty and a confidence number is one more output to be wrong. And tag the escalated call task:escalated so it cannot hide inside the cheap tier’s cost. You will be offered a cleverer version of this, where a model grades the output instead. I don’t believe in them. I have watched too many people build elaborate scoring rigs that mostly measure formatting.

Then we write the keep-list. Which of your work belongs on it, and for which of the two reasons? The first is derived from a measurement: code review stays on the frontier because the best candidate passed 22 of 40 against the baseline’s 31, which is 71%, and the failure mode is a bug reaching production. The second is a policy call made before any measurement: anything irreversible, anything that ships to a customer unreviewed. Keep the two visibly distinct, because in six months somebody will ask whether the list is still accurate, and only the first kind is worth re-testing when a new model appears.

Now go build something this weekend!

John Cook