Can DeepSeek Replace Claude on Your Code Reviews?

Hello builders,

Somebody on your team wants to move the code reviews to DeepSeek, somebody else says quality will drop, and the argument dies there because nobody has a number. It dies in public too: the only evidence is a percentage a vendor measured on somebody else’s codebase. So we are going to compare llm models on your own data instead: replay 40 reviews you really ran, score DeepSeek and Claude with the same pass test, and read the gap in dollars per finished task.

Here’s the bar, from somebody shutting a migration down at their own company:

When a developer is asked “would you rather use Claude or our local LLM”, the answer should not be a definite Claude answer. Then there’s no point in switching.

That’s a completely reasonable standard and it’s unmeasurable as written, which is why that team is still paying full price. “Would you rather” is a survey. So ask the answerable version instead: on your own work, what does the cheap model actually cost you in pass rate?

Replay, do not benchmark

Your instinct is to go find a benchmark. Resist it, and not because benchmarks are bad. Public test sets get published, scraped and swallowed into training data, and from that point they measure memorisation mixed with capability in a ratio nobody can separate.

You already own something better. You own last Tuesday.

Your own trace file cannot have leaked, because it did not exist a month ago. It’s weighted correctly, because it’s your real mix of boring work and hard cases in the proportion you meet them, and that proportion sets your bill. So take a prompt you really sent, send it to a candidate, and grade the answer against the standard you’d have applied to the original.

Score both models

Here’s the step everyone skips, and it decides whether your number means anything. Score your current model on the same traces with the same test. Your frontier model does not pass 100% of its own tasks either. Skip its row and you’ll spend a week blaming a cheap model for failures your test would have flagged on anything.

# score.py - replay YOUR traces against a candidate, score BOTH, print the gap.
import math
from tasks import TASKS         # YOUR pass predicates, one per task type
from providers import complete  # YOUR client, any OpenAI-compatible endpoint
from tracelog import price      # YOUR rate table; nobody hands you a cost

def wilson(k, n, z=1.96):
    """95% interval. Without it, 22 out of 40 reads like a fact."""
    if not n:
        return (0.0, 0.0)
    p, d = k / n, 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))

def score(model, traces, check, replay=True):
    passed, cost = 0, 0.0
    for t in traces:
        if replay:                # candidate: pay for one fresh answer
            r = complete(model, [{"role": "user", "content": t["input"]}])
            text, usage = r.choices[0].message.content, r.usage
        else:                     # baseline: you already bought this one
            text, usage = t["output"], t
        cost += price(model, usage)
        passed += bool(check(text, t))
    n = len(traces)
    return {"model": model, "passed": passed, "rate": passed / n,
            "ci": wilson(passed, n), "cpt": cost / n}

Two calls fill the table, score(baseline, traces, check, replay=False) and score(candidate, traces, check). Only the second costs money: you already bought the first set of answers. Nothing in there is hardcoded to my work. It reads your traces.jsonl, calls your predicate, and prices against your rate table. Nothing leaves your machine.

A correct run looks like this. Numbers illustrative, shape is the point:

task: code-review        traces: 40     candidate: deepseek-v4-flash
                                        baseline:  claude-opus-4-8

                       PASSED   RATE         95% CI    $/TASK
claude-opus-4-8            31    78%     62% - 88%    0.0684
deepseek-v4-flash          22    55%     40% - 69%    0.0031

quality delta  -22 points  (candidate passes 71% of baseline's tasks)
cost delta     -95.5%      (candidate costs 4.5% of baseline)
CI overlap     yes -> at n=40 this gap is suggestive, not established

Read the last two lines before the headline. The delta is measured against your baseline, not against perfection: 22 out of 31 is 71% of what the frontier managed, a different number from the 55% raw rate. And the intervals overlap, so at 40 traces we have established nothing. Printing that is what stops a 22-point headline standing unqualified.

Now fill in the sentence at the top of that scorecard, out loud: on my own work, this model passes this share of the tasks the frontier passes, at this share of the cost. Nobody in any thread you’ve read can say that. You can.

Why per task

We price per finished task because the two rulers disagree, and somebody already published the receipt. Databricks benchmarked coding agents on their own multi-million-line codebase, using tasks built from real internal pull requests, precisely because public benchmarks leak. Cost per finished task came back as $1.28 for GLM 5.2, $1.94 for Opus 4.8 and $2.09 for Sonnet 5:

“Sonnet 5 is ~1.7x cheaper per token than Opus 4.8, but, on our tasks, we found that Sonnet cost $2.09/task vs Opus’s $1.94, while scoring six points lower on task completion (81% vs 87%). This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.”

— Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase

A 1.7x per-token discount, eaten by a 1.9x appetite, and the buyer paid extra for a lower score. GLM 5.2 went the other way, “statistically tied with Opus 4.8 on quality” at two thirds of the price, though the post never puts a completion number on GLM in text, so neither will we.

That’s Databricks’ number, on Databricks’ code, with Databricks’ definition of done. It proves the unit matters. It is not a ranking you inherit, and if you finish this thinking “so GLM is the cheap one” you have taken the wrong lesson twice.

Two warnings, then go run it. A candidate that ties your frontier model exactly should make you suspicious, because identical pass rates usually mean your test checks the format and both models can format. And a candidate that loses is a good day: that’s a documented pass rate instead of a vibe, and it condemns one task type rather than the whole idea of cheaper models.

So read ten outputs by hand. Every time, however good the table looks. I have been wrong in both directions on my own work, and a measurement you trust without eyeballing the raw answers is a measurement of your predicate, not of the model.

Now go build something this weekend!

John Cook