Compared two local 27Bs against a hosted frontier model for agentic code review

[u/mahendragr]

We wanted to see If we can make the code review cheaper and if cheaper model still holds up against a frontier one so we benchmarked two local models against a frontier model using real production PRs.

The Setup

Three models through the exact same review agent—same tools, same prompts, same review code path:

- 1x hosted frontier model
- 2x 27B models on a single H200 (bf16 via vLLM)

Cases were real merge requests from a production codebase, replayed at their original commits with full repo access. It's a proper agentic loop: reads the diff, greps the repo, opens files, and follows references across 12+ turns.

Single-shot prompt benchmarks give a false sense of security here—a model that looks fine on an isolated diff hunk behaves completely differently when it actually has to navigate a tree. Every finding was checked against the code at that commit by a separate model, blind to which reviewer produced it.

What We Measured

Four metrics (because they trade off against each other):

- Detection — does it land on cases that actually contain a bug
- Noise — does it stay quiet on clean code
- Precision — is what it says true when it does speak
- Completion — does the review finish at all, vs. timing out or erroring

What are the different experiments we did:

- Toggling reasoning on vs. off with everything else held constant.
- Forcing the harness to execute more tool calls (opening every changed file before returning a verdict, enforcing minimum call counts, and batching).
- Extracting the step-by-step review procedure the best model visibly followed in its traces and handing that to the weaker models as explicit system instructions.

Key Takeaways

The two local models ended up further apart from each other than the better local model was from the frontier API—despite running on the exact same GPU, harness, and prompts.

The local model's failure mode wasn't context size or speed; it was grounding. Most of its findings didn't cite specific files, making them unactionable. When we forced hard constraints to require file and line citations, it dropped its findings entirely rather than outputting grounded ones.

More details are in the blog: https://getspinal.com/blog/frontier-vs-local-code-review