2026-09-12 · 4 min read

AI code review benchmarks, 2026 update: the numbers got worse, and diff size explains why

In July we wrote that AI code reviewers miss about half the bugs, based on one vendor benchmark of eight tools against 67 production bugs. That post got read more than anything else we've published, mostly by people asking Claude and ChatGPT whether they can trust their review bot, so it seems fair to come back with what's been published since. Short version: three more benchmarks and two academic papers landed, the numbers got worse rather than better, and the most useful finding is about diff size, which nobody was talking about in July.

The updated scoreboard

Entelligence has refreshed the 67-bug set with a full per-tool table. Best recall is now CodeRabbit at 49.2% (32 of 65 bugs it was evaluated on), best F1 is the benchmark's own author at 47.2%, and the field sits between 34% and 49% recall with precision mostly under 50%. Graphite is the outlier again: 7.5% recall at 66.7% precision, the reviewer who says almost nothing and is usually right when it does.

Tenki ran a separate round in May with a bigger bug set, 122 across 50 pull requests from the same open-source repos (cal.com, Sentry, Grafana, Keycloak, Discourse), default configurations, three-model majority-vote judging. Leaving aside Tenki's own number, since it's their benchmark, the rest of the field came in between 3.3% and 36.1% recall: Greptile and Devin at 36.1%, Cursor's Bugbot at 32%, CodeRabbit at 28.7%, Copilot at 24.6%, Graphite at 3.3%. Lower across the board than Entelligence's set, which is what you'd expect from a harder bug set and a stricter judge.

The one first-party figure worth having next to those: Anthropic's hosted Code Review reports that less than 1% of its findings get marked incorrect and that on PRs over a thousand lines it finds an average of 7.5 issues. That's precision and volume, not recall. Nobody publishes their own miss rate, for the obvious reason.

The academic numbers are lower still

Two arXiv papers this spring did the thing the vendors can't, which is evaluate the underlying models on human-annotated ground truth with no product in the race.

SWE-PRBench built 350 pull requests with human-labelled issues and ran eight frontier models against them. The abstract's own sentence: models "detect only 15-31% of human-flagged issues on the diff-only configuration, demonstrating that AI code review remains far below human expert performance despite strong results on code generation benchmarks." And a second finding that matters for anyone tuning these tools: performance degrades as you give the model more surrounding context, which the authors call attention dilution. More context, worse review.

The second paper has the finding I'd put on a slide. "Bigger Isn't Always Better" compared models on 100 synthetic mutations plus 50 real PRs and measured F1 by diff size. On diffs under ten lines: 0.657. On diffs over 150 lines: 0.043. The same models, the same bug classes, and a fifteen-fold collapse once the diff is the size of an actual feature. They also report near-zero recall on performance bugs, zero for four of the five models, which tracks with the N+1 and quadratic-join PRs in our library being the ones nobody's bot ever comments on. And in a small detail that should worry anyone paying for the big model: Haiku 4.5 beat Sonnet 4.6 on F1, with 18% higher recall.

Why the size finding changes the conversation

Every tool benchmark and both papers agree on the shape, once you line them up: the smaller the diff, the better the bot. On a twenty-line change it's genuinely good. On a 400-line feature PR, the kind an agent now produces before lunch, it's close to noise. And 400-line agent PRs are exactly what's flooding review queues, which GitHub's own "agent pull requests are everywhere" post from May is candid about. The tool is weakest on the input that's growing fastest.

That inverts the lazy division of labour, where the bot handles the big boring PR and the human spot-checks. On the evidence, it should be the other way round: let the bot sweep the small diffs where it's strong, and put the human's attention on the large ones, in the specific bug classes where the numbers say the models fail. Concurrency, authorization, anything performance-shaped, and anything whose bug lives in the relationship between two files rather than in one line.

The number that still isn't measured

The July post ended on the point that the bots get a benchmark and the humans get vibes, and after five more publications that's still true. Every one of these studies needed planted ground truth, bugs known in advance, to compute recall. Nobody runs that on the reviewer whose name is on the approve button. Which is a shame, because the human's number is the one that determines whether the 51% to 85% the bots miss gets caught or shipped.

You can find your own in an afternoon. The library has two hundred-odd PRs with planted bugs across the categories above, graded per category, and most people discover that "I'm a careful reviewer" and their recall on authorization slips are further apart than they'd like. The bot's number is public now. Yours can be too, at least to you.