2026-07-28 · 5 min read

AI code reviewers miss about half the bugs, and nobody benchmarks the human catching the rest

Last week Entelligence published a benchmark of eight AI code review tools against 67 real production bugs: race conditions, security holes, breaking API changes and logic errors that actually shipped in Cal.com, Sentry, Discourse, Keycloak and Grafana, each one confirmed by the fix that landed later. The winner found 30 of the 67. The best recall in the field was 49%, which is a careful way of saying the most thorough tool in the lineup missed every other bug. And yes, the winner of the benchmark is the company that ran it, which tells you something about this market too.

The numbers, with the salt they need

Vendor-run benchmarks deserve suspicion, so it matters that the independent data agrees. Martian's larger study, built on 200,000+ real pull requests with open-sourced methodology, put the field's effectiveness at 50 to 60%, with most tools tuned toward precision: they'd rather stay quiet than risk a noisy comment, because noisy comments get tools uninstalled. The tradeoff shows up vividly in the Entelligence lineup: one tool posted 67% precision with 7.5% recall. Almost everything it said was right. It said almost nothing.

To be clear about what this doesn't mean: it doesn't mean the tools are useless. Catching 40% of shipped-to-production bugs before they ship is real value, cheap, and tireless. The problem isn't the tools. The problem is what teams infer from their silence.

Silence is a coin flip

When an AI reviewer leaves no comment, the natural reading is "checked, found nothing". The benchmark says the honest reading is closer to "found nothing, with roughly even odds that something is there". Those are very different statements to attach your approval to, and the human tendency to blur them has a name from aviation: automation complacency. When the bot is right most of the time, the human stops checking exactly where checking matters. We've written before about the trust gap: 61% of engineering leaders shipped an AI-code incident in a quarter while their trust in AI code went up. Reviewer bots with fifty-percent recall, trusted like they had ninety, are part of how that happens.

The operational takeaway is one sentence: an AI reviewer changes what the human reviews, not whether the human reviews. Treat the bot's comments as leads. Treat its silence as no information. The bugs it misses, the benchmark says, are not the exotic ones; race conditions and authorization slips are exactly the classes that hide behind plausible-looking diffs.

The mirror nobody holds up

Here's the part of the benchmark that interests me most, and it isn't a number. It's the method. To measure a reviewer, the authors needed ground truth: bugs known in advance, planted in real code, with recall and precision computed against them. That's the only honest way to measure review, because in a live repo you never learn what a reviewer missed; escaped defects surface months later, unattributed.

Now notice who never gets measured that way: the humans. The reviewer whose approval actually gates the merge has no recall number, no precision number, and no idea which bug categories they systematically miss. The bots get a benchmark; the person responsible for the other half of the bugs gets vibes.

That's the exact gap DiffDojo is built on: realistic AI-written PRs with planted bugs, your comments graded against the rubric, and a profile that accumulates your recall (catch rate per failure category), your precision (false flags, with fair questions costing nothing), and your blind spots. The same axes the bot benchmarks use, pointed at the reviewer for once. Today's PR is free, no signup. It's a genuinely interesting exercise to find your own number and put it next to the chart above.

Read next: Engineering leaders should check AI-generated code more than ever