2026-07-28 · 5 min read

AI code reviewers miss about half the bugs, and nobody benchmarks the human catching the rest

Last week Entelligence published a benchmark of eight AI code review tools against 67 real production bugs, and the bugs aren't toy examples: race conditions, security holes, breaking API changes and logic errors that actually shipped in Cal.com, Sentry, Discourse, Keycloak and Grafana, each one confirmed by the fix that landed later. The winner found 30 of the 67, which sounds respectable until you realize that the best recall in the entire field was 49%, a careful way of saying the most thorough tool in the lineup missed every other bug. And yes, the winner of the benchmark is the company that ran it, which tells you something about this market too.

The numbers, with the salt they need

Vendor-run benchmarks deserve suspicion, so it matters that the independent data agrees. Martian's larger study, built on 200,000+ real pull requests with open-sourced methodology, put the field's effectiveness at 50 to 60%, with most tools deliberately tuned toward precision over recall: they'd rather stay silent on a real bug than risk leaving a noisy comment on clean code, because noisy comments get tools uninstalled faster than missed bugs get noticed. That tradeoff shows up vividly in the Entelligence lineup, where one tool posted 67% precision with 7.5% recall, which means almost everything it said was right and it said almost nothing, the kind of reviewer who shows up to one standup a month but always has a good point when they do.

To be clear about what this doesn't mean: it doesn't mean the tools are useless, because catching 40% of shipped-to-production bugs before they ship is real value, it's cheap, and it's tireless in a way no human reviewer will ever be. The problem isn't the tools themselves. The problem is what teams infer from the absence of a comment, because "the bot didn't flag anything" and "there's nothing wrong" are very different statements and people treat them as the same one.

Silence is a coin flip

When an AI reviewer leaves no comment on a PR, the natural reading, the one your brain defaults to at 4pm on a Thursday, is "checked, found nothing." The benchmark says the honest reading is closer to "found nothing, with roughly even odds that something is there," which is about as useful as a coin flip when you're deciding whether to hit approve. The human tendency to blur those two readings has a name from aviation: automation complacency, the pattern where an automated system is right most of the time and the human gradually stops checking precisely at the moments where checking would have mattered. We've written before about the trust gap, where 61% of engineering leaders shipped an AI-code incident in a single quarter while simultaneously telling a surveyor their trust in AI code had gone up. Reviewer bots with fifty-percent recall, trusted like they have ninety, are a big part of how that gap keeps widening.

The operational takeaway fits in one sentence: an AI reviewer changes what the human reviews, not whether the human reviews. Treat the bot's comments as leads worth investigating, treat its silence as no information whatsoever, and keep in mind that the bugs it misses are not the exotic edge cases you'd expect. The benchmark data says race conditions and authorization slips, the bread and butter of production incidents, are exactly the classes that hide behind plausible-looking diffs and sail past every tool in the lineup.

The mirror nobody holds up

Here's the part of the benchmark that interests me most, and it isn't any of the numbers. It's the method. To measure a reviewer, the authors needed ground truth: bugs known in advance, planted in real code, with recall and precision computed against them. That's the only honest way to measure how good a reviewer is at reviewing, because in a live repo you never find out what a reviewer missed. Escaped defects surface months later, usually during an incident, and by then nobody traces the bug back to the specific PR where someone hit approve and moved on with their day.

Now notice who never gets measured that way: the humans. The reviewer whose approval actually gates the merge, the person whose name is on the approve button, has no recall number, no precision number, and no idea which categories of bugs they systematically walk past. The bots get a benchmark with precision and recall computed to the decimal. The person responsible for catching the other half of the bugs gets vibes and a vague sense that they're "pretty thorough."

That's the exact gap DiffDojo is built on: realistic AI-written PRs with planted bugs, your comments graded against the rubric, and a profile that accumulates your recall per failure category, your precision (where fair questions cost nothing but false flags count), and the blind spots you didn't know you had. The same axes the bot benchmarks use, pointed at the human reviewer for once. Today's PR is free, no signup. It's a genuinely interesting exercise to find your own number and put it next to the chart above, because most people discover that their self-assessed "I'm a careful reviewer" and their actual catch rate are further apart than they'd like.