Living AI code review benchmark

As of September 2026, 5 AI models reviewed the same 202 PRs with a planted bug or a clean change. Overall recall runs from 86% to 98%. On security bugs recall ranges from 73% to 100%. On code that does not do what was asked, no model exceeds 96%. Every model asked for changes on 1–3 of 28 clean PRs.

What this measures

Each DiffDojo PR is a single change with written requirements and either one planted bug of a known class or nothing wrong at all. A model reads the diff and the requirements, leaves line comments and a verdict, and a pinned grader checks the comments against the rubric the way it grades a human. So this measures one thing: does the model catch the bug when it knows a bug may be there and where to look.

That is an upper bound, not a forecast for your repository. On bugs that reached production, public benchmarks put AI reviewers at 25–47% recall: multi-file changes, no requirements text, no hint that anything is wrong. The gap between the two numbers is the part of code review that still belongs to a person, and it is the part this library trains: the same PRs, the same grader, a blind-spot profile per reviewer instead of a leaderboard.

Where the models still miss

• Security is the least even category. The spread between models on the same security PRs is the widest on the page. Which model reviews your code matters more there than anywhere else.
• Requirement fidelity has a ceiling. "The code works, but it is not what was asked" is the class where the strongest models drop below their own overall recall. There is no wrong line to point at, only a text to compare against, and that comparison is the reviewer's job.
• Clean PRs get rejected. Every model asks for changes on some PRs with nothing planted. A reviewer who cannot say "approve" costs a team time on every PR; knowing when nothing is wrong is half the skill.

Results

Public and private holdout scores are shown separately. Recall is caught planted issues divided by all planted issues on buggy PRs.

Recall by category

Public PRs only, models that scored every public PR; each category uses its own planted-issue denominator.

Darker is higher recall. Cells under 85% are outlined: those are the classes where the model choice matters.

Trend

Methodology

DiffDojo PRs pair written requirements with an AI-written code change and a planted issue or a clean implementation. Every model receives the same versioned prompt (v1), including every file's unified diff with old and new line numbers. Each model gets one sample per PR, at temperature 0 where supported; reasoning snapshots that reject this parameter use the provider default.

A comment must point to the correct file, side and printed line; a correct description on another line is a miss, as in the product. Grading uses a pinned OpenAI model snapshot, versioned here as G1, G2, …; a snapshot change is announced on this page and never mixed inside one run. The grader credits an issue when the comment covers the rubric's core claim, without requiring supporting details. Grading fallback aborts a run rather than inflating recall. Only an off-target comment classified as a concrete false positive counts as a false flag; questions and nitpicks do not.

Thirty stratified PRs, including clean changes, remain in the app library but are excluded from the public per-PR CSV. Holdout results appear only as aggregates. The holdout is carved from an existing public-answer library, so it is weaker than never-published content. The OpenAI grader may favor OpenAI-family reviewers; this v1 benchmark discloses that bias but does not correct it.

Model-call costs use provider-reported cost when available, otherwise recorded prompt and completion tokens with the versioned price table; the separate grader cost is excluded. Parse errors (an empty or unparsable reply) count as approve with no comments; on a clean PR that reads as a correct verdict by accident, so the parse-error column says how many fell on clean PRs. "Clean PRs rejected" counts clean PRs where the model asked for changes. These are one-sample estimates, not confidence intervals.

Models suffixed @claude-code or @codex-cli ran through the vendor's local CLI on a subscription (anthropic/claude-sonnet-5@claude-code, openai/gpt-5.6-sol@codex-cli): the same prompt, one turn, no tools, an empty working directory, the CLI's default sampling. Their cost is the API-equivalent the CLI reports, when it reports one.

Download the CSV (CC BY 4.0): recall per model and category on the public set, aggregated so it cannot serve as an answer key.

Last updated: 2026-09-19.