2026-09-16 · 6 min read

I ran Claude Code as a reviewer on six planted-bug PRs. It caught all five bugs, and got one of them wrong.

The fastest-rising search phrase in our corner of the internet is "code review skills," and it doesn't mean what it meant a year ago. Nobody's looking for how to get better at reviewing. They're looking for a SKILL.md file that makes Claude Code do it for them. The superpowers repo is past 280,000 stars, Addy Osmani's skill pack is near 95,000, and Claude Code ships a built-in /code-review. So I did the obvious experiment: I pointed Claude Code at six pull requests from our library, five with a planted bug and one clean, and asked it to review them the way a senior engineer would. It caught all five. Then I read the reviews properly, and one of them is the most interesting thing I've seen a model do in a while, because it found the bug and got the consequence wrong.

The setup, so you can repeat it

Each challenge became a tiny git repo: base commit with the pre-PR state, feature branch with the PR, a PR.md holding the requirements exactly as our users see them. Then, from inside each repo, one non-interactive call to Claude Code on Sonnet with read-only tools and a plain prompt: review the branch against main, list every defect you'd block on with file and line, then give a verdict. No custom skill, no repo instructions, default effort. I wanted the floor, not the ceiling. The six PRs were the ones we modelled on CrowdStrike (a bounds check against the wrong length), Knight Capital (a retired feature flag reused), the Debian OpenSSL entropy removal, a pagination endpoint missing a field the spec asked for, a comment list sorted by the wrong key, and a clean React search component with correct abort handling.

What it caught

Everything, honestly, and with better write-ups than most humans leave. On the CrowdStrike PR it went straight to the line and said what I'd want a reviewer to say: rules.length and INPUT_REGISTER_COUNT "are unrelated quantities (number of rules vs. size of the fixed input array), so the check is wrong in both directions depending on channel size." That "both directions" is a detail our own canonical answer doesn't spell out; it's correct and it's sharper. On the entropy PR, where a cleanup commit removes a randomBytes(32) call the description calls a redundant allocation, it called the removed buffer "load-bearing" and "the opposite of redundant," which is the whole review in five words. On the sort-key PR it noticed not just that the comparator used the author instead of the timestamp but that the timestamp was never referenced anywhere in the file, which is a nice piece of evidence.

On these diffs, twenty to forty lines, a single file, a clear spec, the model is a very good reviewer. I'd take its comments over a rushed human's. That's the honest headline and it would be silly to bury it.

Where it got it wrong while getting it right

The Knight Capital PR gates a new priority-tagging feature behind an existing config key, priorityDispatch, which the requirements say is retired and must stay off. The catch is that a legacy block earlier in the same file still checks that key, and in the dispatch loop the legacy check runs first and ends with continue. Claude found the reuse immediately and blocked the merge. Then it explained the consequence:

As deployed, the flag stays off forever, so the new priority-tagging
feature can never be enabled through its intended path. It's permanently
dead code.

If priorityDispatch were ever accidentally turned on, it would silently
activate the *new* feature too, an unintended coupling of two unrelated
behaviors under one flag name.

Both of those are wrong, and wrong in the direction that matters. Flipping the flag doesn't activate the new feature "too." It activates the legacy dispatcher, which hits continue before the new code is ever reached, and the legacy dispatcher has no ledger, so it resubmits every order to the live carrier on every tick, including the ones already shipped. The feature isn't dead code; the flag is a live wire to a retired routine, which is exactly what happened on Knight's eighth server. Claude identified the right line, blocked on it, recommended the right fix (a fresh key), and described the blast radius as "the feature won't work" when the actual blast radius is "the carrier gets flooded with duplicate shipments, forever."

Would that matter in practice? The PR gets blocked either way. But think about what happens next in a real team: the author reads "dead code, rename the flag," shrugs, renames the flag, and nobody learns that the legacy block is armed. The severity of a review comment decides whether someone opens a ticket to delete the dead path this sprint or in a year. And the model's reasoning error was specifically a failure to trace control flow across two places in the same file, the continue fifteen lines away from the diff, which is the same weakness every benchmark this year points at: single-line findings are great, relationships between lines are not.

And the clean PR

It requested changes on the clean one. Partly my fault: the component imports a search API module that the challenge, like a real PR, doesn't include, and Claude flagged the missing file as a broken import. Fair. What I found more telling is that it then walked the six requirements one by one, debounce timing, stale-response guard, preserving previous results, ARIA roles, stable keys, and confirmed each was correctly implemented, before blocking on the import anyway. It could not bring itself to say "approve" with an unresolved question on the table. That's a defensible default for a bot. It's also the thing that makes teams stop reading bot reviews, and it's the reason our grader scores a false flag on a clean PR as a miss: knowing when to approve is half the skill.

What the experiment is actually evidence for

Not "the bot is bad." On this size of diff it clearly isn't. The evidence is about which half of the job it does. It's excellent at the part you can do with a diff and a spec in one window: is this line consistent with that requirement. It's noticeably weaker at the part that needs you to leave the diff, trace what a value does two functions away, and come back with a severity. And it has a strong bias toward blocking when uncertain, which pushes the calibration problem, when is a PR actually fine, back onto the human every time.

Those two gaps, cross-file consequence and calibrated approval, are the review skills that still belong to you, and the uncomfortable part is that they're also the ones you can't tell you've lost. A skill file gives Claude a checklist. Nothing gives you a recall number. If you want one, the six PRs above are live in the library; the model's answers are in this post, so you're playing against a known score. My bet is you beat it on the clean one and lose to it on the entropy one. Find out.