2026-09-13 · 4 min read

GitHub let Copilot approve pull requests this month. Its recall on real bugs is 25 to 34 percent.

On 1 September GitHub shipped a changelog entry with a title that would have been a joke two years ago: "Copilot code review can now approve pull requests." Off by default, admin-gated, dismissed on new pushes, counts toward branch protection once enabled. Perfectly reasonable rollout. And the same week I was reading two independent benchmarks that put Copilot's bug recall at 34% and 25%. Both things are true at once, and the gap between them is the job you still have.

What actually shipped

Copilot code review has been growing fast. GitHub's own numbers from March: 60 million reviews since launch, one in five code reviews on GitHub involving it, 12,000 organisations running it automatically on every PR. Over the summer it got the ability to review PRs written by other bots, lost its 300-file size limit, and started running with full shell tools behind a firewall. Then the approval button. The docs still carry the line "Copilot is not guaranteed to spot all problems or issues in a pull request. Sometimes it will make mistakes," which is admirable honesty and also the sort of sentence nobody reads twice.

I don't think the feature is a bad idea. If your team already treats a Copilot pass as a prerequisite, having it show up as a formal approval is just accounting. The problem is what a green check does to the second reviewer, the human one, and for that you need to know what the check is actually worth.

What the check is worth

Two vendor-run benchmarks this year evaluated Copilot alongside their own products, and vendor benchmarks deserve suspicion, but here the vendors have no reason to be kind to Copilot and their numbers agree with each other. Tenki's May round, 122 real bugs across 50 pull requests from cal.com, Sentry, Grafana, Keycloak and Discourse: Copilot found 30 of 122, a recall of 24.6%, at 18.9% precision. Entelligence's September update to the 67-bug set we wrote about in July: 23 of 67, recall 34.3%, precision 16.8%. Both benchmarks used default settings, which is also what most teams use.

Read those precision numbers again, because they matter as much as the recall. Roughly one Copilot comment in six points at a real bug. The other five are noise you have to read and dismiss, which GitHub seems to know, because the same summer of changelogs added a "Won't fix" and an "Incorrect" button for resolving its comments. A tool that ships an "Incorrect" button for its own output is telling you something about its output.

So the honest reading of a Copilot approval on a PR is: it read the diff, it left some comments of which maybe one in six is real, and it did not find anything it considered blocking, which on the two available benchmarks means there is somewhere between a 65% and 75% chance that a bug of the kind that actually ships to production is still in there. That is not a reason to turn it off. It's a reason to not change how you read the diff when you see the check.

The part where the human gets worse

Here's the thing I actually worry about, and it isn't the tool. It's automation complacency, the well-documented pattern from aviation where a system that is right most of the time trains the human to stop checking precisely at the moments where checking would have mattered. A Copilot approval is right often enough to feel like information. The bugs it misses aren't exotic; the benchmark data across every round we've seen says race conditions, authorization slips and logic errors that live in the relationship between two files are exactly the classes that sail past. Those are also the classes a rushed human skips when there's already a green check on the PR, because "the bot looked at it" is a comforting thing to think at 5pm.

GitHub's March post has a number I found more interesting than the 60 million: 29% of Copilot reviews return no comments at all. Nearly a third of the time, the bot is silent. With the recall figures above, silence is not "nothing's wrong," it's "nothing I noticed," and the difference between those two sentences is where incidents come from. We wrote about this in July as silence being a coin flip; with an approval attached, the silence now has a checkmark on it.

What to do with this, practically

Keep the tool on. Treat its comments as leads, its silence as no information, and its approval as a statement that a 25 to 35 percent recall reviewer didn't object. Then do the thing the approval is quietly tempting you not to do, which is read the diff for the classes it can't see: the bound checked against the wrong length, the flag that still has old code listening to it, the export that authenticates the caller and never scopes the query. None of those show up as style issues, none of them break tests, and all of them are the kind of thing a human who has practised on a few dozen of them spots in under a minute.

The skill that matters in a world where Copilot can approve is not "reviewing harder." It's knowing, specifically and by category, what the bot misses and what you miss, and closing the second gap since you can't do much about the first. The bot has a recall number. You should too.