2026-07-29 · 5 min read

AI now writes the tests for its own code, and that changes what green means

There's a solid empirical study on this now: Yoshimoto and colleagues analyzed 2,232 test-related commits across ten TypeScript projects for MSR 2026 and found that AI agents author 16.4% of all commits that add tests, ranging from 1.9% in a large enterprise repo to literally 100% in smaller ones. The part that should make you uncomfortable isn't the quantity, it's how good the tests look by every metric a dashboard can show you: coverage gains comparable to human-written tests, sometimes better, and twice the assertion density, a median of two assertions per test against one for humans. If you were reading these numbers in a quarterly report you'd think the machines were doing a better job than your team, which is exactly the kind of conclusion the numbers are designed to support and exactly the conclusion that falls apart the moment you ask what the assertions are actually asserting.

Why the good numbers are the uncomfortable part

Coverage and assertion counts share a blind spot that matters more now than it ever has: they measure whether code paths get exercised and whether something gets checked, but they tell you absolutely nothing about whether the right thing gets checked. A test that pins down the wrong behavior, that faithfully verifies the code does exactly the wrong thing, scores just as well on every dashboard metric as a test that pins down the spec. And when the same model wrote both the code and its tests, that's not some theoretical concern you can wave away at standup. The tests are the code's self-portrait, painted by the same hand, asserting that the code does what the code does, which is a statement that is always true and never useful.

The field data shows where this leads in practice, and it's not pretty. In Checksum's survey of engineering leaders, 74% rolled back AI-generated code that failed in a way its unit tests didn't catch, which means three quarters of them watched a green CI badge sail through review and then had to revert something by hand after production broke. GitClear's repo-scale data adds the darker sibling to that story: a 47% rise in error masking, code that wraps calls in try-catch blocks that swallow failures so nothing looks broken, which is exactly what a model does when it optimizes for "no test should fail" instead of "every test should catch something." Put those next to the doubled assertion density and you get the honest summary: tests got more numerous, more thorough-looking, and less trustworthy as a signal, all at the same time.

Reviewing tests is now reviewing, full stop

Most reviewers treat the test file as the boring half of the diff, the part you scroll through nodding while your real attention stays on the implementation. That habit made sense in an era when a human colleague wrote the tests as a check on their own understanding of the spec, because then the test was a second opinion by default. The generated test file deserves the opposite treatment, and once you start reading tests as code under review rather than evidence about other code, most of the work comes down to a handful of questions you can ask in under a minute per test.

The first one is whether the assertions come from the requirements or from the code. Read the ticket first, then look at the expected values in the test and ask yourself: could these numbers only have been derived from the spec, or are they just the function's current output, frozen in place? An expected value that's computed by reimporting the same constants and running the same formula as the implementation is a regression detector for behavior nobody ever validated in the first place, and it will faithfully track every bug you introduce because the expected value drifts along with the code.

The second question is what a given test would actually fail on. If you can't name a single realistic bug that would turn it red, you're looking at coverage theater. Tautological tests, the discount test that recomputes the discount using the same formula the function uses, are a recurring AI pattern precisely because they always pass on every input, including the broken ones. The third question is what's been mocked away: a test that mocks the business rule it claims to verify doesn't prove the rule works, it proves the mock works, and generated suites lean on mocks heavily because mocks make tests deterministic, and determinism looks like quality to the model that's optimizing for green. And the fourth is whether the awaits are real, because an assertion inside an unawaited promise never runs, the test passes vacuously, and this pattern ships constantly because it produces a green result on the very first run.

None of this requires heroics or an extra hour in your review queue. It requires treating the test half of the diff as code under review rather than as evidence about the other half, and the moment tests stopped being written by a second mind, they stopped being a second opinion.

It's trainable, like the rest of reviewing

Deceptive tests are one of the nine failure categories on DiffDojo, and the challenges there are exactly the patterns above: tautological suites, mocked-away rules, assertions that never execute, all inside realistic PRs where everything is green and the verdict is yours to defend. Your profile tracks whether test-reading is one of your blind spots, and for most reviewers it quietly is, because nobody ever taught them to read tests as anything other than a green badge next to the PR title. Today's PR is free, no signup.