2026-08-17 · 4 min read

Half of Bluesky went down because one endpoint skipped one line. Would you have caught it in review?

In April 2026 Bluesky had its worst outage in recent memory, roughly half of all users lost service for about eight hours, and the public postmortem by systems engineer Jim Calabro is one of those rare write-ups where the engineering team is genuinely, uncomfortably honest about what went wrong and why nobody caught it. The short version of the whole thing fits in one paragraph, which is part of what makes it so painful.

What happened

A newly deployed internal service started calling the GetPostRecord endpoint, and it didn't call it often, under three requests per second, which is the kind of traffic that doesn't even register on a dashboard. But each of those requests could carry a batch of 15 to 20 thousand post URIs, and the handler fanned out one goroutine per URI with no concurrency limit at all. Every other endpoint in the entire system bounded its concurrency with errgroup.SetLimit, because somebody at some point had thought about what happens when you open ten thousand connections at once. This one endpoint, and only this one, was missing the line.

What followed was a textbook death spiral, the kind you draw on whiteboards during incident reviews and hope you never see in production: tens of thousands of simultaneous lookups blew past the memcached pool's 1000-connection cap, closed connections piled up in TCP TIME_WAIT until the machines ran out of ephemeral ports entirely, error logging went vertical and started consuming its own resources, the Go runtime spawned roughly ten times its usual thread count trying to keep up, garbage collection stalled because there was nothing left to collect with, containers ran out of memory one by one, and restarts couldn't reconnect because the ports were still exhausted from the previous round of failures. The whole thing lasted eight hours, and the fix was group.SetLimit(50), a single line that every other endpoint already had.

The uncomfortable part

This bug passed code review, and of course it did, because the diff looks great. Concurrent fan-out over a batch is idiomatic Go, the code is clean and well-structured, and on any reasonable test input, say a handful of URIs instead of twenty thousand, it works instantly and correctly. Nothing about the changed lines says "this needs a number in it," because the information that makes this a ship-blocker doesn't live in the diff at all. It lives in the answer to two questions that the diff never asks: how big can the batch actually get in production, and what sits behind each of those concurrent calls?

That's exactly the kind of reviewing that gets harder every quarter in the AI era, because generated code produces idiomatic, clean, plausible fan-out all day long, and it never pauses to wonder whether the thing it's fanning out across can handle the load. The reviewer is the only person in the entire pipeline who's positioned to ask: what bounds this?

Try it yourself

I rebuilt the shape of this bug in Python, asyncio instead of goroutines but the same mechanism and the same missing line, as a review challenge on DiffDojo, where you review realistic PRs, leave comments, give a verdict, and find out whether you caught the planted bug. See if you would have flagged it without knowing what you know now: the batch fetch challenge.

And the next time you review a PR that uses asyncio.gather, a Promise.all, or an errgroup, ask the Bluesky question before you hit approve: what is the largest input this can receive, and what happens downstream when it all arrives at once?