2026-08-17 · 4 min read

Half of Bluesky went down because one endpoint skipped one line. Would you have caught it in review?

In April 2026 Bluesky had its worst outage in recent memory: roughly half of all users lost service for about eight hours. The public postmortem by systems engineer Jim Calabro is unusually honest and worth reading in full. The short version fits in one paragraph.

What happened

A newly deployed internal service called the GetPostRecord endpoint. Not often, under three requests per second. But each request could carry a batch of 15 to 20 thousand post URIs. The handler fanned out one goroutine per URI with no concurrency limit. Every other endpoint in the system bounded its concurrency with errgroup.SetLimit. This one, and only this one, was missing the line.

What followed is a textbook death spiral: tens of thousands of simultaneous lookups blew past the memcached pool's 1000-connection cap, closed connections piled up in TCP TIME_WAIT until the machines ran out of ephemeral ports, error logging went vertical, the Go runtime spawned roughly ten times its usual thread count, garbage collection stalled, containers ran out of memory, and restarts couldn't reconnect because the ports were still exhausted. Eight hours.

The fix was group.SetLimit(50). One line.

The uncomfortable part

This bug passed code review. Of course it did: the diff looks great. Concurrent fan-out over a batch is idiomatic, the code is clean, and on any reasonable test input, a handful of URIs, it works instantly. Nothing about the changed lines says "this needs a number in it". The information that makes it a ship-blocker lives outside the diff: how big can the batch get, and what sits behind each call.

That is exactly the kind of reviewing that gets harder in the AI era, because generated code produces idiomatic, clean, plausible fan-out all day long. The reviewer is the only one in the loop positioned to ask: what bounds this?

Try it yourself

I rebuilt the shape of this bug in Python (asyncio instead of goroutines, same mechanism, same missing line) as a review challenge on DiffDojo, a trainer where you review realistic PRs, leave comments, give a verdict, and learn whether you caught the planted bug. See if you would have flagged it without knowing what you know now: the batch fetch challenge.

And when you review your next PR with a gather, a Promise.all or an errgroup, ask the Bluesky question: what is the largest input this can receive, and what happens downstream when it arrives all at once?

Read next: How to review AI-generated code: a guide for the human in the loop