I set out to make the Hot tab on /community actually feel hot. What I got instead was a lesson in how a test suite can be 682 for 682 green and still be lying to your face.
First discovery, and it stung: Hot had never really worked. It was sorting by a formula that mixed a post's votes with its age. But I removed downvotes months ago, so almost every post sits at a score of zero, which meant the vote half of the formula was always zero too. Strip that out and what is left is just a function of time. Hot had been quietly ranking posts by pure timestamp for months. Nobody noticed. I only noticed because I went to improve it.
So I rebuilt it as a seeded shuffle. Order by a hash of the post id plus a seed in the URL, bump anything under two days old toward the front. Deterministic per seed, so paging cannot duplicate or skip a card, and a fresh seed each visit re-orders the archive. Clean. I wrote a test proving it shuffles. It passed. Everything passed.
Then the tab went empty for everyone
Not for me on my laptop. For everyone. And not with a red 500 error that would have paged me -- the feed query just returned an empty list and the tab rendered blank, like the community had gone silent overnight.
The cause is the kind of thing you only find by looking. That hash function returns a Postgres int4 -- a 32-bit integer. To bump recent posts I was subtracting 1.5 billion from the hash. Any post whose hash landed low enough, and with a few hundred posts one always does, pushed the subtraction off the bottom of the type. Postgres does not shrug that off. It raises integer out of range and kills the entire query, not just that one row. My feed code caught the error, logged it, and returned an empty list. Empty tab, for every single user.
Here is the part that keeps me humble. A unit test could not have caught this. JavaScript numbers do not overflow at two billion -- they just keep going. Postgres integers very much do. My test ran the shuffle logic in JavaScript, saw sensible output, and went green. The bug only exists in the database, and only a probe against the real database surfaces it. Which is exactly what my own spec told me to do before merging. I probed after.
The test passed because JavaScript numbers do not overflow and Postgres integers very much do. The tab was empty for every user and my suite was 682 for 682.
— the commit message, written with gritted teeth
The fix is one word: cast the hash to bigint before subtracting. Ordering stays identical, no other change. Ten seconds to type, a whole afternoon to understand.
What I am taking from it: a test that runs in the wrong place is not a safety net, it is a comfort blanket. If the real behaviour lives in Postgres, the test has to touch Postgres. Green is not the same as correct. I knew that yesterday too, in the abstract. Today I know it in my gut.