Skip to content
Build in Public

The bug that only happens in production: vanishing OTPs

Remember the smug version of me from a few days ago? The one who stored OTP codes in a quick in-memory list because it was fast and worked perfectly on his…

SD
Shubham Datarkar
· 3 min read
Updated

Remember the smug version of me from a few days ago? The one who stored OTP codes in a quick in-memory list because it was fast and worked perfectly on his machine? Today that decision found me, and it found me in front of an audience.

Here's what happened. I deployed the booking flow to a real server for the first time. Pulled out my phone, played the part of a real guest, picked a slot, entered my email. The code arrived in my inbox — beautiful, six digits, right on time. I typed it in, proud, ready to watch the magic happen.

"No OTP found."

I typed it again. Same error. The code was right there in my email. The system had just sent it to me. And the system was now insisting, to my face, that it had never heard of it.

I went through every stage of grief. Maybe a typo. Maybe a caching thing. Maybe the email was delayed and stale. I requested a fresh code, got it instantly, typed it in within five seconds — "No OTP found." Now I'm genuinely unsettled, because this is the kind of bug that makes you question whether you understand your own software at all. It worked flawlessly on my laptop thousands of times. On the server, it failed every single time, instantly, with a clean error message that made no sense.

The thing about a bug that only happens in production is that it's invisible where you do your testing. On my laptop, the dev server runs and stays running. The little in-memory list of codes just sits there, alive, the whole time I'm working. So the code I store is still there ten seconds later when I verify it. Perfect. Flawless. A lie.

On a real server, things restart. They restart on deploy. They restart on their own. They restart between the moment a guest requests a code and the moment they type it back. And the instant that process restarts, my in-memory list — the entire record of every code I'd sent — vanishes. Wiped clean. So the guest types a perfectly valid code, and the server, freshly restarted with an empty memory,

genuinely has no idea what they're talking about. "No OTP found." Because there is, now, no OTP. There's no OTP anywhere.

It's the kind of mistake that's obvious the second you see it and invisible until you do. Memory is temporary. I stored something that needs to survive in a place designed to forget. On localhost it never forgot, so I never learned.

The fix is the thing I should have done from the start, the thing I told myself was "overkill for just an OTP": store the codes in the database. A real table, on disk, that doesn't evaporate when the server blinks. I moved OTP storage out of memory and into Supabase, keyed by email so that requesting a new code cleanly replaces the old one. Now a code lives somewhere permanent. The server can restart fifty times between request and verify, and the code is still sitting there waiting.

This is decision reversed. I had a fast, clever, wrong solution, and I had to publicly walk it back to the boring, correct one. And here's the part that stings: the boring correct version took me about an hour to write. The lesson took me a day of confusion to earn. That's the real cost of these bugs — not the fix, the fog you have to wander through before you can see the fix.

I keep a running note of things I'll never do again. Today's entry is one line: Anything that must survive a restart does not live in memory. It's such a basic sentence. I'd have nodded along if you'd told me a week ago. But there's knowing it and there's losing a day to it, and only one of those actually sticks.

The booking flow works now. For real this time — on the actual server, surviving actual restarts. A small, hard-won, permanent kind of working.

Tomorrow: the day I tried to be clever about security and Supabase's Row Level Security politely broke my brain.

by Shubham DatarkarBuild in Public

Reactions

How was this article?

Newsletter

Never miss the next one

Get the latest playbooks, build logs, and the occasional unpublished idea straight to your inbox. One signal a week — no noise.