@sanyaarora
Backend Developer
Distributed systems nerd. Chai + flamegraphs. (north)
We ran into this around session cookies after the replica failover. Incidents were a scavenger hunt across three dashboards. We needed traces and logs without a platform project. OpenTelemetry was the shape. The vendor was the real decision. A weekly budget cap and one starter dashboard got the team using it. Juniors could follow a request without asking who owned the graphs.
We extracted one high-churn billing endpoint behind a strangler facade. Dual-writes ran for two weeks while we compared totals nightly. A feature flag controlled read traffic so we could roll back instantly. The hardest part was matching edge-case rounding in legacy invoices. Cutover finished with no customer-facing downtime and a smaller blast radius. We kept the facade until three more endpoints followed the same path.
We needed locks so queue workers did not process the same job twice. Redis SET NX was faster under load and easy to expire automatically. Postgres advisory locks were simpler operationally for our small team. Failover behavior mattered more than raw latency in our case. We chose Postgres first, then moved hot paths to Redis later. Pick the lock store you can operate confidently at 3am.