We ran into this around rate limits during a traffic spike. The error rate looked fine in the average and ugly in one slice. Two percent of requests reset upstream with no application exception. Idle timeouts between the proxy and the app had drifted apart. Aligning them stopped the intermittent failures within an hour. We added a panel for reset reasons so the next page is faster to find.
We ran into this around rate limits after we split the monolith. Release notes used to depend on someone remembering to write them. A small bot grouped merged pull requests and asked for missing summaries. It started as a cron job and later listened to webhooks. Product stopped chasing engineers for copy after each weekly release. The rough edges are labeling discipline and weekend merges.
We ran into this around rate limits while rolling back payments. The error rate looked fine in the average and ugly in one slice. Two percent of requests reset upstream with no application exception. Idle timeouts between the proxy and the app had drifted apart. Aligning them stopped the intermittent failures within an hour. We added a panel for reset reasons so the next page is faster to find.
We ran into this around rate limits on a quiet Sunday incident. Release notes used to depend on someone remembering to write them. A small bot grouped merged pull requests and asked for missing summaries. It started as a cron job and later listened to webhooks. Product stopped chasing engineers for copy after each weekly release. The rough edges are labeling discipline and weekend merges.
We ran into this around rate limits after the replica failover. The error rate looked fine in the average and ugly in one slice. Two percent of requests reset upstream with no application exception. Idle timeouts between the proxy and the app had drifted apart. Aligning them stopped the intermittent failures within an hour. We added a panel for reset reasons so the next page is faster to find.
We ran into this around rate limits during a Friday deploy. Release notes used to depend on someone remembering to write them. A small bot grouped merged pull requests and asked for missing summaries. It started as a cron job and later listened to webhooks. Product stopped chasing engineers for copy after each weekly release. The rough edges are labeling discipline and weekend merges.