Intermittent Login Failure — Investigation Handoff

Your team has been chasing an intermittent login failure in a production web service. The on-call engineer spent six hours investigating and has to hand over to the next shift. You need to produce a document that lets the incoming engineer continue without re-treading old ground.

Context gathered so far

Symptoms Roughly 2–3% of login attempts fail with HTTP 503 during peak traffic hours (09:00–11:00 UTC). Users see "Service temporarily unavailable." Failures started appearing four days ago.

Investigation carried out

• Ruled out the database: connection pool metrics are healthy, query latency is normal
• Ruled out the load balancer: health checks pass on all nodes, logs show no dropped connections
• Identified a correlation: failures only occur on requests routed to auth-node-3 and auth-node-4 (out of six nodes)
• Pulled application logs from those nodes: error trace shows Redis connection timeout in the session store client
• Checked Redis cluster: replication lag on redis-replica-2 is elevated (up to 800 ms at peak)
• Attempted a restart of the Redis client on auth-node-3 — failures continued
• Confirmed that the session store client library version on auth-node-3 and auth-node-4 differs from the other nodes (v2.1.4 vs v2.1.2); this discrepancy was introduced by a partial rollout last Tuesday

Decisions made and not yet validated

• The team agreed to test rolling back the session store client to v2.1.2 on auth-node-3 first, but this has NOT been done yet
• The Redis replication lag may be a red herring caused by the elevated error rate rather than the cause — this has not been confirmed either way

Dead end to flag Restarting the Redis client process does NOT clear the issue. The incoming engineer should not waste time on repeated restarts.

Prepare the handoff document for the incoming shift engineer.