The Database Failover That Didn't Fail Over

This is a story about the day we learned that our "high availability" database setup wasn't. The primary crashed. The replica was supposed to take over. It didn't. We discovered that failover was a theory we'd never actually tested.

Context

Setup:

• Primary PostgreSQL + 2 replicas (sync replication to replica 1, async to replica 2)
• Automatic failover via Patroni/etcd
• Last failover test: 18 months ago (manual, during maintenance)
• Replication lag monitoring: Alert at 10 seconds (we'd never hit it)

Assumptions Made:

• Failover would work when needed
• Replicas were always in sync
• The failover automation was correct (we'd set it up once)

The Incident

Root Cause Analysis

Why failover failed:

1. Replica 1: Under heavy read load, replication had fallen 45 seconds behind. Patroni refused to promote (data loss risk)
2. Replica 2: Async replica—had never been validated for promotion. Script assumed sync replica.
3. Failover script bug: Had a hardcoded assumption about replica roles. Never run in production.
4. No regular testing: Failover hadn't been tested in 18 months. Config had drifted.

Fix & Mitigation

1. Monthly failover drills: Chaos engineer primary kill, validate failover works
2. Replication lag alerting: 1 second lag = warning. 5 seconds = page
3. Failover runbook: Documented manual failover steps. Tested quarterly
4. Replica capacity: Replicas now have headroom for replication during read spikes
5. Automated validation: Health check that verifies failover path weekly

Key Lessons

1. If you haven't tested failover, you don't have failover—you have hope
2. Replication lag is a silent killer—monitor it like your life depends on it
3. Async replicas are for read scaling, not failover—unless you've tested and accepted data loss
4. HA systems rot—config drifts, code changes, assumptions break. Test constantly.