High Availability and Failover in PostgreSQL

Replication gives you standby copies of your database. High availability is what turns those standbys into a system that survives the primary failing without a human scrambling at 3am. When the primary dies (hardware fault, zone outage, a crash), something has to notice, promote a replica to be the new primary, and redirect the application to it, fast enough that users barely notice. This article covers how that happens and the decisions that make it reliable rather than a new source of outages.

This is part 35 of the PostgreSQL Masterclass, following replication and read replicas.

What failover actually involves

Failover is three steps, and each can go wrong:

1. Detect that the primary is down. Not just slow or briefly unreachable, actually down, because promoting a replica while the old primary is still alive is a disaster (more on that below).
2. Promote a replica to become the new primary. A promoted replica stops replaying WAL and starts accepting writes.
3. Redirect the application's writes to the new primary, so traffic flows to the right place.

Doing all three manually during an incident is slow and error-prone, so high availability means automating them with tooling designed for it.

The tools that automate it

You don't build failover from scratch. Several battle-tested systems handle detection, promotion, and often traffic redirection:

• Patroni is the most popular open-source choice. It uses a distributed consensus store (etcd, Consul, or ZooKeeper) to agree on who the leader is, promotes a replica automatically when the primary fails, and prevents split-brain. It's the common answer for self-managed HA.
• repmgr is a lighter-weight tool for managing replication and failover, simpler than Patroni but with fewer guarantees.
• Managed services (AWS RDS Multi-AZ, Aurora, Cloud SQL HA) do all of this for you. You enable Multi-AZ and the provider handles detection, failover, and DNS redirection, typically failing over in a minute or two.

For most teams, the choice is "managed service" or "Patroni." Rolling your own failover logic is a way to turn a database outage into a worse, longer outage.

Split-brain: the failure you must prevent

The dangerous failure mode in any HA setup is split-brain: two servers both believing they're the primary and both accepting writes. This happens when a replica gets promoted because it can't reach the primary, but the primary is actually still alive and serving other clients (a network partition, not a real death). Now you have two diverging databases, and merging them afterward means data loss and manual reconciliation.

Preventing split-brain is the main job of serious HA tooling. The mechanisms:

• Consensus/quorum: a majority of nodes must agree who the leader is, so a partitioned minority can't promote itself. This is why Patroni needs an odd number of consensus nodes (3 or 5).
• Fencing (STONITH): forcibly isolating or shutting down the old primary before promoting a new one, so it can't accept writes.

If you take one thing from this article: never build failover that promotes a replica based only on "I can't reach the primary." That's exactly how split-brain happens. Promotion must be coordinated through a quorum that a lone partitioned node can't satisfy.

Redirecting the application

After promotion, the application has to find the new primary. A few approaches:

• A virtual IP or DNS record that points at the current primary, updated on failover. Simple, but DNS caching can slow the switch.
• A proxy or pooler (like the one from the connection-pooling article, or HAProxy) that always routes to whoever is currently primary. The app connects to a stable endpoint and the proxy tracks the leader.
• Managed endpoints: RDS and Aurora provide a primary endpoint that automatically points at the current writer after failover, so the app's connection string never changes.

The application also needs to handle the brief window during failover gracefully: connections drop, writes fail for a few seconds, and the app should retry rather than surface errors to users. Failover is never truly instant, so retry-on-connection-failure logic is part of an HA-ready application.

Measuring HA: RTO and RPO

Two numbers define your high-availability goals, and they drive the design:

• RTO (Recovery Time Objective): how long you can be down. Automated failover targets an RTO of seconds to a couple of minutes. Manual failover is tens of minutes.
• RPO (Recovery Point Objective): how much data you can afford to lose. With asynchronous replication, RPO is the small window of un-replicated commits (usually sub-second of data). With synchronous replication, RPO is zero, no committed data lost, at the write-latency cost from the last article.

Deciding your RTO and RPO tells you what you need: synchronous replication if RPO must be zero, automated tooling if RTO must be small, and multi-zone or multi-region placement if you need to survive a whole data center failing.

High availability is the discipline of surviving the primary's death automatically and safely. Use proven tooling (a managed service or Patroni) rather than hand-rolled promotion, make preventing split-brain the top priority through quorum and fencing, route the app through a stable endpoint that follows the leader, and build retry logic for the failover window. Set your RTO and RPO consciously, because they decide how much machinery and cost the target requires.

Next, we cover the other half of resilience: backups and point-in-time recovery, because replication protects against a server dying, not against a bad DELETE or a corrupted table.