Skip to content

Deciding If a Node Is Dead

Adoption (in multi-node mode) and leader election (in leader mode) both depend on deciding that a peer node is dead. pg_relay is deliberately conservative about this, because it never assumes a dispatched action is idempotent (safe to run more than once) — a duplicate run could genuinely cause damage, such as charging a customer twice.

  • Liveness is read from the replication connection, not from lag. A peer only counts as live while its replication slot (the PostgreSQL mechanism that tracks what a replica has and has not received yet) is active, as shown in pg_replication_slots. Lag — how far behind a replica is — is directional and ambiguous: a slow-but-alive node and a genuinely dead node can both show growing lag, so lag is never used to decide the question.
  • A connection is not declared dead until wal_receiver_timeout passes (PostgreSQL's default here is 60 seconds). So adoption and leader-takeover latency is set by that timeout, not by the 1-second poll tick — the watcher still runs every second, but it only acts once the connection is definitively gone. This is deliberate: acting sooner risks re-homing a merely-slow node's rows and running them twice. You can tune wal_receiver_timeout to trade off takeover latency against that risk.

The guarantee: every eligible row runs at least once, eventually — or, if it carried an expiry time that lapsed while it was waiting to be adopted, it is expired instead (which is the intended outcome for unimportant or time-sensitive work you would rather drop than run late). Expiry is applied by the new owner's ordinary dispatch process, not by the watcher itself.

Node-restricted rows are never adopted. A row pinned to a specific node (see Node-Restricted Channels) is excluded from both adoption paths — its work is node-local by definition, so moving it to another node would run it against the wrong node's state entirely. A dead node's pinned rows simply wait for that node to come back; see Node-Restricted Channels for the manual clean-up procedure when a node is never coming back.

Continue to Watching Your Cluster for the queries and log messages that tell you what your cluster is doing.