Lifecycle Notifications¶

Every notification pg_relay can send — email, Slack, SMS, a PagerDuty page — travels through the database: a queue row, a claim, an audit record. Which leaves exactly one failure the system can never report about itself: the database being unreachable. And a Processor that is starting up or deliberately exiting has lifecycle news of its own that should not depend on a queue event. Lifecycle notifications close both gaps: contracts held in environment variables, fired directly by the binary — no queue, no notifier SQL — each one free to use its own medium.

Three environment variables¶

The names are fixed. Each variable is independent: unset (or blank) means that notification is off; any combination may be configured, each to a different medium.

Variable	When it fires
`PG_RELAY_OUTAGE_NOTIFY`	The dead-man's switch: `outage_message` after the database has been unreachable past a threshold, once per outage episode; `recovery_message` (optional) when it comes back.
`PG_RELAY_START_NOTIFY`	Its `message`, once, right after the `pg_relay started` log line — connected, preflight passed.
`PG_RELAY_STOP_NOTIFY`	Its `message`, on any deliberate exit: clean `SIGTERM`/`SIGINT` shutdown and fatal error exits (a preflight failure, a failed reload) alike. A `kill -9` or crash can never send anything — inherent, not a limitation of this feature.

The contract¶

Every variable holds one JSON object of the same shape:

{
  "transport": "webhook",          // default; or "smtp" | "m365" | "acs" | "gmail"
  "profile":   { ... },            // that transport's profile — the same schema
                                   // a pg_relay_notifier profile uses
  "message":          { ... },     // START / STOP contracts
  "outage_message":   { ... },     // OUTAGE contract (required)
  "recovery_message": { ... }      // OUTAGE contract (optional)
}

transport picks the medium, profile configures it, the message blocks say what to send. For the email transports (smtp, m365, acs, gmail), both profile and the messages use exactly the schemas documented in the notifications book. The chapter you followed to set up a notifier profile for a provider doubles as the reference here, and the send goes through the very same transport code.

For webhook, the profile carries url, the auth block (bearer_header / custom_header / basic_auth), optional headers, and optional body_merge (below). The message is POSTed as JSON, and success is hard-coded as any 2xx — there is no classify_webhook_response to consult, because the database that holds it is precisely what may be down. Redirects are never followed. profile.timeout_seconds (default 30, capped at 120) bounds each send.

A worked outage contract, PagerDuty via webhook. PagerDuty authenticates with a routing_key inside the request body, so the contract uses the webhook profile's body_merge — the key is declared once in the profile, where _env: resolution applies, and injected into both messages at send time:

export PAGER_DUTY_ROUTING_KEY=R0XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

PG_RELAY_OUTAGE_NOTIFY='{
  "profile": {
    "url": "https://events.pagerduty.com/v2/enqueue",
    "auth": {"style": "bearer_header", "secret": "unused"},
    "body_merge": {"routing_key": "_env:PAGER_DUTY_ROUTING_KEY"}
  },
  "outage_message": {
    "event_action": "trigger",
    "dedup_key": "pgrelay-db-outage",
    "payload": {"summary": "pg_relay cannot reach the database",
                "source": "relay_dev", "severity": "critical"}
  },
  "recovery_message": {
    "event_action": "resolve",
    "dedup_key": "pgrelay-db-outage"
  }
}'

And a start announcement through a local SMTP relay — worth singling out, because with auth: "none" on localhost it is the one medium with no external dependency at all, which is exactly the property you want in the alerting path of a dead-man's switch:

PG_RELAY_START_NOTIFY='{
  "transport": "smtp",
  "profile": {"host": "localhost", "port": 25, "security": "none",
              "auth": "none", "from": "[email protected]"},
  "message": {"to": ["[email protected]"],
              "subject": "pg_relay started on db01",
              "body_text": "The Processor connected and passed preflight."}
}'

Secrets: the profile resolves `_env:`, the messages never do¶

The contract follows the notifier's own rule. Any string in profile written as _env:VAR_NAME is resolved from the Processor's environment (Keeping Secrets Out of the Database). A credential therefore lives in its own environment variable — typically the very one your notifier profile for the same provider already references — and rotating it never means editing the contract. (A Processor restart is still required: a process's environment is fixed when it starts.) The message blocks are sent as written — an _env: string inside one stays literal.

A handful of APIs authenticate inside the request body — PagerDuty's routing_key, as in the example above. For those, the webhook profile's body_merge bridges the gap exactly as it does on the notifier's webhook transport. Its top-level keys are overlaid onto the message immediately before sending, unconditionally overwriting any same-named key, and because the block lives in the profile, its _env: references resolve. Merge values go verbatim ({{variable}} templates are a message-block feature and never touch them). When body_merge is configured, every message block in the contract must be a JSON object for the overlay to land on.

An _env: reference to an unset variable is part of contract validation: it refuses startup, never surfacing mid-outage.

Validation is eager everywhere it can be. Every one of these refuses startup with notify_config_invalid in the log: malformed JSON; an unknown transport; a missing profile; a webhook contract without url, with a bad auth.style, with a non-object body_merge, or with a non-object message while body_merge is configured; an email message without recipients or a body. Broken alerting must fail while a human is watching, not stay silent until the 3 a.m. outage it was meant to catch.

Message variables¶

Message strings may embed {{variable}} tokens — {{host}}, {{time}}, {{outage_duration}}, and more — expanded at send time from facts the Processor knows without asking the database. They have their own chapter: Message Variables.

The outage thresholds¶

Situation	Flag	Default
The Processor has never connected (startup)	`--startup_timeout`	120 s
An established connection is lost mid-run	`--stop_timeout`	10 s

The two exist because the situations differ. At boot — a patched server restarting, say — PostgreSQL legitimately takes a while to come up, and the DBA wants to hear about it only if it doesn't: 120 seconds of grace. Mid-run, a connection that has been up for weeks going dark for even 10 seconds is news. Both measure wall-clock time from the first failed attempt; both are per-instance; zero means "fire on the first failure". A Processor whose database is unreachable at startup does not exit — it waits in-process with the normal reconnection backoff, so the timer can run and the alert can fire from a process that remembers having fired it. (A startup failure with the database reachable — a preflight error, a registry that predates the binary — still exits non-zero: waiting cannot fix a missing grant.)

Alerts fire on the first backoff iteration after the threshold, so they can lag it by up to the current backoff interval (the backoff caps at 30 s); recovery likewise waits for an actual reconnect.

The outage episode lifecycle¶

An episode is one continuous outage, and it drives a simple latch:

The database becomes unreachable; the reconnect loop starts; the clock starts.
At the threshold, the Processor sends outage_message. If delivery itself fails, it is retried on each backoff iteration — a coincidental blip at the alerting endpoint must not mean a silent outage — but once one delivery lands, the outage message never fires again for this episode. Once per outage, as many attempts as it takes to deliver it once.
The database comes back. If and only if the outage message was delivered, the Processor sends recovery_message. A failed recovery delivery keeps the episode open and retries (throttled to every 30 seconds) from the poll loop until it lands.
Delivered recovery resets the latch. The next outage is a fresh episode and alerts anew.

Two honest limitations, both accepted by design. The latch lives in memory: a Processor manually restarted mid-outage fires again after its threshold (one duplicate), and one restarted between outage and recovery loses the pending resolve. And multiple instances each fire their own messages — they cannot coordinate, because the coordination point is the database that is down. Both are absorbed neatly by alerting platforms with a client dedup key: give every instance the same static dedup_key (as in the example above) and PagerDuty collapses all of it into one incident with a correct lifecycle. See the Incident Alerting Platforms chapter for which platforms offer this.

Email gets the same treatment automatically. The Message-ID of an outage email is seeded deterministically from the episode's start time, so every delivery retry of one episode's message carries the identical id and a deduping receiver collapses them. The recovery email pairs with its episode under its own id, and start/stop messages have their own. This mirrors the notifier's own pk-seeded Message-ID rule.

Start and stop semantics¶

The start message is one attempt, best-effort, sent synchronously right after pg_relay started (so it is bounded by the profile timeout): a failure logs start_notify_failed and the Processor runs on — an announcement must never block event processing. It fires once per process: reloads and reconnects do not re-announce (an outage recovery is the outage contract's job).

The stop message fires on every deliberate exit path — the SIGTERM/SIGINT shutdown, a preflight or registry failure at startup, a failed reload — as one bounded attempt on the way out (stop_notify_sent / stop_notify_failed). It does not fire when the contracts themselves failed to parse (there is nothing trustworthy to send), and it cannot fire on a crash or kill -9. The database being down does not stop it: the stop contract's medium is out-of-band by construction.

Testing a contract without a database¶

$ pg_relay --test-notify PG_RELAY_OUTAGE_NOTIFY
sending outage_message from PG_RELAY_OUTAGE_NOTIFY via webhook
http_status: 202
response: {"status":"success", ...}
delivered

--test-notify VAR parses the named variable with exactly the daemon's validation and sends through the same delivery path — any variable name works, so you can stage a contract in a scratch variable before promoting it. For an outage contract, --recovery sends recovery_message instead; for a start/stop contract the single message is sent. Email transports report the provider's reference instead of an HTTP status. The exit code is 0 only on a delivered send, so it doubles as a config linter in scripts.

An administrator's connection test: `--test-notify`¶

--test-notify PG_RELAY_STOP_NOTIFY (or any named variable) sends a production contract exactly as configured — the right final check, but its text is the real page. To prove a new connection delivers with a harmless message instead, an administrator composes a fourth variable, PG_RELAY_TEST_NOTIFY, which is the default when --test-notify is run with no name. It is a complete, self-contained contract: transport, profile, and a single message carrying every key the transport requires ({{variables}} and to_char formats welcome; {{event}} renders test). Then run:

$ pg_relay --test-notify
sending message from PG_RELAY_TEST_NOTIFY via smtp
response: <pgrelay-notify-…@localhost>
delivered

PG_RELAY_TEST_NOTIFY='{
  "transport": "smtp",
  "profile": {"host": "localhost", "port": 25, "security": "none",
              "auth": "none", "from": "[email protected]"},
  "message": {"to": ["[email protected]"],
              "subject": "Test from {{host}} - please ignore",
              "body_text": "Connectivity test at {{time_local:YYYY-MM-DD HH24:MI:SS}}, pid {{pid}}."}
}'

The daemon never reads PG_RELAY_TEST_NOTIFY — a broken or half-finished test contract can never refuse a Processor's startup or leak into production sends. Validation is identical to the daemon's (an outage-family test contract works too — its outage_message is sent, --recovery selecting the other), and the exit code is 0 only on a delivered send. Once the connection is proven, copy the working transport/profile into the real lifecycle variable.

What never happens on this path¶

No queue row, no audit row, nothing in pgrelay.log — this path must work with the database gone. Message content and secrets never appear in the Processor's own log lines, and transport errors are stripped of the request URL before logging (a URL-keyed webhook endpoint's URL is a credential). The registry email transports run with no database connection at all — their OAuth token endpoints are external HTTPS — which is also the honest caveat: m365, acs, gmail, and SMTP's oauth2 mode need their token endpoint reachable during the outage. A database-down incident leaves them fine; a total network outage takes any alerting path down. The local SMTP relay is the only option immune even to that.