Incident Alerting Platforms¶

Paging the on-call engineer is the natural last mile for a database event queue, and pg_relay reaches these platforms the same way it reaches Slack or an SMS gateway: through the webhook transport, with pg_relay_notifier building the platform-correct message and a classify_webhook_response branch reading the reply — zero Processor code per platform. This category fits even better than SMS: alerting platforms are built to ingest machine-generated events over plain HTTP, and every one of them speaks JSON, so the default body_style covers the whole table.

Platform	Supported	`auth.style`	Send endpoint	Success reply → provider ref
PagerDuty	Yes	placeholder (routing key via `body_merge`)	`POST https://events.pagerduty.com/v2/enqueue`	`202` → `dedup_key`
Jira Service Management	Yes	`custom_header` (`GenieKey …`)	`POST https://api.atlassian.com/jsm/ops/integration/v2/alerts`	`202` → `requestId`
Grafana Cloud IRM	Yes	placeholder (key in URL)	per-integration endpoint URL	`2xx`
Better Stack	Yes	`bearer_header`	Incidents API (`uptime.betterstack.com`)	`201` → incident id
incident.io	Yes	`bearer_header`	`POST https://api.incident.io/v2/alert_events/http/{source_id}`	`202` → deduplication key
Rootly	Yes	`bearer_header`	`POST https://api.rootly.com/v1/incidents`	`201` → `data.id`
FireHydrant	Yes	`bearer_header`	Signals ingestion / `POST https://api.firehydrant.io/v1/incidents`	`2xx` → id
Splunk On-Call	Yes	placeholder (key in URL)	`…alert.victorops.com/integrations/generic/20131114/alert/{key}/{routing}`	`200`
OnPage	No	—	—	—
xMatters	Yes	`basic_auth`	`POST https://{company}.xmatters.com/api/xm/1/events`	`202` → `requestId`
Squadcast	Yes	placeholder (key in URL)	per-service Incident Webhook URL	`2xx`
Runframe	Yes	placeholder / `bearer_header`	`POST https://{host}/api/webhooks/{routingKey}` or REST API	`2xx`

Eleven of the twelve work today. Every platform name links to a section below — either its own, or the group whose pattern it shares.

Rules that apply to every platform¶

Dedup keys turn at-least-once into exactly-one-alert. This is the headline of the whole category. pg_relay's delivery guarantee is at-least-once: a crash between a successful send and pg_relay's own commit means the event is sent again. Email and SMS can only mitigate that (the SMTP transport's deterministic Message-ID shifts dedup to the receiver). Alerting ingestion APIs are better: they are designed idempotent. PagerDuty's dedup_key, Splunk On-Call's entity_id, incident.io's deduplication_key — have pg_relay_notifier derive that field deterministically from its own primary key (e.g. pgrelay-{pk}), and a retried send updates the alert it already created instead of paging the engineer twice. Always set the dedup field; never generate it randomly.

The Processor can also page when the database itself is down. Everything in this chapter flows through the queue — which makes the queue's own database the one thing these channels can never alert on. That gap is covered by the Processor's out-of-band lifecycle notifications: a contract held in the PG_RELAY_OUTAGE_NOTIFY environment variable — webhook, SMTP, M365, ACS, or Gmail — fired directly by the binary when the database has been unreachable too long, with PagerDuty's trigger/resolve + dedup_key as its natural webhook target. The lifecycle contract's webhook profile honours body_merge too, so the routing-key-from-environment pattern below carries over unchanged.

A 2xx really is the finish line here. With email and SMS, "accepted" was a weaker promise than "delivered". With an alerting platform, acceptance is the hand-off: escalation, retries, and reaching a human are the platform's whole job from that moment. What pg_relay never sees is the acknowledge/resolve flow — the platforms report those through outbound webhooks of their own, and pg_relay deliberately has no inbound HTTP listener. Closing an alert from the database side still works, because it is just another outbound post: a second channel sending event_action: "resolve" (PagerDuty) or message_type: "RECOVERY" (Splunk On-Call) with the same dedup key.

Classification is uniform. A 2xx is sent (with the id or dedup key from the body as the provider reference), 429 / any 5xx / http_status = 0 are retry, any other 4xx is failed. Unlike SMS's Infobip, no platform in this table hides a failure inside a 200 body.

URL-keyed endpoints follow the established secret rules. Several platforms authenticate by a key embedded in the endpoint URL. _env: references replace whole string values — there is no partial substitution inside a longer string — so the entire URL goes into the environment variable, exactly as the Mattermost and Microsoft Teams recipes do, with the usual placeholder auth block.

Documentation-verified, not live-verified

Endpoints, auth mechanics, and response shapes in this chapter come from each platform's current documentation, not from live sends through pg_relay. Trigger one real (test-service!) alert per platform before relying on a profile in production.

The clean fits: Better Stack, incident.io, FireHydrant¶

A Bearer token, a JSON body, an id in the reply — nothing beyond the table row. incident.io as the worked example, using its HTTP alert source (created under Alerts → Sources; the source's config id lands in the URL):

// profile
{
  "url": "https://api.incident.io/v2/alert_events/http/01HXXXXXXXXXXXXXXXXXXXXXXX",
  "auth": {"style": "bearer_header", "secret": "_env:INCIDENT_IO_TOKEN"},
  "timeout_seconds": 30
}

// message — the alert event, with the dedup field derived from the notifier pk
{
  "title": "disk alert: volume is filling",
  "status": "firing",
  "deduplication_key": "pgrelay-33",
  "description": "relay_dev tablespace at 91%"
}

Better Stack authenticates its Incidents API the same way (bearer_header with an API token); FireHydrant likewise, whether you post to its Signals ingestion or create incidents directly via /v1/incidents. In all three the reply's id is the provider reference.

URL-keyed ingestion: Grafana Cloud IRM, Splunk On-Call, Squadcast¶

These three authenticate by the endpoint URL itself — a per-integration key baked into the address, no auth header at all. Store the whole URL as the secret and carry the placeholder auth block. Splunk On-Call (formerly VictorOps) as the worked example:

export VICTOROPS_URL=https://alert.victorops.com/integrations/generic/20131114/alert/07aba30c-XXXX-XXXX-XXXX-bc10bbd2112a/database-oncall

// profile
{
  "url": "_env:VICTOROPS_URL",
  "auth": {"style": "bearer_header", "secret": "unused"},
  "timeout_seconds": 30
}

// message — entity_id is the dedup key; RECOVERY on the same id resolves
{
  "message_type": "CRITICAL",
  "entity_id": "pgrelay-33",
  "entity_display_name": "disk alert",
  "state_message": "volume is filling on relay_dev"
}

The trailing path segment after the key is the routing key, steering the alert to a team — different channels can point at different routing keys through separate profiles.

Grafana Cloud IRM works identically: create an integration in IRM (a webhook or Alertmanager integration), copy its unique endpoint URL, and that URL is the credential. Grafana's archived OSS OnCall product offered the same pattern, but with the project read-only and development moved to Cloud IRM, IRM is the only variant this book covers.

Squadcast requires one deliberate choice: use its Incident Webhook API — the per-service URL-keyed endpoint made for exactly this — and not its v3 management API. The v3 API authenticates by exchanging a refresh token for a short-lived access token on each session, a multi-step token flow the webhook transport cannot perform (that machinery exists only inside the email transports' OAuth2 support). The Incident Webhook route needs nothing of the sort.

PagerDuty¶

PagerDuty's Events API v2 is the industry's reference alert-ingestion API, and it authenticates with a routing_key inside the JSON body — there is no auth header. That is exactly the case the webhook transport's body_merge profile field exists for: the routing key lives in an environment variable like every other credential, the profile injects it into the outgoing body, and the message data in the database never contains it. The profile carries the familiar placeholder auth block plus the merge:

export PD_ROUTING_KEY=R0XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

// profile
{
  "url": "https://events.pagerduty.com/v2/enqueue",
  "auth": {"style": "bearer_header", "secret": "unused"},
  "body_merge": {"routing_key": "_env:PD_ROUTING_KEY"},
  "timeout_seconds": 30
}

// message — no routing key anywhere; the profile injects it on send
{
  "event_action": "trigger",
  "dedup_key": "pgrelay-33",
  "payload": {
    "summary": "disk alert: volume is filling",
    "source": "relay_dev",
    "severity": "critical"
  }
}

If a producer does put a routing_key in the message, the merged value silently overwrites it — the key is profile-owned, by design. A 202 returns {"status": "success", "dedup_key": ...} — the dedup key is the provider reference. event_action of acknowledge or resolve with the same dedup_key drives the alert's lifecycle from the database side.

The REST API remains an alternative for environments that prefer header auth end to end: POST https://api.pagerduty.com/incidents with custom_header on Authorization holding the full Token token=… value from an env var, plus a static From: [email protected] entry in profile.headers — a heavier API (a full incident object instead of an event) with no body-side auth at all.

Jira Service Management¶

JSM's Operations module is Opsgenie's alert API relocated — Atlassian retired Opsgenie as a product and moved its alerting into JSM, keeping the API shape and the auth scheme. The integration events API still authenticates with Authorization: GenieKey <api-key>, which the custom_header style expresses with the whole value in the env var:

export JSM_AUTH="GenieKey aaaabbbb-cccc-dddd-eeee-ffffgggghhhh"

// profile
{
  "url": "https://api.atlassian.com/jsm/ops/integration/v2/alerts",
  "auth": {"style": "custom_header", "header_name": "Authorization", "secret": "_env:JSM_AUTH"},
  "timeout_seconds": 30
}

// message
{
  "message": "disk alert: volume is filling",
  "alias": "pgrelay-33",
  "priority": "P2"
}

Two things to get right: the key is a per-integration API key, created on a team's Operations page (Integrations → Add integration → API) — not an Atlassian account API token; the two are not interchangeable. And the base URL is region-sensitive (EU sites use a different host), so copy the URL from the integration's setup screen rather than assuming api.atlassian.com. The alias field is JSM's dedup key; a 202 returns a requestId as the provider reference (the API is asynchronous, like ACS's email send — acceptance, then processing).

Rootly¶

Bearer auth, so nearly a clean fit — but Rootly's REST API speaks JSON:API: the body nests under {"data": {"type": "incidents", "attributes": {…}}} and the endpoint expects Content-Type: application/vnd.api+json. Deep nesting is irrelevant to the default body_style (any JSON shape goes), and the content type rides the headers override:

// profile
{
  "url": "https://api.rootly.com/v1/incidents",
  "auth": {"style": "bearer_header", "secret": "_env:ROOTLY_API_KEY"},
  "headers": {"Content-Type": "application/vnd.api+json"},
  "timeout_seconds": 30
}

Rootly also offers per-source alert webhooks (the URL-keyed pattern above) if the JSON:API ceremony is more than the use case needs.

xMatters¶

xMatters (now part of Everbridge) accepts events at POST https://{company}.xmatters.com/api/xm/1/events with Basic authentication — an integration user's credentials map directly onto basic_auth (username + secret, the secret from _env:). A 202 returns a requestId. The alternative wiring is a Flow Designer HTTP-trigger URL carrying an API key — the URL-keyed pattern again — which some xMatters workflows prefer; both fit, so use whichever your workflow exposes. The event body's shape depends on the form/workflow receiving it, which makes the live-send verification especially important here.

Runframe¶

Runframe offers both patterns and either works: each configured integration gets a webhook endpoint at https://{your-host}/api/webhooks/{routingKey}, where the routing key doubles as the credential (URL-keyed pattern — whole URL in an env var, placeholder auth), and its REST API takes a Bearer API key from Settings → API Keys (bearer_header). The webhook route is the natural one for alert traffic; the REST API adds incident management beyond alerting if you need it.

OnPage — why it cannot be integrated¶

OnPage is the one platform in this table pg_relay cannot reach, and the reasons are structural, not a missing recipe:

Its authentication is a flow, not a credential. The webhook transport authenticates by attaching a static value to a single request — a bearer token, a header, a Basic pair, a key in the URL. OnPage's documented API instead requires a login step: client credentials are exchanged for a session/API token, which subsequent requests then present (and which expires and must be re-obtained). The transport sends exactly one request per event; it cannot run a login conversation first, store the issued token, or refresh it when it lapses. pg_relay does have token-exchange machinery, but only inside the email transports' OAuth2 support — deliberately not in the generic webhook path, whose contract is "one uninterpreted request".
Its status model is built on callbacks. OnPage reports what happened to a page (delivered, read, replied) by calling your endpoint back — the send request carries a callbackUri, and API tokens carry a callback-registration permission. pg_relay has no inbound HTTP listener, by design: there is nowhere for those callbacks to land, so the integration's own confirmation model cannot function.
Its API documentation is gated. The full API v2 reference sits behind a document-request form rather than public docs, so this assessment rests on what OnPage publishes openly. That is enough to establish the two blockers above, but not enough to rule out an enterprise arrangement that avoids them.

What would change the verdict: a documented endpoint accepting a static API key in a header or URL, used fire-and-forget without callbacks. If you have access to OnPage's gated enterprise documentation and find such an endpoint, the recipe would be an ordinary custom_header profile like the others in this chapter. Failing that, reaching OnPage means either their email gateway (OnPage accepts pages by email — pg_relay's SMTP transport can drive that today) or a first-class transport in the Processor, which the login-flow auth alone does not justify.

Continue to Jira — creating tickets rather than alerts, with the issue key returned as the provider reference.