Skip to content

What a Compromised Processor Can Do

Every security design should be able to answer "and what if this component is fully compromised?" without flinching. This chapter answers it for the Processor: assume an attacker has complete control of the Processor host — its process, its environment, its files — and ask what that actually buys them.

The honest answer is not much, and this chapter is careful to say precisely how much. It assumes the Processor is deployed as this book recommends: on its own host, connecting as the pgrelay role over TCP. (On a co-located sandbox where the Processor runs as the postgres OS user, the answer is instead everything — which is the whole argument of Where to Run the Processor.)

What the attacker holds

Three things, and only three:

  1. The pgrelay role's database credential — a .pgpass entry or a short-lived cloud identity token — valid only from the Processor host's address if pg_hba.conf and the network are configured as Database-Side Controls describes.
  2. The outbound provider secrets in the Processor's environment — an SMTP password, a Graph or ACS client secret, a Gmail refresh token, webhook bearer tokens, an OTLP receiver token. Whatever _env: variables the notifier profiles reference.
  3. Whatever flows through the Processor while they watch — the content of notifications as they are fetched and sent, and the payloads of events as they are processed.

Not held: any other database credential, any table's contents, any superuser or OS-level access to the database host (it is a different machine, and on a managed platform does not even exist as a host), or any way to make the database do something it would not do for the pgrelay role.

What the pgrelay role can do

The role is granted exactly the "operating set" — the functions the Processor needs to do its job — listed in Granting Permissions. Read as an attacker's toolkit, that set allows:

Capability How Impact
Fire events on any active, unreserved channel pgrelay.notify() — granted to PUBLIC by default, so this is true of any role that can connect, not specifically pgrelay Triggers registered actions with an attacker-chosen payload. For a 'sql' channel the payload is inert $1 data, so this runs the action as designed with unexpected input — the same exposure as any application bug that calls notify() wrongly. For a notification channel the payload is a notifier record id, so an attacker who can guess ids can re-send existing notifications
See every pending event, and process it queue_pending_ids(), _queue_claim(), process_one() This is the Processor's job; an attacker can do it early, selectively, or not at all. Registered SQL actions run as pgrelay (process_one is SECURITY INVOKER), with whatever privileges that role has in your schemas
Discard pending events _queue_mark_done() / _queue_delete() on any id they can see The most significant capability: a denial of processing. Events can be marked done without running, or deleted. Since a rogue Processor could equally just stop processing, this is the same outcome by another route — but it is silent, and it is why the audit trail matters (below)
Write misleading audit rows _write_log() The audit log can be polluted with fabricated ok or error rows. It cannot be edited — rows are insert-only through this function — so genuine history survives; the risk is noise, not erasure
Insert retry rows _queue_insert_retry() Duplicate or spurious retries of events on channels that allow retries
Read every notification's content pgrelay_notifier.fetch() (granted by the notifier's install) Any notification record the attacker can name by id — recipients, subject, body, attachments. Together with the provider secrets, this is the meaningful confidentiality exposure: the Processor necessarily sees what it sends
Trigger fleet reloads, write health rows request_reload(), _write_health() Nuisance-level: a reload storm, or fabricated health snapshots
Run ad-hoc SQL as pgrelay The pg_relay.exec lane, or simply a direct connection with the same credential Bounded entirely by what the pgrelay role — and any role GRANTed to it — can do; see the next section
Use the provider secrets They are in the environment Send mail as your configured sender, post to your webhooks, push metrics — until the secrets are rotated

What the pgrelay role cannot do

This list is the reason the design holds up:

  • Read or modify pg_relay's own tables directly. pgrelay.queue, pgrelay.log, pgrelay.actions, pgrelay.action_types, pgrelay.processor_health, pgrelay.options all have every privilege revoked from PUBLIC and are not granted to pgrelay. The only reads are the narrow ones the functions above perform for the role. An attacker cannot dump the queue's history or enumerate channel configuration.
  • Change any channel or job. register(), update(), unregister(), enable()/disable(), schedule_job(), run_sql(), cancel(), pause_job() and the rest are management functions granted through grant_user() to management roles — never to pgrelay. The attacker cannot repoint an action at different SQL, register a new channel, or schedule new work.
  • Reach the reserved ad-hoc SQL channel. notify() refuses reserved channels, and the functions that can enqueue onto one (schedule_job(), run_sql()) are management-only. The attacker cannot submit new SQL for execution through the queue; they can only run SQL directly as pgrelay, which brings us to:
  • Do anything in your schemas beyond pgrelay's own privileges. Outside the pgrelay schema, the role has whatever PUBLIC has in your database plus anything you granted it explicitly. Keep that at nothing (see Database-Side Controls) and a direct connection as pgrelay can read no application table at all.
  • Become another role — except one you deliberately GRANTed to pgrelay for run_as jobs. This is the single most important knob in the whole design, and it deserves its own section.
  • Connect from anywhere else. With pg_hba.conf restricting pgrelay to the Processor hosts' addresses (and a cloud security group doing the same), the credential is worthless off the host.
  • Reach anything the egress allowlist does not permit. The provider secrets can only be used against the destinations the Processor host can reach.
  • Escalate on the database host. It is a different machine (or, in the cloud, not a machine you can reach at all).

The one knob that sets the blast radius: GRANT … TO pgrelay

Ad-hoc SQL jobs (pg_relay.exec) can run SET ROLE <run_as> so that a maintenance job executes as a role that owns the tables it must VACUUM or REINDEX. For that to work, the DBA must first GRANT <run_as> TO pgrelay — a deliberate, per-role opt-in.

Every role you grant this way is a role a compromised Processor can become, directly, with a plain SET ROLE, no queue involved. So:

  • Grant narrow, purpose-built maintenance roles — one that owns exactly the tables a job maintains — never an application role, and never anything with CREATEROLE, CREATEDB, BYPASSRLS, or ownership of schemas you care about.
  • Treat the list of roles granted to pgrelay as a security-reviewed inventory. SELECT roleid::regrole FROM pg_auth_members WHERE member = 'pgrelay'::regrole; shows it.
  • If no ad-hoc job needs run_as, grant nothing. The Processor then can never be anything but pgrelay.

With that list empty and pgrelay holding no privileges in your schemas, the worst a full Processor compromise can do to your data is: read the notifications it was going to send anyway, fire or suppress events, and pollute the audit log with extra rows. It cannot read your tables, cannot change your data except through actions you registered, cannot alter pg_relay's configuration, and cannot touch the database host.

Detection

A compromised Processor is not silent from the database's point of view — everything it does leaves marks it cannot remove:

  • pgrelay.log is insert-only. Fabricated rows can be added; genuine rows cannot be changed or deleted by pgrelay. Use log_report() and queue_stats() to watch for events resolving as ok implausibly fast, an unexpected volume of error rows, or channels going quiet while their producers are active.
  • pg_stat_activity shows every pgrelay session with its client_addr. A session from an address that is not a Processor host is an alarm. log_connections = on for the role gives you a durable record.
  • Instance slots. Each Processor stamps a slot number (1–64) into every log line; an instance count in journalctl that does not match the fleet you run means something else is holding a slot.
  • Lifecycle notifications. PG_RELAY_STOP_NOTIFY fires on any deliberate exit and PG_RELAY_OUTAGE_NOTIFY when the database becomes unreachable — an attacker stopping the real Processor to substitute their own triggers the first; an attacker locking the real Processor out triggers the second. See Lifecycle Notifications.
  • Provider-side signals. Unexpected sends from your configured sender, webhook posts, or metric points are visible at the provider, and are the reason to scope provider credentials narrowly (a send-only mailbox, a token limited to one webhook) so that the damage is bounded even before you notice.

Containment

Because everything the Processor can do depends on one role and one set of secrets, containment is short:

  1. Lock the role out: ALTER ROLE pgrelay NOLOGIN; then SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = 'pgrelay';. Every Processor — legitimate or not — is now disconnected and cannot return. (Note that pgrelay.stop() is not the kill switch here: pausing the fleet is honoured by the real Processor's code, not by an attacker's own client.)
  2. Rotate the database credential — or, with identity-based cloud auth, revoke the host's identity.
  3. Rotate every provider secret the environment held. They are all listed in the environment file; nothing else references them.
  4. Review the audit log for the compromise window with log_report(), and the queue with queue_pending(), before re-enabling anything.
  5. Rebuild the Processor host from a known image — it holds no state worth preserving, which is the point.
  6. ALTER ROLE pgrelay LOGIN; and restart the fleet.

The queue itself is untouched by all of this: events enqueued during the outage wait durably, and retry rows are already part of the normal flow. Nothing needs to be replayed by hand.

Continue to Hardening the Service.